(Too long for a comment on Gerry Myerson's answer.)

It is clear that edge-transitive implies the OP's property.  But at least for infinite (multi-)graphs, I believe that edge-transitivity is strictly stronger.  Consider the following infinite graph, with countably many vertices and countably many edges.  There are two distinguished vertices, say $A$ and $B$, which are each connected by $\aleph_0$ edges.  Moreover, there exist further vertices $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$, such that each $A_n$ is connected by $\aleph_0$ edges to $A$ and each $B_n$ is connected by $\aleph_0$ edges to $B$.  In particular, whenever there is one edge between two vertices there are infinitely many, so removing any edge leaves a graph which is isomorphic to the one we started with -- *a fortiori* any two choices of edge removal lead to isomorphic graphs.  But the edges joining $A$ to $B$ lie in the middle of a path of length three, whereas the other edges do not, so the graph is not edge-transitive.

Whether there are counterexamples which are not so "cheap" remains to be seen...

<b>Added</b>: I think the following modification of the construction gives a countably infinite simple graph whose isomorphism class does not change upon removal of any one edge but is not edge-transitive.  Consider the following three types of simple graphs:

(i) A single vertex $P$ with no edges.   
(ii) An infinite spoke $S$: i.e., with a central vertex $A$ and peripheral vertices $\{A_n\}_{n=1}^{\infty}$ such that there is an edge joining $A$ to each $A_n$.  
(iii) A double spoke $D$: Take two infinite spokes $S_A$ and $S_B$ and glue two peripheral vertices: say identify $A_1$ with $B_1$.  

Now take the graph $G$ to be the direct sum of countably [any other infinite cardinal $\kappa$ would work as well to give an example of cardinality $\kappa$] copies of each of the graphs $P$, $S$ and $D$.