(Too long for a comment on Gerry Myerson's answer.)

It is clear that edge-transitive implies the OP's property. But at least for infinite (multi-)graphs, I believe that edge-transitivity is strictly stronger. Consider the following infinite graph, with countably many vertices and countably many edges. There are two distinguished vertices, say $A$ and $B$, which are each connected by $\aleph_0$ edges. Moreover, there exist further vertices $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$, such that each $A_n$ is connected by $\aleph_0$ edges to $A$ and each $B_n$ is connected by $\aleph_0$ edges to $B$. In particular, whenever there is one edge between two vertices there are infinitely many, so removing any edge leaves a graph which is isomorphic to the one we started with -- a fortiori any two choices of edge removal lead to isomorphic graphs. But the edges joining $A$ to $B$ lie in the middle of a path of length three, whereas the other edges do not, so the graph is not edge-transitive.

Whether there are counterexamples which are not so "cheap" remains to be seen...

Added: I think the following modification of the construction gives a countably infinite simple graph whose isomorphism class does not change upon removal of any one edge but is not edge-transitive. Consider the following three types of simple graphs:

(i) A single vertex $P$ with no edges.
(ii) An infinite spoke $S$: i.e., with a central vertex $A$ and peripheral vertices $\{A_n\}_{n=1}^{\infty}$ such that there is an edge joining $A$ to each $A_n$.
(iii) A double spoke $D$: Take two infinite spokes $S_A$ and $S_B$ and glue two peripheral vertices: say identify $A_1$ with $B_1$.

Now take the graph $G$ to be the direct sum of countably [any other infinite cardinal $\kappa$ would work as well to give an example of cardinality $\kappa$] copies of each of the graphs $P$, $S$ and $D$.

(Too long for a comment on Gerry Myerson's answer.)

It is clear that edge-transitive implies the OP's property. But at least for infinite (multi-)graphs, I believe that edge-transitivity is strictly stronger. Consider the following infinite graph, with countably many vertices and countably many edges. There are two distinguished vertices, say $A$ and $B$, which are each connected by $\aleph_0$ edges. Moreover, there exist further vertices $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$, such that each $A_n$ is connected by $\aleph_0$ edges to $A$ and each $B_n$ is connected by $\aleph_0$ edges to $B$. In particular, whenever there is one edge between two vertices there are infinitely many, so removing any edge leaves a graph which is isomorphic to the one we started with -- a fortiori any two choices of edge removal lead to isomorphic graphs. But the edges joining $A$ to $B$ lie in the middle of a path of length three, whereas the other edges do not, so the graph is not edge-transitive.

Whether there are counterexamples which are not so "cheap" remains to be seen...

Added: I think the following modification of the construction gives a countably infinite simple graph whose isomorphism class does not change upon removal of any one edge but is not edge-transitive. Consider the following three types of simple graphs:

(i) A single vertex $P$ with no edges.
(ii) An infinite spoke $S$: i.e., with a central vertex $A$ and peripheral vertices $\{A_n\}_{n=1}^{\infty}$ such that there is an edge joining $A$ to each $A_n$.
(iii) A double spoke $D$: we have vertices $A$, $B$, $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$. There is one edge connecting each $A_n$ to $A$ and each $B_n$ to $B$ and finally one edge connecting $A$ to $B$. (Thus removing that last edge results in a disjoint union of two spokes.)

Now take the graph $G$ to be the direct sum of countably [any other infinite cardinal $\kappa$ would work as well to give an example of cardinality $\kappa$] copies of each of the graphs $P$, $S$ and $D$.

(Too long for a comment on Gerry Myerson's answer.)

It is clear that edge-transitive implies the OP's property. But at least for infinite (multi-)graphs, I believe that edge-transitivity is strictly stronger. Consider the following infinite graph, with countably many vertices and countably many edges. There are two distinguished vertices, say $A$ and $B$, which are each connected by $\aleph_0$ edges. Moreover, there exist further vertices $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$, such that each $A_n$ is connected by $\aleph_0$ edges to $A$ and each $B_n$ is connected by $\aleph_0$ edges to $B$. In particular, whenever there is one edge between two vertices there are infinitely many, so removing any edge leaves a graph which is isomorphic to the one we started with -- a fortiori any two choices of edge removal lead to isomorphic graphs. But the edges joining $A$ to $B$ lie in the middle of a path of length three, whereas the other edges do not, so the graph is not edge-transitive.

Whether there are counterexamples which are not so "cheap" remains to be seen...

(Too long for a comment on Gerry Myerson's answer.)

It is clear that edge-transitive implies the OP's property. But at least for infinite (multi-)graphs, I believe that edge-transitivity is strictly stronger. Consider the following infinite graph, with countably many vertices and countably many edges. There are two distinguished vertices, say $A$ and $B$, which are each connected by $\aleph_0$ edges. Moreover, there exist further vertices $\{A_n\}_{n=1}^{\infty}$, $\{B_n\}_{n=1}^{\infty}$, such that each $A_n$ is connected by $\aleph_0$ edges to $A$ and each $B_n$ is connected by $\aleph_0$ edges to $B$. In particular, whenever there is one edge between two vertices there are infinitely many, so removing any edge leaves a graph which is isomorphic to the one we started with -- a fortiori any two choices of edge removal lead to isomorphic graphs. But the edges joining $A$ to $B$ lie in the middle of a path of length three, whereas the other edges do not, so the graph is not edge-transitive.

Whether there are counterexamples which are not so "cheap" remains to be seen...

Added: I think the following modification of the construction gives a countably infinite simple graph whose isomorphism class does not change upon removal of any one edge but is not edge-transitive. Consider the following three types of simple graphs:

(i) A single vertex $P$ with no edges.
(ii) An infinite spoke $S$: i.e., with a central vertex $A$ and peripheral vertices $\{A_n\}_{n=1}^{\infty}$ such that there is an edge joining $A$ to each $A_n$.
(iii) A double spoke $D$: Take two infinite spokes $S_A$ and $S_B$ and glue two peripheral vertices: say identify $A_1$ with $B_1$.

Now take the graph $G$ to be the direct sum of countably [any other infinite cardinal $\kappa$ would work as well to give an example of cardinality $\kappa$] copies of each of the graphs $P$, $S$ and $D$.