Connections having the same holonomy along loops at a point I find myself stuck with the following question, which seems very classical but for which I have not been able to find a reference.
Consider a smooth vector bundle $E$ of rank $r$ over a compact orientable surface $S$ and fix a base point $x \in S$: when do two connections on $E$ have the same holonomy along loops at $x$ ? 
More precisely, I want to fix one connection and study the set of connections with the same holonomy.
If this helps, I am willing to use the standard cellular decomposition of $S$ with $2g$ loops at $x$ and just consider the holonomy around this finite number of  loops. Also, it is OK to fix a frame of the fibre $E_x$, so that the parallel transport operators along loops at x lie in the group $GL(r,\mathbb{C})$.
Have you ever encountered this question or do you know an answer to it? My impression is that the answer should have to do with the single $2$-cell in the fixed cellular decomposition of $S$ but I am unable to formalize that.
Perhaps it is worth emphasizing that I am not assuming that the parallel transport is the same along all paths in $S$ (in which case the two connections should be gauge equivalent) but only along loops at a given point.
Edit: The answers of Vladimir and Tobias have made me realize that my question was not well-posed, to say the least. What I meant is the following. Let $A_E$ be the space of all linear connections on the vector bundle $E$. We fix a frame of $E_x$ and consider the map $$H: \begin{array}{rcl} A_E & \longrightarrow & (GL(r,\mathbb{C}))^{2g} \\ \nabla & \longmapsto & (T^{\nabla}_{\gamma_i})_{1\leq i \leq 2g}\end{array}$$ where $(\gamma_i)_{1\leq i\leq 2g}$ are the loops at $x$ in the standard cellular decomposition of the surface $S$ and $T^{\nabla}_{\gamma}$ is the matrix of the parallel transport operator along $\gamma$ in the given frame. The question is: what is the fibre of the map $H$?
 A: If the holonomies of two connections coincide for all piecewise-smooth loops, then these connections are gauge equivalent. The basic idea is to consider the path bundle $PM$ over $M$, which is a principal bundle with the based loops $LM$ as the structure group. Then the holonomy map $LM \to G$ allows to reduce/extend the structure group to $G$ and thus gives an isomorphism $P = PM \times_{Hol} G$. Moreover, $PM$ carries a natural connection (whose horizontal lift is essentially composition of paths) and it is easy to see that the induced connection on $PM \times_{Hol} G$ coincides up to gauge transformation with the original connection on $P$. This viewpoint is explored in:


*

*Lewandowski, J. Group of loops, holonomy maps, path bundle and path connection

*Barrett, J. Holonomy and path structures in general relativity and Yang-Mills theory


If you just require that the holonomies coincide for the canonical $2g$ loops, then the answer is not that easy. In the abelian case, these holonomies determine the topological structure of the bundle (which is given by an element of $H^1(M, U(1))$) and you need the curvature form to reconstruct the connection up to gauge transformation. I think, this picture generalizes at least to central (Yang-Mills) connections (see the paper by Atiyah and Bott 1983 and some follow-ups by Huebschmann).
A: Most (in the natural sense) connections have the same holonomies, namely the maximal one. 
Most affine connections have the same holonomy, namely the whole $GL_+(n)$. 
Most Levi-Civita connections have the same holonomy, namely $SO(n)$. 
Usually if the holonomy group is not the maximal one then there  exists  an  object parallel with respect to the connection (as the  parallelity of the metric with respect of the Levi-Civita connection implies that the group of holonomy is $ SO$ and not $GL_+$). One can sometimes reconstruct the nature of the object, say if the holonomy of a Levi-Civita connection is $U(n/2)\subset  SO(n)$
 then there exists a parallel complex structure, but in most cases one can not say anything more. Well, sometimes one can say more, say if the holonomy  of a Levi-Civita connection is as of  irreducible symmetric space then the  connection is the Levi-Civita connection of this symmetric space, but this is rather an exception than a rule.     
Added after Oliver reformulated his question:  The fiber of $H$ can be  quite wierd. It is inifinitely-dimensional. To see this  observe that if we change the connections in the interior of cells, we do not change parallel transport along the edges. There is also other freedom since one can change the connection along the first half  arbitrary  of an edge and conpensate the change in the second half. 
If the structure group has complicated topology, one can also have some  topological effects (which make the moduli space not connected), since a connection gives an imbedding of the 2g-polygon into the Lie group and two such imbedding could be not isotop.     
