I find myself stuck with the following question, which seems very classical but for which I have not been able to find a reference.

***Consider a smooth vector bundle $E$ of rank $r$ over a compact orientable surface $S$ and fix a base point $x \in S$: when do two connections on $E$ have the same holonomy along loops at $x$ ?*** 

More precisely, I want to fix one connection and study the set of connections with the same holonomy.

If this helps, I am willing to use the standard cellular decomposition of $S$ with $2g$ loops at $x$ and just consider the holonomy around this finite number of  loops. Also, it is OK to fix a frame of the fibre $E_x$, so that the parallel transport operators along loops at x lie in the group $GL(r,\mathbb{C})$.

Have you ever encountered this question or do you know an answer to it? My impression is that the answer should have to do with the single $2$-cell in the fixed cellular decomposition of $S$ but I am unable to formalize that.

Perhaps it is worth emphasizing that I am not assuming that the parallel transport is the same along all paths in $S$ (in which case the two connections should be gauge equivalent) but only along loops at a given point.

***Edit:*** The answers of Vladimir and Tobias have made me realize that my question was not well-posed, to say the least. What I meant is the following. Let $A_E$ be the space of all linear connections on the vector bundle $E$. We fix a frame of $E_x$ and consider the map $$H: \begin{array}{rcl} A_E & \longrightarrow & (GL(r,\mathbb{C}))^{2g} \\ \nabla & \longmapsto & (T^{\nabla}_{\gamma_i})_{1\leq i \leq 2g}\end{array}$$ where $(\gamma_i)_{1\leq i\leq 2g}$ are the loops at $x$ in the standard cellular decomposition of the surface $S$ and $T^{\nabla}_{\gamma}$ is the matrix of the parallel transport operator along $\gamma$ in the given frame. The question is: what is the fibre of the map $H$?