It is known that any holomorphic bundle of any rank over a noncompact Riemann surface is trivial.

The idea of proof is the following: non-compact curve is actually affine manyfold. On affine manyfolds cohomologies of coherent sheaves vanishes (Serre). Extensions of one bundle by another is controlled by Ext^1(V,W) which is coherent and hence vanishes. So non-triviality of vector bundles reduces to linear bundles. Linear bundles are trivial by exponential sequence (Pay attention here is difference between holomorphic and algebraic situation - algebraic linear bundles can be non-trivial on affine curves).

Let $E$ be a holomorphic vector bundle over a compact Riemann surface $X$ with gauge group $G$. A consequence of the above theorem is the restriction $E|_{X-\{p\}}$ for any point $p\in X$ is a trivial bundle. Thus $E$ can be recovered by specifying the transition function $g: > D\cap (X-\{p\}) \rightarrow G$ where $D$ is a small disk containing $p$.

Is this correct? If not, could you give a counter-example? I am mainly interested in learning about the moduli space of holomorphic bundles over $X$ in a concrete way, e.g. using transition functions.

Yes, that is correct. Such point of view on bundles became very popular since end 1980-ies. It is widely used in reseach related to confomal field theory (Verlinde formula), integrbale systems (Hitchin system) and Langlands correspondence over C.

However it is not something which makes things "explicit".

The goal of that technique is to translate geometric problems about vector bundles to Lie group/algebra questions about loop groups. The benefit is that you have "uniformization" - you can think of transition function function which are actually loop group G((t)) as a kind of "universal" moduli space of vector bundles - universal in very strong way since curves of all genuses "sits inside". In a sense that is in a spirit of global to local principle - geometric questions (global) reduced to questions on G((t)) - which are local.

The Verlinde formula story, Knizhnik-Zamolodchikov equation, Hitchin system etc benefit much from it, see e.g. https://mathoverflow.net/a/316733/10446

But now as we minimally only have one transition function, we have no cocycle conditions to impose. How do I still see that $T_{[E]}^\ast > Bun_G(X) \cong H^0(X,\mathfrak{g}\otimes K_X)$?

I am not sure I understand question correctly. But let me try to answer.

Let us think in terms of group theory. So as we discussed above above G((t)) is a kind of universal moduli space. Tanget space at "e" is Lie algebra g((t)). But we are interested in cotanget space ! The point is that natural pairing is given by $ \int f d g $ , so g((t)) are functions, but $g^*((t))$ are 1-forms ! That is how "K_x" appears on the level of Lie algebra.

(Going further for central extension of g((t)) dual space will be identified with space of connections).

It is known that any holomorphic bundle of any rank over a noncompact Riemann surface is trivial.

The idea of proof is the following: non-compact curve is actually affine manyfold. On affine manyfolds cohomologies of coherent sheaves vanishes (Serre). Extensions of one bundle by another is controlled by Ext^1(V,W) which is coherent and hence vanishes. So non-triviality of vector bundles reduces to linear bundles. Linear bundles are trivial by exponential sequence (Pay attention here is difference between holomorphic and algebraic situation - algebraic linear bundles can be non-trivial on affine curves).

Let $E$ be a holomorphic vector bundle over a compact Riemann surface $X$ with gauge group $G$. A consequence of the above theorem is the restriction $E|_{X-\{p\}}$ for any point $p\in X$ is a trivial bundle. Thus $E$ can be recovered by specifying the transition function $g: > D\cap (X-\{p\}) \rightarrow G$ where $D$ is a small disk containing $p$.

Is this correct? If not, could you give a counter-example? I am mainly interested in learning about the moduli space of holomorphic bundles over $X$ in a concrete way, e.g. using transition functions.

Yes, that is correct. Such point of view on bundles became very popular since end 1980-ies. It is widely used in reseach related to confomal field theory (Verlinde formula), integrbale systems (Hitchin system) and Langlands correspondence over C.

However it is not something which makes things "explicit".

The goal of that technique is to translate geometric problems about vector bundles to Lie group/algebra questions about loop groups. The benefit is that you have "uniformization" - you can think of transition function function which are actually loop group G((t)) as a kind of "universal" moduli space of vector bundles - universal in very strong way since curves of all genuses "sit inside". In a sense that is in a spirit of global to local principle - geometric questions (global) reduced to questions on G((t)) - which are local.

The Verlinde formula story, Knizhnik-Zamolodchikov equation, Hitchin system etc benefit much from it, see e.g. https://mathoverflow.net/a/316733/10446

But now as we minimally only have one transition function, we have no cocycle conditions to impose. How do I still see that $T_{[E]}^\ast > Bun_G(X) \cong H^0(X,\mathfrak{g}\otimes K_X)$?

I am not sure I understand question correctly. But let me try to answer.

Let us think in terms of group theory. So as we discussed above above G((t)) is a kind of universal moduli space. Tanget space at "e" is Lie algebra g((t)). But we are interested in cotanget space ! The point is that natural pairing is given by $ \int f d g $ , so g((t)) are functions, but $g^*((t))$ are 1-forms ! That is how "K_x" appears on the level of Lie algebra.

(Going further for central extension of g((t)) dual space will be identified with space of connections).