Let the spacetime be 4-dimensional.
- In the usual Maxwell theory of Abelian gauge fields $A$, where field strength $F=dA$ one considers the Maxwell action written as
$$ S_{Maxwell}\equiv\int -\frac{1}{2}(F \wedge \star F)=\int -\frac{1}{4}(F_{\mu \nu} F^{\mu \nu}) d^4x. $$ The normalization of $\frac{1}{4}$ has a precise physical meaning for the sake of maintaining the conventions of Maxwell equations. In the presence of source current 1-form $J$, solved from the equations of motions (through variational principles), from, $$ S_{Maxwell+source}\equiv \int-\frac{1}{2}(F \wedge \star F)+A \wedge \star J= \int(-\frac{1}{4}(F_{\mu \nu} F^{\mu \nu}) + A_\mu J^\mu )d^4x, $$ Maxwell equations becomes $$ d (\star F)= J, \quad dF=0, $$ namely there is no further numerical factor in front of the Maxwell equation. This gives a definite reason/answer behind the factor $1/4$.
In the Clay Math Millennium Prizes Yang–Mills and Mass Gap problem, the Yang–Mills action is written as
In other words, we have $$ \boxed{S_{YM,1}\equiv\int \frac{1}{4 g^2}\operatorname{Tr}(F \wedge \star F)}= \int\frac{1}{8 g^2}\operatorname{Tr}(F_{\mu \nu} F^{\mu \nu})d^4x=\int \frac{1}{8 g^2} \operatorname{Tr}[T^a T^b] F^{a\mu \nu} F_{\mu \nu}^b d^4x $$ $$ =\int \frac{1}{8 g^2} \frac{1}{2} \delta^{ab} F^{a\mu \nu} F_{\mu \nu}^b d^4x =\int \frac{1}{16 g^2} \sum_a F^{a\mu \nu} F_{\mu \nu}^a d^4x =\int \frac{1}{16 g^2} F^{a\mu \nu} F_{\mu \nu}^a d^4x, $$ where Einstein summation notation is assumed in the end. Based on the SU(N) Lie algebra, $\operatorname{Tr}[T^a T^b] =\frac{1}{2} \delta^{ab} $. The $a,b$ are the indices for fundamental representation of SU(N).
While in Steven Weinberg's textbook on QFT Volume in (15.2.3), we see that $ S_{YM,2}\equiv -\int \frac{1}{4 } F^{a\mu \nu} F_{\mu \nu}^a d^4x, $ This means that Weinberg's and many other physicists' textbooks have: $$ \boxed{S_{YM,2}\equiv -\int \operatorname{Tr}(F \wedge \star F) =-\int \frac{1}{4 } F^{a\mu \nu} F_{\mu \nu}^a d^4x.} $$ It looks that M Atiyah also uses the similar convention as $S_{YM,2}$.
question: I wonder, whether there is a reason behind the convention factor $\frac{1}{4 g^2}$ in front of Clay Math Millennium Prizes Yang–Mills and Mass Gap note: $S_{YM,1}\equiv\int \frac{1}{4 g^2}\operatorname{Tr}(F \wedge \star F)$? Is there some physical or mathematical reason behind this convention in contrast to this $S_{YM,2}$ convention? Or other math / physics constraints that we should be aware of? (Note: This term is the kinetic term, not the topological term like $F \wedge F$.)
See also a related question on Yang-Mills normalization.
In principle, the partition function $Z$ of quantum Yang-Mills theory will be schematically written as the path integral form over the measure $[DA]$ of gauge connection $A$ $$ Z = \int [DA] \exp(i S_{YM}) $$