Let the spacetime be 4-dimensional. 

 -  In the usual Maxwell theory of Abelian gauge fields $A$, where field strength $F=dA$ one considers the Maxwell action written as

$$
S_{Maxwell}\equiv\int
-\frac{1}{2}(F \wedge \star F)=\int
-\frac{1}{4}(F_{\mu \nu}  F^{\mu \nu}) d^4x.
$$
The normalization of $\frac{1}{4}$ has a precise physical meaning for the sake of maintaining the conventions of Maxwell equations. In the presence of source current 1-form $J$, solved from the equations of motions (through variational principles), from,
$$
S_{Maxwell+source}\equiv
\int-\frac{1}{2}(F \wedge \star F)+A  \wedge \star J=
\int(-\frac{1}{4}(F_{\mu \nu}  F^{\mu \nu}) + A_\mu J^\mu )d^4x,
$$
Maxwell equations becomes
$$
d (\star F)= J, \quad dF=0,
$$
namely there is no further numerical factor in front of the [Maxwell equation](https://en.wikipedia.org/wiki/Mathematical_descriptions_of_the_electromagnetic_field#Current_3-form,_dual_current_1-form). This gives a definite reason/answer behind the factor $1/4$.

 - In the [Clay Math  Millennium Prizes Yang–Mills and Mass Gap](https://www.claymath.org/sites/default/files/yangmills.pdf) problem, the Yang–Mills action is written as

-
>>
[![enter image description here][1]][1]
-

In other words, we have
$$
\boxed{S_{YM,1}\equiv\int
\frac{1}{4 g^2}\operatorname{Tr}(F \wedge \star F)}=
\int\frac{1}{8 g^2}\operatorname{Tr}(F_{\mu \nu}  F^{\mu \nu})d^4x=\int \frac{1}{8 g^2}
\operatorname{Tr}[T^a T^b] F^{a\mu \nu} F_{\mu \nu}^b d^4x
$$
$$
=\int \frac{1}{8 g^2}
\frac{1}{2} \delta^{ab} F^{a\mu \nu} F_{\mu \nu}^b d^4x
=\int \frac{1}{16 g^2} \sum_a F^{a\mu \nu} F_{\mu \nu}^a d^4x
=\int \frac{1}{16 g^2} F^{a\mu \nu} F_{\mu \nu}^a d^4x,
$$
where Einstein summation notation is assumed in the end. Based on the SU(N) Lie algebra, $\operatorname{Tr}[T^a T^b] =\frac{1}{2} \delta^{ab} $.
The $a,b$ are the indices for fundamental representation of SU(N). 

While in Steven Weinberg's textbook on QFT Volume in (15.2.3), we see that
$
S_{YM,2}\equiv
-\int \frac{1}{4 } F^{a\mu \nu} F_{\mu \nu}^a d^4x,
$
This means that Weinberg's and many other physicists' textbooks have:
$$
\boxed{S_{YM,2}\equiv -\int
\operatorname{Tr}(F \wedge \star F)
=-\int \frac{1}{4 } F^{a\mu \nu} F_{\mu \nu}^a d^4x.}
$$
It looks that M Atiyah also uses the similar convention as $S_{YM,2}$.

> question: I wonder, whether there is a reason behind the convention factor $\frac{1}{4 g^2}$ in front of [Clay Math  Millennium Prizes Yang–Mills and Mass Gap note](https://www.claymath.org/sites/default/files/yangmills.pdf): 
$S_{YM,1}\equiv\int
\frac{1}{4 g^2}\operatorname{Tr}(F \wedge \star F)$? Is there some physical or mathematical reason behind this convention in contrast to this $S_{YM,2}$ convention? Or other math / physics constraints that we should be aware of? (Note: This term is the kinetic term, not the topological term like $F \wedge F$.)

See also [a related question on Yang-Mills normalization](https://physics.stackexchange.com/q/413022/12813).

In principle, the partition function $Z$ of quantum Yang-Mills theory will be schematically written as the path integral form over the measure $[DA]$ of gauge connection $A$
$$
Z = \int [DA] \exp(i S_{YM})
$$

  [1]: https://i.sstatic.net/omznr.png