Given a matrix $A \in \mathbb R^{n \times n}$ whose entries are i.i.d. $N(0,1)$, what is the expected value of its largest singular value? Equivalently, what is the expected value of the largest eigenvalue of $A'A$?
If $A$ is a Gaussian random matrix as you describe, then the ensemble of matrices given by $A^TA$ is known as the Wishart ensemble, or the Laguerre ensemble. It has been extensively studied, and you can find information in standard books about random matrix theory.
The average of the largest eigenvalue of $A^TA$ is $4n$. The distribution around the average is given by the TracyWidom function. Distribution of large deviations from the mean are also known.
You can start collecting references about largest eigenvalues by looking here:
Large Deviations of the Maximum Eigenvalue in Wishart Random Matrices, by Pierpaolo Vivo, Satya N. Majumdar, Oriol Bohigas, https://arxiv.org/abs/condmat/0701371

$\begingroup$ I tried to read those articles, but they are hard to understand for a nonmath person. Could you briefly describe to me how one gets the distribution of the eigenvalues of the Wishart matrix? Thanks. $\endgroup$ – wenyuz Dec 2 '17 at 23:08

$\begingroup$ There are different methods. Two popular ones are the "orthogonal polynomials" method and the "Coulomb gas" method. Maybe you can look them up. $\endgroup$ – Marcel Dec 3 '17 at 15:55