Can you elaborate the claim: "if you use this metric, the heat equation is the gradient flow of the entropy"? Also what inner product do you use to obtain the Wasserstein metric? I am currently trying to grasp information geometry and there the "fisher metric" (a riemann metric) acts as the scalar product. Although this induces the fisher distance (not the wasserstein distance) unless those two coincide...? For future readers my reference is "Information Geometry" by Ay et al. (2017)

So if we remain with the first flavour (a mathematical tool to calculate the moment generating function of the logarithm of a random variable), then the steps I put together are alright? Because that is a neat result already. Although my complex analysis is too weak to come up with good sufficient condition for the moment generating function to be holomorphic. I really need to read a book on complex analysis at some point

I see, although the wikipedia article seems to ignore that part as well. I interpret this as: there are two flavours of the replica trick. The first one is described by wikipedia for which "in all cases where the replica method can be compared with other exact solutions, the methods lead to the same results" (wikipedia) and the one you describe of which "the validity of the replica trick is doubtful. The most obvious analytic continuation, obtained under the replica symmetric (RS) ansatz, sometimes leads to the wrong results" (paper you posted)

See also: arxiv.org/pdf/2108.12846.pdf

related: en.wikipedia.org/wiki/Carlson%27s_theorem (condition for when $f$ is uniquely determined by $f(n)$ for $n\in \mathbb{N}$) this only requires exponential type $\pi$ (not $\log(2)$)

I think I figured it out: math.stackexchange.com/a/4720470/445105 (the constant is slightly different)

do you have a reference for the derivation of the autocovariance?

@YaroslavBulatov Wouldn't the largest eigenvalue also have an outsized influence on the value of $Ht$ though? Hm, I mean you can do this entrywise I guess since you can move basis changes out of $\exp$....

@YaroslavBulatov hm, so you are saying the approximation holds for most eigenspaces (which are not the largest eigenvalue)?

@YaroslavBulatov if the eigenvalues were to decay, would it not make sense to increase the learning rate? Due to the optimality arguments in distill.pub/2017/momentum? I am completely unfamiliar with this concept of eigenvalue decay. I will try to read up on it

@YaroslavBulatov But if $h\lambda_d$ is greater than 1, then $(1-h\lambda_d)\not\approx \exp(1-h\lambda_d)$. I am also not sure why decaying eigenvalues would be plausible. I mean you would expect the hessian to be approximately constant close to the minimum due to the second taylor approximation.

@YaroslavBulatov hey, it is been some time and I am not really in-topic anymore. But given the usual derivation of optimal step sizes (e.g. distill.pub/2017/momentum ) I would think that A is not close to zero. I mean the entire point is trying to achieve $(1-h \lambda_d) = (1- h \lambda_1)$ which requies $h \lambda_d$ to be larger than 1 for the largest eigenvalue $\lambda_d$. I mean you could purposefully select smaller step-sizes, but...