Return to Question

I'm trying to model convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$\mathbf{x}\leftarrow (\mathbf{1}-\mathbf{h})^2 \mathbf{x} + \mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, vector multiplication, addition, squaring are applied pointwise and $\mathbf{h}\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

$\mathbf{h}$ is small enough so that continuous approximation $\mathbf{x}_t\approx \exp(At)\mathbf{x}_0$ holds with $\mathbf{x_0}=\mathbf{h}$ and I need to know how trajectory of $\|\mathbf{x}_t\|_1$ depends on $p$ when $d\to\infty$.

Things are easy if we didn't have the the $\mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$ term (mixing term). $A$ is diagonal, and approximating $\|\mathbf{x}_t\|_1$ with an integral I get formulas which match observed behavior very well.

_^Notebook

However, keeping the mixing term makes things much harder to handle. System now evolves as $\exp(At)$ where $A$ is a diagonal+rank1 matrix. Integration approach as before gives formula which is cumbersome. There's also numeric approach which works but doesn't give insight on the role of $p$.

Any advice on the approaches to follow to get a nice upper bound on $\|\mathbf{x}_t\|_1$ in terms of $p$?

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.

I'm trying to analyze convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$\mathbf{x}\leftarrow (\mathbf{1}-\mathbf{h})^2 \mathbf{x} + \mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, vector multiplication, addition, squaring are applied pointwise and $\mathbf{h}\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

I care about L1 norm after $t$ steps which is well approximated by the following:

$$f(t)=\|\mathbf{x}_t\|_1\approx e^{t(\operatorname{diag}[(\mathbf{1}-\mathbf{h})^2]+\mathbf{h}\mathbf{h}^T)}\mathbf{h}$$

For a given $p$, how do I get a nice upper bound on $f(t)$ which holds when $d\to \infty$?

Things are easy if we didn't have the the $\mathbf{h}\mathbf{h}^T$ term: approximating $\|\mathbf{x}_t\|_1$ with an integral I get formulas which match observed behavior very well:

_^Notebook

Keeping $\mathbf{h}\mathbf{h}^T$ term makes things much harder to handle. Straightforward approach gives formula in terms of roots of diagonal + rank-1 matrix, but not practical for large $d$. There's also a numeric approach which works but also leaves unclear the dependence on $p$.

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.

I'm trying to model convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$\mathbf{x}\leftarrow (\mathbf{1}-\mathbf{h})^2 \mathbf{x} + \mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, vector multiplication, addition, squaring are applied pointwise and $\mathbf{h}\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

$\mathbf{h}$ is small enough so that continuous approximation $\mathbf{x}_t\approx \exp(At)\mathbf{x}_0$ holds with $\mathbf{x_0}=\mathbf{h}$ and I need to know how trajectory of $\|\mathbf{x}_t\|_1$ depends on $p$ when $d\to\infty$.

Things are easy if we didn't have the the $\mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$ term (mixing term). $A$ is diagonal, and approximating $\|\mathbf{x}_t\|_1$ with an integral I get formulas which match observed behavior very well.

_^Notebook

However, keeping the mixing term makes things much harder to handle. System now evolves as $\exp(At)$ where $A$ is a diagonal+rank1 matrix. Integration approach as before gives formula which is cumbersome. There's also numeric approach which works but doesn't give insight on the role of $p$.

Any advice on the approaches to follow to get a nice upper bound on $\|\mathbf{x}_t\|_1$ in terms of $p$?

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.

I'm trying to model convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$\mathbf{x}\leftarrow (\mathbf{1}-\mathbf{h})^2 \mathbf{x} + \mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, vector multiplication, addition, squaring are applied pointwise and $\mathbf{h}\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

$\mathbf{h}$ is small enough so that continuous approximation $\mathbf{x}_t\approx \exp(At)\mathbf{x}_0$ holds with $\mathbf{x_0}=\mathbf{h}$ and I need to know how trajectory of $\|\mathbf{x}_t\|_1$ depends on $p$ when $d\to\infty$.

Things are easy if we didn't have the the $\mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$ term (mixing term). $A$ is diagonal, and approximating $\|\mathbf{x}_t\|_1$ with an integral I get formulas which match observed behavior very well.

_^Notebook

However, keeping the mixing term makes things much harder to handle. System now evolves as $\exp(At)$ where $A$ is a diagonal+rank1 matrix. Integration approach as before gives formula which is cumbersome. There's also numeric approach which works but doesn't give insight on the role of $p$.

Any advice on the approaches to follow to get a nice upper bound on $\|\mathbf{x}_t\|_1$ in terms of $p$?

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.

I'm trying to model convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$x\leftarrow (\mathbf{1}-h)^2 x + h\langle x, h\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, $v^2$ means squaring each component of vector $v$ and $h\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

$h$ is small enough so that continuous approximation $x_t\approx \exp(At)x_0$ holds and I need to know how trajectory of $\|x_t\|_1$ depends on $p$ when $d\to\infty$.

Things are easy if we didn't have the the $h\langle x, h\rangle$ term (mixing term). $A$ is diagonal, and approximating $\|x_t\|_1$ with an integral I get formulas which match observed behavior very well.

_^Notebook

However, keeping the mixing term makes things much harder to handle. System now evolves as $\exp(At)$ where $A$ is a diagonal+rank1 matrix. Integration approach as before gives formula which is cumbersome. There's also numeric approach which works but doesn't give insight on the role of $p$.

Any advice on the approaches to follow to get a nice upper bound on $\|x_t\|_1$ in terms of $p$?

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.

I'm trying to model convergence of $x\in \mathbb{R}^{+d}$ which follows the following recurrence

$$\mathbf{x}\leftarrow (\mathbf{1}-\mathbf{h})^2 \mathbf{x} + \mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$$

Here $\mathbf{1}$ indicates a vector of $1$'s, vector multiplication, addition, squaring are applied pointwise and $\mathbf{h}\propto (1^{-p},2^{-p},\ldots ,d^{-p})$ for some $p\in (1,2)$.

$\mathbf{h}$ is small enough so that continuous approximation $\mathbf{x}_t\approx \exp(At)\mathbf{x}_0$ holds with $\mathbf{x_0}=\mathbf{h}$ and I need to know how trajectory of $\|\mathbf{x}_t\|_1$ depends on $p$ when $d\to\infty$.

Things are easy if we didn't have the the $\mathbf{h}\langle \mathbf{x}, \mathbf{h}\rangle$ term (mixing term). $A$ is diagonal, and approximating $\|\mathbf{x}_t\|_1$ with an integral I get formulas which match observed behavior very well.

_^Notebook

However, keeping the mixing term makes things much harder to handle. System now evolves as $\exp(At)$ where $A$ is a diagonal+rank1 matrix. Integration approach as before gives formula which is cumbersome. There's also numeric approach which works but doesn't give insight on the role of $p$.

Any advice on the approaches to follow to get a nice upper bound on $\|\mathbf{x}_t\|_1$ in terms of $p$?

Motivation: this equation models expected value of iterated Gaussian linear system like this for non-isotropic Gaussian case. Used to model training curve of neural network training.