Optimizing over matrices with spectral radius <1? Suppose $F(x)$ is a convex objective function on $n\times n$ matrices, and I need to numerically optimize $F$ with the condition that $x$ has spectral radius less than $1$. This might be too hard, so an approximation would be needed. Has this problem been studied before?
Motivation: Boltzmann machines are hard to evaluate when spectral radius of the weight matrix is large, especially if it's above $1$ so best fit to data subject to this constraint would give a useful model.
Example: Let $X=\{1,-1\}^d$ and $\hat{X}$ some list of $\{1,-1\}$ $d$-tuples. Find $$\max_A \sum_{x\in \hat{X}} \mathbf{x}'A\mathbf{x} - |\hat{X}|\log \sum_{x\in X} \exp(\mathbf{x}'A\mathbf{x})$$
Where $A$ is symmetric real-valued $d\times d$ matrix with spectral radius < 1. This needs to be done in time polynomial in $d$ and linear in $|\hat{X}|$. When spectral radius is <1, belief propagation gives a reasonably accurate way to approximate gradient of this objective in $O(|\hat{X}|d^2)$ time
 A: If your matrices are symmetric, the set of matrices with spectral radius $\le 1$ is convex, and can be modelled using a linear matrix inequality (LMI), see e.g. page 147 in Lectures on Modern Convex Optimization by Ben-Tal and Nemirovski. If you wanted to minimize a convex objective that is also semidefinite-representable, you could in principle formulate and solve your problem as a semidefinite programming problem. However, maximizing a convex objective over a convex set is a much more difficult problem.
A: This started as a comment, but it's too long.
Is the objective function invariant under conjugation?
Spectral radius is far from a convex function of matrices.  If you take two non-negative matrices with 1's on the diagonal, one zero below the diagonal and positive above, the other vice versa, any convex combination has spectral radius bigger than 1. It's easy to make the spectral radius as  large as you like.
The set of characteristic polynomials for matrices of spectral radius 1 isn't convex either.  For example, the average of 
$(x - .99)^2$ and $(x - .99i)^2$ has roots outside the unit circle.
However, every conjugacy class in $GL(n,\mathbb C)$ has a representative that is upper triangular, and the upper triangular matrices of spectral radius 1 form a convex set. This may make it easier to find the optimum (depending what it is, which you didn't say). There are convex sets containing just conjugacy classes of spectral radius < 1 for various other kinds of matrices.  
Even if the function is not invariant under conjugation, it may help to break it up by conjugacy class: for each conjugacy class, find the optimum, then find the optimum among all conjugacy classes.
