I have a general question. Suppose I have the following to optimize $$\|Y-A(\mathbf{x})B(\mathbf{y})\|^2$$ where $Y$ is a vector, $A(\mathbf{x})$ is a matrix that depends on a vector $\mathbf{x}$ in a linear way, i.e., the elements of $A(\mathbf{x})$ are those in $\mathbf{x}$, but the x-values show up "here and there" in $A(\mathbf{x})$. The same holds for $B(\mathbf{y})$. "Here and there" is a in fact a very complicated pattern, and will not help much. In my problem, I'm only interested in the value of $\mathbf{x}$ that minimizes the cost, but both $\mathbf{x}$ and $\mathbf{y}$ are to be optimized over. To make the problem feasible, one element of $\mathbf{x}$ is frozen to an arbitrary value, say 1.
Given a starting position $\mathbf{x}_0$, I can solve for the optimal $\mathbf{y}$ by just applying the Moore-Penrose inverse. Then, with that $\mathbf{y}$, I can update my $\mathbf{x}$ with the Moore-Penrose inverse, and so on and on.
An alternative would to define the optimal $B(\mathbf{y})$ as a function of $\mathbf{x}$, denote this by $B^{\mathrm{opt}}(\mathbf{x})$ and then solve $$\|Y-A(\mathbf{x})B^{\mathrm{opt}}(\mathbf{x})\|^2.$$
In the latter case, I can solve this for example with a gradient method, or with a conjugate gradient method.
Here is now my questions:
Is there a relation between a gradient method and the iterative least squares method above?
In the gradient method, I can speed up by doing a conjugate method. Is there any possibility to carry the "conjugate" framework over to the iterative method, so that I don't undo the improvements I did in iteration k-1 in iteration k?
Is there any other simple good method that is suitable for this type of problems?
This is just a small part of a paper I'm writing, so I am not so much interested in diving deep into this type of optimization problems, but I still need something that works and that shows that doesn't look to stupid…
