Questions tagged [least-squares]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
18 views

Minimum eigenvalue of normal matrix with polynomial basis

For each $n\in \mathbb{N}\cup\{0\}$, let $x_n(t)=\frac{t^n}{n!}$ for all $t\in [0,1]$. As the functions $X_N=(x_0 ,\ldots, x_N)$ are linearly independent, the matrix $ \int_0^1 X_N(t)^\top X_N(t)\,\...
user avatar
  • 423
0 votes
0 answers
12 views

Convergence of solutions of regularised least square problems

Let $\mathcal{H}_1$ and $\mathcal{H}_2$ be two Hilbert spaces ($\mathcal{H_2}$ being finite dimensional, but I don't think that it matters). Consider $(A_{\lambda})_{\lambda>0}$ a familly of ...
user avatar
  • 143
20 votes
4 answers
2k views

Is the pseudoinverse the same as least squares with regularization?

Given a linear system $Ax=b$, the pseudoinverse of $A$ is found as the matrix $A^+$ such that $x=A^+ b$ where $x$ solves the least squares problem $\min \| Ax - b \|^2 $ and $x \perp \mathcal{N}(A)$. ...
user avatar
3 votes
1 answer
200 views

Concentration inequality for norm of solution to nonlinear least-squares problem

Define the piecewise-linear function $\psi(t):=\max(t,0)$ for all $t \in \mathbb R$. Let $d,n,k \to \infty$ at the same rate (i.e $n \asymp k \asymp d$). Let $y_1,\ldots,y_n \in \{-1,1\}$ uniformly ...
user avatar
  • 5,708
1 vote
2 answers
131 views

Robust estimation of $Ax=b$

Problem setting : $ \underset{x}{\text{min}} \|Ax-b\|$, where $A \in \mathcal{R}^{m \times n}, m\gg n $, full rank. L1 loss is used for robust estimation using IRLS. The corresponding equation to ...
user avatar
  • 21
1 vote
1 answer
104 views

Understanding the Time Delay of Arrival trilateration algorithm

I'm trying to algorithmically solve the Time Delay of Arrival problem as part of some mathematics research. The problem is as follows: Given the location of three receivers in a plane (A, B, and C), ...
user avatar
  • 111
1 vote
2 answers
73 views

Non-parametric regression and curvature

Given a finite set of points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ in the plane, Linear Regression tells us how to find the straight line "$y=a+bx$" best approximating the given points, in the ...
user avatar
  • 2,143
1 vote
3 answers
78 views

RKHS/non-parametric regression with missing response values

I am interested in doing RKHS regression with missing response variables. Given input-output pairs $(x_i,y_i)$, I want to estimate a function $f(\cdot)$ as follows \begin{equation}f(x)\approx u(x)=\...
user avatar
  • 441
2 votes
1 answer
294 views

Minimise $\sum_i \begin{Vmatrix}\boldsymbol{x}_i \\ \boldsymbol{y}_i\end{Vmatrix}$

Consider column vectors $\boldsymbol{z}_i$, $\quad i=1,\dots,n$. Each $\boldsymbol{z}_i$ has $j$ elements and can be expressed as $\boldsymbol{z}_i = \begin{bmatrix} \boldsymbol{x}_i \\ \boldsymbol{y}...
user avatar
5 votes
2 answers
546 views

Symmetric linear least-squares solution

Given tall matrices $A$ and $Y$ and the following overdetermined linear system in square matrix $X$ $$AX=Y$$ is there an explicit formula for the least-squares solution if $X$ is constrained to be ...
user avatar
  • 193
0 votes
1 answer
495 views

How to determine the damping factor in Levenberg-Marquardt?

From the algorithm, we can see that it tries different damping factor until it gets a good one by the error. Is the damping factor related to the eigenvalues of the Hessian matrix?
user avatar
2 votes
0 answers
28 views

Difference Between Total Least Squares Plane and Plane Orthogonal to Principal Axis of Inertia Tensor

Given a finite set $P$ of points in $\mathbb{E}^3$ , one can calculate an approximating plane either as the solution of a Total Least Squares problem or by interpreting the problem physically, ...
user avatar
  • 11.1k
1 vote
1 answer
97 views

Minimization Proof of Conditioning on Gaussian is Gaussian

It is well known that $E[X|X+Y]$ is Gaussian if both $X$ and $Y$ are, and the result can be derived using standard density arguments. However, how can one prove it by only resulting to optimization ...
user avatar
  • 5,008
1 vote
0 answers
110 views

Choice of residual function for least squares error minimization

Good morning, I have the a set of data $(\sigma,D,\alpha_0)_i$, $i=1...n$ data. I want to determine two parameters $K_{IC}$, $C_f$ in the basic equation given as $K_{IC} = \sigma \sqrt{D} k_0(\...
user avatar
  • 11
2 votes
2 answers
160 views

In practice, what's the fastest method to find a least square solution rather than using SVD decompostion?

I'm working on a real-time implementation of Lucas-Kanade for optical flow. However, the SVD decomposition to do achieve the least square method to reduce the error seems to take too much time. A ...
user avatar
8 votes
3 answers
293 views

Regularized linear vs. RKHS-regression

I'm studying the difference between regularization in RKHS regression and linear regression, but I have a hard time grasping the crucial difference between the two. Given input-output pairs $(x_i,y_i)...
user avatar
  • 441
5 votes
1 answer
220 views

Rank-constrained least-squares solution of the Sylvester matrix equation

For the Sylvester matrix equation $AX+XB=C$, I want to find the least-squares solution $X$ via $$\begin{array}{ll} \text{minimize} & \| AX + XB - C \|_{\text{F}}^2\\ \text{subject to} & \mbox{...
user avatar
  • 191
4 votes
1 answer
298 views

Least square solution to $AXB+CXD=E$

I am trying to find the least-squares solution $X$ of the following matrix equation $$AXB+CXD=E$$ Of course, I know that this equation can be written in the form $$(B^T \otimes A+D^T \otimes C) \...
user avatar
  • 191
3 votes
2 answers
330 views

Standard solution to semidefinite program [closed]

I have an optimization problem of the following form $$\text{minimize} \,\|Qa-b\|_2 \quad \text{ subject to } Q \succeq 0$$ where $a,b \in \mathbb{R}^n$ are given and the $n \times n$ square matrix ...
user avatar
2 votes
1 answer
173 views

Reconstruct matrix given all differences of neighbors

We have an unknown $m\times n$ matrix $X=(x_{ij})_{i=1,j=1}^{m,n}$. Assume we are given measurements of the differences $$x_{i,j+1}-x_{i,j}$$ and $$x_{i+1,j}-x_{i,j}$$ for all $(i,j)\in \{1,\...
user avatar
6 votes
1 answer
15k views

Gauss-Newton vs gradient descent vs Levenberg-Marquadt for least squared method

I need to clarify some idea I have in my mind about linear and non-linear regressions. Whatever I know about this topic comes from the book of Taylor "Introduction to error analysis": a set ...
user avatar
0 votes
0 answers
73 views

Why is ideal wavelet selection a least-squares estimate?

In their classic paper "Ideal spatial adaptation by wavelet shrinkage" (http://biomet.oxfordjournals.org/content/81/3/425.short?rss=1&ssource=mfr), Donoho and Johnstone make the following ...
user avatar
  • 111
1 vote
1 answer
109 views

MSE of measurable function is still conditional expectation

Motivation Then the usual stochastic filtering problem says that: $$ \operatorname{argmin}_{Z \in L^2(\mathscr{G}_t)}\,\mathbb{E}[(Y_t-Z_t)^2], $$ where $\mathscr{G}_t$ is the $\sigma$-algebra ...
user avatar
  • 5,008
1 vote
1 answer
181 views

Least squares problem with constrained solution [closed]

If $a_{m\times 1}$ and $Q_{m\times n}$ ($m<n $) are known, and we know every element of $b$ is between $[-1\ \ 1]$, how to determine $b$ to minimize $\|a+Qb\|_2$?
user avatar
2 votes
1 answer
271 views

How to force least squares solution matrix to be diagonal? [closed]

I have the following matrix equation $$AX=B$$ given $8 \times 3$ matrices $A$ and $B$. $X$ is a $3 \times 3$ diagonal matrix whose main diagonal contains the $3$ unknowns. Whenever I solve for $X$ ...
user avatar
7 votes
1 answer
183 views

Least-squares solution of systems of Sylvester equations

The Sylvester equation $AX+XB=C$ has been studied quite a lot and there are known algorithms for solving it. But has the situation where (an over-determined) system of equations $A_{i}X+XB_{i}=C_{i}$ ...
user avatar