Questions tagged [least-squares]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

Nonlinear least-squares parameter estimation of a vector-valued function

I'm given a function $f\, (x; \vec{\beta})$, non-linear, where ${x} \in \mathbb{R}$ and $\vec{\beta} \in \mathbb{R}^k$. It models an experiment of a setup characterized by the parameters $\vec{\beta}$....
19
votes
4answers
999 views

Is the pseudoinverse the same as least squares with regularization?

Given a linear system $Ax=b$, the pseudoinverse of $A$ is found as the matrix $A^+$ such that $x=A^+ b$ where $x$ solves the least squares problem $\min \| Ax - b \|^2 $ and $x \perp \mathcal{N}(A)$. ...
2
votes
1answer
142 views

Concentration inequality for norm of solution to nonlinear least-squares problem

Define the piecewise-linear function $\psi(t):=\max(t,0)$ for all $t \in \mathbb R$. Let $d,n,k \to \infty$ at the same rate (i.e $n \asymp k \asymp d$). Let $y_1,\ldots,y_n \in \{-1,1\}$ uniformly ...
1
vote
2answers
114 views

Robust estimation of $Ax=b$

Problem setting : $ \underset{x}{\text{min}} \|Ax-b\|$, where $A \in \mathcal{R}^{m \times n}, m\gg n $, full rank. L1 loss is used for robust estimation using IRLS. The corresponding equation to ...
1
vote
1answer
76 views

Understanding the Time Delay of Arrival trilateration algorithm

I'm trying to algorithmically solve the Time Delay of Arrival problem as part of some mathematics research. The problem is as follows: Given the location of three receivers in a plane (A, B, and C), ...
0
votes
0answers
41 views

unusual least square solution in SINDy algorithm

In the SINDy algorithm, the equation $\dot{X} \approx \Theta\Xi$ has to be solved for $\Xi$. $\dot{X}$ and $\Theta$ are known numerical matricies. The intuitive solution to this problem would be to ...
1
vote
2answers
63 views

Non-parametric regression and curvature

Given a finite set of points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ in the plane, Linear Regression tells us how to find the straight line "$y=a+bx$" best approximating the given points, in the ...
1
vote
3answers
70 views

RKHS/non-parametric regression with missing response values

I am interested in doing RKHS regression with missing response variables. Given input-output pairs $(x_i,y_i)$, I want to estimate a function $f(\cdot)$ as follows \begin{equation}f(x)\approx u(x)=\...
2
votes
1answer
288 views

Minimise $\sum_i \begin{Vmatrix}\boldsymbol{x}_i \\ \boldsymbol{y}_i\end{Vmatrix}$

Consider column vectors $\boldsymbol{z}_i$, $\quad i=1,\dots,n$. Each $\boldsymbol{z}_i$ has $j$ elements and can be expressed as $\boldsymbol{z}_i = \begin{bmatrix} \boldsymbol{x}_i \\ \boldsymbol{y}...
5
votes
2answers
361 views

Symmetric linear least-squares solution

Given tall matrices $A$ and $Y$ and the following overdetermined linear system in square matrix $X$ $$AX=Y$$ is there an explicit formula for the least-squares solution if $X$ is constrained to be ...
0
votes
1answer
313 views

How to determine the damping factor in Levenberg-Marquardt?

From the algorithm, we can see that it tries different damping factor until it gets a good one by the error. Is the damping factor related to the eigenvalues of the Hessian matrix?
2
votes
0answers
19 views

Difference Between Total Least Squares Plane and Plane Orthogonal to Principal Axis of Inertia Tensor

Given a finite set $P$ of points in $\mathbb{E}^3$ , one can calculate an approximating plane either as the solution of a Total Least Squares problem or by interpreting the problem physically, ...
1
vote
1answer
85 views

Minimization Proof of Conditioning on Gaussian is Gaussian

It is well known that $E[X|X+Y]$ is Gaussian if both $X$ and $Y$ are, and the result can be derived using standard density arguments. However, how can one prove it by only resulting to optimization ...
1
vote
0answers
98 views

Choice of residual function for least squares error minimization

Good morning, I have the a set of data $(\sigma,D,\alpha_0)_i$, $i=1...n$ data. I want to determine two parameters $K_{IC}$, $C_f$ in the basic equation given as $K_{IC} = \sigma \sqrt{D} k_0(\...
2
votes
2answers
148 views

In practice, what's the fastest method to find a least square solution rather than using SVD decompostion?

I'm working on a real-time implementation of Lucas-Kanade for optical flow. However, the SVD decomposition to do achieve the least square method to reduce the error seems to take too much time. A ...
8
votes
3answers
262 views

Regularized linear vs. RKHS-regression

I'm studying the difference between regularization in RKHS regression and linear regression, but I have a hard time grasping the crucial difference between the two. Given input-output pairs $(x_i,y_i)...
5
votes
1answer
192 views

Rank-constrained least-squares solution of the Sylvester matrix equation

For the Sylvester matrix equation $AX+XB=C$, I want to find the least-squares solution $X$ via $$\begin{array}{ll} \text{minimize} & \| AX + XB - C \|_{\text{F}}^2\\ \text{subject to} & \mbox{...
4
votes
1answer
266 views

Least square solution to $AXB+CXD=E$

I am trying to find the least-squares solution $X$ of the following matrix equation $$AXB+CXD=E$$ Of course, I know that this equation can be written in the form $$(B^T \otimes A+D^T \otimes C) \...
3
votes
2answers
310 views

Standard solution to semidefinite program [closed]

I have an optimization problem of the following form $$\text{minimize} \,\|Qa-b\|_2 \quad \text{ subject to } Q \succeq 0$$ where $a,b \in \mathbb{R}^n$ are given and the $n \times n$ square matrix ...
2
votes
1answer
171 views

Reconstruct matrix given all differences of neighbors

We have an unknown $m\times n$ matrix $X=(x_{ij})_{i=1,j=1}^{m,n}$. Assume we are given measurements of the differences $$x_{i,j+1}-x_{i,j}$$ and $$x_{i+1,j}-x_{i,j}$$ for all $(i,j)\in \{1,\...
6
votes
1answer
13k views

Gauss-Newton vs Gradient Descent vs Levenberg-Marquadt for least squared method

I need to clarify some idea I have in my mind about linear and non-linear regressions. Whatever I now about this topic comes from the book of Taylor "Introduction to error analysis": a set of ...
0
votes
0answers
68 views

Why is ideal wavelet selection a least-squares estimate?

In their classic paper "Ideal spatial adaptation by wavelet shrinkage" (http://biomet.oxfordjournals.org/content/81/3/425.short?rss=1&ssource=mfr), Donoho and Johnstone make the following ...
1
vote
1answer
103 views

MSE of measurable function is still conditional expectation

Motivation Then the usual stochastic filtering problem says that: $$ \operatorname{argmin}_{Z \in L^2(\mathscr{G}_t)}\,\mathbb{E}[(Y_t-Z_t)^2], $$ where $\mathscr{G}_t$ is the $\sigma$-algebra ...
1
vote
1answer
172 views

Least squares problem with constrained solution [closed]

If $a_{m\times 1}$ and $Q_{m\times n}$ ($m<n $) are known, and we know every element of $b$ is between $[-1\ \ 1]$, how to determine $b$ to minimize $\|a+Qb\|_2$?
2
votes
1answer
228 views

How to force least squares solution matrix to be diagonal? [closed]

I have the following matrix equation $$AX=B$$ given $8 \times 3$ matrices $A$ and $B$. $X$ is a $3 \times 3$ diagonal matrix whose main diagonal contains the $3$ unknowns. Whenever I solve for $X$ ...
7
votes
1answer
167 views

Least-squares solution of systems of Sylvester equations

The Sylvester equation $AX+XB=C$ has been studied quite a lot and there are known algorithms for solving it. But has the situation where (an over-determined) system of equations $A_{i}X+XB_{i}=C_{i}$ ...