Questions tagged [linear-regression]
The linear-regression tag has no usage guidance.
35
questions
-2
votes
0
answers
123
views
Is this regression problem solvable? [duplicate]
I have a random vector $\pmb{x}=(X_1,...,X_p)^T\in \mathbb{R}^p$, a symmetric matrix
$$\Theta = \left(\begin{matrix}0 & \theta_{12} & \theta_{13} & \cdots & \theta_{1p}\\
\theta_{12} &...
1
vote
0
answers
22
views
How to extract 'top k' multiple solutions from a quadratic optimization problem?
Imagine we are interested in the following problem:
$$
\min_{w} \left( w^T V w + \lambda \|w\| \right) \\
\text{s.t. } w^T R \geq c
$$
Where 𝑤 is an $N \times 1$ vector, $V$ is an $N \times N$ ...
0
votes
1
answer
61
views
Handling the $\ell^2$ norm of a matrix expression in a linear regression
I am reading a scientific article in which matrices are handled (which I do not use often). We consider a matrix $X\in\mathbb R^{n\times p}$ and a vector $y\in\mathbb R^n$. The authors show that the ...
0
votes
0
answers
88
views
Weighted least squares regression: Iterative modeling of variance
In chemical analysis, the instrument's signal are plotted as a function of chemical concentration. In general, higher the concentration higher is the response and the relationship is linear. At ...
0
votes
0
answers
28
views
Adjust X to strengthen the linearity to Y, in regression model
Assume that we have 2 series X and Y, and obvious we can fit a linear regression model and get all the statistics. I am seeking for some transformation / adjustment which will adjust the value of X, ...
2
votes
1
answer
119
views
Justification of the use of residual plot
$\DeclareMathOperator\Cov{Cov}$Backround of my Question
Let $Y$ be the response variable, $\mathbb{X}$ be the explanatory variables. The ultimate goal of prediction is finding a function $f^{*}$ that ...
1
vote
0
answers
61
views
Least squares regression with nonnegative error
I'm looking for algorithms to solve a special quadratic programming problem, but I don't know its name or related keywords. Can anyone give me some clues? The problem reads
\begin{equation}
{\min}_x \...
4
votes
1
answer
166
views
Least squares problem with left and right unknowns
For $i=1,...,n$, let $b_i$ be a scalar and $A_i$ be an $k\times l$ matrix. Is there a closed form solution for the following problem assuming $n>k+l$?
$$\min_{x\in \mathbb{R}^k ,y\in \mathbb{R}^l} \...
2
votes
0
answers
67
views
Under what conditions is the least-squares approximation bounded with the same Lipschitz gradient constants?
Let $f(x):\mathbb{R}^K\Longrightarrow \mathbb{R}^L$ denote a multivariate continuously differentiable function. All the partial derivatives of $f$ (all its Jacobian elements) are bounded from above ...
0
votes
0
answers
33
views
Normalizing a parameter in a regression
I am thinking about the possibility of making a parameter in my regression, let's say the $\lambda$ in a ridge regression, somehow, inside a range : $\lambda \in [0,1]$. Do you have any ideas how I ...
1
vote
0
answers
69
views
Fitting a model
I have a function expressed as the ratio of two exponential series with certain parameters
$$\frac{\sum\limits_{j=1}^{i-1} \frac{e^{-ar_jt}}{\prod_{l=1\\l \ne j}^{i-1} (b^j-b^l)}}{\sum\limits_{j=1}^{i}...
1
vote
0
answers
48
views
semi-parametric regression
Suppose the observation $(X_1, Y_1), \ldots, (X_n, Y_n)$ satisfies the following semi-parametric model
$$Y_t = m(X_t, \alpha) + \sigma(X_t, \beta) U_t,$$
where $U_t$ is independent with $X_t$ with ...
0
votes
1
answer
38
views
The nonparametric estimation in generalized regression model
Let $Y_t \in \mathbb{R}$ be a response variable and $X_t$ a $d$-dimensional explanatory variable. Assume we observe the process that $(X_1, Y_1), \cdots, (X_n, Y_n)$.
\begin{equation}
Y_{t} = \mu(...
1
vote
1
answer
487
views
Hanson-Wright inequality with random matrix
I'm interested in bounding the tail probabilities of a quadratic form
$x^t A x$ where $x\in \mathbb{R}^n$ is a sub-Gaussian vector with independent entries. $A\in \mathbb{R}^{n\times n}$ is a matrix. ...
1
vote
3
answers
86
views
RKHS/non-parametric regression with missing response values
I am interested in doing RKHS regression with missing response variables.
Given input-output pairs $(x_i,y_i)$, I want to estimate a function $f(\cdot)$ as follows
\begin{equation}f(x)\approx u(x)=\...
1
vote
0
answers
128
views
How to compress variables in a linear regression
I have a large linear regression where all the independent variables are logical (ie TRUE/FALSE) and sparse. The data has roughly 10,000 variables and 10 million observations, on average around 20 ...
1
vote
1
answer
119
views
Does the intercept converge if we fit a best fit line to points with prime coordinates?
A few months ago I asked this question on Mathematics Stack Exchange but it has received little attention. Perhaps the question is more applicable here.
Let $p_k$ denote the $k$th prime such that $...
8
votes
3
answers
341
views
Regularized linear vs. RKHS-regression
I'm studying the difference between regularization in RKHS regression and linear regression, but I have a hard time grasping the crucial difference between the two.
Given input-output pairs $(x_i,y_i)...
0
votes
2
answers
350
views
Unique parameterization of size MxN matrices of rank k
Any rank k matrix $Y\in\mathbb{R}^{m\times n}$ can be written as:
$$ Y = UV'$$
Where $U\in \mathbb{R}^{m\times k}, V\in \mathbb{R}^{n\times k}$. This factorization is not unique since for any ...
4
votes
2
answers
1k
views
Given the joint probability distributions of $X$ and $Y$ for $Y = R\,X+C$, find the probability distributions of $R$ and $C$
Let $R$, $C$, and $X$ be independent random variables defined on $(0,\infty)$ and
$$Y=\underbrace{R\, X}_{Z}+C.$$
We are given the joint probability distribution of $X$ and $Y$, $P_{XY}(x,y)$ and ...
3
votes
1
answer
121
views
Why to multiply the penalty by $n$ in the penalized least squares and likelihood?
In the SCAD paper by Fan and Li (2001), there exist two forms of penalized least squares as follows:
$$\frac{1}{2}\left \| y-X\beta \right \|^2+\lambda \sum_{j=1}^{d}p_j (\left | \beta _j \right |),$$
...
2
votes
0
answers
46
views
increasing inter-class distances results in decreasing linear regression error
Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set.
Define $\mathbf{...
2
votes
1
answer
56
views
matrix regression under side conditions
I want to solve the folowing problem B*M=V, where B is the unknown of size 3x3, M of size 3xN and V of size 3xN. The difficulty is, that B has to be unitary.
N is in the range of 500. All matrices ...
1
vote
0
answers
92
views
A different objective function in liner regression analysis
I'm an undergraduate student who is green in statistics. I have a problem in the chose of objective function when estimating the parameters.
Let $Y = \beta^TX + \epsilon $ be the standard liner ...
9
votes
1
answer
19k
views
Gauss-Newton vs gradient descent vs Levenberg-Marquadt for least squared method
I need to clarify some idea I have in my mind about linear and non-linear regressions. Whatever I know about this topic comes from the book of Taylor "Introduction to error analysis": a set ...
1
vote
0
answers
64
views
Posterior consistency of non linear model
This is possibly a reference request. Let $G$ : $\mathbb{R}^p \to \mathbb{R}^q$ be a continuous injective/bijective function. Let $\mu$(we may also assume this to be a non degenerate Gaussian) be ...
2
votes
0
answers
147
views
Derivation of gradient of SSE in Geodesic Regression
On page 79 (or page 5) of this this paper the gradient of the SSE of the Geodesic model is described explicitly. My question is how are these equitations derived in detail; where can I find the ...
3
votes
3
answers
2k
views
When does a Vandermonde-like matrix have full rank
I have a matrix which is similar to Vandermonde matrix except that the entries are monomials of degree $d$ polynomial in 2 variables. Each row has the following form:
$X_{i}= [1, x_{i}, y_{i}, x_{i}^...
2
votes
0
answers
314
views
Is there an efficient way to compute the "complete subset regression"?
Background: Let $X \in \mathbb{R}^{N\times K}$ and $y \in \mathbb{R}^{N\times 1}$ be data for a regression problem. The aim is to find $\beta \in \mathbb{R}^{K\times 1}$ such that $X\beta \approx y$ ...
1
vote
1
answer
167
views
Checking the intersection of two sets
Let $E\subset{\mathbb R}^n$ be a set of the type $I_1\times \dots \times I_n$, where $I_k$ are real intervals, and $X$ be and $n\times p$ real matrix. Suppose also that $rank(X)=p$ and $n>p$. Is ...
-1
votes
1
answer
648
views
Fitting a quadratic using regression when the y-intercept needs to be 0 [closed]
I'm trying to fit a quadratic $a_0 + a_1x + a_2x^2$ by Polynomial Regression:
$$
\begin{pmatrix}
n & \Sigma x_i & \Sigma x_i\\
\Sigma x_i & \Sigma x_i^2 & \Sigma x_i^3\\
\Sigma ...
1
vote
0
answers
273
views
How to find all least-square solutions [closed]
I was looking at numpy's lstsq to find a least squares solution of an equation system when the following occurred to me:
Given the points (0,0), (3,4), (4,3), if I ...
4
votes
1
answer
211
views
Regression with correlation structure
I have a theoretical question about regression models.
Let's say I measured multiple responses from $n$ subjects and these responses are correlated with each other. For example, let's say I measured ...
1
vote
1
answer
3k
views
Minimizing sum of absolute deviations
Suppose we want to find coefficients $b$ in $\underset{b}{\operatorname{argmin}} \displaystyle\sum\limits_{i=1}^n | y_{i}-b_{1}x_{i}-b_{0}\mid$.
If we rewrite this problem in terms of linear ...
10
votes
0
answers
300
views
Testing contrasts in statistics: Is this provably a hard problem, or not?
Scheffé's method for identifying statistically significant contrasts is widely known. A contrast among the means $\mu_i$, $i=1,\ldots,r$ of $r$ populations is a linear combination $\sum_{i=1}^r c_i \...