In my research on neural networks and learning theory I have recently come across the following problem dealing with gradient descent:

We consider a given column vector $ x=[x_1,x_2,...,x_{d}]^T \in \mathbb{R}^{d} $ and we define a general column vector $ w = [w_1,w_2,...,w_{d}]^T \in \mathbb{R}^{d} $, and we define the following function of w only as x is given $f(w)=\max{\{w^Tx,0\}}$ and we define the loss function in terms of the weight $ MSE(w) = (y-f(w))^2 $ where y is a constant know to be either 0 or 1. We denote by $ t \in \mathbb{N} $ the index of iteration and we optimize the weights via gradient descent: $ w^{(t+1)}=w^{(t)}-\frac{1}{t}(\nabla MSE(W))|_{W=W^{(t)}} $ where the gradient of the MSE function is with respect to the w vector which is the only variable in the problem, and we know of course that these iterations converge to a critical point where the gradient is defined and zero, or where the gradient is undefined when the max function is not differentiable where $w^Tx=0$

I was wondering if someone could please tell me if there is some way to determine via the initial setting of the $w$ vector if we converge to a gradient-zero point or a non-differentiable point? Is there a criterion of sorts on the initial choosing of $w$ to help us determine which regions in $ \mathbb{R}^{d} $ converge to which sort of points? Maybe starting with scalar w values may simplify the problem at hand? I thank all helpers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.