Welcome to MathOverflow

Question

1

I came across this paper but the smooth approximation for the hinge loss function is wrong. Can someone guide me to the proper smooth approximation (using polynomials) of the function $h(x)=max(0,1-x)$ which is exact when $|x|>=\rho$, where $\rho$ can be made arbitrarily small?

Thanks

flag	asked Jan 7 2011 at 5:02 Bernard 63●3

	You might as well work with $max(0,-x)$, which makes your bound involving $\rho$ make sense. And then you can work with a fixed $\rho$ ( $= 1$, say) – David Roberts Jan 7 2011 at 5:35
	But I don't know if you can do it with polynomials. – David Roberts Jan 7 2011 at 5:35
	The paper that you have linked works only for UIUC; please fix the link so that others can see what paper you are referring to. – S. Sra Jan 7 2011 at 10:03
	Also, what about the approximation in: people.csail.mit.edu/jrennie/writing/… – S. Sra Jan 7 2011 at 10:06
	Sorry about the link. The paper can be viewed here: cs.uiuc.edu/homes/afarhad2/index_files/…. Actually, I found the proper coefficients for the order 4 polynomial. The paper has a typo on page 5. Instead of $\frac{3}{2}x$ it should be $\frac{1}{2}x$. The link for the smooth quadratic hinge function was a great help. Thanks Suvrit. – Bernard Jan 7 2011 at 11:59

show 2 more comments

S. Sra · Answer 1 · 2011-01-07 14:43:42Z UTC

Here are the details for those who might feel too lazy to chase the links in the comments above.

The Hinge loss for $x \in R$ is defined as $$H(x) = \max(0, 1-x)$$

Jason Rennie in his article "Smooth Hinge Classification" describes the following smooth version of the Hinge loss (a smoothed version was being sought because of discontinuity in the derivative at $x=1$).

Rennie defines (the definition seems natural enough that somebody might have also found a similar one; I will be happy to be corrected) the smoothed Hinge loss: $$H_s(x) = \begin{cases} \tfrac{1}{2}-x & x \le 0,\\ \tfrac{1}{2}(1-x)^2 & 0 < x < 1\\ 0 & x \ge 1 \end{cases}$$

This loss is smooth, and its derivative is continuous (verified trivially).

Rennie goes on to discuss a parametrized family of smooth Hinge-losses $H_s(x; \alpha)$. Additionally, several other variations are possible, depending on what numerical behavior seems more appropriate for an application.

smooth approximation of the hinge loss function

Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

1 Answer

You can accept an answer to one of your own questions by clicking the check mark next to it. This awards 15 reputation points to the person who answered and 2 reputation points to you.