You may also be interested in knowing about Exponentiated Gradient +/- (EG+/-), described by Warmuth and Kivinen on page 15 here (https://users.soe.ucsc.edu/~manfred/pubs/J36.pdf).

In EG+/- you maintain two sets of weights, one representing the positive side of the weights and a second representing the negative side of the weight. In use you combine the positive and negative weights, but maintain the separately for the updates.

The update is described quite succinctly there on page 15, the intuition is that the update method generates a set of values a little above or below 1 based on the gradient which is multiplied into the weights to draw them up or down as appropriate.

I've applied the update method in a fully connected feedforward network in this code: http://github.com/davidparks21/experimental_neural_network_matlab

At present I've demonstrated, and a few other papers on EG applied to neural networks have also found, that it tends to out perform standard additive gradient descent updates in the presence of heavy amounts of noise, however additive GD tends to out perform EG+- on noise-free datasets, MNIST as an example as such. The data sets I've demonstrated the improvement on have had noise added, I haven't yet demonstrate the same on a real world data set (thought it's on my todo list).