MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

## Smoothing out Noisy Data

I recently launched a rocket with an altimeter that is accurate to roughly 10 ft. The recorded data is in time increments of 0.05 sec per sample and a graph of altitude vs. time looks pretty much like it should when zoomed out.

The problem is when I try to calculate other values such as velocity or acceleration from the data, the accuracy of the measurements makes the calculated values pretty much worthless. What techniques can I use to smooth out the data so that I can calculate (or approximate) reasonable values for the velocity and acceleration? It is important that major events remain in place in time, most notably the 0 for for the first entry and the highest point during flight (2707).

The altitude data follows and is measured in ft above ground level. The first time would be 0.00 and each sample is 0.05 seconds after the previous sample. The spike at the beginning of the flight is due to a technical problem that occurred during liftoff and removing the spike would be best.

All help is greatly appreciated.

00000
00000
00000
00076
00229
00095
00057
00038
00048
00057
00057
00076
00086
00095
00105
00114
00124
00133
00152
00152
00171
00190
00200
00219
00229
00248
00267
00277
00286
00305
00334
00343
00363
00363
00382
00382
00401
00420
00440
00459
00469
00488
00517
00527
00546
00565
00585
00613
00633
00652
00671
00691
00710
00729
00759
00778
00798
00817
00837
00856
00885
00904
00924
00944
00963
00983
01002
01022
01041
01061
01080
01100
01120
01139
01149
01169
01179
01198
01218
01238
01257
01277
01297
01317
01327
01346
01356
01376
01396
01415
01425
01445
01465
01475
01495
01515
01525
01545
01554
01574
01594
01614
01614
01634
01654
01664
01674
01694
01714
01724
01734
01754
01764
01774
01794
01804
01814
01834
01844
01854
01874
01884
01894
01914
01924
01934
01954
01954
01975
01995
01995
02015
02015
02035
02045
02055
02075
02075
02096
02096
02116
02126
02136
02146
02156
02167
02177
02187
02197
02207
02217
02227
02237
02237
02258
02268
02278
02278
02298
02298
02319
02319
02319
02339
02349
02359
02359
02370
02380
02380
02400
02400
01914
02319
02420
02482
02523
02461
02502
02543
02564
02595
02625
02666
02707
02646
02605
02605
02584
02574
02543
02543
02543
02543
02543
02543
02554
02543
02554
02554
02554
02554
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02543
02533
02543
02543
02543
02543
02543
02543
02543
02543
02533
02523
02523
02523
02523
02523
02523
02523
02523
02543
02523
02523
02523
02523
02523
02523
02523
02523
02513
02513
02502
02502
02492
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02482
02472
02472
02472
02461
02461
02461
02461
02451
02451
02451
02461
02461
02451
02451
02451
02451
02451
02451
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02441
02431
02441
02431
02441
02431
02420
02431
02420
02420
02420
02420
02420
02420
02420
02420
02420
02420
02420
02420
02410
02420
02410
02410
02410
02410
02400
02400
02410
02400
02400
02400
02400
02400
02400
02400
02400
02400
02400
02400
02400
02390
02390
02390
02380
02380
02380
02380
02380
02380
02380
02380
02380
02380
02380
02380
02380
02370
02370
02380
02370
02359
02359
02359
02359
02359
02359
02359
02359
02359
02359
02359
02359
02359
02359
02349
02349
02349
02349
02349
02339
02339
02339
02339
02339
02339
02339
02339
02339
02339
02339
02339
02339

-
Down vote without explanation? – NickLarsen Dec 10 2009 at 20:29
I don't think this question is appropiate for MO. Please read the FAQ: mathoverflow.net/faq – Alberto García-Raboso Dec 10 2009 at 20:30
@NickLarsen: Perhaps one reason that some people don't believe this is an appropriate question for this forum is because your question is extremely focused on a very specific event. If you had asked your question in a way that could apply to a more arbitrary setting, perhaps people might find it more appropriate. That said, I won't down vote as the question itself is not uninteresting, it does motivate other things in mathematics, and in general I don't like the overuse of the down vote. – MLevi Dec 10 2009 at 23:09
There's a big difference to my mind between a "localized" question and a general question with an instance attached (which is what this is). The real question here is how to do numerical differentiation, which seems perfectly appropriate here (and isn't as easy as some people might think). – Darsh Ranjan Dec 11 2009 at 0:56
@Darsh: you are right. I do feel that the question is posed too tightly tied to the instance though, in the sense that it may be difficult to see the forest because of the large tree in the front. I apologize if Nick felt dismissed by my earlier comment. Also, a suggestion: it might have been better to include a graph instead of the raw data. – Alberto García-Raboso Dec 11 2009 at 1:28

If you have data for the whole flight available to you then a good approach is Kalman smoothing. If you want estimates during the flight you want Kalman filtering. Seems like you're interested in the former. The difference is that Kalman smoothing uses data from the entire flight to estimate values at each point in time while Kalman filtering only uses the "past" to make its estimates. Some googling found a readable looking paper on both Kalman filtering and smoothing. But you need to tune it with an estimate of the properties of the noise from your sensors. I think that in the real world engineers often guess these values. (Or maybe just the ones I know...)

-
 This looks the most promising, and has the ability to preserve the 'known' events (marked correct for this reason). I also like the ability to use this in flight as the rocket may one day have a reason to know its flight properties in flight. – NickLarsen Dec 11 2009 at 16:30 Kalman filtering and smoothing is also very good at combining data from multiple sources eg. altimeter+GPS+accelerometer. – Dan Piponi Dec 11 2009 at 20:00

Convolve the data (simplest thing is to use a box) and renormalize. Use the conv command in MATLAB.

EDIT: You might also consider splines. Keep the dicontinuity of higher derivatives in mind if you do this.

-

The simplest strategy that comes to mind is just to replace each data point with the average of its closest neighbors. (This is a simple kind of convolution, but it might be good enough that you don't have to do anything more complicated. Apparently lots of computer algebra programs have convolution algorithms for exactly this purpose. Did you try Google or Wikipedia?)

-
I actually did this before ever asking the question, and the numbers were a little choppy, so I wrote a program which repeatedly does this until the numbers smooth out enough to get decent data, but it also caused the curve to flatten at the top and bottom of the flight, so I think my values are off a bit from the real answers. Is there some kind of normalization that can be run after averaging the values to bring the curve back to its original height? – NickLarsen Dec 10 2009 at 22:15
Also, thanks for the wiki link, I didn't find that in my search. – NickLarsen Dec 10 2009 at 22:16
When you switch it with the average (Say a small Fejer kernel, which is certainly better than average) you get a_{n-1}+2 a_n +a_{n+1}, so velocity is now a_n+2a_{n+1}+a{n+2} - a_{n-1}-2a_n-a_{n+1} = -a_{n-1}-a_n+a_{n+1}+a_{n+2}, and acceleration is a_{n+2}-a_{n-2}. Doesn't make much sense. If you don't have a noise model, then smoothing out just brings some extra noise in. – David Lehavi Dec 11 2009 at 9:26

have you heard of the Kalman filter: that is a very popular approach for this kind of problems (eg:satellite tracking) where you just have noisy observations, and a known dynamics (Newton's laws of motion + the different characteristics of your rocket). This is also very easily implemented - see wikipedia

-
Kalman filters are usually what is used when you need at estimate right now of what is happening now, ie. you don't have any measurements from the future. This means that a Kalman filter produces a value that is a function of data points up to the present point. But if you have all the data available to you, like the example given above, then you can use measurements from time t'>t to improve your estimate of what happened at time t. – Dan Piponi Dec 10 2009 at 23:13
Thanks sigfpe: this is actually what I meant, but I always use the words smoother/filter interchangeably, which is a bad habit (and wrong). Btw, the Kalman smoother should give reasonable results for your data since it is pretty stable if you get your dynamics right. And this is nothing more than a quick way (recursive) to compute the conditional expectation E[position,velocity|noisy observation] without inverting a huge matrix, if you assume that everything is Gaussian. – Alekk Dec 11 2009 at 1:53

http://math.lanl.gov/Research/Highlights/PDF/tvdiff.pdf

This paper