Smoothing out Noisy Data

Question

I recently launched a rocket with an altimeter that is accurate to roughly 10 ft. The recorded data is in time increments of 0.05 sec per sample and a graph of altitude vs. time looks pretty much like it should when zoomed out.

The problem is when I try to calculate other values such as velocity or acceleration from the data, the accuracy of the measurements makes the calculated values pretty much worthless. What techniques can I use to smooth out the data so that I can calculate (or approximate) reasonable values for the velocity and acceleration? It is important that major events remain in place in time, most notably the 0 for for the first entry and the highest point during flight (2707).

The altitude data follows and is measured in ft above ground level. The first time would be 0.00 and each sample is 0.05 seconds after the previous sample. The spike at the beginning of the flight is due to a technical problem that occurred during liftoff and removing the spike would be best.

All help is greatly appreciated.

I don't think this question is appropiate for MO. Please read the FAQ: mathoverflow.net/faq — Alicia Garcia-Raboso, Commented Dec 10, 2009 at 20:30
@NickLarsen: Perhaps one reason that some people don't believe this is an appropriate question for this forum is because your question is extremely focused on a very specific event. If you had asked your question in a way that could apply to a more arbitrary setting, perhaps people might find it more appropriate. That said, I won't down vote as the question itself is not uninteresting, it does motivate other things in mathematics, and in general I don't like the overuse of the down vote. — MLevi, Commented Dec 10, 2009 at 23:09
There's a big difference to my mind between a "localized" question and a general question with an instance attached (which is what this is). The real question here is how to do numerical differentiation, which seems perfectly appropriate here (and isn't as easy as some people might think). — Darsh Ranjan, Commented Dec 11, 2009 at 0:56
@Darsh: you are right. I do feel that the question is posed too tightly tied to the instance though, in the sense that it may be difficult to see the forest because of the large tree in the front. I apologize if Nick felt dismissed by my earlier comment. Also, a suggestion: it might have been better to include a graph instead of the raw data. — Alicia Garcia-Raboso, Commented Dec 11, 2009 at 1:28

Dan Piponi · Accepted Answer · 2009-12-10 23:26:17Z

6

If you have data for the whole flight available to you then a good approach is Kalman smoothing. If you want estimates during the flight you want Kalman filtering. Seems like you're interested in the former. The difference is that Kalman smoothing uses data from the entire flight to estimate values at each point in time while Kalman filtering only uses the "past" to make its estimates. Some googling found a readable looking paper on both Kalman filtering and smoothing. But you need to tune it with an estimate of the properties of the noise from your sensors. I think that in the real world engineers often guess these values. (Or maybe just the ones I know...)

answered Dec 10, 2009 at 23:26

Dan Piponi

8,2715 gold badges64 silver badges92 bronze badges

$\begingroup$ This looks the most promising, and has the ability to preserve the 'known' events (marked correct for this reason). I also like the ability to use this in flight as the rocket may one day have a reason to know its flight properties in flight. $\endgroup$
– NickLarsen
Commented Dec 11, 2009 at 16:30
$\begingroup$ Kalman filtering and smoothing is also very good at combining data from multiple sources eg. altimeter+GPS+accelerometer. $\endgroup$
– Dan Piponi
Commented Dec 11, 2009 at 20:00

Add a comment |

Alekk · Accepted Answer · 2009-12-10 22:08:58Z

3

have you heard of the Kalman filter: that is a very popular approach for this kind of problems (eg:satellite tracking) where you just have noisy observations, and a known dynamics (Newton's laws of motion + the different characteristics of your rocket). This is also very easily implemented - see wikipedia

answered Dec 10, 2009 at 22:08

community wiki

Alekk

1

$\begingroup$ Kalman filters are usually what is used when you need at estimate right now of what is happening now, ie. you don't have any measurements from the future. This means that a Kalman filter produces a value that is a function of data points up to the present point. But if you have all the data available to you, like the example given above, then you can use measurements from time t'>t to improve your estimate of what happened at time t. $\endgroup$
– Dan Piponi
Commented Dec 10, 2009 at 23:13
$\begingroup$ Thanks sigfpe: this is actually what I meant, but I always use the words smoother/filter interchangeably, which is a bad habit (and wrong). Btw, the Kalman smoother should give reasonable results for your data since it is pretty stable if you get your dynamics right. And this is nothing more than a quick way (recursive) to compute the conditional expectation E[position,velocity|noisy observation] without inverting a huge matrix, if you assume that everything is Gaussian. $\endgroup$
– Alekk
Commented Dec 11, 2009 at 1:53

Add a comment |

Steve Huntsman · Accepted Answer · 2009-12-11 00:28:52Z

2

Convolve the data (simplest thing is to use a box) and renormalize. Use the conv command in MATLAB.

EDIT: You might also consider splines. Keep the dicontinuity of higher derivatives in mind if you do this.

edited Dec 11, 2009 at 0:28

answered Dec 10, 2009 at 20:24

Steve Huntsman

15.4k7 gold badges75 silver badges130 bronze badges

Add a comment |

alex · Accepted Answer · 2009-12-11 01:23:10Z

2

http://math.lanl.gov/Research/Highlights/PDF/tvdiff.pdf

This paper

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.102.6671&rep=rep1&type=pdf

contains a brief survey of the literature on stable numerical differentiation with refernces.

answered Dec 11, 2009 at 1:23

alex

9764 gold badges14 silver badges26 bronze badges

$\begingroup$ Thanks a lot for the linked papers. I'll have a good amount of reading to do this weekend. $\endgroup$
– NickLarsen
Commented Dec 11, 2009 at 15:28

Add a comment |

Qiaochu Yuan · Accepted Answer · 2009-12-10 21:16:49Z

0

The simplest strategy that comes to mind is just to replace each data point with the average of its closest neighbors. (This is a simple kind of convolution, but it might be good enough that you don't have to do anything more complicated. Apparently lots of computer algebra programs have convolution algorithms for exactly this purpose. Did you try Google or Wikipedia?)

answered Dec 10, 2009 at 21:16

Qiaochu Yuan

118k40 gold badges447 silver badges741 bronze badges

1

$\begingroup$ I actually did this before ever asking the question, and the numbers were a little choppy, so I wrote a program which repeatedly does this until the numbers smooth out enough to get decent data, but it also caused the curve to flatten at the top and bottom of the flight, so I think my values are off a bit from the real answers. Is there some kind of normalization that can be run after averaging the values to bring the curve back to its original height? $\endgroup$
– NickLarsen
Commented Dec 10, 2009 at 22:15
$\begingroup$ Also, thanks for the wiki link, I didn't find that in my search. $\endgroup$
– NickLarsen
Commented Dec 10, 2009 at 22:16
$\begingroup$ When you switch it with the average (Say a small Fejer kernel, which is certainly better than average) you get a_{n-1}+2 a_n +a_{n+1}, so velocity is now a_n+2a_{n+1}+a{n+2} - a_{n-1}-2a_n-a_{n+1} = -a_{n-1}-a_n+a_{n+1}+a_{n+2}, and acceleration is a_{n+2}-a_{n-2}. Doesn't make much sense. If you don't have a noise model, then smoothing out just brings some extra noise in. $\endgroup$
– David Lehavi
Commented Dec 11, 2009 at 9:26

Add a comment |

user14717 · Accepted Answer · 2019-02-16 21:12:41Z

0

A paper on this topic question was recently published: Discrete Lanczos Derivatives of Noisy Data. I found it to work very well on real data. An implementation is given in Boost.Math.

answered Feb 16, 2019 at 21:12

user14717

2212 silver badges6 bronze badges

Add a comment |

Stack Exchange Network

Smoothing out Noisy Data

6 Answers 6

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
na.numerical-analysis
.

Smoothing out Noisy Data

6 Answers 6

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged na.numerical-analysis.

Related

Not the answer you're looking for? Browse other questions tagged
na.numerical-analysis
.