1
$\begingroup$

Hi,

I'm trying to study a list of vectors. The vectors represent particular states of an organic system over time. The vectors grow in size (relatively fast) and (absolute) value of entries (relatively slowly) over time.

A sequence of these vectors displays the characteristic that adjacent vectors are quite similar. So the sequence of 'first derivatives' of these vectors display mostly 0s and small numbers.

However, once I 'differentiate' this sequence this 'similarity property' (my one piece of evidence for a persistent underlying pattern) disappears.

For e.g.

The list of vectors may be

[5, -2, 2, -3, 4, -1, 3, -2 ]

[4, -3, 3, -2, 3, -1, 3, -4 ]

[6, -3, 2, -1, 3, -1, 3, -3, 2, -2 ]

So the sequence of first derivatives is

[4-5, -3--2, 3-2, -2--3, 3-4, -1--1, 3-3, -4--2 ] =

[ -1, -1, 1, 1, -1, 0, 0, -2 ]

[2, 0, -1, 1, 0, 0, 0, 1, 2, -2 ]

The sequence of second derivatives (only 1 ) is

[3, 1, -2, 0, 1, 0, 0, 3 2, -2 ]

The thing is I am hoping to find a pattern where ( like polynomials ) when I keep 'differentiating' the vectors, I find smaller and smaller and more homogenous vectors that result. This however does not happen.

A typical sequence of 1000 vectors differentiated to the 100th degree, displays a sequence of (1ooth differentials) vectors with entries of order 10^29.

What I am wondering is -- What am I dealing with?

The systems states are very closely related at the first level. My attempts to use this 'differentiation' to discover an underlying structure (or 'function' as one can do with polynomials) is not working.

So I wonder

-- What does it mean about the system that these 'derivatives' although very similar at first, from then on keep growing?

-- What other options do I have for studying, searching for, finding patterns that could explain / model the way the vectors change over time?

Any fresh ideas are welcome in this case.

Best,

Cris

Improve this question
$\endgroup$
2
  • $\begingroup$ The analytic rule is: integration is a smoothing operator, and differentiation (almost by definition) removes one degree of differentiability. When working with discrete data with noise, you need to smooth out the noise before differentiating. The basic linear method is to choose a wider constellation (i.e., use more than 2 steps of information) for your derivative. More sophisticated methods include Kalman filtering: en.wikipedia.org/wiki/Kalman_filter $\endgroup$
    – S. Carnahan ♦
    Mar 4, 2011 at 13:24
  • $\begingroup$ wow that is great. thank you for the new ideas and ways to think about this. i'm sure i can use this to investigate further. $\endgroup$
    – Cris Stringfellow
    Mar 6, 2011 at 4:08

1 Answer 1

Reset to default
2
$\begingroup$

Just some notes (I am not an expert on this, and a different unknown):

Phenomena like this are not unique to a discrete model.

Consider a function, $f$ on say $(c,1)$ with $c>0$ but small whose first derivative is

$x sin (1/x)$ this is small for small $x$.

But, if you differentiate again you get $sin(1/x) - x^{-1} cos(1/x)$ and this is not at all small for all small $x$.

To put it differently, if your data is determined, say, by some linear function plus noise, you detect this pattern by 'differentiating' (forming the difference). But by doing so you also 'remove' the pattern, so what remains after 'differentiating' is the noise only, and there is nothing more to be found.

There is a lot to be said on problems like this a key-word is 'time-series analysis'.

While there are certainly people on this site more knowledgeable than me, you might get even better results on that site https://stats.stackexchange.com/ , which is similar to this one but focused on statistics (dislaimer: I never used that other site).

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.