6
$\begingroup$

This is not actually a research question. It is more an exercise which I posed myself in mathematical/statistical modelling. I have some Whatsapp data of a chat with someone. I want to find a mathematical model to describe the data. I have manually cut the chat into meaningful conversation pieces. So far I have the following Ansatz: Let $t_{j,i}$ be the time at which something is said by Person A or Person B in the whatsapp-chat at conversation j. We have the following "waiting times": $0=t_{11}<t_{12}<\cdots<t_{1,a_1}<t_{2,1}<t_{2,2}<\cdots<t_{2,a_2}<\cdots<t_{n,1}<\cdots<t_{n,a_n}$ So we have $n$ "conversations" in this chat by two people. Now my modeling Ansatz is that we have between each conversation a pause $P_j$:

$t_{1,a_1}+P_1 = t_{2,1}$

$t_{2,a_2}+P_2 = t_{3,1}$

$\cdots$

$t_{n-1,a_{n-1}}+P_{n-1} = t_{n,1}$

I have verified with the Kolmogorov-Smirnov Test all my assumptions concerning distribution of variables. Now we have

$P_j \sim Exp(\lambda_P)$

$d_{j,i} = t_{j,i+1}-t_{j,i} \sim Exp(\lambda_d)$ "interarrival times"

$a_j \sim Pois(\lambda_a)$

Now one could think of this as a "nested Poisson process", by which I mean, that we have a Poisson Process which governs the distributions of the conversations, and in each conversation we have a homogeneous Poisson process. Two conversations might have different parameters.

Ok, so in reality we can not observe when one conversation ends and when it starts. So the question is, given the data $t_1 < \cdots < t_m$ is it possible to calibrate the above model to find out how many conversations there are in this chat and when a conversation ends / starts, or are there to many parameters in the model, which need to be estimated?

If it is of help: We also observe at each timestamp who is chatting (Person A / Person B).

We have

$t_{n,a_n} = \sum_{j=1}^n P_j + \sum_{j=1}^n\sum_{i=1}^{a_j-1}d_{j,i}$

From this I have computed the expected value and the variance of $t_{n,a_n}$:

$E(t_{n,a_n}) = n/\lambda_P + n(\lambda_a-1)/\lambda_d$

$Var(t_{n,a_n}) = n/\lambda_P^2 + n(\lambda_a-1)/\lambda_d^2$

Now the question is, given the data $t_1<\cdots<t_m$ how to estimate the parameters: $n, \lambda_P, \lambda_d, \lambda_a$?

EDIT: (by suggestion of Bjørn Kjos-Hanssen):

One idea, as suggested by Bjørn Kjos-Hanssen is to plot the differences (pauses) and then to cut them off at the mean of pauses:

diff-times

The number of times the pauses are above the mean, could be estimated as $n$ the number of conversations. So to make it more precise let $d_i = t_{i+1}-t_i$ $i=1,\cdots,m-1$ Then $\widehat{d} = 1/(m-1) \sum_{i=1}^{m-1} d_i$. Now let $n = $ number of times we have $d_i > \widehat{d}$. What assumptions should I make to justify this procedure?

Suppose, that the above procedure can distinguish between a conversation and a pause, then we have $E(m) = \sum_{i=1}^nE(a_i) = n \lambda_a$ hence we can estimate $\lambda_a$ as $\widehat{\lambda_a} = m / n$. On the other hand we can estimate $\lambda_P$ as $\widehat{\lambda_P} = \frac{1}{1/n \sum_{d_j>\widehat{d}}d_j}$

And the Ansatz

$t_m = n/\widehat{\lambda_P}+n(\widehat{\lambda_a}-1)/\widehat{\lambda_d}$

gives an estimate of $\widehat{\lambda_d}$ as:

$\widehat{\lambda_d} = \frac{m/n-1}{t_m/n-1/n \sum_{d_j>\widehat{d}}d_j}$

So in order to make this argumentation more valid, my question is: What assumptions should I make to justify the procedure above?

The data is:

conversation   time person
         1      0      A
         1      1      A
         1     34      B
         1     35      A
         1     36      B
         2   5585      B
         2   5586      B
         2   5911      A
         3   8837      B
         3   8838      A
         3   8839      B
         3   8840      B
         3   8841      B
         3   8850      A
         3   8851      A
         3   8870      A
         3   8947      B
         3   8948      B
         3   9592      A
         4  14406      B
         4  14430      A
         4  14435      B
         4  14443      B
         4  14446      A
         4  14447      B
         5  14857      B
         5  15834      B
         5  17125      A
         5  17162      B
         5  17163      A
         5  17165      B
         6  17251      A
         6  17253      A
         7  23330      B
         7  23999      A
         8  32968      A
         8  32969      A
         8  32970      B
         8  32971      B
         8  32972      B
         8  32973      B
         8  32988      B
         9  39365      A
         9  39742      B
         9  46310      A
         9  46330      B
         9  46331      A
         9  50791      A
         9  50866      B
         9  51368      A
         9  51429      B
         9  51441      A
         9  51459      B
         9  51461      A
         9  51462      B
         9  51467      A
         9  51468      A
        10  52890      A
        10  52891      B
        11  54825      B
        11  54830      A
        11  54831      A
        11  54842      A
        11  54843      B
        11  54844      A
        11  54859      B
        11  54860      A
        11  54861      A
        11  54863      B
        11  54865      A
        12  70562      A
        12  70566      B
        12  70568      A
        12  70570      A
        12  70571      A
        12  70572      B
        12  70586      A
        12  70587      B
        13  71609      B
        13  71611      A
        13  71613      B
        13  71617      A
        13  71618      B
        13  71619      A
        14  96595      A
        14  96625      A
        14  96626      A
        14  96627      A
        14  96632      B
        14  96633      B
        14  96634      A
        14  96635      A
        15  96755      B
        15  96782      A
        15  96787      A
        15  96792      B
        15  96794      A
        15  96867      A
        15  96869      B
       15  96870      B
       15  96871      A
       15  96873      B
       15  96905      A
       15  96911      A
       15  96921      B
       16 102817      A
       16 102940      B
$\endgroup$
  • $\begingroup$ It looks like you only plotted the 15 pauses between what you already decided were the conversations. I rather meant to plot the pauses between all messages, and see if you can clearly distinguish inter-conversation and intra-conversation pauses that way. $\endgroup$ – Bjørn Kjos-Hanssen Jun 14 '17 at 19:11
4
$\begingroup$

I think you'll want some further assumptions like $\lambda_j>\lambda$ (inter-conversation pauses are longer than intra-conversation pauses).

Also, it should simplify the model to let $\lambda_j=\lambda_1$ for all $j$.

Finally, to determine the number of conversations you could naively just plot all the pauses and look for bimodality in the histogram.

$\endgroup$
1
$\begingroup$

Actually I do not think it is a good idea to model this with nested Poisson processes. The major problem occurs at the very beginning of your analysis.

...I want to find a mathematical model to describe the data. I have manually cut the chat into meaningful conversation pieces.

Usually such a parsing process should be conducted using some sort of parser, whose mathematical description can be automas like [Brill]. When you tried to parse it by hand, it usually introduces unwanted bias that is not analyzable.

I have verified with the Kolmogorov-Smirnov Test all my assumptions concerning distribution of variables

When you try to test all your assumptions, did you carry out the KS test over all variables or just variables separately? I have to admit that this is the first time I saw KS test on a stochastic process model so I was curious how you carry out your test?

...is it possible to calibrate the above model to find out how many conversations there are in this chat and when a conversation ends / starts, or are there to many parameters in the model, which need to be estimated?

I think a more appropriate model is some sort of Bayesian model using Dirichlet process prior with the number of conversations as a variable; in that way you can use stick-breaking process/beta process with flexible choice of base measures.

And this nonparametric Bayesian model is capable of catching the change of break intervals of "switch of topics" better than estimating $\lambda$'s jointly by allocating new conversations to new urns with new base measure.

Also, this model will catch the rare events relatively well. Say if you got a long break after you sent a message to a lady; Nested Poisson process will generally rule out the possibility that the conversation is still going on, while the Dirichlet mixture model will still retain a relatively small possibility that the conversation is going on...one of "heavy-tail" phenomena [Hong et.al] that indicates the Poisson model is inappropriate.

Those empirical statistics clearly indicate the invalidity of Poisson process in mimicking the human dynamics in many real-life systems.[Hong et.al]

[Brill]Brill, Eric. "Automatic grammar induction and parsing free text: A transformation-based approach." Proceedings of the workshop on Human Language Technology. Association for Computational Linguistics, 1993.

[Hong et.al]Wei, Hong, et al. "Heavy-tailed statistics in short-message communication." Chinese Physics Letters 26.2 (2009): 028902.

$\endgroup$
  • $\begingroup$ Thank you for your answer. Well you have to manually cut the messages into meaningfull "conversations". So I don't think this can be automated. For your later suggestion: I am not familiar with Dirichlet process. Also my main intention was to have a measure on when to remind person A to write to person B after a "long" period of time. I would be interested to hear if you have a model worked out. $\endgroup$ – orgesleka Jun 21 '17 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.