Suppose I have some random variable X with probability distribution P(.;theta). Suppose I have a single sample x from this distribution.

Does it make any sense to ask for the derivative of x with respect to theta?

I believe it does not, but am having trouble convincing some of my colleagues.

  • $\begingroup$ in the end, what did you end up telling them to convince them? $\endgroup$ – Pinocchio Aug 19 '16 at 2:08
  • $\begingroup$ actually Ian, I don't understand your question. Are you taking the derivative of P with respect to x (the sample) or $\theta$? I can't conceive how it makes sense to take the derivative wrt to x if x is a sample. If its a sample, then its observed and fixed, so taking the derivative wrt to it is simply zero. Or am I missing something? $\endgroup$ – Pinocchio Aug 19 '16 at 23:11

It does not because $x$ is just some element of your state space. You could conceivably choose $x$ as your sample point for any or all of the $\theta$'s. So what it would make sense to differentiate with respect to $\theta$ is $$\mathbb{P}_{\theta}(X=x)$$ as long as you have a discrete distribution. For a continuous distribution you could substitute $x$ with some interval.

  • $\begingroup$ I read your answer but I didn't understand it. Why is my counter example wrong. Consider $P_{\theta}(X = x) = \theta^x (1 - \theta)^{1-x}$ and let $x=1$ be the sample observed. Then $P_{\theta}(X = 1) = \theta $. One can easily take the derivative of that function, it has $\theta$ as a variable, its derivative is simply 1. Why is that not correct? $\endgroup$ – Pinocchio Aug 19 '16 at 2:12
  • $\begingroup$ @Pinocchio because you did not take the derivative of little x as a function of theta but rather of the probability that big X = 1 as a function of theta. $\endgroup$ – BSteinhurst Aug 19 '16 at 22:59
  • $\begingroup$ now that Im re-reading the question I think I misunderstood it. It seems that the issue was that he couldn't take the derivative of of x, of course if x is an observed sample its a fixed number so taking a derivative of it leads to 0. It seems that taking the derivative of $\theta$ remains sensible even with samples observed (i.e. its similar to MLE, maximum likelihood estimation). $\endgroup$ – Pinocchio Aug 19 '16 at 23:08

What might make sense is if $X$ is a function $g(U,\theta)$ of some underlying random variable $U$ (with distribution not depending on $\theta$) and $\theta$, where the differentiable function $g$ is chosen so that $g(U,\theta)$ has the given (continuous) distribution. Then you could say $\dfrac{dX}{d\theta} = \dfrac{\partial g}{\partial \theta}(U,\theta)$. Of course this depends on the "implementation", i.e. the choice of $g$ and $U$, rather than just on the distribution of $X$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.