Derivatives through random variables?

Suppose I have some random variable X with probability distribution P(.;theta). Suppose I have a single sample x from this distribution.

Does it make any sense to ask for the derivative of x with respect to theta?

I believe it does not, but am having trouble convincing some of my colleagues.

• in the end, what did you end up telling them to convince them? – Pinocchio Aug 19 '16 at 2:08
• actually Ian, I don't understand your question. Are you taking the derivative of P with respect to x (the sample) or $\theta$? I can't conceive how it makes sense to take the derivative wrt to x if x is a sample. If its a sample, then its observed and fixed, so taking the derivative wrt to it is simply zero. Or am I missing something? – Pinocchio Aug 19 '16 at 23:11

It does not because $x$ is just some element of your state space. You could conceivably choose $x$ as your sample point for any or all of the $\theta$'s. So what it would make sense to differentiate with respect to $\theta$ is $$\mathbb{P}_{\theta}(X=x)$$ as long as you have a discrete distribution. For a continuous distribution you could substitute $x$ with some interval.
• I read your answer but I didn't understand it. Why is my counter example wrong. Consider $P_{\theta}(X = x) = \theta^x (1 - \theta)^{1-x}$ and let $x=1$ be the sample observed. Then $P_{\theta}(X = 1) = \theta$. One can easily take the derivative of that function, it has $\theta$ as a variable, its derivative is simply 1. Why is that not correct? – Pinocchio Aug 19 '16 at 2:12
• now that Im re-reading the question I think I misunderstood it. It seems that the issue was that he couldn't take the derivative of of x, of course if x is an observed sample its a fixed number so taking a derivative of it leads to 0. It seems that taking the derivative of $\theta$ remains sensible even with samples observed (i.e. its similar to MLE, maximum likelihood estimation). – Pinocchio Aug 19 '16 at 23:08
What might make sense is if $X$ is a function $g(U,\theta)$ of some underlying random variable $U$ (with distribution not depending on $\theta$) and $\theta$, where the differentiable function $g$ is chosen so that $g(U,\theta)$ has the given (continuous) distribution. Then you could say $\dfrac{dX}{d\theta} = \dfrac{\partial g}{\partial \theta}(U,\theta)$. Of course this depends on the "implementation", i.e. the choice of $g$ and $U$, rather than just on the distribution of $X$.