# Concentration Inequality for Score Functions of Exponential Familty

Let $p$ be the density of a continuous one-parameter exponential family distribution on $\mathbb{R}$. We assume that $$p(x) = c(x)\cdot \exp\bigl [ x \cdot \theta - b(\theta ) \bigr ],$$ where $\theta \in \mathbb{R}$ is the parameter. The score function is defined as $$S(x) = \frac{\mathrm{d}( \log p(x))}{\mathrm{d}x} = \frac{ p'(x)}{p(x) } = \theta + c'(x) / c(x).$$ It can be shown that for $X \sim p$, $\mathbb{E} S(X) = 0$ and $\text{Var} [ S(X)] = I_p$, which is the Fisher's information of $p$.

Now we set $X_1, \ldots, X_n \sim p$ to be $n$ i.i.d. random variables. It is easy to see that $$\frac{1}{n} \sum_{i=1}^n S(X_i) \xrightarrow{a.s} 0,~~\frac{1}{\sqrt{n} } \sum_{i=1}^n S(X_i)\xrightarrow{p} N(0, I_p).$$ I was wondering if there is any result on the concentration behavior of $\frac{1}{n} \sum_{i=1}^n S(X_i)$. Is it possible to obtain $$\mathbb{P} \biggl [ \frac{1}{n} \sum_{i=1}^n S(X_i) > t \biggr ] \leq \exp( - n \cdot C \cdot t^{\alpha} )$$ for some constants $C>0$ and $\alpha >0$? In particular, if $\alpha = 2$, we just show that $\frac{1}{n} \sum_{i=1}^n S(X_i)$ has a Gaussian-type tail.

• Do you want $\frac {\partial }{\partial \theta}$ in the score function or what you have ? – user83457 Jan 20 '17 at 13:09
• @michael the score function does involves $\partial / \partial \theta$. For instance, if $p$ is the density of $N(0,1)$ distribution, then the score function is $s(x) = x$. – Steve Jan 20 '17 at 23:12