Is there a differentiable random walk? Is there a random walk which is differentiable or smooth? Like brownian motion except smoothed out on small distances. I was wondering if there is a "natural" or "canonical" analogue of brownian motion for differentiable curves, it will obviously have some scale parameter associated to it.
 A: Actually if all you are concerned with is the smoothness of the sample path, the smoothness of a Gaussian process is completely characterized by its covariance function. The following result provides an insight into this issue.
In this aspect we can discuss smoothness with probability one, and the sample path smoothness in this sense is relatively clear in stationary case.

Theorem 3.4.1[Adler] Let
  $X(\boldsymbol{t}),\boldsymbol{t}\in\mathbb{R}^N$ be a real-valued
  zero mean Gaussian random field (in your case take $N=1$ as a special
  case of this result) with a continuous covariance function. Then if
  for some $0<C<\infty$ and some $\epsilon>0$
  $$\boldsymbol{E}|X(\boldsymbol{t}-X(\boldsymbol{s}))|\leq\frac{C}{|\log\|\boldsymbol{t-s}\||^{1+\epsilon}}$$
  for all $\boldsymbol{t},\boldsymbol{s}\in I_0\subset\mathbb{R}^N$.
  Then $X$ has a continuous sample path over $I_0$ with probability one.

In this spirit of "being continuous with probability one", you can actually control the smoothness by choosing different covariance kernels. For example the Matérn covariance function will allow you to have a control over the sample path smoothness by varying its degree of freedom $\nu$; i.e. if you choose a stationary Matérn covariance realization of GP, then the sample path will be in $\mathcal{C}^{[\nu]-1}$ with probability one.
Reference
[Adler] Adler, Robert J. The geometry of random fields. Society for Industrial and Applied Mathematics, 2010.
[Rasmussen] http://ml.dcs.shef.ac.uk/gpip/slides/rasmussen.pdf
A: The non-differentiability of the stochastic process is needed for the Markov assumption. If you relax the Markov assumption and allow for non-delta-function correlated noise, the process can be differentiable. This has been worked out, for example, in Non-Markovian Effects on the Brownian Motion of a Free Particle (2010). The noise is correlated over a time $t_c$, and the root-mean-square displacement
$$\Delta x(t)=\langle[X(t)-X(0)]^2\rangle^{1/2}=c\sqrt{t+t_ce^{-t/t_c}}$$
is differentiable at $t=0$, acquiring the $\sqrt t$ increase only for times $t\gg t_c$.
For a comprehensive treatment, see the thesis I mentioned in an earlier comment, Differentiable approximations to Brownian motion on manifolds: We consider the family of $C^n$ approximations to Brownian motion on 2-uniformly smooth Banach spaces, obtained by iterating the construction of the Ornstein-Uhlenbeck process.
