How can the Kalman filter be adapted to handle binary observations? Imagine a coin with a time-varying probability of coming up heads.  (For example, perhaps the probability follows a random walk that is constrained to live in $[0, 1]$.  And say we have some information about how quickly we expect the true probability to vary over time (analogous to the standard deviation of the random walk described above).
At discrete time points, we are given data about the outcome of a given coin flip.  The goal is to come up with a robust procedure to estimate that probability over time.
This sounds like a task for a Kalman filter.  But the literature seems to assume that the observations will be normally distributed around a linear function of the state.  That feels quite violated by the assumptions above.
What's a reasonable way to proceed?
 A: In principle, this is what nonlinear filtering does. Check this out, also under the name "hidden Markov model". Particle filters can be adapted to deal with
this setup.
In a nutshell, here is what hidden Markov processes are.
You have a two component Markov chain $(X_n,Y_n)$ (I write it in discrete time, there is an analogue in continuous time). You observe only 
$\{Y_i\}_{i=1}^n$ and want to estimate $X_n$, i.e. construct the conditional pdf
of $X_n$ given the observations.
In your case, the law of $Y_n$ given the state history $\{X_i\}_{i=1}^n$ depends only on $X_n$ and is a Bernoulli variable with mean depending on $X_n$.  $X_n$ itself is what you call $p$.
PS I just noted that my answer is closely related to that of @passerby51.
A: The Kalman filter is based on an assumption of Gaussian noise in both the observations and process. As I read your problem statement you have no observation noise. Given that, I don't think the KF is the correct choice. 
"Yes, well, I'll just set the measurement noise to zero". The problem is that the KF performs the update step in measurement space. If you have no measurement noise, then you have no uncertainty in the normal to the measurement hyperplane. The system covariance ends up being singular (the math blows up).
Off the top of the head I don't know the correct approach to your question, but this does not sound like a KF application to me.
A: What I would try when facing such a problem:
1°) Minimize $\lambda\int p'(t)^2\ dt +$ the sum of informations $\log (1/p(t_k))$ if heads at $t_k$, $\log (1/[1-p(t_k)])$ if tails.
2°) Chose $\lambda$ using (a suitable form of) cross-validation.
3°) Keep searching how people did before. 
