Question

I am wondering what is the derivative of the following function with respect to $x(t)$ in sense of distributions. $$ I\left(\int_0^t x(\tau)d\tau \leq c\right) $$ where $I$ is the indicator function and $c$ is a constant.

Answer 1

You are asking for the derivative of a non-linear function on an infinite dimensional space. (You do not specify the latter but the space locally integrable functions seems a natural candidate). The derivative can only exist in a very weak sense so it is natural to go for the directional derivative which has two inputs---the point $ x $ where the derivative is computed and the direction $ y $ in which the rate of change takes place. A back of the envelope calculation suggests that the derivative should then be $$-\sum \frac {Y(a)}{x(a)}\delta_a$$ where we use capitals to indicate the primitives which occur in the formulation and the sum is taken over the $a$ in the pre-image of $c$ under $X$.

Some general remarks: the question was posed in such a vague manner that it is not really possible to give a precise, rigorous answer. Presumably you have some concrete application in view and I suggest that you check the above formula there to see if it leads to the expected result (ones I looked at were the functions $t$---with primitive $t^2$---and $\sin t$ for the function $x$).

As is evident from the above formula, the derivative is in a much weaker sense than the usual concepts of functional analysis---one requires special conditions on $x$ (beyond just smoothness) for the above expression to make sense and the limit of the difference quotients used to define the derivative does not take place in the underlying function space but in a larger space (of distributions---hence the Dirac functions in the formula. I presume that this is the reason for the reference to distributions in your question).

Answer 2

Let me try. Your mapping is:
$1 - H(x-c)$ , where $H$ is the Heaviside function, derivative is $-\delta(x-c)$.
composed with
$\int_0^t : C^\infty(\mathbb R) \to \mathbb R$, which is bounded and linear, surjective.
No way with the chain rule, in fact, there is no chain rule in this situation.

Simpler approach, but it might not answer your question: Suppose $s\mapsto x(\tau,s)$ is a smooth variation of $x(\tau)$, i.e., $x\in C^\infty(\mathbb R^2)$.
Then $$s\mapsto 1-H\Big(\int_0^t x(\tau,s)d\tau -c\Big)$$ can be viewed as a distribution in $s$. Take a test function $f(s)$ and consider $$ \int \Big(1-H\Big(\int_0^t x(\tau,s)d\tau -c\Big)\Big)(-f'(s))ds $$ Now try formal partial integration with respect to $s$. Since the left hand integrand is 1 or 0, by the fundamental theorem it will be the difference of the values of $f$ at those $s$ where this jumps. If you fix $x(\tau,s)$ you can work this out.