A convenient way to think about it is to represent a convex body in terms of its support function (restricted to the unit sphere). Minkowski addition corresponds to the addition of support functions. The perimeter is the integral of the support function: $$L(h) = \int_{\mathbb{S}^1} h\, dx = \langle h, 1 \rangle_{L^2}.$$ So, the gradient of the perimeter functional is the constant function $1$, that is the unit disk centered at the origin, as you have assumed.