I think in order to answer this usefully, we need more details about what you want to do. Which pieces of the formula do you want to be continuous, and which can stay discrete? Is there an easy example which you could include?

At the bare minimum, if $X$ isn't finite, then you'll have to replace the power set of $X$ with something else in order to make any sense of that sum.

(edit) Okay, so as far as i can see you want to find a replacement for the mobius transform, but for a $\sigma$-algebra. In fact I'm going to guess that your $\sigma$-algebra is the measurable sets in the unit interval, based on what you've said.

The most general setting I know of in which you can define a Möbius function is a locally finite, partially ordered set (see, for example, http://en.wikipedia.org/wiki/Incidence_algebra). So it sounds like you're out of luck. The measurable sets definitely don't form a locally finite poset.

However, I really don't think you've asked the right question yet. You probably would get better answers than mine if you frame your question in terms of measure theory, rather than Möbius inversion. For instance, the wikipedia article seems to imply that I should think of Möbius inversion as "analagous to differentiation", and convolution with the zeta function as "analagous to integration". I don't really find this too helpful, but that's what it says. Maybe you're looking for some kind of derivative? Just a guess.