I tried calculus of variations. It's been a while since I've done calculus of variations, so I could be messing it up completely. Also, this isn't completely rigorous. There are ways of making calculus of variations rigorous, but I don't know them, and they're a lot harder than just doing calculations.

By my comment above, the $Q(\frac{1}{2})=0$ constraint is worthless, so we will ignore it.

Assume $Q$ is the minimum function (this is the first, and most important, non-rigorous step, since we are assuming a minimum exists). Now, let's plug in $Q+\epsilon$, where $Q$ and $\epsilon$ are both functions of $x$. We have

$$M=\frac{1}{12} + \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right)^2 (Q+\epsilon)^2 \ dx - 4 \left[ \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) (Q+\epsilon) \  dx\right]^2 .$$

Extracting the first-order terms in $\epsilon(x)$, we get

$$\Delta M = 2\int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right)^2 Q \epsilon \ dx -8 \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) Q \  dx \  \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) \epsilon \  dx .$$

We still have to take care of the condition $\int_0^\frac{1}{2}  Q(x)^2 \ dx = 1$. We do this using Lagrange multipliers, and so we get the expression

$$2\int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right)^2 Q \epsilon \ dx -8 \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) Q \  dx \  \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) \epsilon \  dx + \lambda \int_0^\frac{1}{2}  Q \epsilon\ dx .$$

This has to be zero for all functions $\epsilon(x)$. To simplify things further, let

$$C =  \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) Q \  dx,$$

since it's a constant independent of $\epsilon$. Now, we have

$$2 \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right)^2 Q \epsilon \ dx -8 C \  \int_0^\frac{1}{2} \left( \tfrac{1}{2}-x \right) \epsilon \  dx + \lambda \int_0^\frac{1}{2}  Q \epsilon\ dx $$

or, putting everything in the same integral sign, 


$$ \int_0^\frac{1}{2} 2 \left( \tfrac{1}{2}-x \right)^2 Q \epsilon -8 C \left( \tfrac{1}{2}-x \right) \epsilon  + \lambda   Q \epsilon\ dx $$

and for this to be $0$ for all functions $\epsilon$, we must have

$$Q = \frac{ (\frac{1}{2}-x)}{\alpha(\frac{1}{2}-x)^2+\beta}$$

for some $\alpha$, $\beta$. 

Note that, if $\beta \neq 0$, we get $Q(\frac{1}{2}) = 0$, as desired.

A Maple or Mathematica program should at least let you calculate $\alpha$ and $\beta$ numerically. When I do this, I seem to be getting values of $M$ which are slightly less than 0, but I guess I could have mistyped my Maple formulas.