One can get a bound which is within a constant of the optimal bound using the following
Paley-Zygmund type inequality Let $X$ be a real random variable with mean zero and finite fourth moment, that is not identically zero. Then $$ {\bf P}(X > 0) \geq \frac{({\bf E} X^2)^2}{4 {\bf E} X^4}.$$
Proof
By Holder we have
$$ {\bf E} X^2 1_{X>0} \leq ({\bf E} X^4)^{1/2} {\bf P}(X>0)^{1/2} \quad (1)$$
and
$$ {\bf E} X 1_{X>0} \leq ({\bf E} X^4)^{1/4} {\bf P}(X>0)^{3/4}$$
and hence by the mean zero hypothesis
$$ {\bf E} |X| 1_{X<0} \leq ({\bf E} X^4)^{1/4} {\bf P}(X>0)^{3/4}.$$
Hence by Holder again
$$ {\bf E} X^2 1_{X<0} \leq ({\bf E} |X| 1_{X<0})^{2/3} ({\bf E} |X|^4)^{1/3} \leq ({\bf E} X^4)^{1/2} {\bf P}(X>0)^{1/2} $$
which on summing with (1) gives
$$ {\bf E} X^2 \leq 2 ({\bf E} X^4)^{1/2} {\bf P}(X>0)^{1/2}$$
hence the claim. $\Box$
(It should be possible to improve the constant $4$ a bit by using the fact that the fourth moment has to be shared between the positive and negative components of $X$, but I have not tried to optimise this. The extremal relationship between ${\bf P}(X>0)$, ${\bf E} X^2$, and ${\bf E} X^4$ is probably coming from the case $X = \xi - p$ of a normalised Bernoulli random variable $\xi$. One can also obtain a comparable bound by applying the usual Paley-Zygmund inequality to $(X - \sqrt{\theta {\bf E} X^2})^2$ for some parameter $0 < \theta < 1$ that one can optimise in.)
In your situation, writing $X = \sum_i a_i (\xi_i - p)$ for the normalised sum of Bernoulli variables $\xi_i$, $X$ has mean zero, variance $p(1-p) \sum_i a_i^2$, and fourth moment $$ 6 \sum_{i<j} a_i^2 a_j^2 (p(1-p))^2 + \sum_i a_i^4 (p (1-p)^4 + (1-p) p^4)$$ $$ \leq \max( 3(p(1-p))^2, p (1-p)^4 + (1-p) p^4) \sum_{i,j} a_i^2 a_j^2$$ $$ = \max( 3p^2 (1-p)^2, p(1-p)(1-3p+3p^2)) (\sum_i a_i^2)^2$$ and hence $$ {\bf P}(X>0) \geq \frac{1}{4 \max( 3, (1-3p+3p^2)/p(1-p) )}$$ which is asymptotic to $p/4$ as $p \to 0$, or $(1-p)/4$ as $p \to 1$. One should be able to improve the constant $4$ with a bit more effort.