I don't know a good answer, but the following seems to give some lower bound.
First, define $\rho(n,\alpha) \equiv \inf_{|S| = n} \rho_{S}(\alpha)$. Next, fix a set $S$, and for $T \subset S$ define $L_{T}(\alpha)$, $L_{T}'(\alpha)$ in the obvious way. Then for fixed $3 \leq k \leq |S|$, I believe that
$$ \rho_{S}(\alpha) = \frac{L_{S}(\alpha)}{L_{S}'(\alpha)} = \frac{\sum_{|T| = k, T \subset S} L_{T}(\alpha)}{\sum_{|T| = k, T \subset S} L_{T}'(\alpha)} \geq \rho(k,\alpha),$$
where the second equality follows from the fact that each triple is counted exactly ${k \choose 3}$ times in both numerator and denominator.
But this tells us that $\rho(n,\alpha)$ is monotone-increasing in $n$, and that we can get a lower bound on your limit by getting a lower bound for any finite $n$ (and of course this is "sharp" in the sense that it will eventually recover the right answer).
Getting some lower bound for fixed $n$ and $\alpha$ doesn't seem as hard as the original problem. For example, for $n = 4$, we observe the following. Let's write $S = \{0,a,b,1\}$ and assume $L_{S}(\alpha) = 0$. Looking at the various triples, we find:
$a \in (\alpha b, (1-\alpha)b)$ and $a,b \in (\alpha, 1-\alpha)$. But then combining these we see $a \in (\alpha^{2}, (1-\alpha)^{2})$ as well. So $a \in (\alpha, 1-\alpha) \cap (\alpha^{2}, (1-\alpha)^{2})$. But this is impossible if $\alpha > (1-\alpha)^{2}$, which happens for $\alpha > \frac{3 - \sqrt{5}}{2} \approx 0.382$. Thus, $\rho(n,\alpha) \geq \frac{1}{3}$ for all $\alpha \geq 0.382$ and all $n \geq 4$. For any fixed $\alpha$, basically the same calculation immediately gives some bound for sufficiently large $n$, and thus a nondegenerate bound on $\rho(\alpha)$.
Edited to add: since this was unclear, I'll do the same calculation for large $n$. Consider $S = \{0,s_{1},\ldots,s_{n},1\}$. Then to have $L_{S}(\alpha) =0$, we must have $s_{n} \in (\alpha,1-\alpha)$, $s_{n-1} \in (\alpha s_{n}, (1-\alpha)s_{n}) \subset (\alpha^{2}, (1-\alpha)^{2})$, $s_{n-2} \in (\alpha^{3},(1-\alpha)^{3})$, and iterating $s_{1} \in (\alpha^{n}, (1-\alpha)^{n})$. But of course we must also have $s_{1} \in (\alpha,1-\alpha)$. This is impossible if $\alpha > (1-\alpha)^{n}$, and of course for any fixed $\alpha > 0$ this inequality is violated for all $n > \frac{\log(1-\alpha)}{\log(\alpha)}$ sufficiently large. Set $N(\alpha) = \frac{\log(1-\alpha)}{\log(\alpha)}$; this shows that $\rho(\alpha) > {N(\alpha) + 2 \choose 3}^{-1}$.
This is of course a terrible bound, and going further by hand seems fairly tedious. However I'd guess that doing it by a computer is probably not too bad for moderate $n$ - as with the above calculation, you end up with some finite list of conditions, and checking whether any particular set is simultaneously-satisfiable is just checking to see if the intersection of a bunch of half-spaces is empty.
This naive algorithm does have running time that is exponential in $n$, so e.g. getting the right answer for $n=40$ is probably not feasible this way... but of course if you just want lower bounds you don't actually need to check everything, and I would guess that you could do a faster and more targeted search with a few minutes thinking about the "bad" cases.
As an aside: there is a famous problem about "arithmetic progressions," and people often talk about "approximate" arithmetic progressions in this context. What you're doing here sounded somewhat familiar, though I don't know if their techniques work when $\alpha$ is quite small (those arithmetic progressions are very approximate). Unfortunately I'm nothing like an expert, it has been a long time since I've looked at the subject, and my old notes are in my office so I can't check very quickly right now. There is a nice book on additive combinatorics which relates (approximate) arithmetic progressions to Fourier analysis, and it might be worth taking a look.
