Let $1\leq \ell \leq \binom{k}{2}$. It seems that there should exist some set $\mathcal{P}$ of $kd-\ell+1$ points in $\mathbb{R}^d$ which do not lie on the union of any $k$ hyperplanes $H_1,\ldots, H_k$, exactly $\ell$ of which are orthogonal. Indeed, this should be generic.
If $\ell=1$, this is easily seen: put the points in general position and chosen so that the affine hulls of any two subsets of the points are not orthogonal. But this argument doesn't seem to work for 7 points in $\mathbb{R}^3$.
Any suggestions, or reference for something similar?