Denote by $\mathbb{F}_q$ a finite field with $q$ elements. For $\mathcal{P}$ be arbitrary subset of $\mathbb{F}_q^d.$ We define the set

$$S:= \left\{ \det([x_1,x_2,\dots,x_d]): x_1,x_2,\dots,x_d \in \mathcal{P}\right\},$$

where $[x_1,x_2,\dots,x_d]$ is $d \times d$ matrix with $d$ column vectors $x_1,x_2,\dots,x_d.$

What is the largest cardinality of $\mathcal{P}$ such that $S \neq \mathbb{F}_q$?

(equivalently, What is the minimum value of $|\mathcal{P}|$ to make sure that $S=\mathbb{F}_q?$)

Comment: If we consider a subset $\mathcal{Q}$ of $M_d(\mathbb{F}_q)$ and $$S:=\left\{\det(A): A \in \mathcal{Q}\right\}.$$ Then, the minimum of cardinality of $\mathcal{Q}$ such that $S=\mathbb{F}_q$ for sure is $$N = q^{d^2} - \dfrac{|\mathrm{GL}_d(\mathbb{F}_q)|}{q-1} + 1.$$

How about the previous question?