Let me write a partial answer: We have $\|a\| = 1$. For $b$, we have $\|b\| = \sqrt{1-q^2}$ if $q < \sqrt{\frac{1}{2}}$. If $q \geq \sqrt{\frac{1}{2}}$ is of the form $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$, then we have $\|b\| = \frac{1}{2q}$. Otherwise, we still have $\|b\| \leq \frac{1}{2q}$ whenever $q \geq \sqrt{\frac{1}{2}}$.

We first show that $\|a\| \leq 1$. Observe that $a$ is self-adjoint and $a(1-a) = q^2bb^\ast$ is positive, so by functional calculus, $0 \leq a \leq 1$. Thus, $\|a\| \leq 1$. Now, $\|b\|^2 = \|b^\ast b\| = \|a(1-q^2a)\|$. Since $0 \leq a \leq 1$, by functional calulus,

$$0 \leq a(1-q^2a) \leq \begin{cases}\frac{1}{4q^2}, &\text{ if }\frac{1}{2} \leq q^2\text{, i.e., }q \geq \sqrt{\frac{1}{2}}\\ 1-q^2, &\text{ if } q < \sqrt{\frac{1}{2}}\end{cases}$$

Thus, $\|b\| \leq \begin{cases} \frac{1}{2q}, &\text{ if }q \geq \sqrt{\frac{1}{2}}\\ \sqrt{1-q^2}, &\text{ if }q < \sqrt{\frac{1}{2}} \end{cases}$. For the inequalities in the other direction, we exhibit some actual operators $A, B$ on a Hilbert space $H$ that satisfies the required relations. Let $H = \ell^2(\mathbb{Z}_{\geq 0})$. Let $A$ be the diagonal operator defined by,

$$A(e_n) = q^{2n}e_n$$

And $B$ be defined by,

$$B(e_n) = q^n\sqrt{1-q^{2n+2}}e_{n+1}$$

We verify that $A$ and $B$ satisfy the conditions defining $a$ and $b$. Clearly, $A$ is self-adjoint. And,

$$AB(e_n) = q^n\sqrt{1-q^{2n+2}}A(e_{n+1}) = q^{3n+2}\sqrt{1-q^{2n+2}}e_{n+1}$$

Meanwhile,

$$q^2BA(e_n) = q^{2n+2}B(e_n) = q^{3n+2}\sqrt{1-q^{2n+2}}e_{n+1}$$

So $AB = q^2BA$. Taking adjoints immediately yields $AB^\ast = q^{-2}B^\ast A$. Moreover, we may verify that,

$$B^\ast(e_n) = \begin{cases} 0, &\text{ if }n = 0 \\ q^{n-1}\sqrt{1-q^{2n}}e_{n-1}, &\text{ if }n \geq 1\end{cases}$$

So,

$$BB^\ast(e_n) = \begin{cases} 0, &\text{ if }n = 0 \\ q^{n-1}\sqrt{1-q^{2n}}B(e_{n-1}) = q^{2(n-1)}(1-q^{2n})e_n, &\text{ if }n \geq 1\end{cases}$$

Note that when $n = 0$, we have $1-q^{2n} = 0$, so $BB^\ast(e_n) = q^{2(n-1)}(1-q^{2n})e_n$ is true for all $n$. Meanwhile,

$$q^{-2}A(1-A)(e_n) = q^{-2}q^{2n}(1-q^{2n})e_n = q^{2(n-1)}(1-q^{2n})e_n$$

So $BB^\ast = q^{-2}A(1-A)$. Finally,

$$B^\ast B(e_n) = q^n\sqrt{1-q^{2n+2}}B^\ast(e_{n+1}) = q^{2n}(1-q^{2n+2})e_n$$

Meanwhile,

$$A(1-q^2A)(e_n) = q^{2n}(1-q^{2n+2})e_n$$

So $B^\ast B = A(1-q^2A)$. Thus, there is a $\ast$-homomorphism $\pi: A_q \to \mathbb{B}(H)$ with $\pi(a) = A$ and $\pi(b) = B$. In particular, $\|A\| \leq \|a\|$, $\|B\| \leq \|b\|$. It is easy to see that $\|A\| = 1$, which shows $\|a\| = 1$. When $q < \sqrt{\frac{1}{2}}$, we have,

$$\|B\| = \sup_{n \geq 0} q^n\sqrt{1-q^{2n+2}} = \sqrt{1-q^2}$$

So $\|b\| = \sqrt{1-q^2}$ in this case. If $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$, then,

$$\|B\| = \sup_{n \geq 0} q^n\sqrt{1-q^{2n+2}} = q^m\sqrt{1-q^{2m+2}} = \frac{1}{2q}$$

So $\|b\| = \frac{1}{2q}$ in this case.

----------

For the remaining cases, I'm honestly not sure. I would find it surprising that the norm of $b$ somehow depends on whether $\log_q (\frac{1}{2})$ is an integer or not, so I suspect that $\|b\| = \frac{1}{2q}$ is true for all $q \geq \sqrt{\frac{1}{2}}$. On the other hand, I tried a few times and couldn't find any representation of $A_q$ that's different from just a direct sum of copies of the representation I wrote down above, and while I couldn't formalize the argument, I did think of some informal arguments that seem to show that those are actually the only representations. If that is the case, then one would have, whenever $q \geq \sqrt{\frac{1}{2}}$,

$$\|b\| = q^{n_q}\sqrt{1-q^{2n_q+2}}, \text{ where }n_q = \text{argmin}_{n \geq 0} |q^{2n+2} - \frac{1}{2}|$$

which is distinct from $\frac{1}{2q}$ unless $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$.

----------

Let me also mention that the norms are continuous in the limits $q \to 0$ and $q \to 1$. When $q = 1$, the relations define an abelian $C^\ast$-algebra with self-adjoint $a$ and an element $b$ s.t. $|b|^2 = a(1-a)$. If I'm not mistaken, this is simply the algebra of continuous functions on the unit sphere, with $a = \frac{1+z}{2}$ and $b = \frac{x+iy}{2}$ where $(x,y,z)$ are Cartesian coordinates functions. Thus, we have $\|a\| = 1$ and $\|b\| = \frac{1}{2}$. In $A_q$, when $q \geq \sqrt{\frac{1}{2}}$, recall that we have,

$$\frac{1}{2q} \geq \|b\| \geq q^{n_q}\sqrt{1-q^{2n_q+2}}, \text{ where }n_q = \text{argmin}_{n \geq 0} |q^{2n+2} - \frac{1}{2}|$$

So $\|b\| \to \frac{1}{2}$ as $q \to 1$, consistent with the case where $q = 1$. For $q = 0$, we need to move $q^{-2}$ to the other sides of the equations in some relations for this to make sense, but we would end up with,

$$a = a^\ast, ab = b^\ast a = 0, a(1-a) = 0, b^\ast b = a$$

Thus, $a$ defines a self-adjoint projection and $b$ is a partial isometry whose initial space is $a$ and whose final space is orthogonal to $a$. Hence, we easily have $\|a\| = \|b\| = 1$ in this case. On the other hand, in $A_q$ when $q < \sqrt{\frac{1}{2}}$, we have $\|b\| = \sqrt{1-q^2}$, so indeed $\|b\| \to 1$ as $q \to 0$, consistent with the case where $q = 0$.