We have $\|a\| = 1$. For $b$, we have $\|b\| = \sqrt{1-q^2}$ if $q < \sqrt{\frac{1}{2}}$. If $q \geq \sqrt{\frac{1}{2}}$ is of the form $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$, then we have $\|b\| = \frac{1}{2q}$. Otherwise, we still have $\|b\| \leq \frac{1}{2q}$ whenever $q \geq \sqrt{\frac{1}{2}}$. (See the second and third parts of this answer for what exactly $\|b\|$ is in this case.)

We first show that $\|a\| \leq 1$. Observe that $a$ is self-adjoint and $a(1-a) = q^2bb^\ast$ is positive, so by functional calculus, $0 \leq a \leq 1$. Thus, $\|a\| \leq 1$. Now, $\|b\|^2 = \|b^\ast b\| = \|a(1-q^2a)\|$. Since $0 \leq a \leq 1$, by functional calulus,

$$0 \leq a(1-q^2a) \leq \begin{cases}\frac{1}{4q^2}, &\text{ if }\frac{1}{2} \leq q^2\text{, i.e., }q \geq \sqrt{\frac{1}{2}}\\ 1-q^2, &\text{ if } q < \sqrt{\frac{1}{2}}\end{cases}$$

Thus, $\|b\| \leq \begin{cases} \frac{1}{2q}, &\text{ if }q \geq \sqrt{\frac{1}{2}}\\ \sqrt{1-q^2}, &\text{ if }q < \sqrt{\frac{1}{2}} \end{cases}$. For the inequalities in the other direction, we exhibit some actual operators $A, B$ on a Hilbert space $H$ that satisfies the required relations. Let $H = \ell^2(\mathbb{Z}_{\geq 0})$. Let $A$ be the diagonal operator defined by,

$$A(e_n) = q^{2n}e_n$$

And $B$ be defined by,

$$B(e_n) = q^n\sqrt{1-q^{2n+2}}e_{n+1}$$

We verify that $A$ and $B$ satisfy the conditions defining $a$ and $b$. Clearly, $A$ is self-adjoint. And,

$$AB(e_n) = q^n\sqrt{1-q^{2n+2}}A(e_{n+1}) = q^{3n+2}\sqrt{1-q^{2n+2}}e_{n+1}$$

Meanwhile,

$$q^2BA(e_n) = q^{2n+2}B(e_n) = q^{3n+2}\sqrt{1-q^{2n+2}}e_{n+1}$$

So $AB = q^2BA$. Taking adjoints immediately yields $AB^\ast = q^{-2}B^\ast A$. Moreover, we may verify that,

$$B^\ast(e_n) = \begin{cases} 0, &\text{ if }n = 0 \\ q^{n-1}\sqrt{1-q^{2n}}e_{n-1}, &\text{ if }n \geq 1\end{cases}$$

So,

$$BB^\ast(e_n) = \begin{cases} 0, &\text{ if }n = 0 \\ q^{n-1}\sqrt{1-q^{2n}}B(e_{n-1}) = q^{2(n-1)}(1-q^{2n})e_n, &\text{ if }n \geq 1\end{cases}$$

Note that when $n = 0$, we have $1-q^{2n} = 0$, so $BB^\ast(e_n) = q^{2(n-1)}(1-q^{2n})e_n$ is true for all $n$. Meanwhile,

$$q^{-2}A(1-A)(e_n) = q^{-2}q^{2n}(1-q^{2n})e_n = q^{2(n-1)}(1-q^{2n})e_n$$

So $BB^\ast = q^{-2}A(1-A)$. Finally,

$$B^\ast B(e_n) = q^n\sqrt{1-q^{2n+2}}B^\ast(e_{n+1}) = q^{2n}(1-q^{2n+2})e_n$$

Meanwhile,

$$A(1-q^2A)(e_n) = q^{2n}(1-q^{2n+2})e_n$$

So $B^\ast B = A(1-q^2A)$. Thus, there is a $\ast$-homomorphism $\pi: A_q \to \mathbb{B}(H)$ with $\pi(a) = A$ and $\pi(b) = B$. In particular, $\|A\| \leq \|a\|$, $\|B\| \leq \|b\|$. It is easy to see that $\|A\| = 1$, which shows $\|a\| = 1$. When $q < \sqrt{\frac{1}{2}}$, we have,

$$\|B\| = \sup_{n \geq 0} q^n\sqrt{1-q^{2n+2}} = \sqrt{1-q^2}$$

So $\|b\| = \sqrt{1-q^2}$ in this case. If $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$, then,

$$\|B\| = \sup_{n \geq 0} q^n\sqrt{1-q^{2n+2}} = q^m\sqrt{1-q^{2m+2}} = \frac{1}{2q}$$

So $\|b\| = \frac{1}{2q}$ in this case.

----------

**Edit:** I've now found an argument that deals with the remaining cases, as follows.

Here is a proof that any faithful representation of $A_q$ on a separable Hilbert space is a direct sum of copies of the representation in the first part of my answer (which, for clarity's sake, we shall call the *standard representation*) and possibly a zero part (i.e., a part on which $a, b$ both act as $0$. The identity operator still acts as the identity).

We first consider $\ker(a)$. On $\ker(a)$, $bb^\ast = q^{-2}a(1-a)$ and $b^\ast b = a(1-q^2a)$ both act as the zero operator, so $b$ and $b^\ast$ both act as the zero operator on $\ker(a)$ and $\ker(a)^\perp$ is invariant under both $b$ and $b^\ast$. This is the zero part of the representation. By cutting down by $p_{\ker(a)^\perp}$, we may now assume WLOG that $a$ is injective.

The spectrum of $a$ now consists of a discrete part and a diffuse part, and we correspondingly decompose our Hilbert space into the discrete part $H_d$ and the diffuse part $H_c$. We first consider $H_d$, which is spanned by eigenvectors of $a$. If $h$ is a unit eigenvector of $a$ with eigenvalue $\lambda$, then,

$$ab^\ast(h) = q^{-2}b^\ast a(h) = q^{-2}\lambda b^\ast(h)$$

So $b^\ast(h)$ is either zero, or an eigenvector of $a$ with eigenvalue $q^{-2}\lambda$. But $a \leq 1$, so all eigenvalues of $a$ are bounded above by $1$, i.e., if $\lambda > q^2$, we must have $b^\ast(h) = 0$. However,

$$0 = \|b^\ast(h)\|^2 = \langle h, bb^\ast h\rangle = \langle h, q^{-2}a(1-a)h\rangle = q^{-2}\lambda(1-\lambda)$$

As $0 \leq a \leq 1$ and $a$ is assumed injective, this implies $\lambda = 1$. Thus, the only eigenvalue of $a$ that is larger than $q^2$ is $1$. This also shows that, when $\lambda \neq 1$, $b^\ast(h)$ is nonzero, so for any eigenvalue $\lambda$ of $a$ that is not $1$, we have $q^{-2}\lambda$ is also an eigenvalue.

On the other hand, again suppose $h$ is a unit eigenvector of $a$ with eigenvalue $\lambda$, then,

$$ab(h) = q^2ba(h) = q^2\lambda b(h)$$

And,

$$\|b(h)\|^2 = \langle h, b^\ast bh\rangle = \langle h, a(1-q^2a)h\rangle = \lambda(1-q^2\lambda) \neq 0$$

As $\lambda > 0$ and $q^2\lambda \leq q^2 < 1$. Hence, $b(h)$ is an eigenvector of $a$ with eigenvalue $q^2\lambda$, so for any eigenvalue $\lambda$ of $a$, $q^2\lambda$ is an eigenvalue as well.

Combining the above, we see that, if $a$ admits any eigenvalue, the set of eigenvalues must be exactly,

$$\sigma_e(a) = \{1, q^2, q^4, \cdots\}$$

We now show that eigenspaces of different eigenvalues are actually isomorphic to each other. Indeed, for $n \geq 0$, the map,

$$\pi_{0n}: 1_{\{1\}}(a) \to 1_{\{q^{2n}\}}(a), \pi(h) = \frac{1}{\sqrt{\prod_{i=0}^{n-1} q^{2i}(1-q^{2i+2})}}b^n(h)$$

is a well-defined isometry. (This easily follows from the fact that whenever $h$ is a unit eigenvector of $a$ with eigenvalue $\lambda$, then $\|b(h)\|^2 = \lambda(1-q^2\lambda)$ and $b(h)$ is an eigenvector of $a$ with eigenvalue $q^2\lambda$, as we have seen before.) Its conjugate is,

$$\pi_{0n}^\ast: 1_{\{q^{2n}\}}(a) \to 1_{\{1\}}(a), \pi^\ast(h) = \frac{1}{\sqrt{\prod_{i=1}^n q^{2i-2}(1-q^{2i})}}(b^\ast)^n(h)$$

We note that $\pi_{0n}^\ast$ is also an isometry. (This follows from the fact that whenever $h$ is a unit eigenvector of $a$ with eigenvalue $\lambda \neq 1$, then $\|b^\ast(h)\|^2 = q^{-2}\lambda(1-\lambda)$ and $b^\ast(h)$ is an eigenvector of $a$ with eigenvalue $q^{-2}\lambda$, as we have seen before.) But this means $\pi_{0n}$ is a unitary, so we may identify all eigenspaces with $1_{\{1\}}(a)$. Thus,

$$H_d = \ell^2(\mathbb{Z}_{\geq 0}) \otimes 1_{\{1\}}(a)$$

And both $a$ and $b$ only act on the first tensor component, with $a$ acting as the diagonal operator $\text{diag}(1, q^2, q^4, \cdots)$ while,

$$\begin{split}
b(e_n \otimes h) &= b\pi_{0n}\pi_{0n}^\ast(e_n \otimes h)\\
&= \frac{\sqrt{\prod_{i=0}^n q^{2i}(1-q^{2i+2})}}{\sqrt{\prod_{i=0}^{n-1} q^{2i}(1-q^{2i+2})}}\pi_{0(n+1)}\pi_{0n}^\ast(e_n \otimes h)\\
&= (q^n\sqrt{1-q^{2n+2}}e_{n+1}) \otimes h
\end{split}$$

So $H_d$ is just the direct sum of $\dim(1_{\{1\}}(a))$ copies of the standard representation.

Note that this entire argument has also shown that $H_d$ is a reducing subspace of $A_q$, so $H_c = H_d^\perp$ is reducing as well, and we shall now restrict to $H_c$. Our claim is that, in fact, $H_c = 0$. To prove this, assume to the contrary that $H_c \neq 0$. We first note that, by tensoring with $\ell^2$ and applying the spectral theorem (recall that $0 \leq a \leq 1$), we may assume $H_c = L^2([0, 1], \mu) \otimes \ell^2$ for some atomless probability measure $\mu$ on $[0, 1]$, where $a$ acts only on the first tensor component as multiplication by $x$. Fix $n > 0$ and $f \in H_c$ of unit length. Then for each $1 \leq i \leq n$, let $f_i = f|_{[\frac{i-1}{n}, \frac{i}{n}]}$. As $\mu$ is atomless, $f = \sum_{i=1}^n f_i$. Moreover, $1 = \|f\|^2 = \sum_{i=1}^n \|f_i\|^2$. Also observe that $\|a(f_i) - \frac{i}{n}f_i\| \leq \frac{1}{n}\|f_i\|$, so, if we let $g_i = b(f_i)$,

$$a(g_i) = ab(f_i) = q^2ba(f_i) = q^2\frac{i}{n}b(f_i) + q^2b(a(f_i) - \frac{i}{n}f_i) = q^2\frac{i}{n}g_i + q^2b(a(f_i) - \frac{i}{n}f_i)$$

So as $\|b\| \leq 1$, we have $\|(a - q^2\frac{i}{n})g_i\| \leq q^2\frac{1}{n}\|f_i\|$. But then,

$$q^2\frac{1}{n}\|f_i\| \geq \|1_{[\frac{q^2+1}{2}, 1]}(a - q^2\frac{i}{n})g_i\| \geq \frac{1-q^2}{2}\|1_{[\frac{q^2+1}{2}, 1]}b(f_i)\|$$

So $\|1_{[\frac{q^2+1}{2}, 1]}b(f_i)\| \leq \frac{2q^2}{n(1-q^2)}\|f_i\|$. Thus,

$$\begin{split}
\|b(f) - 1_{[0, \frac{q^2+1}{2}]}b(f)\| &= \|\sum_{i=1}^n 1_{[\frac{q^2+1}{2}, 1]}b(f_i)\|\\
&\leq \frac{2q^2}{n(1-q^2)}\sum_{i=1}^n \|f_i\|\\
&\leq \frac{2q^2}{n(1-q^2)}\sqrt{n}\sqrt{\sum_{i=1}^n \|f_i\|^2}\\
&= \frac{2q^2}{n^{\frac{1}{2}}(1-q^2)}\\
&\to 0, \text{ as }n \to \infty
\end{split}$$

Hence, $b(f) = 1_{[0, \frac{q^2+1}{2}]}b(f)$. In particular, $b(f) \perp L^2([\frac{q^2+1}{2}, 1], \mu) \otimes \ell^2$ for all $f$. Thus, $b^\ast(L^2([\frac{q^2+1}{2}, 1], \mu) \otimes \ell^2) = 0$. But for $h \in L^2([\frac{q^2+1}{2}, 1], \mu) \otimes \ell^2$,

$$\|b^\ast(h)\|^2 = \langle h, bb^\ast(h)\rangle = q^{-2}\langle h, a(1-a)h\rangle$$

which can only be zero if $h$ is supported on $\{1\}$ - which, as $\mu$ is atomless, has measure zero. Whence, we must have $L^2([\frac{q^2+1}{2}, 1], \mu) = 0$, i.e., $\|a\| \leq \frac{q^2+1}{2}$.

At this point, one may repeat the argument above once more. This time, instead of dividing the interval $[0, 1]$ into $n$ subintervals, we divide $[0, \frac{q^2+1}{2}]$. Then the same argument shows $\|a\| \leq (\frac{q^2+1}{2})^2$. By repeatedly applying the argument, we then have $\|a\| \leq (\frac{q^2+1}{2})^m$ for all $m \geq 1$, which means $a = 0$ on $H_c$. As $a$ is assumed injective, this must mean $H_c = 0$. Thus, $H = H_d$ is a direct sum of copies of standard representations.

**Remark:** This also provides a concrete description of $A_q$, namely, as $A_q$ is separable, it admits a faithful representation on a separable Hilbert space, so $A_q = C^\ast(A \oplus 0, B \oplus 0, 1)$ for $A, B$ in the standard representation. In fact, as $A, B$ are compact, picking out the constant term is a character on $C^\ast(A, B, 1)$, so $A_q = C^\ast(A, B, 1)$ is also true. Also note that this means the standard representation is faithful. It is easy to see that the standard representation is irreducible, as $W^\ast(A, B, 1) = \mathbb{B}(H)$, so $A_q$ is primitive.

----------

As $A_q$ is separable, by the argument in the previous part, one would have the norm of $b$ is just the norm of $B$ in the standard representation, i.e., whenever $q \geq \sqrt{\frac{1}{2}}$,

$$\|b\| = q^{n_q}\sqrt{1-q^{2n_q+2}}, \text{ where }n_q = \text{argmin}_{n \geq 0} |q^{2n+2} - \frac{1}{2}|$$

which is distinct from $\frac{1}{2q}$ unless $q = \sqrt[2m+2]{\frac{1}{2}}$ for some integer $m \geq 0$.

----------

Let me also mention that the norms are continuous in the limits $q \to 0$ and $q \to 1$. When $q = 1$, the relations define an abelian $C^\ast$-algebra with self-adjoint $a$ and an element $b$ s.t. $|b|^2 = a(1-a)$. It is not hard to check that this is simply the algebra of continuous functions on the unit sphere, with $a = \frac{1+z}{2}$ and $b = \frac{x+iy}{2}$ where $(x,y,z)$ are Cartesian coordinates functions. Thus, we have $\|a\| = 1$ and $\|b\| = \frac{1}{2}$. In $A_q$, when $q \geq \sqrt{\frac{1}{2}}$, recall that we have,

$$\|b\| = q^{n_q}\sqrt{1-q^{2n_q+2}}, \text{ where }n_q = \text{argmin}_{n \geq 0} |q^{2n+2} - \frac{1}{2}|$$

So $\|b\| \to \frac{1}{2}$ as $q \to 1$, consistent with the case where $q = 1$. For $q = 0$, we need to move $q^{-2}$ to the other sides of the equations in some relations for this to make sense, but we would end up with,

$$a = a^\ast, ab = b^\ast a = 0, a(1-a) = 0, b^\ast b = a$$

Thus, $a$ defines a self-adjoint projection and $b$ is a partial isometry whose initial space is $a$ and whose final space is orthogonal to $a$. (The algebra is just $M_2(\mathbb{C}) \oplus \mathbb{C}$, with $a = e_{11} \oplus 0$ and $b = e_{21} \oplus 0$.) Hence, we easily have $\|a\| = \|b\| = 1$ in this case. On the other hand, in $A_q$ when $q < \sqrt{\frac{1}{2}}$, we have $\|b\| = \sqrt{1-q^2}$, so indeed $\|b\| \to 1$ as $q \to 0$, consistent with the case where $q = 0$.