MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

2 added 11 characters in body

Well, Peter's answer is overkill for this particular problem. While this zeta-function will certainly be a Burgess zeta-function, the study of the zeta-function of this particular kind will be much simpler, and its properties can be directly deduced from properties for the Dirichlet L-functions. For simplicity I will show how to do this in the case $\chi(n)=1$ in your question, although the general character case can be treated similarly, since if we assume that $\chi$ is a character mod $N$ then $\chi \chi_1$ will be a character mod $Nq$ whenever $\chi_1$ is a character mod $q$.

Let $$B(s)=\prod_{p \equiv a \pmod q} (1-p^{-s})^{-1}.$$ Taking the logarithm we find that $$\log B(s)= \sum_{n=1}^\infty \frac{B_0(ns)} n,$$ where $$B_0(s)= \sum_{p \equiv a \pmod q} p^{-s}$$ is some variant of the prime zeta-function. For the half plane Re$(s)>1/2$ the terms when $n \geq 2$ will be absolutely convergent and the main term will be $B_0(s)$. For the Dirichlet $L$-series $L(s,\chi)$ we have similarly that $$\log L(s,\chi) = \sum_{n=1}^\infty \frac{L_0(ns,\chi^n)} n,$$ where $$L_0(s,\chi)= \sum_{p} \chi(p) p^{-s}.$$ By Möbius inversion we get $$L_0(s,\chi)= \sum_{n=1}^\infty \frac{\mu(n)}{n} \log L(ns,\chi^n).$$ It is simple to see from the definitions of the Dirichlet series and using the fact that $\sum_{\chi \pmod q}\chi(a)=\phi(q)$ if $a \equiv 1 \pmod q$ and 0 otherwise that $$B_0(s)= \frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)} L_0(s,\chi).$$ By combining these results we find that $$\log B(s)= \frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)} \sum_{n=1}^\infty \frac 1 n \sum_{d|n} \mu \left(\frac n d \right) \log L(ns,\chi^d).$$

The most important term will come from $n=1$ since the other terms will be absolutely convergent for Re$(s)>1/2$. Thus we have that $$\log B(s)=\frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)}\log L(s,\chi)+ R(s),$$ where $R(s)$ is absolutely convergent for Re$(s)>1/2$. This means that we can write $$B(s)= \prod_{\chi \pmod q} L(s,\chi)^{\chi(a)/\phi(q)L(s,\chi)^{\overline{\chi(a)}/\phi(q)} A(s),$$ where $A(s)$ is a Dirichlet series absolutely convergent and nonvanishing for Re$(s)>1/2$. In particular it means that under the Generalized Riemann hypothesis $(s-1)B(s)^{\phi(q)}$ will be a holomorphic nonvanishing function for Re$(s)>1/2$. By this method it will be possible to get an analytic continuation up to Re$(s)=0$ (its natural boundary should be Re$(s)=0$ since singularities coming from the zeros of the L-functions will be dense close to that line), with exeption for singularities at $\rho/n$ where $\rho$ is a zero of some Dirichlet L-function and $1/n$.

Thus the study of the analytic properties of this zeta-function will be simple consequences of the properties of the Dirichlet L-functions.

1

Well, Peter's answer is overkill for this particular problem. While this zeta-function will certainly be a Burgess zeta-function, the study of the zeta-function of this particular kind will be much simpler, and its properties can be directly deduced from properties for the Dirichlet L-functions. For simplicity I will show how to do this in the case $\chi(n)=1$ in your question, although the general character case can be treated similarly, since if we assume that $\chi$ is a character mod $N$ then $\chi \chi_1$ will be a character mod $Nq$ whenever $\chi_1$ is a character mod $q$.

Let $$B(s)=\prod_{p \equiv a \pmod q} (1-p^{-s})^{-1}.$$ Taking the logarithm we find that $$\log B(s)= \sum_{n=1}^\infty \frac{B_0(ns)} n,$$ where $$B_0(s)= \sum_{p \equiv a \pmod q} p^{-s}$$ is some variant of the prime zeta-function. For the half plane Re$(s)>1/2$ the terms when $n \geq 2$ will be absolutely convergent and the main term will be $B_0(s)$. For the Dirichlet $L$-series $L(s,\chi)$ we have similarly that $$\log L(s,\chi) = \sum_{n=1}^\infty \frac{L_0(ns,\chi^n)} n,$$ where $$L_0(s,\chi)= \sum_{p} \chi(p) p^{-s}.$$ By Möbius inversion we get $$L_0(s,\chi)= \sum_{n=1}^\infty \frac{\mu(n)}{n} \log L(ns,\chi^n).$$ It is simple to see from the definitions of the Dirichlet series and using the fact that $\sum_{\chi \pmod q}\chi(a)=\phi(q)$ if $a \equiv 1 \pmod q$ and 0 otherwise that $$B_0(s)= \frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)} L_0(s,\chi).$$ By combining these results we find that $$\log B(s)= \frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)} \sum_{n=1}^\infty \frac 1 n \sum_{d|n} \mu \left(\frac n d \right) \log L(ns,\chi^d).$$

The most important term will come from $n=1$ since the other terms will be absolutely convergent for Re$(s)>1/2$. Thus we have that $$\log B(s)=\frac 1 {\phi(q)} \sum_{\chi \pmod q} \overline{\chi(a)}\log L(s,\chi)+ R(s),$$ where $R(s)$ is absolutely convergent for Re$(s)>1/2$. This means that we can write $$B(s)= \prod_{\chi \pmod q} L(s,\chi)^{\chi(a)/\phi(q)} A(s),$$ where $A(s)$ is a Dirichlet series absolutely convergent and nonvanishing for Re$(s)>1/2$. In particular it means that under the Generalized Riemann hypothesis $(s-1)B(s)^{\phi(q)}$ will be a holomorphic nonvanishing function for Re$(s)>1/2$. By this method it will be possible to get an analytic continuation up to Re$(s)=0$ (its natural boundary should be Re$(s)=0$ since singularities coming from the zeros of the L-functions will be dense close to that line), with exeption for singularities at $\rho/n$ where $\rho$ is a zero of some Dirichlet L-function and $1/n$.

Thus the study of the analytic properties of this zeta-function will be simple consequences of the properties of the Dirichlet L-functions.