I'm wondering how Riemann knew that $\zeta(z)$ could be extended to a larger domain. In particular, who was the first person to explicitly extend the domain of a complex valued function and what was the function?

8$\begingroup$ I guess that this is a very old story. When Euler wrote a series of the form $\sum z^n$, that converges for $z<1$, he was aware of the fact that it can be analytically extended to $\mathbb{C}\{1\}$. $\endgroup$ – Francesco Polizzi Jan 25 at 6:09

4$\begingroup$ This text claims that it was a different function but also Euler: en.wikipedia.org/wiki/… $\endgroup$ – Mark Sapir Jan 25 at 6:48

3$\begingroup$ Riemann knew there was an analytic continuation because he proved it, in fact in two ways. Nobody before him had considered the zetafunction or any other Dirichlet series as a function of a complex variable, so he had to figure out the analytic continuation on his own. $\endgroup$ – KConrad Jan 25 at 10:33

4$\begingroup$ The earliest examples occur in the work of Euler on summation of series, though he did not formulate the definition explicitly. For example he used $1/(1+z)=\sum_n(1)^nz^n$ to sum the series $11+11+\ldots=1/2$. $\endgroup$ – Alexandre Eremenko Jan 25 at 14:24

4$\begingroup$ @WlodAA I have in front of me two books on the development of number theory in Russia: "The St. Petersburg school of number theory" by Delone and "Development of number theory in Russia" by Ozhigova. Both have sections on the work of Chebyshev and neither says anything about him working with the zetafunction in the complex plane. If Chebyshev had used complex variables in connection with the zetafunction before Riemann, then I think Delone would not have hesitated to mention it (his book came out in 1947). $\endgroup$ – KConrad Jan 26 at 4:57
(Expanded 1/26/21
First let me point out for nonnative English speakers that the use of the article 'a' in the phrase 'a complexvalued function' means that the question is not solely in reference to the Riemann or any other zeta function. It includes any function whose domain is some set of the reals, so I interpret the question as "Who is the first to have published an extension of the domain of a significant function from some set of the reals to some continuous domain of the complex, and what was that function?" To me, the exact meaning of the term analytic continuation and whether it is unique or not is a different question.
The first sentence and several of the comments focus on the Riemann zeta function. Riemann did not stand alone and his interests were much broader than the sometimes almost obsessive focus today on the RH might imply. His interests encompassed pretty much all of complex analysis, so it was natural for him to consider extensions of real functions to complex functions.
Hard to believe (smacks of some type of regional bias) that no mathematician before Euler, woke up one morning and thought, "What if I modify my real formulas to include that crazy square root of 1?" Roger Cotes was primed to meaningfully do so with his interest in astronomy and celestial mechanics; familiarity with the work of his colleague Newton on the series reps of the trig functions, their inverses, the calculus, and Newtonian mechanics; use of the logarithmic tables introduced at the beginning of the 1600s by Napier to deal with computations with large numbers encountered in surveying the Earth and the skies; and work on interpolation (Cotes' and Newton's).
Let me stress again that Cotes was familiar with Newton's compositional inversion of power series (one formula includes the associahedron version of the Lagrange inversion formula for formal series, see Ferraro below), including that for the exponential function, and, as noted by Griffiths' comment to the post "The making of the logarithm" by Freiberger: Without these tables of logarithms there would be no theory from Nicholas Mercator of the area under a symmetrical hyperbola equaling the log of the distance along the x axis, nor of Isaac Newton's reversion of the hyperbola formula to achieve the infinite series for the antilogarithm $e^x$. (Mercator maps, beginning to see the dots?) In fact, Ferraro discusses on pages 74 and 75 of "The Rise and Development of the Theory of Series up to the Early 1820s" how Newton inverted the power series for the logarithm $\ln(1x)$ to obtain the power series of the antilogarithm $1 e^{x}$. (Newton with his superb mastery of geometry and analysis would surely have noted the simple inverse function theorem relation here between the derivatives of the two series as well.)
Consequently, it seems natural that at the birth of calculus and its association with power series and compositional inverses, Cotes wrote down in 1714, when Euler was seven years old,
$$ ix = \ln[ \;\cos(x) + i \sin(x) \;]$$
a nascent version of Euler's 1748 fabulous formula (cf. Wikipedia)
$$ e^{i\theta} = \cos(\theta) + i \sin(\theta).$$
An obvious check with the derivative (or fluxions) verifies the formula without explicit use of the exponential
$$ \frac{d}{dx} (ix +constant) = i = \frac{d}{dx} \; \ln[ \;\cos(x) + i \sin(x) \;]= \frac{\sin(x) + i \cos(x)}{\cos(x) + i \sin(x)},$$
which I'm sure was SOP for Newton and Cotesapplication of the chain rule, a.k.a. inverse function theorem in this case, $dx = df(f^{1}(x)) = f'(f^{1}(x)) \; (f^{1})'(x) \; dx$, which indeed makes the formula obvious.
In "The history of the exponential and logarithmic concepts," Cajori explains how John Bernoulli considered the solutions of a differential equation transformed from the reals to the imaginary in 1702 and gives Cotes' derivation of his formula, which Cotes published in 1714 and 1722. Cajori also claims that subsequently Euler did not shy from using imaginary numbers.
Euler's formula as written today had to wait for the development by Euler and colleagues of the symbolic rep of the exponential function $\exp(z) = e^z$ with $e$ being Euler's constant, sometimes referred to as Napier's constant since it occurred in Napier's log tables. This was after much calculus underlying the log had been explicated by Huygens and others. The exponential function was sometimes even referred to as the 'antilogarithm', reflecting the log's priority, as noted in the log post.
Cote's logarithmic formula is an extension from the positive reals to the realm of complex numbers of the argument of the logarithm in a rather more difficult way than simply replacing $n$ in the series rep of $\zeta(n)$ by real numbers on the real line and then to other numbers in the complex plane.
According to the Wikipedia article on Cotes, he published an important theorem on the roots of unity (and gave the value of one radian for the first time) in 1722 in "Theoremata tum logometrica tum triogonometrica datarum fluxionum fluentes exhibentia, per methodum mensurarum ulterius extensam" (Theorems, some logorithmic, some trigonometric, which yield the fluents of given fluxions by the method of measures further developed). He understood trig rather well, and from this perspective, both Cotes and Euler's formulas can be regarded as the continuation of the solutions of $x = 1$ into the complex plane. The solutions define the very simple function with domain 1 and 1 and range 1, which is then analytically continued as a circle of radius 1 in the complex domaina type of interpolation (hover over the interpolation link in the Wiki on Roger Cotes) satisfying a simple functional equation $f(x)=1$. (Other examples of types of interpolation/analytic continuation from functions with discrete integer domains to those with continuous complex domains (related to Newton and sinc/cardinal series interpolations) are given in this MOQ and this MSEQ.)
From a broader perspective Cotes' log formula is a clear example of analytic continuation of the log as a mapping from the real numbers to the real to a mapping of the complex to the complex. Cotes was, of course, aware that (indeed utilized, and would have taken for granted that anyone familiar with the log knew also), for $u,v > 0$,
$$\ln(u)+\ln(v) = \ln(uv),$$
so he wrote down the most difficult part of the analytic continuation of the log from the positive reals to the complex (albeit not explicitly accounting for multiplicity)
$$\ln(r) + ix = \ln[\; r\; (\;\cos(x) + i \; \sin(x)\;) \;].$$
Refs in Wikipedia: John Napier,The History of Logarithms, Logarithm, Roger Cotes, Euler's identity, Euler's Formula.
In addition to Euler summation with complex arguments, Euler was the first to extend the factorial to the gamma function for complex arguments to develop a fractional calculus with his hybrid MellinLaplace integral rep for the gamma function (see "The Euler legacy to modern physics" by Dattoli and Del Franco and the MSEQ noted above). Euler's integral for the beta function allows the same for the generalized binomial coefficients, which Newton (again, colleague of Cotes) had done for the extension to the reals of the integer binomial coefficients. Unfortunately, Euler didn't fully understand the extension to complex numbers (Argand and Wessel come later) otherwise he would have scooped Cauchy, Liouville, and Riemann on the calculus of complex analysis.
For a prehistory of the Riemann zeta function, see "Aspects of ZetaFunction Theory in the Mathematical Works of Adolf Hurwitz" by Oswald and Steuding. The authors don't say whether 's` is real or complex in their discussion of the prehistory of zeta. It would have been natural for Euler and others before Riemann to consider $s$ complex. Euler had the association to powers of pi for even integer arguments of zeta that would have suggested a connection to the complex via both his fabulous formula and his reflection formula for the gamma function, but then he had nothing much to glean from this perspective without Riemann's Mellin transform rep. through which Riemann was the first to really tease out new properties of zeta, to apply Euler's reflection formula to give the Hankel contour continuation of zeta from the right halfplane to the full complex plane, and to develop a clever algorithm to determine the nontrivial zeros, among other developments.
A red herring seems to be some shortsighted effort to force an artificial dichotomy between interpolation and analytic continuation. I use Cotes'(and Newton's) interest and skill at interpolation in the real realm (certainly related to approximating celestial orbits) to indicate he was predisposed to make analytic continuations. In addition, there is no dichotomy. In several MO and MSE questions, I show how interpolation is related to analytic continuation of the factorial to the gamma function, the Bernoulli numbers to the Riemann zeta, the Bernoulli polynomials to the Hurwitz zeta, and the classic calculus of integer powers of the derivative op to complex noninteger values, among other interpolations/ACs (e.g., start at this MOQ or this MOQ). These can be related to sinc function/cardinal series interpolations, binomial expansion interpolation, and/or Newton interpolation and probably others (e.g., this MOQ). Some more sophisticated associations are related to Mahler's theorem and the ref in the answer to this MOQ. One aspect of Riemann's gifts was his insight on how this is related to the Mellin transform.
(For accessibility bias, see Khaneman and Tversky.)

1$\begingroup$ The Wiki article notes that one of the Bernoullis had trouble interpreting complex logarithms, but I also remember reading, maybe in Kline, that the mage Euler waffled on the interpretation at times also (one of my consolations when I discover my multitude of math errors). $\endgroup$ – Tom Copeland Jan 25 at 17:48

1$\begingroup$ Ahh, Cotes worked with Newton, but was one of those brilliant mathematicians who died at a young age. Btw, the function continued above has domain 1 and 1 and range 1. Cotes did work on interpolation, which is one aspect of analytic continuation. $\endgroup$ – Tom Copeland Jan 25 at 18:45

$\begingroup$ Where is the analytic continuation in Cotes ? en.wikipedia.org/wiki/Euler%27s_formula#cite_note4 makes it clear that he had a geometric argument and that he was confusing $i$ with $i$ in his calculations. I think it all depends on his definitions of $\ln,\cos,\sin$ and the note suggests they were sloppy. $\endgroup$ – reuns Jan 26 at 5:07

4$\begingroup$ "Cotes did work on interpolation, which is one aspect of analytic continuation." Sure, and Euclid did work on straight lines, which is one aspect of the Riemann Hypothesis. But there's a lot more to RH than just straight lines, and there's a lot more to analytic continuation than just interpolation. $\endgroup$ – Gerry Myerson Jan 26 at 22:19

1