This question has been out for a while, and I think it deserves a thorough answer, even tho I came quite late to this party.

As both the classical Legendre-Jacobi theory and the Carlson theory have been mentioned, I'll treat the OP's integral in both viewpoints.

---

## Legendre-Jacobi

The OP came pretty close to using the correct substitution. One thing that could have been done instead is to recall the Pythagorean identity $1+\tan^2 u=1$, so that the substitution that should have been used is $x=a \tan^2 u$, where $a=2$ or $a=3$. Taking the smaller value of $a$, and after some amount of algebra, we obtain

$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=\int_0^{\pi/2}\frac{2}{\sqrt{3-\sin^2u}}\mathrm du\\
&=\frac2{\sqrt{3}}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{1-\frac13\sin^2u}}\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$

where I use the parameter convention for elliptic integrals. (Relatedly, see [here](https://math.stackexchange.com/a/108659) for an extended discussion on the notational confusion surrounding elliptic integrals.)

Had we chosen the substitution with $a=3$ instead, we would have instead obtained the result $\sqrt{2}K\left(-\frac12\right)$, which is equivalent through the **[imaginary modulus transformation](https://dlmf.nist.gov/19.7.E5)**

$$K(-m)=\frac1{\sqrt{1+m}}K\left(\frac{m}{m+1}\right)$$

---

## Carlson

In Carlson's theory, there is the [general hypergeometric function](https://dlmf.nist.gov/19.16.ii)

$$R_{-a}(b_1,\dots,b_k;z_1,\dots,z_k)=\frac1{\mathbf B\left(a,-a+\sum_j b_j\right)}\int_0^\infty u^{-a-1+\sum_j b_j}\prod_j \left(u+z_j\right)^{-b_j}\mathrm du$$

where $\mathbf B(x,y)=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$ is the usual Euler beta function.

This multivariable hypergeometric function is a homogenized/symmetrized version of the classical Lauricella $F_D$ function (see e.g. [this paper](https://doi.org/10.1016/0022-247X(63)90067-2) where Carlson introduced his function, tho that reference uses an opposite sign convention for $a$).

With this consideration, the OP's original integral can be expressed either as a three-variable Carlson integral, or a two-variable Carlson integral:

$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=2\,R_{-\frac12}\left(\frac12,\frac12,\frac12;0,2,3\right)\\&=\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)\end{align*}$$

The three-variable (incomplete) case occurs often enough that it is [given the notation](https://dlmf.nist.gov/19.16.E1)

$$\begin{align*}R_F(x,y,z)&=\frac12\int_0^\infty\frac{\mathrm du}{\sqrt{(u+x)(u+y)(u+z)}}\\&=R_{-\frac12}\left(\frac12,\frac12,\frac12;x,y,z\right)\end{align*}$$

and the two-variable form is the so-called ["complete case"](https://doi.org/10.1215/S0012-7094-64-03139-4),

$$\begin{align*}R_K(x,y)&=R_{-\frac12}\left(\frac12,\frac12;x,y\right)\\&=\frac2{\pi}R_F(0,x,y)\end{align*}$$

In fact, the two-variable form has the [integral representation](https://dlmf.nist.gov/19.23.E1)

$$R_K(x,y)=\frac2{\pi}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{x\cos^2 u+y\sin^2 u}}$$

which one recognizes to be related to the integral representation of [Gauss's arithmetic-geometric mean (AGM)](https://dlmf.nist.gov/19.8.E4):

$$R_K(x,y)=\frac1{\operatorname{agm}(\sqrt{x},\sqrt{y})}$$

Thus, the OP's integral is $\pi/\operatorname{agm}(\sqrt{2},\sqrt{3})$.

Additionally, the two-variable Carlson function is also [related to the Gauss hypergeometric function](https://dlmf.nist.gov/19.25.E43), being a homogeneous version of it:

$$R_{-a}(b_1,b_2;x,y)=y^{-a}{}_2 F_1\left({{a,b_1}\atop{b_1+b_2}}\middle|1-\frac{x}{y}\right)$$

so one has

$$\begin{align*}\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)&=\frac{\pi}{\sqrt{3}}{}_2 F_1\left({{\frac12,\frac12}\atop{1}}\middle|\frac13\right)\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$

where we have used the [hypergeometric representation of](http://functions.wolfram.com/HypergeometricFunctions/Hypergeometric2F1/03/07/07/01/0009/) [the complete elliptic integral of the first kind](http://functions.wolfram.com/HypergeometricFunctions/Hypergeometric2F1/03/07/07/01/0008/).