Here's a proof of the statement in Micheal's answer (repeated below) that conceptually has more steps but hopefully more illuminating than the computation found in Huybrechts - we hope to make clear why the $\mathbb{P}^n$ should have the opposite orientation.

Proposition: Let $x\in X$ be a point in a complex manifold $X$. Then the blow-up $\text{Bl}_x(X)$ is \emph{diffeomorphic as an oriented differentiable manifold} to $X \# \overline{\mathbb{P}^n}$.

Proof: We work locally, assuming $X$ is $\mathbb{C}^n$.

Claim 1: The blow-up at the origin $\text{Bl}_0(\mathbb{C}^n)$ is biholomorphic to the total space $E$ of the tautological bundle $\mathcal{O}(-1)$ over $\mathbb{P}^{n-1}$. The exceptional fibre of the blow-up is the zero section of the bundle.

Claim 2: For a complex line bundle $L$ over some smooth manifold, $L^* \cong \overline{L}$ as complex bundles, where $L^*$ is the dual bundle, while $\overline{L}$ has the same underlying real bundle as $L$ but with the negative complex structure, i.e. the conjugate bundle. (Follows by choosing some fibrewise Hermitian form)

Claim 3: $\mathbb{P}^n \backslash \{x\}$ is a line bundle over $\mathbb{P}^{n-1}$ by projecting from the point $x$, and is isomorphic as complex line bundles to $\mathcal{O}(1)$.

Proof of claims are left to the reader.

Now let us assemble the pieces. By claim 1, we have orientation-preserving (in fact holomorphic) inclusions $\mathbb{C}^n \hookleftarrow \mathbb{C}^n \backslash \{0\} \cong E \backslash \{\text{zero section}\} \hookrightarrow E$. Let $F$ be the total space of $\mathcal{O}(1)$. Then by claim 2, in particular, $E \cong \overline{F}$ as differential manifolds, and sends the zero section to the zero section. By claim 3, the one-point compactification of $F$ is $\mathbb{P}^n$. All of this amounts to saying that (real) rays going into the origin of $\mathbb{C}^n$ get magically transformed (under the identification in claim 1) into rays going out of $x$ in $\mathbb{P}^n$. This allows us to say that the blow-up $\text{Bl}_0(\mathbb{C}^n)$ is the connect sum of $\mathbb{C}^n$ with $\overline{\mathbb{P}^n}$.

Here's a proof of the statement in Michael's answer (repeated below) that conceptually has more steps but hopefully more illuminating than the computation found in Huybrechts - we hope to make clear why the $\mathbb{P}^n$ should have the opposite orientation.

Proposition: Let $x\in X$ be a point in a complex manifold $X$. Then the blow-up $\text{Bl}_x(X)$ is diffeomorphic as an oriented differentiable manifold to $X \# \overline{\mathbb{P}^n}$.

Proof: We work locally, assuming $X$ is $\mathbb{C}^n$.

Claim 1: The blow-up at the origin $\text{Bl}_0(\mathbb{C}^n)$ is biholomorphic to the total space $E$ of the tautological bundle $\mathcal{O}(-1)$ over $\mathbb{P}^{n-1}$. The exceptional fibre of the blow-up is the zero section of the bundle.

Claim 2: For a complex line bundle $L$ over some smooth manifold, $L^* \cong \overline{L}$ as complex bundles, where $L^*$ is the dual bundle, while $\overline{L}$ has the same underlying real bundle as $L$ but with the negative complex structure, i.e. the conjugate bundle. (Follows by choosing some fibrewise Hermitian form)

Claim 3: $\mathbb{P}^n \backslash \{x\}$ is a line bundle over $\mathbb{P}^{n-1}$ by projecting from the point $x$, and is isomorphic as complex line bundles to $\mathcal{O}(1)$.

Proof of claims are left to the reader.

Now let us assemble the pieces. By claim 1, we have orientation-preserving (in fact holomorphic) inclusions $\mathbb{C}^n \hookleftarrow \mathbb{C}^n \backslash \{0\} \cong E \backslash \{\text{zero section}\} \hookrightarrow E$. Let $F$ be the total space of $\mathcal{O}(1)$. Then by claim 2, in particular, $E \cong \overline{F}$ as differential manifolds, and sends the zero section to the zero section. By claim 3, the one-point compactification of $F$ is $\mathbb{P}^n$. All of this amounts to saying that (real) rays going into the origin of $\mathbb{C}^n$ get magically transformed (under the identification in claim 1) into rays going out of $x$ in $\mathbb{P}^n$. This allows us to say that the blow-up $\text{Bl}_0(\mathbb{C}^n)$ is the connect sum of $\mathbb{C}^n$ with $\overline{\mathbb{P}^n}$.