$\newcommand{\N}{\mathbb N} \newcommand{\R}{\mathbb R} \newcommand{\B}{\mathcal B} \newcommand{\F}{\mathcal F} \newcommand{\la}{\lambda} \newcommand{\si}{\sigma} \newcommand{\Si}{\Sigma} \renewcommand{\c}{\circ} \newcommand{\tr}{\operatorname{tr}}$

There can be no reasonable notion of the conditional expectation without the sigma-finiteness condition, even in the discrete setting. E.g., let $X=\N$, $\B=2^\N$, and let $\F$ be any sigma-algebra over $\N$ containing an infinite atom $A\subseteq\N$; for instance, one may take $\F=\{\emptyset,\N\}$, with $A=\N$. Let $\nu$ be the counting measure on $\B=2^\N$, and let $f(x)=1/2^x$ for $x\in\N$. Then $E_\nu f=1\in\R$.

However, on the atom $A$ one cannot reasonably ascribe any value to the conditional expectation $E_\nu(f|\F)$, because such a value (say $v$) could reasonably be only the $\nu$-average of $f$ on $A$. Indeed, if you take $v=0$, this would imply $\int_A f\,d\nu=0$, which is false; if you take $v\ne0$, this would imply $|\int_A f\,d\nu|=|v|\nu(A)=\infty$, which is also false.

The problem here is that, while the measure $\nu$ is sigma-finite, its restriction $\nu|_\F$ to $\F$ is not.

$\newcommand{\N}{\mathbb N} \newcommand{\R}{\mathbb R} \newcommand{\B}{\mathcal B} \newcommand{\F}{\mathcal F} \newcommand{\la}{\lambda} \newcommand{\si}{\sigma} \newcommand{\Si}{\Sigma} \renewcommand{\c}{\circ} \newcommand{\tr}{\operatorname{tr}}$

The definition you quoted is correct.

However, there can be no reasonable notion of the conditional expectation without the sigma-finiteness condition, even in the discrete setting. E.g., let $X=\N$, $\B=2^\N$, and let $\F$ be any sigma-algebra over $\N$ containing an infinite atom $A\subseteq\N$; for instance, one may take $\F=\{\emptyset,\N\}$, with $A=\N$. Let $\nu$ be the counting measure on $\B=2^\N$, and let $f(x)=1/2^x$ for $x\in\N$. Then $E_\nu f=1\in\R$.

However, on the atom $A$ one cannot reasonably ascribe any value to the conditional expectation $E_\nu(f|\F)$, because such a value (say $v$) could reasonably be only the $\nu$-average of $f$ on $A$. Indeed, if you take $v=0$, this would imply $\int_A f\,d\nu=0$, which is false; if you take $v\ne0$, this would imply $|\int_A f\,d\nu|=|v|\nu(A)=\infty$, which is also false.

The problem here is that, while the measure $\nu$ is sigma-finite, its restriction $\nu|_\F$ to $\F$ is not.