Topology on Minkowski space $\mathbb{R}^{4}$ and Lorentz invariant measure Let us consider the Minkowski space $(\mathbb{R}^{4},\eta)$ and the mass shell $H_{m}$, $m\ge 0$, given by:
\begin{eqnarray}
H_{m}:=\{x=(x_{0},x_{1},x_{2},x_{3}) \in \mathbb{R}^{4}: \hspace{0.1cm} x\cdot \tilde{x} = m^{2}, x_{0}>0\} \tag{1}\label{1}
\end{eqnarray}
where $\tilde{x}$ is defined by $x=(x_{0},x_{1},x_{2},x_{3}) \mapsto \tilde{x} := (x_{0},-x_{1},-x_{2},-x_{3})$ and $x\cdot \tilde{x}$ is the usual Lorentz inner product:
$$x \cdot \tilde{x} = \eta(x,x).$$
Further, define $j_{m}: \mathbb{R}^{4}\to \mathbb{R}^{3}$ (if $m=0$, take $\mathbb{R}^{3}\setminus \{0\}$ instead) as the homeomorphism $x=(x_{0},x_{1},x_{2},x_{3}) \mapsto (x_{1},x_{2},x_{3})$. The set function:
\begin{eqnarray}
\mu_{m}(E) := \int_{j_{m}(E)} \frac{d^{3}x}{\sqrt{m^{2}+|x|^{2}}} \tag{2}\label{2}
\end{eqnarray}
defines a measure on $H_{m}$.
In the literature, $\mu_{m}$ is called Lorentz invariant measure because it is said to be invariant under Lorentz transformations on $\mathcal{L}_{+}^{\uparrow}$.
I have some really basic questions about this settings, but which are not explicitly stated anywhere I know.
Q1: The "Lorentz inner product" $x\cdot\tilde{x}$ might be misleading since $x\cdot \tilde{x}$ is not positive-definite, so it does not define an inner product in mathematical terms. Does it play any role in defining the topology on $\mathbb{R}^{4}$, or is $\mathbb{R}^{4}$ equipped with its usual Euclidean topology? Note that the topology is important because $j_{m}$ is assumed to be an homeomorphism and, also, because $\mu_{m}$ is a measure on $H_{m}$ which, I assume, becomes a measure space when equipped with the Borel $\sigma$-algebra inherited from $\mathbb{R}^{4}$.
Q2: I know that $H_{m}$ is invariant under $\mathcal{L}_{+}^{\uparrow}$ because, if $\Lambda \in \mathcal{L}_{+}^{\uparrow}$ we have: (a) $(\Lambda x)_{0} > 0$ whenever $x_{0}>0$, and (b) $\Lambda x \cdot \tilde{\Lambda x} = x\cdot \tilde{x} = m^{2}$, since it is a Lorentz transformation. But what is the precise meaning of $\mu_{m}$ being Lorentz invariant? Is it as usually stated in ergodic theory, i.e. $\mu_{m}(\Lambda^{-1}E) = \mu_{m}(E)$ for every measurable $E$?
 A: Judging by your notation, I reckon you are getting the background for your questions from the Appendix to Section IX.8 of the book by M. Reed and B. Simon, Methods of Modern Mathematical Physics II: Fourier Analysis, Self-Adjointness (Academic Press, 1975). That being said, let us proceed to answering your questions (I wasn't as fast as Willie Wong and gmvh, but I'll make up for it by providing more details):
Q1: The term "inner product" is being used in a broader sense here. Indeed it would have been more precise to call $\eta(x,x)$ a "nondegenerate quadratic form of Lorentz signature", but that is a bit long, so it is custom to include possibly indefinite but still nondegenerate quadratic (or, more precisely, symmetric bilinear) forms into the term "inner product". The topology of $\mathbb{R}^4$ can indeed be recovered from $\eta$, but instead of Euclidean balls one uses the so called causal diamonds: given $x=(x_0,x_1,x_2,x_3)=(x_0,\mathbf{x})\in\mathbb{R}^4$, $r>0$, the causal diamond centered at $x$ with spatial radius $r$ is the open subset (w.r.t. the usual Euclidean topology of $\mathbb{R}^4$) $$\mathscr{O}_{x,r}=\{y=(y_0,\mathbf{y})\in\mathbb{R}^4\ |\ |y_0-x_0|+\|\mathbf{y}-\mathbf{x}\|<r\}\ .$$ An interpretation of causal diamonds is provided by the causal structure induced by $\eta$: $\mathscr{O}_{x,r}$ is the set $\mathscr{D}_{x_-,x_+}$ of all points $y\in\mathbb{R}^4$ through which one can draw a timelike curve segment (with respect to $\eta$) from the past endpoint $x_-=x-(r,\mathbf{0})$ of $\mathscr{O}_{x,r}$ to the future endpoint $x_+=x+(r,\mathbf{0})$ of $\mathscr{O}_{x,r}$: $$\mathscr{O}_{x,r}=\mathscr{D}_{x_-,x_+}=\{y\in\mathbb{R}^4\ |\ \exists\gamma:[0,1]\rightarrow\mathbb{R}^4\text{ piecewise smooth such that }$$ $$\phantom{\mathscr{O}_{x,r}=\mathscr{D}_{x_-,x_+}=}\eta(\dot{\gamma}(t),\dot{\gamma}(t))>0\text{ if defined, }\gamma(0)=x_-,\gamma(1)=x_+\}\ .$$ Notice that $x=\frac{1}{2}(x_-+x_+)$ and $r=\frac{1}{2}\sqrt{\eta(x_+-x_-,x_+-x_-)}$.
If you take the family of all regions $\mathscr{D}_{x_-,x_+}$ defined above, with $x_-,x_+$  running through all pairs of points in $\mathbb{R}^4$ such that $x_-$ chronologically precedes $x_+$ with respect to $\eta$ (these regions can be obtained by e.g. applying all composites of Poincaré and scale transformations to a fixed $\mathscr{O}_{x,r}$), as a basis for a topology on $\mathbb{R}^4$ (which is called the Alexandrov topology and can be understood as the order topology induced by the chronology preorder associated to $\eta$), it is easy to see that you just get the Euclidean topology.
(Edit: in connection to Willie Wong's answer, the fact that the Alexandrov and the Euclidean = manifold topologies coincide on $\mathbb{R}^4$ means that the latter, endowed with $\eta$ as a Lorentz metric, is what is called a strongly causal Lorentzian manifold. This is also equivalent to requiring the Alexandrov topology to be Hausdorff, by the way - particularly, the Hausdorff property implies metrizability for the Alexandrov topology)
Q2: Yes, the Lorentz invariance of $\mu_m$ means precisely what you wrote.
A: Q1 The topology on $\mathbb{R}^4$ is the usual one. This is the general case for Lorentzian geometry: the topology is the one defined by the charts in your atlas.
Q2 Given a fixed Lorentz transformation $\Lambda$, it sends $H_m\to H_m$ as you observed. It is a linear transformation of $\mathbb{R}^4$, and hence restricts to a diffeomorphism of $H_m$ to itself. Invariance of $\mu_m$ is simply that it agrees with its pushforward by $\Lambda$, which is the same as what you wrote.
Remark: when $m > 0$, your $\mu_m$ is the volume measure associated to the induced Riemannian metric on $H_m$, and so invariance can also be seen by the fact that $\Lambda$ is an isometry of $\mathbb{R}^4$ and hence restricts to an isometry of $H_m$.
A: Q1: You use the usual topology on $\mathbb{R}^4$. (If you were to use $\eta$ to naively define a topology, you couldn't separate points on the lightcone $\eta(x,x)=0$, which would not be terribly physical, since these correspond to distinct spacetime events.)
Q2: Yes. (In fact, I'm not sure what else it could possibly be taken to mean.)
