$\newcommand{\Si}{\Sigma}\newcommand{\PP}{\mathscr P}\newcommand{\PPP}{\mathfrak P}$Let $X$ be a discrete random variable (r.v.) taking distinct values $x_1,x_2,\dots$, and let $p_i:=P(X = x_i)$. Then the entropy of $X$ is defined by the formula
\begin{equation*}
H(X): = \sum_i p_i \log \frac1{p_i}. \tag{1}
\end{equation*}
Note that it does not matter whatsoever in what set/space the values $x_1,x_2,\dots$ are assumed to be; it does not matter if some of these values are close to, or far away from, one another -- in any sense.
What matters is that the $p_i$'s are the probabilities of distinct values of the r.v. $X$. Clearly, if we aggregate some of these values, then the entropy will go down; and if we split some of these values, then the entropy will go up.
Therefore, the formula $H(X) = \int dx\, p(x) \log \frac1{p(x)}$ will hardly make sense if, say, the integral is understood in the Riemann sense, implying the rather arbitrary grouping of the values $x$ according to the standard metric on $\mathbb R$. Moreover and much more importantly, the Riemann sums $\sum_i p(x_i)\Delta x_i \log \frac1{p(x_i)}\,$ for this integral are quite different from the sums $\sum_i p(x_i)\Delta x_i \log\frac1{p(x_i)\Delta x_i}$ that would genuinely correspond to the reality-based definition (1). Also, these latter sums will usually be very large if the $\Delta x_i$'s are very small, and the values of these sums may fluctuate wildly depending on the choice of the $\Delta x_i$'s.
The more general formula $H(X) = \int p(x) \log \frac1{p(x)}\, \mu(dx)$, where $\mu$ is a measure and the grouping of the values $x$ occurs according to the closeness of the corresponding values of $p(x)$ (!!), will hardly make more sense than the Riemann integral.
The only exception here would be when $\mu$ is the counting measure, with which no actual grouping (or splitting) of any values occurs. Then for the density (say $p$) of the distribution of the r.v. $X$ with respect to the counting measure $\mu$, the condition $\int p\,d\mu=1$ can be rewritten as $\sum_x p(x)=1$, which will imply that $p(x)\ne0$ only for (at most) countably many values of $x$, so that the r.v. $X$ is necessarily discrete -- and then we can write
\begin{equation*}
H(X) = \int p(x) \log \frac1{p(x)}\,\mu(dx)
=\sum_x p(x) \log \frac1{p(x)},
\end{equation*}
which is the same as (1), up to the change in notation.
So, if the r.v. $X$ is not discrete, then the only reasonable value to assign to the entropy of $X$ appears to be $\infty$, at least from the viewpoint of information theory.
As for the integral $\tilde H(X):=\int p(x) \log \frac1{p(x)}\, \mu(dx)$ in the case when $\mu$ is the Lebesgue measure, the main interest to it seems to be the easily seen fact (see e.g. Barron) that the maximum of $\tilde H(X)$ over all absolutely continuous r.v.'s $X$ with a fixed variance is attained when the distribution of $X$ is normal; moreover, the absolute value of difference of $\tilde H(X)$ from its maximum equals the relative entropy $\int p(x)\log\frac{p(x)}{\varphi(x)}\, dx$, where $\varphi$ is the normal density with the same mean and variance as $p$.
However, what is actually used in the proofs is the relative entropy $D(P\parallel Q):=\int_\Omega dP\,\log\frac{dP}{dQ}$ (also known as the Kullback--Leibler divergence), which is well defined for any probability measures $P$ and $Q$ such that $P$ is absolutely continuous with respect to $Q$. One can show that (with appropriate conventions about the division and multiplication by $0$)
\begin{equation*}
D(P\parallel Q)=\sup\sum_j P(A_j)\log\frac{P(A_j)}{Q(A_j)}
=\lim\sum_j P(A_j)\log\frac{P(A_j)}{Q(A_j)},\tag{2}\label{2}
\end{equation*}
where the $\sup$ is taken over the set (say $\mathcal P$) of all finite (or, alternatively, at most countable) measurable partitions $(A_j)$ of $\Omega$ and the $\lim$ is taken over the set $\mathcal P$ of partitions directed by the refinement relation. (Please let me know if details on \eqref{2} are needed). So, the notion of the relative entropy in general reduces to this notion for discrete distributions, for which axiomatic approaches are well known.
Added on 7/16/23: This can be done as follows. Assume the following axioms:
s1: The (measure of) surprise contained in an event is nonnegative and depends only of the probability of the event. That is, there is a function $f\colon[0,1]\to[0,\infty]$ such that for any probability space $(S,\Si)$, any probability measure $P$ on $\Si$, and any $A\in\Si$ the surprise $s_P(A)$ contained in the event $A$ with respect to (w.r.t.) the probability measure $P$ is $f(P(A))$. Let us refer to $s_P(A)$ as the $P$-surprise of $A$.
s2: The function $f$ is measurable and nontrivial (that is, not everywhere zero and not everywhere $\infty$).
s3: If two independent events occur, their $P$-surprises get added: If events $A$ and $B$ are independent w.r.t. a probability measure $P$, then $s_P(A\cap B)=s_P(A)+s_P(B)$.
rs1: For any probability space $(S,\Si)$, any probability measures $P$ and $Q$ on $\Si$, and any $A\in\Si$, the relative $(P\parallel Q)$-surprise $s_{P\parallel Q}(A)$ of $A$ is the difference $s_Q(A)-s_P(A)$.
rs2: For any probability space $(S,\Si)$, any probability measures $P$ and $Q$ on $\Si$, and any $\Si$-measurable partition $\PP=(A_1,\dots,A_n)$ of $S$, the $(P\parallel Q)$-surprise of $\PP$, denoted by $s_{P\parallel Q}(\PP)$, is the (discrete) random variable (r.v.) that is the simple function $\sum_{j=1}^n s_{P\parallel Q}(A_j)\,1_{A_j}$.
rs3: For any probability space $(S,\Si)$, any probability measures $P$ and $Q$ on $\Si$, and any $\Si$-measurable partition $\PP=(A_1,\dots,A_n)$ of $S$, the entropy $D_\PP(P\parallel Q)$ of $P$ relative to $Q$ w.r.t. $\PP$ is the expectation w.r.t. to $P$ of the r.v. $s_{P\parallel Q}(\PP)$ (that is the $(P\parallel Q)$-surprise of the partition $\PP$):
\begin{equation*}
D_\PP(P\parallel Q):=Es_{P\parallel Q}(\PP).
\end{equation*}
(So, $D_\PP(P\parallel Q)$ is the average additional surprise w.r.t. the partition $\PP$ that one will get assuming that the probability distribution is $Q$ whereas the actual probability distribution is $P$.)
rs4: For any probability space $(S,\Si)$ and any probability measures $P$ and $Q$ on $\Si$, the entropy $D_\PP(P\parallel Q)$ of $P$ relative to $Q$ is the supremum of the entropy $D_\PP(P\parallel Q)$ of $P$ relative to $Q$ w.r.t. $\PP$ over the set (say $\PPP$) of all finite $\Si$-measurable partitions $\PP$ of $S$:
\begin{equation*}
D(P\parallel Q):=\sup_{\PP\in\PPP}D_\PP(P\parallel Q).
\end{equation*}
It follows easily from Axioms s1, s2, s3 that $f(p)=\log_b\frac1p$ for some real $b>1$ and all $p\in(0,1]$, with $f(0)=\infty$. So, by Axioms rs1, rs2, rs3, for any probability space $(S,\Si)$, any probability measures $P$ and $Q$ on $\Si$, and any $\Si$-measurable partition $\PP=(A_1,\dots,A_n)$ of $S$,
\begin{equation*}
D_\PP(P\parallel Q)=\sum_{j=1}^n P(A_j)\log_b\frac{P(A_j)}{Q(A_j)}, \tag{3}\label{3}
\end{equation*}
assuming the conventions $p\log_b\frac pq:=0$ whenever $p=0$ and $p\log_b\frac pq:=\infty$ whenever $p>0=q$.
Then it is easy to see that $D_{\PP_1}(P\parallel Q)\le D_{\PP_2}(P\parallel Q)$ if a partition $\PP_2$ is a refinement of a partition $\PP_1$. Now, with the set $\PPP$ directed by refinement, one can show that
\begin{equation*}
D(P\parallel Q)
=\lim_{\PP\in\PPP}D_\PP(P\parallel Q)=\int_S dP\,\log_b\frac{dP}{dQ};
\end{equation*}
the latter integral is defined as $\infty$ if $P$ is not absolutely continuous w.r.t. $Q$.
That is, if $\mu$ is any measure w.r.t. which $P$ and $Q$ are absolutely continuous (for instance, one may take $\mu=P+Q$) with respective densities $p$ and $q$, then
\begin{equation*}
D(P\parallel Q)=\int_S d\mu\,p\log_b\frac pq. \quad\Box
\end{equation*}
