Letting $|a|_p = 1/p^m$, so $m<n$, the Mellin transform $(M\phi)(\chi)$ of a character $\chi$ of $\mathbf Q_p^\times$ is the following integral $$ (M\phi)(\chi) = \int_{\mathbf Q_p^\times} \xi_{a + p^n\mathbf Z_p}(x)\chi(x)d^\times x, $$ where I write $d^\times x$ for the (multiplicative) Haar measure on $\mathbf Q_p^\times$ that you write as $p/(p-1) dx/|x|_p$. It is the Haar measure on $\mathbf Q_p^\times$ that gives the compact open subgroup $\mathbf Z_p^\times$ measure 1. After some calculation that I omit (tell me if you can work this out), we get $$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} \chi(y)d^\times y, $$ which is $\chi(a)$ times the integral of the multiplicative character $\chi$ over the compact multiplicative group $1 + p^{n-m}\mathbf Z_p$. The value of this is determined by whether or not $\chi$ is trivial on $1 + p^{n-m}\mathbf Z_p$: if $\chi \not\equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = 0, $$ and if $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = \chi(a)\int_{1 + p{n-m}\mathbf Z_p} d^\times y = \frac{\chi(a)}{[\mathbf Z_p^\times:1 + p^{n-m}\mathbf Z_p]} = \frac{\chi(a)}{p^{n-m-1}(p-1)}. $$$$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} d^\times y = \frac{\chi(a)}{[\mathbf Z_p^\times:1 + p^{n-m}\mathbf Z_p]} = \frac{\chi(a)}{p^{n-m-1}(p-1)}. $$ There are only finitely many connected components containing the $\chi$ where $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ since such $\chi$ are determined on $\mathbf Z_p^\times$ (not on $\mathbf Q_p^\times$!) by their values on $\mathbf Z_p^\times/(1 + p^{n-m}\mathbf Z_p)$, which is a finite group (so it has only finitely many homomorphisms to $\mathbf C^\times$). Thus $M\phi$ is a "polynomial" in the sense defined above Prop. 4.6.
Letting $|a|_p = 1/p^m$, so $m<n$, the Mellin transform $(M\phi)(\chi)$ of a character $\chi$ of $\mathbf Q_p^\times$ is the following integral $$ (M\phi)(\chi) = \int_{\mathbf Q_p^\times} \xi_{a + p^n\mathbf Z_p}(x)\chi(x)d^\times x, $$ where I write $d^\times x$ for the (multiplicative) Haar measure on $\mathbf Q_p^\times$ that you write as $p/(p-1) dx/|x|_p$. It is the Haar measure on $\mathbf Q_p^\times$ that gives the compact open subgroup $\mathbf Z_p^\times$ measure 1. After some calculation that I omit (tell me if you can work this out), we get $$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} \chi(y)d^\times y, $$ which is $\chi(a)$ times the integral of the multiplicative character $\chi$ over the compact multiplicative group $1 + p^{n-m}\mathbf Z_p$. The value of this is determined by whether or not $\chi$ is trivial on $1 + p^{n-m}\mathbf Z_p$: if $\chi \not\equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = 0, $$ and if $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = \chi(a)\int_{1 + p{n-m}\mathbf Z_p} d^\times y = \frac{\chi(a)}{[\mathbf Z_p^\times:1 + p^{n-m}\mathbf Z_p]} = \frac{\chi(a)}{p^{n-m-1}(p-1)}. $$ There are only finitely many connected components containing the $\chi$ where $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ since such $\chi$ are determined on $\mathbf Z_p^\times$ (not on $\mathbf Q_p^\times$!) by their values on $\mathbf Z_p^\times/(1 + p^{n-m}\mathbf Z_p)$, which is a finite group (so it has only finitely many homomorphisms to $\mathbf C^\times$). Thus $M\phi$ is a "polynomial" in the sense defined above Prop. 4.6.
Letting $|a|_p = 1/p^m$, so $m<n$, the Mellin transform $(M\phi)(\chi)$ of a character $\chi$ of $\mathbf Q_p^\times$ is the following integral $$ (M\phi)(\chi) = \int_{\mathbf Q_p^\times} \xi_{a + p^n\mathbf Z_p}(x)\chi(x)d^\times x, $$ where I write $d^\times x$ for the (multiplicative) Haar measure on $\mathbf Q_p^\times$ that you write as $p/(p-1) dx/|x|_p$. It is the Haar measure on $\mathbf Q_p^\times$ that gives the compact open subgroup $\mathbf Z_p^\times$ measure 1. After some calculation that I omit (tell me if you can work this out), we get $$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} \chi(y)d^\times y, $$ which is $\chi(a)$ times the integral of the multiplicative character $\chi$ over the compact multiplicative group $1 + p^{n-m}\mathbf Z_p$. The value of this is determined by whether or not $\chi$ is trivial on $1 + p^{n-m}\mathbf Z_p$: if $\chi \not\equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = 0, $$ and if $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} d^\times y = \frac{\chi(a)}{[\mathbf Z_p^\times:1 + p^{n-m}\mathbf Z_p]} = \frac{\chi(a)}{p^{n-m-1}(p-1)}. $$ There are only finitely many connected components containing the $\chi$ where $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ since such $\chi$ are determined on $\mathbf Z_p^\times$ (not on $\mathbf Q_p^\times$!) by their values on $\mathbf Z_p^\times/(1 + p^{n-m}\mathbf Z_p)$, which is a finite group (so it has only finitely many homomorphisms to $\mathbf C^\times$). Thus $M\phi$ is a "polynomial" in the sense defined above Prop. 4.6.
The connected components of the group of characters $\Omega(\mathbf Q_p^\times)$ are entirely determined by how the characters look on $\mathbf Z_p^\times$: two characters of $\mathbf Q_p^\times$ are in the same connected component exactly when they are equal on $\mathbf Z_p^\times$, and by continuity a character $\mathbf Z_p^\times \rightarrow \mathbf C^\times$ is trivial on some neighborhood $1 + p^n\mathbf Z_p$ of 1 (a subgroup!) since $\mathbf C^\times$ has no subgroup in a neighborhood of 1 other than $\{1\}$. Therefore a character on $\mathbf Z_p^\times$ is a homomorphism to $\mathbf C^\times$ on some quotient group $\mathbf Z_p^\times/(1 + p^n\mathbf Z_p)\cong (\mathbf Z/p^n\mathbf Z)^\times$$\mathbf Z_p^\times/(1 + p^k\mathbf Z_p)\cong (\mathbf Z/p^k\mathbf Z)^\times$, which is finite. Going the other way, for each homomorphism $(\mathbf Z/p^n\mathbf Z)^\times \rightarrow \mathbf C^\times$$(\mathbf Z/p^k\mathbf Z)^\times \rightarrow \mathbf C^\times$ we can lift it to a character $\eta$ on $\mathbf Z_p^\times$ using the composite map $$ \mathbf Z_p^\times \rightarrow \mathbf Z_p^\times/(1+p^k\mathbf Z_p) \cong (\mathbf Z/p^k\mathbf Z_p)^\times \rightarrow \mathbf C^\times $$ (automaticallythis is automatically continuous there since it is a homomorphism and it is trivial on the neighborhood $1 + p^n\mathbf Z_p$$1 + p^k\mathbf Z_p$ of 1) and then we can multiply this by an unramified character $\chi$ to get a character of $\mathbf Q_p^\times$: $p^nu \mapsto \chi(p)^n\eta(u)$. In other notation, since unramified characters of $\mathbf Q_p^\times$ are just complex powers $|\cdot|_p^s$, each character of $\mathbf Q_p^\times$ is $|\cdot|_p^s\eta$ where $s \in \mathbf C$ and $\eta$ is a character of $\mathbf Z_p^\times$: each connected component can be labeled by the common $\eta$ (restriction to $\mathbf Z_p^\times$) for all characters in that component. (The choice of $s$ for a character really is in $\mathbf C/(2\pi i/\log p)\mathbf Z$, a cylinder, which topologically is the same as $\mathbf C^\times$ using $s + (2\pi i/\log p)\mathbf Z \mapsto 1/p^s$.)
For $x \in \mathbf Q_p^\times$, written as $p^nu$ where $n \in \mathbf Z$ and $u \in \mathbf Z_p^\times$, write $u$ as $u_x$ to indicate its dependence on $x$. Then for each character $\eta$ of some $(\mathbf Z/p^k\mathbf Z)^\times$ and $s \in \mathbf C$, we get a character $\chi$ of $\mathbf Q_p^\times$ by $\chi(x) = |x|_p^s\eta(u_x\bmod p^k)$. (Note that $\eta(p)$ makes no sense.) All characters of $\mathbf Q_p^\times$ look like this.
The connected components of the group of characters $\Omega(\mathbf Q_p^\times)$ are entirely determined by how the characters look on $\mathbf Z_p^\times$: two characters of $\mathbf Q_p^\times$ are in the same connected component exactly when they are equal on $\mathbf Z_p^\times$, and by continuity a character $\mathbf Z_p^\times \rightarrow \mathbf C^\times$ is trivial on some neighborhood $1 + p^n\mathbf Z_p$ of 1 (a subgroup!) since $\mathbf C^\times$ has no subgroup in a neighborhood of 1 other than $\{1\}$. Therefore a character on $\mathbf Z_p^\times$ is a homomorphism to $\mathbf C^\times$ on some quotient group $\mathbf Z_p^\times/(1 + p^n\mathbf Z_p)\cong (\mathbf Z/p^n\mathbf Z)^\times$, which is finite. Going the other way, for each homomorphism $(\mathbf Z/p^n\mathbf Z)^\times \rightarrow \mathbf C^\times$ we can lift it to a character $\eta$ on $\mathbf Z_p^\times$ (automatically continuous there since it is a homomorphism and is trivial on the neighborhood $1 + p^n\mathbf Z_p$ of 1) and then we can multiply this by an unramified character $\chi$ to get a character of $\mathbf Q_p^\times$: $p^nu \mapsto \chi(p)^n\eta(u)$. In other notation, since unramified characters of $\mathbf Q_p^\times$ are just complex powers $|\cdot|_p^s$, each character of $\mathbf Q_p^\times$ is $|\cdot|_p^s\eta$ where $s \in \mathbf C$ and $\eta$ is a character of $\mathbf Z_p^\times$: each connected component can be labeled by the common $\eta$ (restriction to $\mathbf Z_p^\times$) for all characters in that component. (The choice of $s$ for a character really is in $\mathbf C/(2\pi i/\log p)\mathbf Z$, a cylinder, which topologically is the same as $\mathbf C^\times$ using $s + (2\pi i/\log p)\mathbf Z \mapsto 1/p^s$.)
The connected components of the group of characters $\Omega(\mathbf Q_p^\times)$ are entirely determined by how the characters look on $\mathbf Z_p^\times$: two characters of $\mathbf Q_p^\times$ are in the same connected component exactly when they are equal on $\mathbf Z_p^\times$, and by continuity a character $\mathbf Z_p^\times \rightarrow \mathbf C^\times$ is trivial on some neighborhood $1 + p^n\mathbf Z_p$ of 1 (a subgroup!) since $\mathbf C^\times$ has no subgroup in a neighborhood of 1 other than $\{1\}$. Therefore a character on $\mathbf Z_p^\times$ is a homomorphism to $\mathbf C^\times$ on some quotient group $\mathbf Z_p^\times/(1 + p^k\mathbf Z_p)\cong (\mathbf Z/p^k\mathbf Z)^\times$, which is finite. Going the other way, for each homomorphism $(\mathbf Z/p^k\mathbf Z)^\times \rightarrow \mathbf C^\times$ we can lift it to a character $\eta$ on $\mathbf Z_p^\times$ using the composite map $$ \mathbf Z_p^\times \rightarrow \mathbf Z_p^\times/(1+p^k\mathbf Z_p) \cong (\mathbf Z/p^k\mathbf Z_p)^\times \rightarrow \mathbf C^\times $$ (this is automatically continuous since it is a homomorphism and it is trivial on the neighborhood $1 + p^k\mathbf Z_p$ of 1) and then we can multiply this by an unramified character $\chi$ to get a character of $\mathbf Q_p^\times$: $p^nu \mapsto \chi(p)^n\eta(u)$. In other notation, since unramified characters of $\mathbf Q_p^\times$ are just complex powers $|\cdot|_p^s$, each character of $\mathbf Q_p^\times$ is $|\cdot|_p^s\eta$ where $s \in \mathbf C$ and $\eta$ is a character of $\mathbf Z_p^\times$: each connected component can be labeled by the common $\eta$ (restriction to $\mathbf Z_p^\times$) for all characters in that component. (The choice of $s$ for a character really is in $\mathbf C/(2\pi i/\log p)\mathbf Z$, a cylinder, which topologically is the same as $\mathbf C^\times$ using $s + (2\pi i/\log p)\mathbf Z \mapsto 1/p^s$.)
For $x \in \mathbf Q_p^\times$, written as $p^nu$ where $n \in \mathbf Z$ and $u \in \mathbf Z_p^\times$, write $u$ as $u_x$ to indicate its dependence on $x$. Then for each character $\eta$ of some $(\mathbf Z/p^k\mathbf Z)^\times$ and $s \in \mathbf C$, we get a character $\chi$ of $\mathbf Q_p^\times$ by $\chi(x) = |x|_p^s\eta(u_x\bmod p^k)$. (Note that $\eta(p)$ makes no sense.) All characters of $\mathbf Q_p^\times$ look like this.
In those notes, taking $F = \mathbf Q_p$, the scary term "unramified character" of $\mathbf Q_p^\times$ means a continuous homomorphism $\chi \colon \mathbf Q_p^\times \rightarrow \mathbf C^\times$ that is trivial (equal to $1$) on the units $\mathbf Z_p^\times$. The simplest example of such a character is the $p$-adic absolute value: $x \mapsto |x|_p$. This is continuous on $\mathbf Q_p^\times$ and it is definitely trivial on $\mathbf Z_p^\times$ since those are exactly the $p$-adic number of $p$-adic absolute value 1. A complex power $x \mapsto |x|_p^s$ for $s \in \mathbf C$ is also an unramified character of $\mathbf Q_p^\times$, and he is saying all unramified characters of $\mathbf Q_p^\times$ look like this for some $s$. Why is that?
Every nonzero $p$-adic number has the form $p^nu$ for some $n \in \mathbf Z$ and $u \in \mathbf Z_p^\times$. For an unramified character $\chi$ of $\mathbf Q_p^\times$, we have $\chi(u) = 1$, so $\chi(p^nu) = \chi(p^n) = \chi(p)^n$. The number $\chi(p)$ is in $\mathbf C^\times$, so we can write $\chi(p)$ as $1/p^s$ for some $s \in \mathbf C$. (This $s$ is not unique, but is well-defined up to adding an integer multiple of $2\pi i/\log p$ on account of looking at the complex solutions $s$ to $p^s = 1$.) Then $\chi(p^nu) = \chi(p)^n = (1/p^s)^n = (1/p^n)^s = |p^nu|_p^s$, so $\chi(x) = |x|_p^s$ for all $x \in \mathbf Q_p^\times$: $\chi$ is the $s$-th power of the basic unramified character $x \mapsto |x|_p$, where $s$ satisfies $\chi(p) = 1/p^s$. When a homomorphism $\mathbf Q_p^\times \rightarrow \mathbf C^\times$ is trivial on $\mathbf Z_p^\times$, it is continuous since it is locally constant (it is constant near 1 and a homomorphism) and is completely determined by its value at $p$.
The value of $\chi(p)$ can be arbitrary in $\mathbf C^\times$: for each $t \in \mathbf C^\times$ set $t = 1/p^s$ for some $s \in \mathbf C$ and define $\chi_t \colon \mathbf Q_p^\times \rightarrow \mathbf C^\times$ by the rule $\chi_t(p^nu) = t^n$ for $u \in \mathbf Z_p^\times$ and $n \in \mathbf Z$. This is a homomorphism, its value at $p$ is $t$, it is trivial on $\mathbf Z_p^\times$ ($\chi_t$ is "unramified"), and it is continuous since it is locally constant. Since $\chi_t(p^nu) = t^n = (1/p^s)^n = (1/p^n)^s = |p^nu|_p^s$, we have $\chi_t(x) = |x|_p^s$ for all $x \in \mathbf Q_p^\times$. This is why he says for each $t \in \mathbf C^\times$ there is a unique unramified character $\chi$ of $\mathbf Q_p^\times$ with $\chi(p) = t$: that $\chi$ is $\chi_t$.
The connected components of the group of characters $\Omega(\mathbf Q_p^\times)$ are entirely determined by how the characters look on $\mathbf Z_p^\times$: two characters of $\mathbf Q_p^\times$ are in the same connected component exactly when they are equal on $\mathbf Z_p^\times$, and by continuity a character $\mathbf Z_p^\times \rightarrow \mathbf C^\times$ is trivial on some neighborhood $1 + p^n\mathbf Z_p$ of 1 (a subgroup!) since $\mathbf C^\times$ has no subgroup in a neighborhood of 1 other than $\{1\}$. Therefore a character on $\mathbf Z_p^\times$ is a homomorphism to $\mathbf C^\times$ on some quotient group $\mathbf Z_p^\times/(1 + p^n\mathbf Z_p)\cong (\mathbf Z/p^n\mathbf Z)^\times$, which is finite. Going the other way, for each homomorphism $(\mathbf Z/p^n\mathbf Z)^\times \rightarrow \mathbf C^\times$ we can lift it to a character $\eta$ on $\mathbf Z_p^\times$ (automatically continuous there since it is a homomorphism and is trivial on the neighborhood $1 + p^n\mathbf Z_p$ of 1) and then we can multiply this by an unramified character $\chi$ to get a character of $\mathbf Q_p^\times$: $p^nu \mapsto \chi(p)^n\eta(u)$. In other notation, since unramified characters of $\mathbf Q_p^\times$ are just complex powers $|\cdot|_p^s$, each character of $\mathbf Q_p^\times$ is $|\cdot|_p^s\eta$ where $s \in \mathbf C$ and $\eta$ is a character of $\mathbf Z_p^\times$: each connected component can be labeled by the common $\eta$ (restriction to $\mathbf Z_p^\times$) for all characters in that component. (The choice of $s$ for a character really is in $\mathbf C/(2\pi i/\log p)\mathbf Z$, a cylinder, which topologically is the same as $\mathbf C^\times$ using $s + (2\pi i/\log p)\mathbf Z \mapsto 1/p^s$.)
Prop. 4.6 is about continuous functions $\mathbf Q_p^\times \rightarrow \mathbf C$ with compact support. Note that your test-run example of the characteristic function of $\mathbf Z_p$, viewed as a function on $\mathbf Q_p^\times$ by taking $0$ out of its domain, does not have compact support in $\mathbf Q_p^\times$: the set $\mathbf Z_p - \{0\}$ is not compact in $\mathbf Q_p^\times$ just as $(0,1]$ and $[-1,1] - \{0\}$ are not compact in $\mathbf R^\times$. Therefore this is not a good example for a test-run for Prop. 4.6 (unlike Prop. 4.7).
For a better choice of test-run for Prop. 4.6, let $\xi_A$ be notation for the characteristic function of a set $A$ (1 if the variable is in $A$ and 0 otherwise). For $a \in \mathbf Q_p^\times$ and $n \in \mathbf Z$ chosen large enough so that $|a|_p > 1/p^n$, set $\phi = \xi_{a + p^n\mathbf Z_p}$: this is the characteristic function of the ball $a + p^n\mathbf Z_p$, which is a subset of $\mathbf Q_p^\times$ since we can't have $a + p^nx = 0$ for $x \in \mathbf Z_p$, as $|p^nx|_p \leq 1/p^n < |a|_p$. (If you don't like the characteristic function of a general ball in $\mathbf Q_p$ not containing $0$, consider the special case $a = 1$: $\xi_{1 + p^n\mathbf Z_p}$ for $n \geq 1$.)
Letting $|a|_p = 1/p^m$, so $m<n$, the Mellin transform $(M\phi)(\chi)$ of a character $\chi$ of $\mathbf Q_p^\times$ is the following integral $$ (M\phi)(\chi) = \int_{\mathbf Q_p^\times} \xi_{a + p^n\mathbf Z_p}(x)\chi(x)d^\times x, $$ where I write $d^\times x$ for the (multiplicative) Haar measure on $\mathbf Q_p^\times$ that you write as $p/(p-1) dx/|x|_p$. It is the Haar measure on $\mathbf Q_p^\times$ that gives the compact open subgroup $\mathbf Z_p^\times$ measure 1. After some calculation that I omit (tell me if you can work this out), we get $$ (M\phi)(\chi) = \chi(a)\int_{1 + p^{n-m}\mathbf Z_p} \chi(y)d^\times y, $$ which is $\chi(a)$ times the integral of the multiplicative character $\chi$ over the compact multiplicative group $1 + p^{n-m}\mathbf Z_p$. The value of this is determined by whether or not $\chi$ is trivial on $1 + p^{n-m}\mathbf Z_p$: if $\chi \not\equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = 0, $$ and if $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ then $$ (M\phi)(\chi) = \chi(a)\int_{1 + p{n-m}\mathbf Z_p} d^\times y = \frac{\chi(a)}{[\mathbf Z_p^\times:1 + p^{n-m}\mathbf Z_p]} = \frac{\chi(a)}{p^{n-m-1}(p-1)}. $$ There are only finitely many connected components containing the $\chi$ where $\chi \equiv 1$ on $1 + p^{n-m}\mathbf Z_p$ since such $\chi$ are determined on $\mathbf Z_p^\times$ (not on $\mathbf Q_p^\times$!) by their values on $\mathbf Z_p^\times/(1 + p^{n-m}\mathbf Z_p)$, which is a finite group (so it has only finitely many homomorphisms to $\mathbf C^\times$). Thus $M\phi$ is a "polynomial" in the sense defined above Prop. 4.6.
I have not yet addressed your question about the Mellin inversion formula. Consider this a partial answer so far. I will save what I have written and come back to this later.