I shall give two different interpretations of the question. (The second interpretation using true arithmetic is modified in this update.)

**Using arbitrary nonstandard models of PA.** Let us say that a Turing machine program $p$ computes
a function $f$ on the natural numbers *with nonstandard help*, if
for any $a\in\mathbb{N}$ and any nonstandard $N\models\text{PA}$
and any nonstandard number $H\in N$, if we run $p$ inside $N$ with
input $(n,H)$, then the program halts and gives output $f(n)$, if
$n$ is in the domain of $f$, and otherwise it does not halt in $N$.

That is, we run the program inside $N$ with $H$ as an augmented
input to $n$, and the program must always compute $f(n)$ in such a
situation. The program $p$ must work correctly with any nonstandard
help number $H$ and in any nonstandard model $N$.

In this case, we can say exactly what the extra power of this
situation is.

**Theorem.** Assume PA is consistent. Then a total function $f$ is
computable with nonstandard help if and only if it is computable.

**Proof.** Clearly, every computable total function is computable
with nonstandard help, since we can simply ignore the extra input
$H$.

Suppose conversely that $f$ is a total function computed by program
$p$ with nonstandard help. Let $T$ be the theory $\text{PA}$ plus
the assertion $0<H$, $1<H$, $2<H$ and so on, asserting that $H$ is
a nonstandard number. I claim that $f(n)=m$ just in case $T$ proves
that program $p$ on input $(n,H)$ gives output $m$. If $T$ proves
this, then in any nonstandard model $N$ of $\text{PA}$ and every
nonstandard $H$, the program $p$ on input $(n,H)$ must give output
$m$, which must be $m=f(n)$ since $p$ computes $f$ with nonstandard
help. And if $m=f(n)$, then in every nonstandard model and every
nonstandard $H$, it must be that program $p$ halts on input $(n,H)$
with output $m$, and so $T$ will prove this.

It now follows that $f$ is computable, since the graph of $f$ will
be computably enumerable, by searching for proofs in $T$. On input
$n$, search for a proof in $T$ that $p$ halts on $(n,H)$ with some
specific output $m$, and when this is found, output $m$. $\Box$

This argument shows that your remarks about computing the halting
problem and whatnot are incorrect. The reason is that you get false
positives with that argument in some nonstandard models of PA. It
can happen that some models of PA think a program halts, but it
doesn't really halt at any standard stage.

The argument does not work with partial functions.

**Theorem.** A partial function $f$ on the natural numbers is
computable with nonstandard help if and only if it is computable
and has decidable domain.

In particular, there are computable functions that are not
computable with nonstandard help.

**Proof.** If the function is computable and has decidable domain,
then we can see that it is computable with nonstandard help by the
algorithm that first decides if $n$ is in the domain, and then if
it is, computes $f(n)$, while ignoring $H$. Since that computation
is standard finite, it will work inside a nonstandard model $N$.

Conversely, suppose that $f$ is computable with nonstandard help,
by some program $p$. The argument of the theorem above shows that
$f(n)=m$ just in case theory $T$ proves that $p$ gives output $m$
on input $(n,H)$. So the graph of $f$ is c.e. and so $f$ is
computable. So the domain of $f$ is a c.e. set, enumerated by some
program $q$. Let $T^+$ be the theory of $T$ above, plus the
assertion that the function $n\mapsto f(n)$ computed by $p$ via
$(n,H)$ has domain enumerated by $q$. If the domain of the actual
$f$ is not decidable, then there must be a model of $T^+$ in which
additional standard elements are enumerated by $q$, for otherwise
we could decide the set. Thus, there must be some nonstandard
models of PA in which the function computed by $p$ has the wrong
domain, a contradiction. $\Box$

So it is a little surprising that not every computable partial
function is computable with nonstandard help, since the nonstandard
models can get the domain wrong and there is no way to avoid this.

**Using only nonstandard models of true arithmetic.** A better interpretation is obtained by considering only models of true arithmetic, which I think is closer to what you may have meant by referring to nonstandard analysis.

Let us define that a (partial) function $f$ on the natural numbers is computable by program $p$ with *true nonstandard help*, if in any nonstandard model $N$ of true arithmetic (that is, an elementary extension of the standard model) and any nonstandard number $H$ in $N$, the program $p$ on input $(n,H)$ halts in $N$ with output $f(n)$, if $n$ is in the domain of $f$, and otherwise does not halt.

One can easily see that $0'$ is computable with true nonstandard help, since one can simulate any Turing machine program for $H$ steps inside any model of true arithmetic, and this will get the right answer for standard input, precisely because the model is a model of true arithmetic. At first, I had thought that we could use that nonstandard fake version of $0'$ and do the same thing to compute $0''$ and $0'''$ and more, but I now think this is wrong. Rather, what is going on is the following:

**Theorem.** A total function on $\mathbb{N}$ is computable with true nonstandard help if and only if the graph of $f$ has complexity $\Sigma^0_2$.

**Proof.** If the graph of $f$ is $\Sigma^0_2$, then $f$ is computable from $0'$. Since in any nonstandard model of true arithmetic, we can from any nonstandard number compute an approximation to $0'$ by simulating for $H$ steps, and this will be correct on the standard part. We can therefore use that oracle we generated to compute $f$, and since $f$ is total, we will never need to reach into the nonstandard part of the oracle for any standard input. So $f$ will be computable with true nonstandard help.

Conversely, suppose that $f$ is a total function and computable with true nonstandard help by program $p$. It follows that $p$ on input $(n,H)$ gives output $f(n)$ with any nonstandard model of true arithmetic and any nonstandard $H$. Since any larger $H$ would also work, it follows that $p$ with input $(n,k)$ must give output $f(n)$ for all sufficiently large $k$, including sufficiently large standard $k$. It follows that $f(n)=m$ just in case there is $K$ such that for all $k\geq K$ and all halting computations of program $p$ on input $(n,k)$ gives output $m$. This is a $\Sigma^0_2$-expressible property, and so the theorem is proved. $\Box$

2more comments