Question

In many papers about dynamical system, I found the word " ideal boundary". T don't know what is the definition of ideal boundary.

Answer 1

For a Hadamard space $X$ there are two kinds of ideal boundaries, the set $Bd(X)$ of horofunctions up to additive constants, and the set $X(\infty)$ of equivalence classes of rays. These two are homeomorphic by the correspondence:

$$\gamma \text{ (a ray)}\to \text{the Busemann function of } \gamma$$ $$h \text{ (a horofunction)}\to \text{the gradient of } h$$

Here you can find lecture notes by Ballman, with chapter two treating the boundary at infinity via Buseman functions and via rays. You will also find definitions in this paper and this paper.

Answer 2

There are infinitely (uncountably:-) many definitions of the ideal boundary. So you have to specify which paper you are reading:-)

See, for example M. Brelot, On topologies and boundaries in potential theory and Constantinescu, Cornea, Ideale Ränder Riemannscher Flächen.

The simplest example of an ideal boundary is one point compification. But depending on context, one usually has a space of functions, and one wants to add ideal bundary points so that the functions of your space have limits. (Examples: Prime ends, Martin boundary, etc.)