Let $J(c)$ be the Julia set of $f(z)=z^2 +c$ defined as the closure of repelling periodic orbits. Is there a way to prove that $J(c)$ is the boundary of the basin of attraction of attractive fix points of $f$ without using normal families and Montel's theorem. What I would like to have is a proof that uses only elementary analysis and topology.
1 Answer
All right, let's assume that $c$ is in the main cardioid of the BrooksMatelskiMandelbrot set, so that $f(z)=z^2+c$ has only one finite attracting fixpoint (there is always an attracting fixpoint at infinity). Equivalently, the critical point of $f$ is in the basin of the finite attracting fixpoint. An elementary proof of your statement seems a liitle too much to ask, but it seems possible to prove it avoiding Montel's theorem and the concept of a normal family. (However, one cannot completely bypass referrring to limits of sequences of iterates.) Here is a sketch; hopefully the details can be filled in.
First of all, one has to prove without normal families that your Julia set is nontrivial. One can do that using the residue fixed point index as defined in the book ``Dynamics in One Complex Variable" by J. Milnor, published by Vieweg 1999, 2nd edition 2000 (available as a preprint here: http://www.math.sunysb.edu/cgibin/preprint.pl?ims905; beware that the preprint might be missing a few developments and the numeration might be different). Using this notion, the corollary 12.8 says that the Julia set is nonempty for a nonlinear rational map. (There is a consideration of parabolic fixed points as well, but we do not have to worry about them, since there are none.)
For the density of repelling periodic points in the complement of the union of two basins of attraction, you can try to proceed as in MR0226009 Baker, I. N. Repulsive fixpoints of entire functions. Math. Z. 104 1968 252–256.
This argument cannot be called elementary, as it relies on Ahlfors's five island theorem. Still, no normality (at least in this article; there were published variants of it by Schwick as well as by Berteloot+Duval using Zalcman's normality criterion).
The converse, i.e., the fact that $J$ is contained the boundary of the basin of attraction, can be obtained in a more elementary, but also somewhat messier way (maybe because I do not have any better idea).
First, in section 2 of MR0194595 Brolin, Hans Invariant sets under iteration of rational functions. Ark. Mat. 6 1965 103–144 (1965) you can find a proof of the fact that if the map $f$ has exactly two attracting fixpoints then the common boundary of their basins is a Jordan curve. Then you can use some definitions following Milnor's book:
A fixed point $p$ of $f$ is topologically attracting if it has a neightborhood $U$ such that the iterates $f^{\circ n}$ are all defined thorughout $U$ and so that the sequence $\{f^{\circ n}\mid_U$ converges uniformly to the constant map $U \to p$. Te basin of attraction of $p$ is the (open) sen $A$ consisting of all points $z \in \hat{\mathbb{C}}$ for which the sequence of iterates $f(z), f^{\circ 2}(z),...$ converges to $p$. A fixed point $q=f(q)$ (of a continuous map) is called topologically repelling if there is a neighborhood $V$ of $q$ so that for every $p \ne q$ in $V$ there exists some $n \geq 1$ so that the $n$th forward image $f^{\circ n}(p)$ lies outside of $V$.
These definitions agree with the usual FatouJulia ones. Lemma 8.1 in Milnor says that a fixed point for a holomorphic map $f$ is topologically attracting if and only if its multiplier $\lambda$ satisfies $\lambda<1$. Lemma 8.8 says that a fixed point for a holomorphic map $f$ is topologically repellng if and only if its multiplier satisfies $\lambda>1$. The proofs involve Taylor's expansion, nested sets property and Schwarz lemma; no Montel theorem or equivalent thereof.
By these definitions, a repelling periodic point does not belong to any of the two basins of attraction, so your Julia set is contained in the complement of their union.
Edit: I slided in a default mode a few times. E. g., I said "there are no parabolic periodic points", which is true by FatouShishikura inequality. I did not check thoroughly any of the available proofs of this inequality, but it is likely that Douady+Hubbard (the polynomial case) and Shishikura himself use some facts whose standard proofs are based on Montel's theorem. There is also a proof by Adam Epstein using infinitesimal Thurston rigidity, where such statements might be less likely to occur (but I cannot say now for sure). Oh, for the last part of my argument it would be good to know that the Julia set has empty interior but here I give up without normal families...
This Gedankexperiment only confirms how ubiquitous normal families are in complex dynamics of one variable so much so that any attempt of bypassing them (nobody was seriously proposing "replacing" them, right?) seems laughable to some mathematicians.

$\begingroup$ You propose to replace normal families with Ahlfors 5 islands? LOL. $\endgroup$ Jan 12, 2014 at 20:42

$\begingroup$ Not I, but I. N. Baker :) If anyone provides an elementary proof of this (quite standard) statement without Montel's theorem and normal families (in whatever incarnation Marty's criterion, Zalcman's lemma, etc.) the way OP is asking I'd also be happy to know. $\endgroup$ Jan 12, 2014 at 22:02

$\begingroup$ I will be also happy to know:) $\endgroup$ Jan 12, 2014 at 22:46