Here are interesting reviews of russian math circles. Friends who experienced such math circles (in the USSR and copies in other countries) told that it works great, even if one later decides for not becoming a mathematician.
In general, I guess one should have the time frame of brain development in mind: One 'developmental window' is ca. from birth to age 4, the other around puberty. In both, many new nerves/nerve-connections build up, but untrained ones are then deleted. Reports of recent studies suggest that pre-birth risks fro the brain like stress, lead exposure, malnutrition are important too and widely underestimated.