Personally I was not homeschooled, but from around 4th grade I attended a math circle in Boston. Many of the other students were homeschooled, and I think this sort of thing fills exactly the niche you are asking about. So I suggest looking into math circles in your area, although with the caveat that depending on where you are, the local math circles may be targets at older students than the one in Boston (which is particularly welcoming for kindergarten-aged kids) and/or more focussed on problem-solving. The Boston math circle is collaborative and inquiry based. As an example, a class of kindergarten kids might spend 10 weeks playing with math, starting with a question like "Are there numbers between numbers?" or "How many squares fit in a circle?"

Another recommendation: Conway's "The book of numbers" and Courant & Robbins' "What is Mathematics?" both have a wealth of material of sort you're interested in, perfectly suited to elementary school students (although perhaps only after they have some foundation in basic arithmetic).