I remember attending a lecture given by an ultrafinitist who denied that curves are a set of points, he would only say that any particular point may or not be on the curve. Similarly for algebraic or analytic objects, the only way I know how to define them is as a set with operations on the elements of that set. Since ultrafinitists cannot use definitions with infinite sets, what sort of definitions do they use?

2$\begingroup$ I'm not an ultrafinitist, but I would imagine they would say that the whole idea of considering an infinite set is misguided according to their philosophy; "potentiallyinfinite" (as in, unbounded/unlimited) is OK, but a completed infinite set is not. $\endgroup$ – Zen Harper Jul 18 '11 at 3:05

1$\begingroup$ @Zen Harper: I am not an ultrafinitist either, but that sounds more like a finitist viewpoint. As I understand ultrafinitism, it wouldn't even allow "potential infinity". Troelstra and van Dalen (Constructivism in Mathematics) succinctly summarise ultrafinitism as rejecting the idea that we can view 2^1024 as a sequence of units, even in principle. I have argued with a few people calling themselves ultrafinistists, I think they generally accept that, and those (one?) that don't will still agree that "potential infinity" is not kosher. $\endgroup$ – Daniel Mehkeri Jul 18 '11 at 4:52

5$\begingroup$ @Daniel Mehkeri, I remember reading somewhere that one of the famous mathematicians and logicians argued with Alexander EseninVolpin about it. At the end he decided to try to find what is the largest natural number that Alexander EseninVolpin was ready to admit that it exists. After sometime the famous mathematicians noticed that this is not going to work, because as the numbers increased Alexander EseninVolpin waited proportionally more time before replying with yes, he accepts that number exists. $\endgroup$ – Kaveh Jul 18 '11 at 6:44

6$\begingroup$ @Kaveh: I told the story here: mathoverflow.net/questions/44208/… $\endgroup$ – Todd Trimble♦ Jul 18 '11 at 6:49

1$\begingroup$ @Todd, :) also here: scottaaronson.com/blog/?p=103 or even maybe on FOM mailing list. $\endgroup$ – Kaveh Jul 18 '11 at 7:57
Pointset definitions are the most common modern way of defining a geometrical object like a line, but they're not the only way.
In Euclid, lines and circles are primitives.
The axiomatization of Euclidean geometry in Tarski 1999 has only points, betweenness, and congruence as primitives, and sets are not even referred to in the axioms except for the axiom of continuity, which basically says lines are Dedekindcomplete. I don't imagine that ultrafinitists would even want this kind of continuity, so they'd probably leave this axiom out. (Even if you interpret the axiom as a sheaf of axioms over firstorder formulas defining the relevant sets, that would be an infinite sheaf, which I think would be unacceptable.) Tarski's axioms, interpreted using classical logic, imply that there are infinitely many points, but classical logic isn't the appropriate logic for ultrafinitism. So I don't really see any reason to believe that there's any problem with doing an ultrafinitist geometry this way.
As an example, take the unit circle in the Cartesian plane, defined by a firstorder formula (not as a set). I can definitely prove to an ultrafinitist's satisfaction that it contains at least four points (i.e., there are at least four points satisfying that formula). Given a line through the origin O (defined by picking some point P outside the circle that the line goes through), the lack of the axiom of continuity means that it's probably not possible to give an ultrafinitist proof that it intersects the circle (i.e., that there is a point between O and P that satisfies the formula defining the circle). However, one can certainly prove that there exist a point on the line and a point on the circle such that the distance between them is no more than 0.0001. There will be points such that it's neither true nor false that the point is on both the line and the circle.
Even if you're working in a system that has infinities, it's not necessary to describe geometry in terms of point sets. The original presentation of the surreal line didn't use sets, and the surreals are too big to be a ZFC set. In smooth infinitesimal analysis, a curve is not representable as a set of points.
Tarski and Givant, 1999, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.27.9012