This is not what I would call "nonstandard", but you may be interested in surreal numbers. They were originally developed for combinatoric game theory but they allow somewhat unique circumstances in treating infinity and infinitesimals; like the paper below explains, using surreal numbers you can discuss "the square root of infinity" without it being complete nonsense.

Link to fairly good introduction

Here's an brief summary:

A surreal number is a pair of sets of surreal numbers L (the "left set") and R (the "right set") such that no member of R is less than or equal to any member of L. Traditionally the numbers are written { L | R }.

Given surreal numbers x and y, $x \leq y$ if and only if y is less than or equal to no member of x’s left set, and no member of y’s right set is less than or equal to x.

We define the surreal number {|} to be 0.

These definitions (along with logical ones for addition, subtraction, etc.) spin out an entire number system, and it ends up that one version of $\epsilon$ is {0 | 1, 1/2, 1/4, 1/8, ...} and one version of $\omega$ is {$\mathbb{Z}$ | 0}.

While having infinity and infinitesimals as actual numbers in the system sounds like a good deal, it makes integration or differentation hard (I don't believe anyone has yet found a method).

standardset theory). – Yemon Choi Nov 23 '09 at 8:27