There are many "strange" functions to choose from and the deeper you get involved with math the more you encounter. I consciously don't mention any for reasons of bias. I am just curious what you consider strange and especially like.
Please also give a reason why you find this function strange and why you like it. Perhaps you could also give some kind of reference where to find further information.
As usually: Please only mention one function per post - and let the votes decide :-)
A Brownian motion sample path.
These are about the most bizarrely behaved continuous functions on $\mathbb{R}^+$ that you can think of. They are nowhere differentiable, have unbounded variation, attain local maxima and minima in every interval... Many, many papers and books have been written about their strange properties.
Edit: As commented, I should clarify the term "sample path". Brownian motion is a stochastic process $B_t$. We say a sample path of Brownian motion has some property if the function $t \mapsto B_t$ has that property almost surely. So, run a Brownian motion, and with probability 1 you will get a function with all these weird properties.
The empty function $\emptyset:\emptyset\to\emptyset$ is quite strange when you first meet it.
The Busy Beaver function
Let Σ be a finite alphabet, for instance {0, 1}; let M be the set of Turing machines with alphabet Σ, and let H ⊆ M be the set of Turing machines that halt when given the empty string ε as input.
For each M ∈ H, Let s(M) be the number of steps performed by M before halting (when given ε as input).
Finally, let S : ℕ → ℕ be the function defined by
S(n) = max {s(M) : M ∈ H and M has n states}
Notice that S is well-defined, since only finitely many Turing machines with n states exist.
In other words, S(n) is the maximum number of steps performed on ε among all halting Turing machines with n states. S is called the Busy Beaver function.
It turns out that S is uncomputable because it grows faster than any computable function, that is, for all recursive functions f : ℕ → ℕ we have S(n) > f(n) for large enough n, and in particular f is o(S).
I like the Cantor function. A continuous, increasing function $f:[0,1]\rightarrow[0,1]$ with derivative $0$ almost everywhere. See wiki article here.
The Conway base 13 function has to be the weirdest function I know. This function is continuous nowhere, yet it satisfies the intermediate value theorem. Only John Conway...
The Ackermann function $A(n,m)$ is defined on the natural numbers by a very simple recursion, but the values grow enormously, almost beyond conception. This function completely transcends any simple-minded system of rates-of-growth based on polynomial, exponential, double-exponential and so on.
The first few values of the diagonal function $A(n) = A(n,n)$ are:
The levels of the Ackerman function $A_n(m)=A(n,m)$ stratify the primitive recursive functions, in the sense that they are each primitive recursive, but every primitive recursive function is bounded by such a level of the Ackermann function. Thus, the Ackermann function itself is not primitive recursive, although it is computable in the sense of computability theory.
I suppose the strangest function in mainstream mathematics is Riemann zeta function http://en.wikipedia.org/wiki/Riemann_zeta_function
$ \zeta(s) = \sum_{n=1}^\infty \frac{1}{n^s} = \frac{1}{1^s} + \frac{1}{2^s} + \frac{1}{3^s} + \cdots \;\;\;\;\;\;\;{Re}(s) >1. $
It is part of one of the most important hypothesis and is very influential in many branch of moder mathematics. It is actively used in many areas and is researched in many ways, it is not curiosity, or exotic example, but important mathematical being!
And is mysterious and strange! Take a look:
I'm still quite impressed about $f(x)=\mathrm e^x$ …
Thomae's function, also called the "popcorn function". It's continuous at all irrationals and discontinuous at all rationals. Here a picture:
.svg/200px-Thomae_function_(0,1).svg.png)
The Weierstrass function is particularly intriguing, as it's a function that's everywhere continuous, but nowhere differentiable.
$f(x)= \sum_{n=0} ^\infty a^n \cos(b^n \pi x)$
where 0<a<1, and b is a positive odd integer such that $ab > 1 + \frac{3\pi}{2}$.
It challenges the notion that, just because a function is continuous, it must also be differentiable in most places, which I think is pretty cool.
Please excuse me if I include two related functions in one answer. Any space filling curve is rather strange, at least for me. Let $\gamma\colon[0,1]\to[0,1]^2$ be such a curve, that is, $\gamma$ is continuous and surjective. Let $\gamma(t)=(x(t),y(t))$; then $x(t)$ (or $y(t)$) is my other candidate for strangest function: given any $z\in[0,1]$, $x^{-1}(z)$ has the cardinality of the continuum.
An unbounded operator with dense graph.
In functional analysis, one deals with unbounded operators on Hilbert space, but usually ones that are closed, or are at least closable. At the opposite end of the spectrum, one can construct linear operators whose graph is dense: for any pair $(x,y)$, there is a sequence $x_n$ such that $x_n \to x$ and $A x_n \to y$ ! It's not so easy even on $\mathbb{R}$ to come up with a function whose graph is dense, and the examples I think of aren't measurable. But in infinite dimensions, you can find one that is linear! It's just an illustration that facts that are trivial in finite dimensions can be horribly, horribly false in infinite dimensions.
A family of examples of this is constructed in
MR0782615 (86i:47052) Lindsay, J. M. A family of operators with everywhere dense graphs. Exposition. Math. 2 (1984), no. 4, 375--378.
Interestingly, as an appplication, Lindsay uses such operators to prove that a Brownian motion sample path is nowhere differentiable --- which is my other favorite strange function!
The function defined by the power series $f(x)=x-x^2+x^4-x^8+x^{16}-\dots$ What is its limit as $x$ approaches $1$ from below? EDIT (This answer is a trick question.)
Minkowski's question mark function if only for the strange $?(\cdot)$ notation.
It is pretty obvious after you've seen it, but I like the crinkled curve from Halmos's Hilbert Space Problem book:
Let $f:\mathbb{R}\rightarrow(0,\infty)$ be an $L^2$ function, and define $t\mapsto g_t:\mathbb{R}\rightarrow L^2(\mathbb{R})$ by $$g_t(x)=\chi_{(-\infty,t)}(x) \times f(x).$$
Then $g_t$ has the property that for all $t_1 < t_2 < t_3$ the secants $g_{t_2}-g_{t_1}$ and $g_{t_3}-g_{t_2}$ are mutually orthogonal. (The curve turns a corner at every point.)
Dirac delta function seems strange to me since the first time I saw it. http://en.wikipedia.org/wiki/Dirac_delta_function
Interpreting your questions a bit liberally, I suggest the Goodstein sequence:
f(x) = sin (1/x): (x not 0); f(x) = 0 (x equals 0)
These functions like the Cantor function and the continuous-but-not-differentiable function are all well and good, but contrived - the only place you ever see them is as counterexamples. Here is a function that has many uses in Number Theory, and still manages to have a strange property or two. Let $x=h/k$ with $h$ and $k$ integers, $k>0$. Define $$s(x)=\sum_{c=1}^{k-1}((c/k))((ch/k))$$ where $((y))=0$ if $y$ is an integer, $((y))=\lbrace y\rbrace-1/2$ otherwise. It is easily proved that the sum depends only on the ratio of $h$ and $k$, not on their individual values, so $s$ is a well-defined function from the rationals to the rationals. It is known as the Dedekind sum; it came up originally in Dedekind's study of the transformation formula of the Dedekind $\eta$-function.
Now for the strange properties.
Hickerson, Continued fractions and density results for Dedekind sums, J Reine Angew Math 290 (1977) 113-116, MR 55 #12611, proved that the graph of $s$ is dense in the plane.
With Nick Phillips, I proved (Lines full of Dedekind sums, Bull London Math Soc 36 (2004) 547-552, MR 2005m:11075) that, with the exception of the line $y=x/12$, every line through the origin with rational slope passes through infinitely many points on the graph of $s$. We suspect that the points are dense on those lines, though we could only prove it for the line $y=x$.
I like the Theta functions which are given by Fourier-type series. They show up in many areas in mathematics. For example:
i)They are very important in the study of abelian varieties in algebraic geometry (for example, in the case of elliptic curves they are used in the proof of Abel's theorem and are related to Weierstrass $\mathcal{P}-$function).
ii) They satisfy a number of interesting indentities. For example, in the one-dimensional case, they satisfy Jacobi's triple product identity which can be used to show Jacobi's four square theorem
iii) They can be used to solve algebraic equations degree equations explicitly (see this link)
iv) In the one-dimensional case, they solve the heat equation.
The Osgood curve ("A Jordan Curve of Positive Area") is an injective map from [0,1] into $\mathbb{R}^2$ which traces out an image of positive area. (This differs from standard space-filling curves, which are not injective.)
The Banach limit assigns to every bounded sequence of real numbers a real number "limit" in a way that is linear, shift invariant, and agrees with the usual limit whenever it exists. Banach limits are among the mysterious examples of continuous linear functionals on $\ell^\infty$ that aren't represented by elements of $\ell^1$. Unfortunately, the Hahn-Banach theorem is used in the construction of the Banach limit, and the values aren't canonical. There's a precise definition at http://en.wikipedia.org/wiki/Banach_limit .
A canonical example from elementary real analysis - the Blancmange function. Consider $f$ defined piecewise by
$f(x) = x - [x], \quad \text{if} \quad 0 \leq x- [x] \leq \frac{1}{2}$,
and
$f(x) = 1 - (x - [x]), \quad \text{if} \quad \frac{1}{2} < x - [x] < 1$,
(where $[x]$ is the integer part of $x$). Then define the Blancmange function, $B$
$B(x) = \sum_{n=0}^{\infty}\dfrac{1}{2^n} f(2^{n}x)$.
The series converges by the Comparison Test, since $|f(2^{n}x)| \leq \frac{1}{2}$, for all $x \in \mathbb{R}$, and it can be shown that $B$ is uniformly continuous but nowhere differentiable. Here a picture of the function:
A tasty counterexample to the converse of "differentiability $\implies$ continuity".
Among the "special" functions encountered in analysis in my view Ingrid Daubechies' waveletes with compact support are the strangest.
One can construct a natural 'metric' for the Riemann sphere which is equivalent to the spherical metric but which is singular on a dense set of points of the Riemann sphere though remains $L^1$ integrable.
These are built from degree 2 rational maps (first constructed by Mary Rees) which have the whole Riemann sphere as their Julia sets, and have the orbits of their critical points also dense. The Carlesson-Jones-Yoccoz construction of a expanding metric for critically-finite rational maps actually extends to this case, and we get a metric in which this Julia set actually looks as if it was hyperbolic!
[The details are worked out in my PhD thesis, never published as I decided that computer algebra suited me better than complex dynamics].
Since Mariano took my favorite already, I'll go with the stopping time function for the 3x+1 problem: http://www.ieeta.pt/~tos/3x+1.html
Fix a probability $p < 1/2$ of winning an unfair coin toss. For $x \in [0,1]$ rational, let $f(x)$ be the probability that, if you started with $x$ dollars, you could make it to 1 dollar through optimal betting* on the outcome of these coin flips. This function $f(x)$ is obviously weakly increasing on $[0,1]$ (in fact strictly). Less obvious is that it extends to a continuous function on $[0,1]$, whose derivative exists almost everywhere, but that derivative is $0$.
I can't believe no one has mentioned the Dirichlet function:
(I guess it's up to me to bring it up...)
Nonconstant continuous locally recurrent functions are quite unintuitive. A real-valued function is locally recurrent on $\mathbb R$ if for every $x_0\in\mathbb R$ and every deleted neighborhood $N(x_0)$ of $x_0$, there exists $x\in N(x_0)$ for which $f(x)=f(x_0)$. Thus in some sense a nonconstant continuous locally recurrent function looks everywhere like $x\sin(1/x)$ at $x=0$. See papers in the American Math. Monthly of Bush (1962), Marcus (1963), and Mauldon (1965).