I'm not sure if this is the sort of thing that you have in mind, but there's a text-based interactive adventure game called *[A Beauty Cold and Austere][1]*. It was written by Mike Spivey and is highly mathematical.

  [1]: http://ifdb.tads.org/viewgame?id=y9y7jozi0l76bb82