Apologies if this answer is nitpicky and diverges slightly from the questions intent, but this particular soft question appears to warrant a soft answer. 

Implicit in your term "mathcoin" are assumptions that the problem:  

 1. Allows new solutions to be found in regular *tunable* time intervals,  
 2. Is memory hard so that specialized ASICs cannot speed up the computations, and 
 3. Any significantly faster algorithms represent the sort of dramatic advancement that it'd emerge from across the mathematical community simultaneously and not from a secret project. 

Of course, collisions in cryptographic hashed satisfies 1 and 3, while a memory hard cryptographic hash like `scrypt` or `argon2` satisfies 2 as well.

Almost all the answers proposed here thus far fail even 1 and maybe all here fail 2.  Worse, there are probably no calculations that both pass 3 and represent useful mathematical (resp. scientific) research, at least not for 2 (resp. 10) years.  

In short, a naive "mathcoin" as suggested here sounds impossible to do correctly.  There might be less naive approaches where perhaps any problem goes and the problem's difficulty is established by previous attempts, but that's actually solving a social problem to achieve 1 with widely varied math problems as simply a source of randomness that defeats ASICs.

As an aside, there are proof-of-useful-work systems that could possibly support a bitcoin like currency without wasting resources, but again they all provide useful social services like file storage (filecoin) or anonymity via onion routing.