Statistics for mathematicians I'm looking for an overview of statistics suitable for the mathematically mature reader: someone familiar with measure theoretic probability at say Billingsley level, but almost completely ignorant of statistics.
Most texts I've come across are either too basic, or are monographs focused on a specific area or technique.
Any suggestions?
 A: Probability Theory: the Logic of Science by physicist E. T. Jaynes is a must.
A: This won't directly answer the question, but here are some things a mathematician who wants to learn about statistics should learn:

*

*When is a random variable a statistic and when is it not?  (A statistic is an observable random variable.  For example $X - E(X)$ is not a statistic if the "population average" $E(X)$ is not observable.

*Fisher's concept of sufficiency.  Examples, characterizations, theorems.  In particular, the Rao--Blackwell theorem and examples of its use.  That's way cool.

*So is the concept of completeness and the Lehmann--Scheffe theorem.

*If you think that linear regression is called linear because you're fitting a line, then you are naive.  If you're fitting, e.g., a parabola by finding least-squares estimators of three parameters, then you're doing linear regression.  There is also such a thing as non-linear regression.

*Learn the Gauss--Markov theorem on Best Linear Unbiased Estimators (BLUEs).

*Look at my recent answer to a question on prediction intervals.  Why do you need the (finite-dimensional version of) the spectral theorem to understand linear regression?  (Look at the aforementioned answer and consider this question an exercise.)

*As long as we're on linear regression (the topic of the three bullets immediately above this one), look at the Wikipedia article titled "errors and residuals in statistics" (written mostly by me).  Learn the difference between an error and a residual.  Maybe look at "Studentized residual" as an afterthought.

*....and then at "lack-of-fit sum of squares".

*If you think linear regression is child's play rather than something to which the most brilliant person could devote a long career in research, grow up.

*Learn the difference between frequentism and Bayesianism.  In fact, look at the rant I posted on nLab about this.  (The essence of Bayesianism is that probabilities are taken to be epistemic.  Bayesianism is not more subjective than frequentism; rather Bayesians and frequentists put their subjectivity in different places.  (A really glaring example is the 5% critical value legendarily used in medical journals.  Why 5%?  Because that's a subjective economic choice.))

*Learn design of experiments.  Learn why Latin squares and a myriad of other combinatorial designs are used.

*OK, maybe a small and incomplete but nonetheless direct answer to the original question: perhaps Hocking's book on linear models.

*Learn to use the word "sample" correctly.  If you ask the next 100 people you meet whether they intend to vote "yes" or "no", that's not 100 samples; that's one sample.

*Another thing that will give you some idea of the distinct flavor of the subject, and how it differs from probability theory and some other fields, is books on sampling.

*Learn about the Wishart distribution.

*And the multivariate normal distribution.

*Exercise: How do you prove that every non-negative-definite matrix is the variance of some random vector?

*Learn why the Behrens--Fisher problem cannot be regarded as a math problem.  It belongs up there with Hilbert's problems as one of the great challenges, but it's not mathematics for this reason: One can model it as a math problem in any of a variety of different non-equivalent ways.  One can solve those math problems.  But which one is the "right" model?  That's essentially a philosophical question.  And that question, not the math problems, is the Behrens--Fisher problem.  (The Behrens--Fisher problem is this: how do you draw inferences about the difference between the means of two normally distributed populations which may have different variances?  "Inferences" can mean point-estimates or interval estimates or perhaps other things.)

This is just a sampling of the first things that come immediately to mind.  It leans toward showing you what the subject tastes like rather than what it's important to know to do theoretical or applied research.
Statistics is an immensely broader field than mathematical probability theory.
A: I'm currently working my way through Cramér's Mathematical methods of statistics. It starts out with a half-book primer on all measure theory, Lebesgue integration et.c. you might possibly need for anything, and then goes through first probability and then statistics with this backdrop.
Skipping the introductory analysis it might be what you're looking for. 
A: The Book of Van der Vaart is exactly what you are searching for! 
Asymptotic statistics 
A: It's old fashioned, but it's written by a master and most of it may just be what you're looking for: Mathematical Statistics by S.S.Wilks. It's long out of print and hard to find, but worth hunting down.
A: Mark J. Schervish "Theory of Statistics" perhaps is the one closest to your needs since it 


*

*is graduate-student level and proves everything rigorously from probability measure perspective 

*has a rather broader view comparing with other monographs

*gets 5 stars on Amazon


But as the some ppl pointed out, statistics are too broad and have been under a intensive development till now. This book missed some modern results as expected.  
A: I liked Mathematical Statistics by Jun Shao for the classical theory. There is also a companion 'Problems & Solutions' book. 
I second Wasserman's 'All of Statistics' for a quick overview.
A: Kiefer's "Introduction to Statistical Inference" is a particularly nice book.  It crams an incredibly amount of perspective into a very easy to read package.  You'll want to supplement it with volumes like Lehmann's "Point Estimation" and "Testing Statistical Hypotheses."
One thing I particularly like is that he starts at the bottom with the decision theory structure, and imposing other criteria to be able to choose from the plethora of admissible procedures before diving into things like linear estimators and hypothesis tests.
A: There is a book I love: "Statistical inference" by S.D. Silvey. Short, elementary, but perfectly clear and suited for the mathematical mind.  Impossible not to like statistics after that.
A: I like "Statistics" by Freedman, Pisani, Purves, and Adhikari
A: For a very mathematical version of statistics, my favorite is online lecture notes from two MIT courses.  The instructor is named Panchenko and the course is called 'Statistics for Applications'.  There are course notes that read like a book for the course in 2003 and 2006.  I have enjoyed browsing through both of them.  Here is a link for the 2006 course: https://ocw.mit.edu/courses/18-443-statistics-for-applications-fall-2006/
A: Pestman, Wiebe R., Mathematical statistics., de Gruyter Textbook. Berlin: de Gruyter (ISBN 978-3-11-020852-8/hbk; 978-3-11-020853-5/ebook). ix, 597 p. (2009). ZBL1251.62001.
It can't be said to be advanced in that it is intended for undergraduates, but it is written in the style of a book for the mathematically mature. I.E. Definition, Theorem, Proof, Example. It even begins with a brief introduction to measure theory. That's a big improvement over most!
A: Wasserman's All Of Statistics seems concise and useful. 
A: Mathematical Statistics, by Jun Shao, the most mathematical statistics book that I know.
This book is mainly focused on classical statistics. If you want to proceed further into classical statistics, E.L. Lehmann's two books Theory of Point Estimation and Testing Statistical Hypothesis are bibles of classical statistics, worth your reading.
High-Dimensional Statistics: A Non-Asymptotic Viewpoint, by A.M. Wainwright, another popular book, also quite mathematical, mainly focused on high dimensional statistics.
Enjoy!
A: David Williams "Weighing the Odds" is a book about probability and statistics by a distinguished probability theorist. He has a great sense of humour and the book is a lot of fun. The exercises can be demanding, but they are also interesting. (I have latex solutions to many of them if you want to ask me after first trying an exercise.) Professional statisticians think this book is more maths than stats, but it does contain a lot of stats. It's an introduction that will hold your attention if you put effort into the exercises. It's aimed at advanced undergraduates, but in my opinion is also very suitable for research mathematicians (though I have my doubts as to its suitability for a professional probability theorist).
After reading and enjoying this book, a mathematician will find it much easier to get to grips with a good book on stats. But there's no substitute for trying to handle large amounts of real data, when nothing works out as it "ought" to. Learning stats without handling data is like learning maths without working out anything for yourself---it's too easy to kid yourself that you understand.
