Good introduction to statistics from a algebraic point of view? There are already lots of questions on this subject like 
Is there an introduction to probability theory from a structuralist/categorical perspective?
Is there a combinatorial/topological treatment of statistical independence?
What is the algebraic equivalent of independent elements?
and related field called ergodic theory which in fact study different things.
However, as a new category theorist with almost no statistics background I don't aim to learn these advanced topics, but to understand very basic notions like random variable and expectation from a algebraic perspective.
For example, we can define a type family 

Rand: Type->Type

, and a real random variable can be defined as 

randnum : Rand Real

and Expectation as

E : Rand a -> Real

It seems that statistics is the one of the  most recalcitrant subject for  algebraic approach, but I think it is not the case, we can just treat it as any other abstract object, and define axioms on this abstract random type. The notations and formulas in every introduction book of statistics I have read soon become utterly ugly due to lack of a proper foundation, which is really painful for someone ingrained with abstract algebra and functional programming. However,statistics is extremely useful for machine learning and the modelling of human brain and many others.
I have created a repo for a basic type-directed understanding of statistics in haskell https://github.com/doofin/alg-statistics
 A: P. Diaconis "Group representations in probability and statistics, Chapter 6. Metrics on Groups, and Their Statistical Use" pdf projectEuclid is something eye-opening and must-and-pleasure to read, giving group theoretic look on basic tools in statistics like 
Mann-Whitney, Kolmogorov-Smirnov tests, Kendall and Spearmen correlations coefficients.
Let me give brief outlook:   
The central ideas are related to symmetric group $S_n$, and metrics on it, which give a clue to measuring "disorder" in  samples, thus related to main statistics questions. 
1) Kendall rank correlation  coefficient (tau) is closely related to 
number of inversions of permutations.
2) Spearman's rank correlation coefficient is closely related to $L_2$-metric on the permutation group.
3) On Mann–Whitney U test and Kolmogorov–Smirnov test let me quote:

Example 13. Rank tests. Doug Critchlow (1986) has recently found a
  remark able connection between metrics and nonparametric rank tests.
  It is easy to describe a special case: consider two groups of people —
  m in the first, n in the second. We measure something from each person
  which yields a number, say
  $x_i$, $y_i$ We want to test if the two sets of numbers are "about the same."
This is the classical two-sample problem and
  uncountably many procedures have been proposed. The following common
  sense scenario leads to some of the most widely used nonparametric
  solutions.
Rank all n + m numbers, color the first sample red and the
  second sample blue, now count how many moves it takes to unscramble
  the two populations. If it takes very few moves, because things were
  pretty well sorted, we have grounds for believing the numbers were
  drawn from different populations. If the numbers were drawn from the
  same population, they should be well intermingled and require many
  moves to unscramble. 
To actually have a test, we have to say what we
  mean by "moves" and "unscramble." If moves are taken as "pairwise
  adjacent transpositions," and unscramble is taken as "bring all the
  reds to the left," we have a test which is equivalent to the popular
  Mann-Whitney statistic. If m = n, and moves are taken as the basic
  insertion deletion operations of Ulam's metric (see Section B below)
  we get the Kolmogorov-Smirnov statistic

A: See also Lectures on Algebraic Statistics by Drton, Sturmfels, Bernd, Sullivant:
http://www.springer.com/gp/book/9783764389048
A: Lucien Le Cam developed an approach to statistics that largely disposes of measure-theoretic probability and replaced probability measures and random variables with certain Banach lattices. The approach can be found in Le Cam's book Asymptotic Methods in Statistical Decision Theory and the more accessible Comparison of Statistical Experiments by Torgersen.
Keeping the traditional measure theoretic approach to statistics but studying it by a category-theoretic approach is Statistical Decision Rules and Optimal Inference by Cencov.
For basic material on linear regression, there is also The Coordinate-Free Approach to Linear Models by Wichura; this is an area amenable to an approach that is likely to be more comfortable for an algebraist. This is the only book in the list that might be said to be introductory.
That being said, anyone who actually wants to work in statistics needs to be familiar with the standard literature and approach. Warts and all. Much of statistical theory is about inequalities; more analysis than algebra. 
A: A not-so-fancy book is Heyer: Theory of Statistical Experiments, Springer. 
However, it's not category theoretic and gives a rather traditional picture.
It is not true that statistics is not well founded (that's maybe only the case in some textbooks in English speaking countries, because they have rather a tradition to teach statistics in an "applied" and "practical" way). 
There is literature about statistics (in the traditional way, based on probability theory), which gives thorough foundations, e.g. in German the book(s) by Witting (and Mueller-Funk in vol 2) "Mathematische Statistik" (which to my knowledge has never been translated into English) or Schmetterer. Both Witting and Schmetterer are rigorously formalized literature.
