# Is ERNIE output skewed by statistical tests?

ERNIE is a hardware random number generator used to generate winning Premium Bond numbers in the UK. Wikipedia says: "ERNIE's output is independently tested each month by an independent actuary appointed by the government, and the draw is only valid if the output passes tests that indicate it is statistically random." But doesn't it mean that, if ERNIE output was truly and fully random (i.e. generated from a unitary distribution over a set of integers from 1 to N), rejecting some of the draws would actually distort the randomness?

-

No.

The final 'it' in the quoted section refers to the word 'output' not to the word 'draw'. Thus the "independent actuary" tests the output of the machine to check that it is working properly. If so, the draw is made. They do not test the actual draw - that would be pointless!

Note: I know nothing about ERNIE or the process itself. I am inferring this from the wikipedia page together with a basic assumption - which may well be unfounded - that the British government couldn't be that stupid. Combining this with the fact that if the sentence is grammatically correct - perhaps another unfounded assumption - then the 'it' refers, as I said, to the 'output' not to the 'draw', I conclude that the 'output' and the 'draw' are two separate things.

-

Yes. In the same way that flipping a fair coin (with equal probabilities of getting heads of tails) eight times in a row is likely to come up all heads 1/256 times, or all tails 1/256 times. The psychological perception of a sequence with a run of 8 heads or 8 tails is that it is so unlikely as to never occur at all; whereas we mathematicians see the likelihood of a run of 8 in 8 flips as occuring with 2/256 or just under 1% of the time.

The opposite error is true, and also occurs with some frequency in biomedical experiments and medical experiments. The standard for accepting a result in a clinical or medical trial is for $p<0.05$: that there is less than a 5% probability that the results occured by chance. Thus, one in twenty times, it is possible that a random occurence or set of occurences will be perceived or accepted as being statistically valid when it is not.

But it also depends on how much data (how many draws) are in the sample being gauged for randomness. The smaller the sample size, the more likely you are to discard a valid but unreasonable appearing "true" random sequence. So my answer is really a qualified "maybe".

Shouldn't the validity or "true randomness" of the method be the gauge, along with a check to see that the algorithm is properly implemented? The problem, of course, with software is that bugs can creep into the implementation at any point:

• the compiler could be messed up, generating incorrect code,

• the ALU (the arithmetic logic unit) in the central processing unit could be incorrectly implemented, e.g. the Pentium chip had the floating-point co-processor which incorrectly calculated some particular multiply operations depending upon the operands,

• the algorithm specified in the program requirements may be correct, but may be implemented incorrectly,

• the algorithm may be correct for 32-bit integer math calculations, but incorrect if the system uses 64-bit arithmetic, or vice versa,

I wonder if the software code and hardware is vetted as well there as it is in Las Vegas by the Nevada Gaming Commision which oversees gambling and the electronic machinery for slots and electronic poker, etc.

-
So you're actually saying "YES" and "NO" based on how the actuary performs the output test? – ABh Aug 28 '10 at 20:58

I would think not, but there is not really enough information to be sure.

A) Has the output ever been rejected? If not then no bias!

B) Suppose the US government gave out prizes by a 3 step process.

1. a set of 10,000 9 digit numbers is generated.
2. The set is tested for statistical randomness
3. If it passes, find which are valid social security numbers and split the prize money among those people (or give each \$X) OTHERWISE go back to step 1.

That would be fair if the tests were not biased against particular individual numbers. Tests such as those described in http://en.wikipedia.org/wiki/Diehard_tests would be fine.

It seems as if maybe that is the kind of thing going on (based on a quick perusal of the Wikipedia article on ERNIE)

I suppose that in my example every individual would have a fair chance of winning but (if it is the case that the social security number indicates age and/or birthplace) there might be non-randomness in that a case of almost all winners born in UTAH might be rejected for bunching and thus never happen.

-
"Has the output ever been rejected? If not then no bias!" This is not true -- the bias (in the underlying distribution) is still there, it's just not detectable from the data. – JBL Aug 28 '10 at 0:12
(Of course my previous comment is based on the assumption that rejecting output actually does alter the underlying distribution.) – JBL Aug 28 '10 at 0:13
The question isn't that clearly defined. My, not that well expressed points were A) If rejection only happens on a machine malfunction or a 1 in 10^10 coincidence, then no distortion of randomness has happened in our lifetime (and any that did would be less than epsilon) B) Suppose a machine on each run generates 1000 huge numbers one after another. The machine is actually perfect but we reject a set if the numbers come out in increasing order (or with too few or too many inversions). Then we would reject valid runs but would have a uniform distribution on 1000 integer sets. – Aaron Meyerowitz Aug 28 '10 at 5:22
It also seems to depend strongly on exactly what sort of test or audit is going to be performed by the actuary to assess this RNG (random number generator). Measure the smoothness of the distribution? Cross-correlation? How many data items are in the output to be assessed? Is there absolute certainty that the machine performs the same way and utilizes exactly the same algorithm and code steps when it is creating output for testing as opposed to creating output to be an actual draw? Is the code itself audited? Has the code been verified to be correct and random by specific criteria? – ABh Aug 28 '10 at 21:03
Aaron, I don't think we disagree much (if at all). I just intended to point out that if the test being applied alters, say, every one in 1000! draws but does so in a way that destroys uniformity over the set of interest (so, not as in your example), then the result is biased, even though in the lifetime of the universe this bias will never result in any outcomes that differ from what would have happened if the selection were truly uniform. – JBL Aug 28 '10 at 22:58