Is ERNIE output skewed by statistical tests?

Question

ERNIE is a hardware random number generator used to generate winning Premium Bond numbers in the UK. Wikipedia says: "ERNIE's output is independently tested each month by an independent actuary appointed by the government, and the draw is only valid if the output passes tests that indicate it is statistically random." But doesn't it mean that, if ERNIE output was truly and fully random (i.e. generated from a unitary distribution over a set of integers from 1 to N), rejecting some of the draws would actually distort the randomness?

Answer 1

No.

The final 'it' in the quoted section refers to the word 'output' not to the word 'draw'. Thus the "independent actuary" tests the output of the machine to check that it is working properly. If so, the draw is made. They do not test the actual draw - that would be pointless!

Note: I know nothing about ERNIE or the process itself. I am inferring this from the wikipedia page together with a basic assumption - which may well be unfounded - that the British government couldn't be that stupid. Combining this with the fact that if the sentence is grammatically correct - perhaps another unfounded assumption - then the 'it' refers, as I said, to the 'output' not to the 'draw', I conclude that the 'output' and the 'draw' are two separate things.

Answer 2

Yes. In the same way that flipping a fair coin (with equal probabilities of getting heads of tails) eight times in a row is likely to come up all heads 1/256 times, or all tails 1/256 times. The psychological perception of a sequence with a run of 8 heads or 8 tails is that it is so unlikely as to never occur at all; whereas we mathematicians see the likelihood of a run of 8 in 8 flips as occuring with 2/256 or just under 1% of the time.

The opposite error is true, and also occurs with some frequency in biomedical experiments and medical experiments. The standard for accepting a result in a clinical or medical trial is for $p<0.05$: that there is less than a 5% probability that the results occured by chance. Thus, one in twenty times, it is possible that a random occurence or set of occurences will be perceived or accepted as being statistically valid when it is not.

But it also depends on how much data (how many draws) are in the sample being gauged for randomness. The smaller the sample size, the more likely you are to discard a valid but unreasonable appearing "true" random sequence. So my answer is really a qualified "maybe".

Shouldn't the validity or "true randomness" of the method be the gauge, along with a check to see that the algorithm is properly implemented? The problem, of course, with software is that bugs can creep into the implementation at any point:

• the compiler could be messed up, generating incorrect code,

• the ALU (the arithmetic logic unit) in the central processing unit could be incorrectly implemented, e.g. the Pentium chip had the floating-point co-processor which incorrectly calculated some particular multiply operations depending upon the operands,

• the algorithm specified in the program requirements may be correct, but may be implemented incorrectly,

• the algorithm may be correct for 32-bit integer math calculations, but incorrect if the system uses 64-bit arithmetic, or vice versa,

I wonder if the software code and hardware is vetted as well there as it is in Las Vegas by the Nevada Gaming Commision which oversees gambling and the electronic machinery for slots and electronic poker, etc.

Answer 3

I would think not, but there is not really enough information to be sure.

A) Has the output ever been rejected? If not then no bias!

B) Suppose the US government gave out prizes by a 3 step process.

1. a set of 10,000 9 digit numbers is generated.
2. The set is tested for statistical randomness
3. If it passes, find which are valid social security numbers and split the prize money among those people (or give each $X) OTHERWISE go back to step 1.

That would be fair if the tests were not biased against particular individual numbers. Tests such as those described in http://en.wikipedia.org/wiki/Diehard_tests would be fine.

It seems as if maybe that is the kind of thing going on (based on a quick perusal of the Wikipedia article on ERNIE)

I suppose that in my example every individual would have a fair chance of winning but (if it is the case that the social security number indicates age and/or birthplace) there might be non-randomness in that a case of almost all winners born in UTAH might be rejected for bunching and thus never happen.