1
$\begingroup$

I have been asked to assess the accuracy and precision of a new measurement method (Let's call it method B). This new method is compared to an existing one (A) that has its own specifications in terms of stdev of a single measurement. What we do is to measure several samples with method A and then with method B. Since A is very expensive, only one A measurement per case is available. Method B is cheaper, so we measure each sample with method B for several times.

Another problem is that we are unable to find samples that would span across the entire legal measurement range, resulting in several samples in the first quartile of the range, several in the last range quartile and almost no in between.

How can I assess the accuracy and precision of method B? Any help or link will be appreciated.

Thank you very much

P.S. This is not a homework.

P.P.S I admit, I don't know statistics well

$\endgroup$
2
  • 2
    $\begingroup$ When you say method A "has its own specifications in terms of stdev of a single measurement", do you mean that A is known to satisfy those specifications? Is A unbiased in the sense of giving the right answer on average? $\endgroup$ Jul 8, 2010 at 21:16
  • $\begingroup$ ....and also, is the standard deviation of the answer given by A independent of the quantity being measured? (This would imply sometimes giving a negative number even when the answer is always positive; that's part of the downside of unbiasedness.) Or could it be that the standard deviation of the logarithm of A is independent of the size of the quantity being measured? $\endgroup$ Jul 8, 2010 at 21:18

2 Answers 2

1
$\begingroup$

I have asked this question on Allstat and currently considering adopting this suggestion

From your question I understand that there is only one measurement per case with method A. This means that assessing the agreement between A and B reduces to showing that the mean differences between A and B (accuracy)($\overline{\Delta_{A,B}} > \approx 0$) is as close to 0 as possible and the standard deviation of these differences (precision) is as low as possible.

I also learn that the measurements you take can take values between two numbers. This might lead to the situation where the distribution of $\Delta_{A,B}$ is far from being normal. On the other hand, you have multiple measurements of several cases. Now, here comes the tricky part. Assume that the real mean difference between A and B readings is $\mu$ with standard deviation of $\sigma$. We may treat those multiple measurements as different samplings from the overall distribution. Each sampling $i$ has its own mean difference $\overline{\Delta_i}$. According to the central limit theorem, the mean of means ($\mu_{\overline{\Delta}}$) is a good approximation of real $\mu$ and the standard deviation of deltas is connected to the real standard deviation $sigma$ as follows: $\sigma_{\overline{\Delta}} = > \frac{\sigma}{\sqrt{n}}$. You will be also able to calculate the 95\% confidence interval of the difference estimate using either Z or t distribution (depending on the number of cases you have measured)

Having all this information you will be able to conclude that B agrees with A within $\mu_{\overline{\Delta}}$ with standard deviation of $\sigma_{\overline{\Delta}} \times > \sqrt{n}$ or that B agrees with A within $\pm CI_{95\%}$

Is there any reason not to?

$\endgroup$
1
  • 1
    $\begingroup$ I'm not sure if this applies to your problem, but the case of quantized data (which you usually got after a measurement) is often treated wrongly in the literature. Take a look at statistik.tu-dortmund.de/fileadmin/user_upload/Lehrstuehle/… to get an idea of the pitfalls. $\endgroup$
    – j.p.
    Aug 12, 2010 at 13:52
0
$\begingroup$

I assume that the output is continious. Whatever you do, don't try calculating the correlation between the methods. I think that the paper "Statistical methods for assessing agreement between two methods of clinical measurement" is what you need

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.