2. Statistical Tools in Analytical Method Validation

Statistical analysis of data obtained during a method validation should be performed to demonstrate validity of the analytical method. The primary parameters used for the interpretation of analytical method validation results are the calculation of the mean (or average), standard deviation, relative standard deviation, confidence intervals, and regression analysis and the main tools are the F-test, t-test, and regression and correlation analysis. These calculations are characteristically performed using statistical software packages such as SPSS, R, and Minitab. The goal of statistical analysis is to summarize a collection of data that provides an understanding of the examined method characteristic. The statistical tools are described in the following.


2.1 Mean

The mean or average of a data set is the basic and the most common statistics used. The mean is calculated by adding all data points and dividing the sum by the number of samples. It is typically denoted by \(\overline X\) (X bar) and is computed using the following formula: where Xi are individual values and n is the number of individual data points.

\( \overline X = \frac{\Sigma X_i }{n} = \frac{X_1 + X_2 + ... + X_n}{n} \),  where \(X_i\) are individual values and \(n\) is the number of individual data points.


2.2 Standard deviation

The standard deviation of a data set is the measure of the spread of the values in the sample set and is computed by measuring the difference between the mean and the individual values in a set. It is computed using the following formula: where Xi is individual value,  is the sample mean, and n is the number of individual data points.

\(s = \sqrt{\frac{\Sigma_i (X_i - \overline X)^2}{n-1}} = \sqrt{\frac{(X_1 - \overline X)^2 + .... + (X_n - \overline X)^2}{n-1}}\), where \(X_i\) are individual values and \(n\) is the number of individual data points.


2.3 Relative standard deviation

The relative standard deviation is computed by taking the standard deviation of the sample set multiplied by 100% and dividing it by the sample set average. The relative standard deviation is expressed as percent. Generally, the acceptance criteria for accuracy, precision, and repeatability of data is expressed in %RSD:

\(%RSD = \frac{s}{\overline X} \times 100%\)

\(V = s^2\)


2.4 Confidence interval


Confidence intervals are used to indicate the reliability of an estimate. Confidence intervals provide limits around the sample mean to predict the range of the true population of the mean. The prediction is usually based on probability of 95%. The confidence interval depends on the sample standard deviation and the sample mean.

Confidence interval for: \( \mu = \overline X  \pm \frac{zs}{\sqrt n}\),

where \(s\) is the sample deviation,  \(\overline X\) is the sample mean, \(n\) is the number of individual data points, and \(z\) is constant obtained from statistical tables for \(z\).

The value of \(z\) depends on the confidence level listed in statistical tables for \(z\). For 95%, \(z\) is 1.96. For small samples, \(z\) can be replaced by t-value obtained from the Student's t-distribution tables. The value of \(t\) corresponds to \(n−1\).

For example, for:

  •     \(n = 6\),
  •     mean = 101.82,
  •     standard deviation = 1.2,
  •     95% confidence interval = 0.49.


So, this calculated confidence interval indicates that the range of the true population of the mean is between 101.33 and 102.31.

In brief, confidence interval can be applied in various ways to determine the size of errors (confidence) in analytical work or measurements: single determinations in routine work, certain calibrations, determinations for which no previous data exist, etc.


2.5 Regression analysis

Regression analysis is used to evaluate a linear relationship between test results. A linear relationship is, in general, evaluated over the range of the analytical procedure. The data obtained from analysis of the solutions prepared at a range of different concentration levels is habitually investigated by plotting on a graph.

Linear regression evaluates the relationship between two variables by fitting a linear equation to observed data. A linear regression line has an equation of the form \(Y = b_0 + b_1 × X\), where \(X\) is the independent variable and \(Y\) is the dependent variable. The slope of the line is \(b_1\), and \(b_0\) is the intercept (the value of \(y\) when \(x = 0\)). The statistical procedure of finding the "best-fitting" straight line is to obtain a line through the points to minimize the deviations of the points from the prospective line. The best-fit criterion of goodness of the fit is known as the principle of least squares. In mathematical terms, the best fitting line is the line that minimizes the sum of squares of the deviations of the observed values of \(Y\) from those predicted.

In Figure 3, the data clearly shows a linear relationship.

figure 3

Figure 3 Linear regression analysis of the calibration curve of Ecstasy in a seizure by UV-Vis spectrophotometer. Each point is the mean ∓SD of three experiments.

Calculations used to compute y-intercept and the line slope are as follows:

\(b_1 = \frac{S_{xy}}{S_{xx}}, b_0 = \overline Y - b_1 \times \overline X\),
\(S_{xx} = \Sigma_{i=1} ^n X_i ^2, S_{xy} = \Sigma_{i=1}^n (X_iY_i)^2 - \frac{\Sigma_i ^2 X_i \times \Sigma_i ^n y_i}{n}\)


So, the equation of the line, for data listed in Figure 2, is \(Y = −0.2147 + 0.0225 X\).

figure 2

Figure 2 A Gaussian or normal distribution.


Once a regression model has been fit to a group of data, examination of the residuals (the deviations from the fitted line to the observed values) allows investigation of the validity of the assumption that a linear relationship exists. Plotting the residuals on the y-axis against the independent variable on the x-axis reveals any possible non-linear relationship among the variables or might alert to investigate outliers.

The other important calculations that are ordinarily reported are the coefficient of determination (\(r^2\)) and linear correlation coefficient (\(r\)). The coefficient of determination (\(r^2\)) measures the proportion of variation that is explained by the model. Ideally, \(r^2\)should be equal to one, which would indicate zero error. The correlation coefficient (\(r\)) is the correlation between the predicted and observed values. This will have a value between 0 and 1; the closer the value is to 1, the better the correlation. Any data that form a straight line will give high correlation coefficient; therefore, extra caution should be taken when interpreting correlation coefficient. Additional statistical analysis is recommended to provide estimates of systematic errors, not just the correlation or results. For example, in method comparison studies, if one method gives consistently higher results than the other method, the results would show linear correlation and have a high correlation coefficient, although a difference between the two methods.

Equations used to determine the coefficient of determination (\(r^2\)) and the correlation coefficient (\(r\)) are listed in Table 3.

Table 3 Regression summary.

Data Descriptions Mathematical expressions
Equation of the line The relationship between the concentration (\(X\)) and the response (\(Y\)) \(Y = b_0 + b_1 \times X\)
Intercept (\(b_0\)) The value of \(Y\) when \(X\) equals zero \(b_0 = y − b_1 \times X\)
Slope (\(b_1\)) The slope of the line relate to the relationship between concentration and response \(1b_1 = \frac{S_{xy}}{S_{xx}}\)
Standard error (\(b_0\)) (\(SE\) intercept) The standard error of the intercept can be used to calculate the required confidence interval \(SE_{b_0} = s \sqrt {\frac{1}{n} + \frac{\overline X^2}{\Sigma (X_i + \overline X)^2}}\)
  95% confidence interval –
Standard error (\(b_1\)) (\(SE\) slope) The standard error of the slope can be used to calculate the required confidence interval \(SE_{b_1} = \frac{s}{\Sigma )X_i + \overline X)^2}\)
  95% confidence interval –
Coefficient of determination (\(r^2\)) The square of the correlation coefficient \(r^2 = \frac{SSM}{SST}\)
Correlation coefficient (\(r\)) The correlation between the predicted and observed values. This will have a value between 0 and 1; the closer the value is to 1, the better the correlation \(\sqrt {r^2}\)
Regression \(SS\) The regression sum of squares is the variability in the response that is accounted for by the regression line SS total − \(∑(X_i)^2\)
Residual \(SS\) (the error sum of squares) The residual sum of squares is the variability about the regression line (the amount of uncertainty that remains) SS total − SS regression
Total \(SS\) The total sum of squares is the total amount of variability in the response \(∑(Y_i − Y)^2\)

2.6 The hypothesis tests

The hypothesis tests are intended to verify if the experimental data are consistent with certain theoretical hypothesis.

  • The null hypothesis symbolized by \(H_0\) considers that the two elements or series of elements are equal.
  • The second step consists in measuring the deviation between different characteristics.
  • The third step is to calculate the probability \(P\) to have this deviation if \(H_0\) is true.
  • The fourth step is to draw conclusions that are required:

If\(P\) is large, we admit that \(H_0\) is plausible, on the other side if \(P\) is small, the deviation is incompatible with \(H_0\). The value limit of \(P\) that is fixed to determine if \(P\) is large or small is the level of confidence or significance level (usually we chose \(P = 0.95\) as level of confidence (\(α = 0.05\) as significance level)).

Four situations are possible:

  • Acceptance of \(H_0\)true.
  • Rejecting true \(H_0\): first error species (α).
  • Acceptance false \(H_0\): second error species (β).
  • Rejecting false \(H_0\).


2.7 Other statistical tools

Other statistical tools used in method validation include comparative studies using Student's t-test, Fisher's test, analysis of variation (ANVA), design of experiments, and assessment of outliers. Information on these statistical tools can be obtained from references on statistics suggested in the reference section.