Do the four methods have the same mean bias?
We have developed three alternative analytical methods — A, B and C — and want to compare them with the currently used Reference method.
The scientific question is deliberately simple:
Do the four methods have the same mean relative bias?
If they do not, we will eventually want to know which methods differ, by how much, and whether those differences matter chemically.
But those are different questions.
For now, we ask only the global one.
This is why we start with a one-way ANOVA.
Not because ANOVA is simply “the test for more than two groups”, but because we want to start with the simplest model that can answer the question we have asked.
The response variable is RelativeBias_pct, the relative bias of the analytical result.
The dataset contains:
The four measurements from a sample are obtained from separate aliquots and are treated as independent analytical observations for the purpose of this introductory model. This is an assumption about the error structure of the model, not a claim that measurements from the same sample are unrelated. Later in the series, we will revisit what changes when sources of shared variation or repeated measurements become part of the question we want to answer.
For this first analysis, however, we deliberately ignore Matrix.
This is an important modelling decision.
Matrix is present in the data, and we know that it may matter. We are not claiming otherwise.
We are asking a deliberately narrower question first:
If we look only at Method, is there evidence that the four methods have different mean relative bias?
Later, we will add the information that we are ignoring here.
library(here)
library(data.table)
library(ggplot2)
dat <- fread(
here("data", "dataset_v01_clean_balanced.csv")
)
dat[, Method := factor(
Method,
levels = c(
"Reference",
"Method_A",
"Method_B",
"Method_C"
)
)]
dat[, .N, by = Method] Method N
<fctr> <int>
1: Reference 60
2: Method_A 60
3: Method_B 60
4: Method_C 60The design is balanced:
The balanced design is convenient for this first chapter.
It will become deliberately unbalanced later in the series, because one of the most important questions in practical ANOVA is what happens when the clean textbook design disappears.
A statistical model should not be the first thing we look at.
Start with simple descriptive statistics.
method_stats <- dat[, .(
Mean = mean(RelativeBias_pct),
SD = sd(RelativeBias_pct),
SE = sd(RelativeBias_pct) / sqrt(.N),
N = .N
), by = Method]
method_stats Method Mean SD SE N
<fctr> <num> <num> <num> <int>
1: Reference -9.522000 3.226779 0.4165753 60
2: Method_A -17.627167 4.060234 0.5241740 60
3: Method_B 1.952717 3.775214 0.4873780 60
4: Method_C -2.600867 3.673626 0.4742631 60The estimated mean biases are approximately:
The differences are already substantial.
But descriptive statistics alone cannot tell us whether the observed differences are compatible with a common population mean.
That is the role of the model.
Our question concerns mean bias.
The plot should therefore show both the individual observations and the estimated means.
method_stats[, `:=`(
CI_low = Mean - qt(0.975, N - 1) * SE,
CI_high = Mean + qt(0.975, N - 1) * SE
)]
ggplot(dat, aes(x = Method, y = RelativeBias_pct)) +
geom_jitter(
width = 0.12,
alpha = 0.35,
size = 1.5
) +
geom_errorbar(
data = method_stats,
aes(
y = Mean,
ymin = CI_low,
ymax = CI_high
),
width = 0.12,
linewidth = 0.7
) +
geom_point(
data = method_stats,
aes(y = Mean),
size = 3
) +
geom_hline(
yintercept = 0,
linetype = "dashed"
) +
geom_hline(
yintercept = c(-5, 5),
linetype = "dotted"
) +
labs(
x = NULL,
y = "Relative Bias (%)",
title = "Relative bias by analytical method",
subtitle = "Points = observations; symbols and intervals = mean ± 95% CI"
) +
annotate(
"text",
x = 4.45,
y = 5,
label = "+5 pp",
hjust = 1,
vjust = -0.4,
size = 3.5
) +
annotate(
"text",
x = 4.45,
y = -5,
label = "−5 pp",
hjust = 1,
vjust = 1.4,
size = 3.5
) +
theme_minimal(base_size = 12)
The horizontal dashed line represents zero bias.
The dotted lines at ±5 percentage points represent the predefined threshold for chemical relevance used in this project.
They are not confidence limits.
They are not statistical significance thresholds.
They simply provide a scientific reference against which the magnitude of the observed bias can be considered.
This distinction will become increasingly important as we move through the series.
The four observed means are different.
The ANOVA asks whether those differences provide enough evidence to reject the hypothesis that the corresponding population means are all equal.
Our null hypothesis is:
\[ H_0: \mu_\mathrm{Reference} = \mu_A = \mu_B = \mu_C \]
The alternative hypothesis is:
\[ H_1: \text{at least one population mean differs} \]
Notice what this means.
The ANOVA does not test whether every individual observation is equal.
It does not tell us which methods differ.
It does not tell us which method is closest to zero.
And it does not tell us whether a difference is chemically important.
Those are separate questions.
One-way ANOVA can be written as a linear model:
\[ Y_{ij} = \mu + \alpha_i + \varepsilon_{ij} \]
where:
For the classical model we assume:
\[ \varepsilon_{ij} \sim N(0,\sigma^2) \]
with independent errors and a common residual variance.
These assumptions are part of the model.
They are not optional conditions that can be ignored once the F-statistic has been calculated.
We will investigate them much more carefully in Chapter 07 and deliberately introduce assumption violations in Chapter 08.
The basic idea behind ANOVA is straightforward.
The total variation in the response can be decomposed into:
How much do the method means differ from the overall mean?
How much do individual observations vary around their own method mean?
Conceptually:
\[ SS_\mathrm{Total} = SS_\mathrm{Method} + SS_\mathrm{Residual} \]
The F-statistic compares the corresponding mean squares:
\[ F = \frac{MS_\mathrm{Method}} {MS_\mathrm{Residual}} \]
If the method means are sufficiently different relative to the residual variation, the observed F-statistic becomes difficult to explain under the null hypothesis.
This is the core of one-way ANOVA.
The mathematics is simple.
The interpretation becomes more interesting when we start asking what the model assumes and what exactly its output means.
lm()Because one-way ANOVA is a linear model, we can fit it directly with lm().
fit_oneway <- lm(
RelativeBias_pct ~ Method,
data = dat
)
summary(fit_oneway)
Call:
lm(formula = RelativeBias_pct ~ Method, data = dat)
Residuals:
Min 1Q Median 3Q Max
-9.6991 -2.3251 0.0914 2.8272 8.6320
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -9.5220 0.4772 -19.95 <2e-16 ***
MethodMethod_A -8.1052 0.6748 -12.01 <2e-16 ***
MethodMethod_B 11.4747 0.6748 17.00 <2e-16 ***
MethodMethod_C 6.9211 0.6748 10.26 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 3.696 on 236 degrees of freedom
Multiple R-squared: 0.8028, Adjusted R-squared: 0.8003
F-statistic: 320.3 on 3 and 236 DF, p-value: < 2.2e-16With Reference as the baseline level, the fitted model is parameterized as:
\[ Y = \beta_0 + \beta_A I_A + \beta_B I_B + \beta_C I_C + \varepsilon \]
The intercept is therefore the estimated mean bias of the Reference method.
The other coefficients are differences from Reference:
\[ \beta_A = \mu_A-\mu_\mathrm{Reference} \]
\[ \beta_B = \mu_B-\mu_\mathrm{Reference} \]
\[ \beta_C = \mu_C-\mu_\mathrm{Reference} \]
This is our first encounter with an important feature of linear models:
The scientific question and the model parameterization are not the same thing.
R has chosen a baseline because we told it to use Reference as the first factor level.
That does not mean that Reference is inherently the most interesting comparison.
It simply determines how the coefficients are expressed.
We will return to this distinction in the next chapter.
anova(fit_oneway)Analysis of Variance Table
Response: RelativeBias_pct
Df Sum Sq Mean Sq F value Pr(>F)
Method 3 13127.4 4375.8 320.31 < 2.2e-16 ***
Residuals 236 3224.1 13.7
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1The resulting F-test is approximately:
\[ F(3,236)=320 \]
with:
\[ p < 0.001 \]
The evidence against the null hypothesis is therefore very strong.
We reject:
\[ H_0: \mu_\mathrm{Reference} = \mu_A = \mu_B = \mu_C \]
There is strong evidence that the four methods do not share the same population mean relative bias.
That answers the question we asked.
aov()The traditional R interface for ANOVA is aov().
fit_aov <- aov(
RelativeBias_pct ~ Method,
data = dat
)
summary(fit_aov) Df Sum Sq Mean Sq F value Pr(>F)
Method 3 13127 4376 320.3 <2e-16 ***
Residuals 236 3224 14
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1For this simple one-way balanced design, the ANOVA table is the same.
aov() and lm() are therefore not two competing statistical methods here.
The important conceptual point is that ANOVA belongs to the linear-model framework.
We will continue using lm() because the same framework naturally extends to:
Later we will also encounter anova() and car::Anova(), where the differences between functions become more consequential.
The fitted model gives:
summary(fit_oneway)$r.squared[1] 0.8028269with:
\[ R^2 \approx 0.80 \]
In this dataset, Method accounts for approximately 80% of the observed variability in relative bias.
That tells us something useful about the model.
It does not tell us that Method is “80% scientifically important”, nor does it establish a universal threshold for what constitutes a large effect.
\(R^2\) describes the proportion of variance explained by this model for this dataset.
Scientific importance still requires us to look at the magnitude of the effects and the context in which the measurements will be used.
This distinction will be one of the recurring themes of the series.
A statistically significant difference is not automatically a scientifically important difference.
With sufficiently precise measurements, a very small difference can be statistically significant.
Conversely, a scientifically important difference may fail to reach statistical significance in a small or highly variable experiment.
For this project we define in advance:
\[ \boxed{5\text{ percentage points}} \]
as the minimum difference in relative bias considered chemically relevant.
This is a scientific decision criterion.
It is not a property of ANOVA.
It is not a replacement for a p-value.
It gives us a way to interpret the size of estimated effects after considering their statistical uncertainty.
The estimated means are approximately:
| Method | Mean relative bias |
|---|---|
| Reference | −9.5% |
| Method A | −17.7% |
| Method B | +2.0% |
| Method C | −2.6% |
Several observations stand out.
The Reference method has a substantial negative bias.
This is important scientifically: the reference method is not assumed to be unbiased simply because it is called “Reference”.
Its performance is part of the problem we are investigating.
Method A is substantially more negatively biased than Reference.
The difference is large enough to be both statistically and chemically relevant.
Method B is closest to zero.
Its mean bias is approximately +2.0%.
Method C is also close to zero, at approximately −2.6%.
The estimated difference between B and C is therefore:
\[ 2.0 - (-2.6) \approx 4.6 \]
percentage points.
That is close to our 5-percentage-point threshold, but below it.
Whether that difference is statistically significant is a separate question from whether it is chemically relevant.
This distinction is precisely why we need to look beyond the global ANOVA p-value.
Suppose the true concentration of an analyte were 100 mg/kg.
A relative bias of −17.7% would correspond to an average result of approximately:
\[ 100(1-0.177)=82.3\text{ mg/kg} \]
The average difference would therefore be about 18 mg/kg.
A bias of this magnitude could be consequential when analytical results are compared with regulatory thresholds.
This does not mean that bias alone determines a regulatory decision.
Measurement uncertainty, the position of the result relative to the decision threshold and the applicable decision rule all matter.
The example simply illustrates why effect magnitude matters independently of statistical significance.
We have fitted a classical one-way ANOVA.
That means we have made assumptions about the error structure.
The observations contributing to the residual error should be independent, given the experimental design.
In our analytical scenario, each sample is homogenized and aliquoted, with separate aliquots analysed using the four methods.
For this dataset, we treat those analytical observations as independent replicates.
That assumption is based on the experimental process, not on a diagnostic plot.
This distinction matters.
Independence is fundamentally a property of how the observations were generated.
Later in the series we will examine situations involving shared batches, analysts, instruments or days, and consider when those sources of variation require a different model.
The classical F-test assumes normally distributed errors.
This does not mean that every group of raw observations must itself look perfectly normal.
The relevant assumption concerns the residuals of the fitted model.
The classical model also assumes a common residual variance across the methods.
If one method is substantially more variable than another, the classical ANOVA inference may no longer have the properties we expect.
Again, we will investigate this properly later.
Because the purpose of this chapter is to understand the basic model and its global test.
A complete analysis of real data should not stop here.
But there is also a pedagogical danger in beginning with a long list of diagnostic tests before understanding what the model is doing.
So we will proceed incrementally.
For now:
In Chapter 07 we will inspect residuals and diagnostics in detail.
In Chapter 08 we will deliberately violate important assumptions and investigate what happens to the inference.
The global F-test has answered our first question.
It has not answered several other important questions.
The F-test tells us that the four population means are not all equal.
It does not tell us which pairs differ.
The coefficient table from lm() gives comparisons with Reference because of the chosen parameterization.
But perhaps the scientifically important comparison is B versus C.
That is not a reason to change the scientific question simply because R has chosen a convenient baseline.
It is a reason to formulate the appropriate comparison explicitly.
That is the subject of Chapter 02.
We deliberately ignored Matrix.
But the dataset contains three environmental matrices.
If the mean bias differs between Fill, Soil and Sediment, the one-way model is hiding potentially important structure.
We will introduce Matrix as a second factor in Chapter 03.
This is a stronger question.
A method might perform well in one matrix and poorly in another.
If that happens, there may be no single method that can simply be labelled “best” independently of matrix.
This leads to interactions.
ANOVA does not answer this.
The p-value concerns evidence against a statistical null hypothesis.
Chemical relevance requires a separate scientific criterion.
For this dataset, we have deliberately defined a 5-percentage-point threshold so that this distinction is concrete rather than merely theoretical.
At first glance, the workflow seems almost trivial:
fit <- lm(RelativeBias_pct ~ Method, data = dat)
anova(fit)But those two lines hide several decisions.
We decided:
The function call is the easy part.
The statistical reasoning happened before it.
ANOVA is not a black box that outputs “significant” or “not significant”.
It is a linear model that compares variation between groups with variation within groups.
The F-test answers one specific global question:
Are the population means all equal?
For our dataset, the answer is clearly no.
But that is only the beginning.
We still need to determine:
The statistical analysis becomes useful only when those questions remain connected to the scientific problem.
We now know that the four methods do not have the same mean relative bias.
But the global F-test does not tell us which comparisons matter.
R has already produced three comparisons against the Reference method.
But are those the comparisons we actually want?
And what happens when we want to compare several methods while controlling the uncertainty associated with multiple comparisons?
Next, we move from the global ANOVA to contrasts and multiple comparisons.
The question becomes:
Which specific differences should we estimate, and how should we test them?