Aspen biomass data

Twenty aspen trees from Minnesota, USA are used as example data. On these trees, diameter at breast height ranged from 6.7 to 13.0 inches. We seek to compare predictions of aboveground tree biomass from two different models. The first model makes predictions from the Component Ratio Method (CRM). A set of regional volume equations drive the CRM approach, which are then converted to tree biomass. The second model makes predictions from the National Scale Volume-Biomass Estimators (NSVB), a recently developed set of equations that replaced the CRM equations in estimates provided by the USDA Forest Service.

Here are the 20 aspen trees and their biomass predictions from the two models:

You’ll see below that the NSVB predictions for these aspen trees are greater than the CRM predictions for each tree. The finding that NSVB biomass predictions are greater than CRM ones was also described when these equations were expanded across the US:

t-test

First, let’s conduct a two-sample t-test using the t.test() function in R. We can specify a two-sided two-sample t-test at a level of significance of \(\alpha = 0.05\) that determines whether or not the two predictions of biomass are equal with the following hypotheses:

• The null hypothesis states that the mean of the CRM biomass is equal to the NSVB biomass.
• The alternative hypothesis states that the mean of the CRM biomass is not equal to the NSVB biomass.

We’ll assume the variances of both predictions are not equal using the var.equal = F statement:

t_test <- t.test(tree$AGB_NSVB, tree$AGB_CRM, var.equal = F)

print(t_test)

    Welch Two Sample t-test

data:  tree$AGB_NSVB and tree$AGB_CRM
t = 1.2281, df = 34.652, p-value = 0.2277
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -57.6855 234.1755
sample estimates:
mean of x mean of y 
  492.225   403.980 

Note that the calculated t-statistic is 1.2281. The p-value of 0.2277 is greater than our level of significance of \(\alpha = 0.05\), hence, we have evidence to accept the null hypothesis and conclude that the CRM biomass predictions are equal to the NSVB biomass predictions.

Equivalence test

For conducting equivalence tests in R, the TOSTER package provides functions for computing two one-sided tests that can compared with the t-test. We’ll set the equivalence bounds for the test, or the regions where we would expect differences. This value can be subjective and is similar to the level of significance that is specified for a hypothesis test. However, it’s different because it is expressed in the units of the variable that we’re testing (i.e., in pounds of a tree in our example).

A common approach is to specify the magnitude of this region of similarity as a percentage of the standard deviation of the difference in the two values. This allows for some disagreement while still holding the model to strict accuracy standards. For example, we’ll assign the region of similarity to be +/- 25% of the standard deviation of the difference between CRM and NSVB biomass predictions, which is approximately 19 pounds:

tree <- tree |> 
  mutate(model_diff = AGB_NSVB - AGB_CRM)

sd_25 <- sd(tree$model_diff)*0.25

low_eqbound <- -sd_25
high_eqbound <- sd_25
low_eqbound; high_eqbound

[1] -19.12677

[1] 19.12677

The tsum_TOST() function available in the TOTSER package can perform the two one-sided test of equivalence. We first specify the mean, standard deviation, and total number of observations for each variable (e.g., m1, sd1, and n1), then we specify the equivalence bounds:

library(TOSTER)

eq_test <- tsum_TOST(m1 = mean(tree$AGB_NSVB),
                     sd1 = sd(tree$AGB_NSVB), 
                     n1 = length(tree$AGB_NSVB),
                       
                     m2 = mean(tree$AGB_CRM), 
                     sd2 = sd(tree$AGB_CRM), 
                     n2 = length(tree$AGB_CRM),
                       
                     low_eqbound = -sd_25, 
                     high_eqbound = sd_25)
print(eq_test)

Welch Two Sample t-test

The equivalence test was non-significant, t(34.65) = 0.962, p = 8.29e-01
The null hypothesis test was non-significant, t(34.65) = 1.228, p = 2.28e-01
NHST: don't reject null significance hypothesis that the effect is equal to zero 
TOST: don't reject null equivalence hypothesis

TOST Results 
                t    df p.value
t-test     1.2281 34.65   0.228
TOST Lower 1.4942 34.65   0.072
TOST Upper 0.9619 34.65   0.829

Effect Sizes 
               Estimate      SE                 C.I. Conf. Level
Raw             88.2450 71.8573 [-33.1963, 209.6863]         0.9
Hedges's g(av)   0.3799  0.3276    [-0.1371, 0.8914]         0.9
Note: SMD confidence intervals are an approximation. See vignette("SMD_calcs").

You’ll first notice that the output provides results from a t-test, confirming the t-value of 1.2281 that we found above. The TOST results indicate a similar result to the t-test, that is we would not reject the null equivalence hypothesis and conclude the model results are similar. Note the output contains two p-values, one each for the lower and upper part of the test, along with the effect sizes.

I suspect the model predictions turn out to be equivalent here because of the relatively small difference we set in the equivalence bounds. After all, 19 pounds of biomass is not much of a difference in trees that are over 10 inches in diameter. You can continue to change the region of dissimilarity in the TOST function by modifying the low_eqbound and high_eqbound values. The smallest value in which the equivalence test result “flips” to indicate dissimilarity is termed the minimum detectable negligible difference. In this example, it happens if the equivalence bounds between CRM and NSVB predictions are set to approximately 210 pounds.

Instead of t-tests, give equivalence tests a try when evaluating models. In forestry we often have plenty of models available to us to compare predictions of volume and growth, either available in the literature or used in our own organization. Equivalence tests are an excellent approach the compare “old” models with a new and improved ones.

–

By Matt Russell. For more, subscribe to my monthly email newsletter to stay ahead on data and analytics trends in the forest products industry.