I have given talks and I have listened to talks where the following statement is given: “Despite apparent differences, there are no statistical or significant differences among any of the treatments.” However, for people looking at the data chart, table or graph, it appears many of the treatments are indeed different. The presenter disagrees and says there is no significant difference. Many in the audience question whether the presenter has actually examined the data, as the differences among the treatments appear obvious.
So what is all this “significant difference” jargon? If the numbers or results are different, then aren’t the treatments different? Well, it all depends, and the focus is primarily based on the confidence we have in the data. How do we have confidence in data? Basically, if the experiment is well designed, we gain confidence in the data when there is consistency across replicates. In other words, when we see similar numbers in the replicates of the same treatment, we believe the study was well conceived, fully executed according to plan, and that the data reflect the results of the main effect (the treatment) with few outside influences, biases or factors external to the treatments that impact the outcome.
Statistical methods often assume that variability across groups or replicates are equal. In statistical terms, this is called homoscedasticity or homogeneity of variance. Some of the more commonly used statistical tests, such as analysis of variance, assume that the variance across groups or replicates is equal. There is a test that we often use called Bartlett’s test, and there are various methods to “normalize” data, which allow us to be more confident in the results.
Let me be perfectly clear: When I say we test the data or normalize the data, I am NOT referring to trying to massage or manipulate the data so that it gives us the results we want. Rather, it is to ensure that when we subject the data to statistical analysis, the analysis will provide us with a clear picture of what the trial results really mean.
As an example of variability across replicates, we will use a golf course fairway. Across the fairway, there may be variability in soil types, soil moisture, turf health and other factors, including management practices. We will use this fairway to test products for performance. Since I am an entomologist, we will use a trial to evaluate insecticides for mole cricket control.
We find a fairway that has an existing population of mole crickets, but the infestation is not consistent across the area where we will lay out our plots. The variability in the population across the fairway means we already have a factor that will influence variability across replicates. There are variations across the fairway in soil types and soil moisture, and since mole crickets are soil insects, we have already added more factors that will influence mole cricket behavior and insecticide effectiveness. I could keep adding to the list of factors across the fairway that can add variability to a large-scale field trial. The same would be true if we were doing a trial on weeds, disease or nematodes.
As a result of these real world factors, we see some variability in our data that is not due exclusively to the performance of the product, but possibly due to factors in the field. Even when comparing the best products to the least effective, the numbers we obtain aren’t always as clear as night and day in our field trials.
As scientists, we want to be confident in our data, so when we make recommendations, we feel certain about the statements we make. Conducting statistical analyses of our data helps us gain confidence.
A typical stat evaluation will use a 95% confidence interval. In other words, if we see a difference in treatment effects and the statistical analysis confirms that the difference is indeed due to the treatment (and not due to in-field variability), we can say that with a 95% level of confidence. However, since variation across plots is a real thing in the field, the statistical analysis sometimes tells us we cannot say one treatment is different from another based solely on the treatment effect. When that happens, we have to make the statement “despite apparent differences, there are no statistical or significant differences among any of the treatments.”
Such is the world of field trials and sometimes even laboratory research.
Rick Brandenburg, Ph.D., is a turfgrass entomology professor at North Carolina State University in Raleigh, a post he’s held since 1985. The 29-year GCSAA member is also a frequent presenter for GCSAA, both in webinars and at the annual GCSAA Conference and Trade Show.