Principles of Experimental Design and Statistical Inference

Most empirical studies fall into two groups: experimental studies and observational studies.

Experimental studies

In an experiment, a researcher manipulates a variable (e.g., imposes some treatment) to study its effect. An experiment usually involves:

  • randomization – to ensure groups being compared are similar apart from the treatment
  • control and compare – to study the difference with and without treatment
  • replication – to minimize the variation that results from taking samples
  • (in some cases) a matched pair/blocked design – to control for likely sources of variation in the data

Example experiment: A chemist wants to obtain the calibration curve of the ability of an instrument to detect the concentration of a chemical in a sample. They test the response of the instrument for a series of samples with known concentrations of the chemical in a random order.

Important definition: “randomization”: The random allocation of experimental units to experimental treatments. To allocate at random, each unit must have had the same chance of receiving any of the possible treatments.

To randomize 20 units between 2 experimental groups: (1) assign each experimental unit a unique number from 1 to 20; (2) use a random numbers table or computer software e.g., the function RAND () in MS Excel to also assign a random number to each unit (3) sort the data in order of the random number (4) the first 10 units in the ordered list are allocated to group 1 and the remaining 10 units to group 2.

Statistical inference and analysis

A well-designed experiment can often be analyzed using simple statistical methods (e.g. a t-test). If randomization and control has been sufficient then we do not need to worry about any additional variables affecting the results. Therefore, we can attribute any change in the response to be due to the experimental treatment. If we have sufficient replication, we should be able to detect an effect if it exists.

Observational studies

Because it is not always possible (for ethical or practical reasons) to conduct an experiment, observational studies are also common. In an observational study, a researcher does not manipulate any variable. Instead, they investigate the natural variation in a population.

There is a range of different types of observation studies, such as:

  • A longitudinal study
  • A cross-sectional study
  • A case-control study

Things to remember in an observational study include defining the population of interest and, when taking a sample, using random sampling to ensure the sample is representative of the population. Because in a natural population there will be multiple factors affecting the data, it can be useful to collect data on potentially confounding variables to help understand the patterns.

Important definition: “confounding variable”: a variable that varies with a proposed explanatory variable and therefore whose effect it cannot be separated from.

Example observational study: A researcher might be interested in the link between the number of hours spent watching television and the exam scores of students. They conduct a questionnaire survey to obtain data on exam scores and TV watching hours from a random sample of students at a university and investigate the association. Other variables (such as time spent studying and type of TV programs watched) need to be included in the analysis to disentangle the multiple factors that might affect exam scores.

Statistical inference and analysis

In an observational study, we are only investigating associations, which we cannot assume are causal relationships because of possible confounding variables. We need to investigate the effects of possible confounding variables by data analysis, for instance, multiple regression, to infer the factors likely to be causing the patterns in the data.

Interval versus external validity

Well-designed experiments are less biased than observational studies because they eliminate the effects of additional variables in the data and allow estimation of the true effect in the context being studied. Because of this, experiments can be considered to have high internal validity. By contrast, the external validity of a study is determined by whether its results can be generalized to other contexts of interest.

For instance, imagine we are interested in the effects of increasing air temperature due to climate change on plant growth in forests. We might:

(a) Perform a mesocosm chamber experiment in which we tightly control temperature and measure the growth of saplings.

(b) Perform an observational study and collect data on air temperature and forest growth from a sample of plants mapped to unique locations in a forest.

What might affect the internal and external validities of these studies?

If need you need scientific assessment of if you’ve established validity, or on any other aspect of your manuscript, check our wide range of research assistance services.

Scroll to Top