Experimental Design¶

Introduction to Experiments¶

If 10 swimmers from a club are given a new type of swim cap to see if it improves the time it takes to swim one lap

An experiment unit is "a swimmer from the club"

The treatment is "being given a new type of swim cap"

An observational study is a way of finding out information from a sample by simply recording you what you observe naturally

An observational study is either

Retrospective, where data is collected then analyzed
Prospective, where a sample is observed for a period of time into the future

Experiment vs Observational study:

In an observational study, you do not influence or change the individuals
In experiments you impose a treatment on them

Well-Designed Experiments¶

Confounding Variables¶

A confounding variable is a variable that

You are not interested in

But can affect the results of your experiment

As the explanatory variable changes, the confounding variable also changes
In an experiment, it is important to identify and control confounding variables
In an observational study, you cannot control confounding variables

A Well Designed Experiment¶

At least two treatment groups
- A control group counts as a treatment group
Treatment groups are formed by randomly assigning treatments to the experimental units
- Randomization
Treatment groups have more than one experimental unit each
- Replication
Confounding variables are identified and controlled
- Stay the same across all treatment groups

Statistically Significant Experiment Results¶

The results of an experiment are called statistically significant if the changes in the response variable (or the differences between treatment groups) are so large that they are unlikely to be down to chance

It suggests that there is a relationship between the treatment and the response

When can the word "cause" be used:

Treatments are randomly assigned to experimental units
Results are statistically significant

When can the word "generalize" be used:

The sample is randomly selected

Do not confuse the process of randomly selecting experimental units from a population to use in your experiment, with the subsequent process of randomly assigning your selected experimental units the different treatments!

A control group is a treatment group that is purposefully given either

Inactive form of the treatment

Pre-existing/Baseline treatment

A placebo is a dummy (inactive) treatment that the experimental units believe is an active treatment

A placebo effect is the type of response that experimental units show to a placebo

e.g. Telling individuals they are taking a pain-relief drink when in fact it is water

Control groups can use placebos as inactive forms of treatment, though not necessary

A single-blind experiment is one in which the participants do not know which treatment they are receiving

Helps reduct the placebo effect

A double-blind experiment is one in which participants and the researchers (in particular those interacting with participants and measuring their responses) do not know which groups are assigned which treatments

This helps to reduce the placebo effect

and also helps to reduce any bias introduced by the researchers themselves

There will be an experimental designer (who is not a participant, nor a researcher measuring the responses of the participants) who will know which participants received which treatments

Completely Randomized Design¶

To randomly assign two different treatments to, say, 100 experimental units
- Label the experimental units from 1 to 100
- Use a random number generator to generate random numbers between 1 and 100 until 50 unique numbers are obtained
  - Note how this is half of the sample size
  - Unique means no duplicate numbers (repeats)
- Assign the experimental units who have those 50 unique numbers the first treatment and the remaining 50 receive the second treatment
To randomly assign four different treatments to, say, 100 experimental units
- Label the experimental units from 1 to 100
- Use a random number generator to generate random numbers between 1 and 100 until 75 unique numbers are obtained
  - Note how this is three-quarters of the sample size
- Assign the experimental units who have
  - the first 25 unique numbers the first treatment
  - the second 25 unique numbers the second treatment
  - the third 25 unique numbers the third treatment
  - then the remaining 25 the fourth treatment

Randomized Block Design¶

A block is group of experimental units who have something in common (are similar) that may affect how they respond to a treatment - e.g. a group of participants who are smokers

Blocking is the act of dividing up the experimental units into different blocks

e.g. separating participants out into smokers and non-smokers, smoking or not" is the blocking variable

Blocking should only be done if the researcher believes the blocking variable could affect the results

An experiment that has a randomized block design is one in which experimental units are separated out into blocks based on an identified blocking variable that could cause an issue

then experimental units are randomly assigned the different treatments within each block