Professor George gave another seminar this afternoon, looking at how to carry out experimental research. Joey is self-taught in experimental research methodology (as opposed to being taught through a class).

Agenda:
– Definition
– Pros and Cons
– Checklist
– Execution
– Data Analysis and Write-Up

Definition
– Lab experiments … take place in an especially created setting to investigate a phenomenon. The researcher has control over the independent variable(s) and the assignment of random participants to various treatment and non-treatment conditions. From AIS.

Pros and Cons
Pros:
– a narrow focus.
– control – high levels of internal validity
– established data analysis methods, eg, ANOVA (traditional)
– … can be an issue with reviewers though, if you haven’t done the analysis correctly.
– established format for writing up results. There is pretty much a checklist to go through.
– student subjects are readily available

Cons:
– lack of realism
– limited generalization or external validity. Generally is a group of researchers with students that are easily found.
– results should be viewed as a single data point. Need to run multiple experiments … replications, others with changes
– … or you could mix it with other methods, eg, start with an experiment and then explore with case study or an ethnography
– students subjects are not always the best choice, given the research question. Need to justify the use of student subjects.
– it is not always fast and easy
– … it takes time to get it done right.

Checklist
1. Research Question
2. Research Model and Hypotheses
3. Dependent Variables (and measurement instruments)
4. Independent Variables (manipulations)
5. Research Design (controls)
6. Subjects
7. Tasks
8. Procedures
9. Institutional Approval

Example
Joey went through a recent paper, The Effects of Warnings, Computer-Based Media, and Probing Activity on Successful Lie Detection.

Some pointers:
– experiments tend to be factorial designs, eg, 2×4
– balance is important. Eg, for training vs. no training, you need to have something for those not getting trained something to do while the others are being trained.
– you need to anticipate all possible threats to internal validity. See the chapter on Internal Validity in Cook and Campbell for more.
– some designs include explicit controls where there is no manipulations
– on using students: hard to get them to the experimental location; and how you do keep them engaged in the task at hand?
– there is a repository of tasks for experiments, but sometimes you have to create them.
– piloting the experiment is essential.