Empirical Research Methods - Introduction to Computer Science - Lecture Slides, Slides of Computer Science

Download Empirical Research Methods - Introduction to Computer Science - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Empirical Research Methods in

Human Computer Interaction

Part I – The Short Answer

Why?

• Two answers…

• First, we conduct empirical research to…

– answer (or raise!) questions about a new or

existing UI design or interaction method.

• Second… (we’ll get to this later)

How?

• We conduct empirical research through…

– a program of inquiry conforming to “the scientific

method”.

Three Themes

• Answer and raise questions

• Observe and measure

• User studies

Observe and Measure

• Observations are gathered…

– Manually (human observers)

– Automatically (computers, software, sensors, etc.)

• A measurement is a recorded observation

When you cannot measure, your knowledge is

of a meager and unsatisfactory kind.

Kelvin, 1883

Ratio Measurements

• Preferred scale of measurement
• With ratio measurements summaries and comparisons are

strengthened

• Report “counts” as ratios where possible
• Example – a 10-word phrase was entered in 30 seconds
  • Bad: t = 30 seconds
  • Good: Entry rate = 10 / 0.5 = 20 wpm
• Example – two errors were committed while entering a 10-

word (50 character) phrase

• Bad: n = 2 errors
• Good: Error rate was 2 / 50 = 0.04 = 4%

Research Questions

• Why do we conduct empirical research?
• Simply…
  • To answer (or raise!) questions about a new or existing UI design or interaction technique!
• Questions include…
  • Is it viable?
  • Is it as good as or better than current practice?
  • Which of several design alternatives is best?
  • What are its performance limits and capabilities?
  • What are its strengths and weaknesses?
  • Does it work well for novices, for experts?
  • How much practice is required to become proficient?

Research Questions (2)

• Weak question…

• Is the new technique better than multi-tap?

• Better…

• Is the new technique faster than multi-tap?

• Better still…

• Is the new technique faster than multi-tap within one hour of use?

• Even better…

• If error rates are kept under 2%, is the new technique faster than multi-tap within one hour of use?

A Tradeoff

Breadth of Question

Narrow Broad

Accuracy of Answer

High

Low

Is the new technique better than multi-tap?

If error rates are kept under 2%, is the new technique faster than multi-tap within one hour of use?

Internal validity

External validity

External Validity

• Definition: The extent to which results are

generalizable to other people and other

situations

• Statistically…

– Re people, the participants are representative of

the broader intended population of users

– Re situations, Test environment and experimental

procedures are representative of real world

situations where the UI/technique will be used

Test Environment Example

• Scenario…
  • You wish to compare two input devices for remote pointing (e.g., at a projection screen)
• External validity is improved if the test environment mimics

expected usage

• Test environment should probably…
  • Use a projection screen (not a CRT)
  • Position participants at a significant distance from screen (rather than close up)
  • Have participants stand (rather than sit)
  • Include an audience!
• But… is internal validity compromised?

The Tradeoff

• There is tension between internal and external validity
• The more the test environment and experimental procedures

are “relaxed” (to mimic real-world situations), the more the

experiment is susceptible to uncontrolled sources of variation,

such as pondering, distractions, or secondary tasks

Internal

validity

External

validity

Strive for the Best of Both Worlds

• Internal and external validity are increased by…
  • Posing multiple narrow ( testable ) questions that cover the range of outcomes influencing the broader ( untestable ) questions - E.g., a technique that is faster , is more accurate , takes fewer steps , is easy to learn , and is easy to remember , is generally better
• The good news
  • There is usually a positive correlation between the testable and untestable questions - I.e., participants generally find a UI better if it is faster , more accurate , takes fewer steps , etc.