HCI: Evaluating designs

There are several methods to evaluate a design:

Usability studies

  • This could be informal, watch and learn or formal usability labs.
  • They offer good learning and uncover quirks, bugs or false assumtions.
  • it is not the same as user’s using it to perform real work in real environment
  • difficult to compare alternatives
  • experimental bias


  • quickly gets feedback from a large number of users
  • relatively easy to compare alternatives
  • No need to build prototype, a screenshot or mockup will do.
  • There is a difference between what people say they’ll do and what they actually do.

Focus groups

  • gather a small group of people to discuss a design or idea.
  • could be difficult due to group dynamics or if the subject makes people uncomfortable

Feedback from Experts

  • peer review
  • dogfooding
  • heuristic evaluation

Comparative experiments

  • taking two or more distinct options and comparing their performance to each other.
  • observe actual behaviour as opposed to self report (in surveys)
  • it can be better than usability studies since it compares multiple alternatives.
  • but you can’t observe people like in usability study.

Participant observation

  • observe people in their actual work environment


  • Useful when alternatives can be mathematically evaluated against design goals
  • Allows lots of alternatives to be compared


The method of evaluation to be used for the specific design goal depends on these (often conflicting) parameters:

  • Reliability: could be reproduce the results
  • Generalizability: applicability to larger set of people
  • ¬ěRealism: do the observations hold good in real world situations
  • Comparison: can we easily compare different (new of existing) designs
  • Work involved: the effort to get the feedback