Bias & Ethics in User Research
Bias in surveys
If you're conducting user research, you should be seeking the truthful answers – not just what "sounds best" or aligns with your goals. Unfortunately, it's very easy to guide users to "telling us what we want to hear" by asking the wrong questions.
Leading Questions
To avoid leading questions, think like a lawyer. From The Legal Seagull:
Example #1: “You were at Seagull’s Pub the night of October 31st, right?”
Example #2: "Didn't you only drink water all night while the plaintiff had four pints of beer?"
Example #3: "You told the bartender that you would follow Jim home, didn't you?"
Do you see the answers embedded in these questions? According to the examiner, the witness:
- Was at Seagull’s Pub on October 31st;
- Only drank water (but the plaintiff had four beers); and
- Told the bartender that he'd follow Jim home.
Isn't that so? Objection. Leading the witness. In all three examples, the opposing attorney should (and probably would) state an objection.
You might be wondering: what is wrong with leading a witness? Leading questions can be problematic because they allow the examiner to unduly influence or control the witness’ testimony. Non-leading questions provide a more “natural” flow of testimony based on the witness’ personal knowledge and recollection of the events.
One of the easiest ways to overcome the objection is to rephrase the question to get the testimony you need without putting words in the witness's mouth.
Example #1
Original: “You were at Seagull’s Pub the night of October 31st, right?”
Rephrased: "Where were you on the night of October 31st?"
Example #2
Original: "Didn't you only drink water all night while the plaintiff had four pints of beer?"
Rephrased: "What did you drink during your time at the Pub?"
Follow up: "What did the plaintiff drink?"
Example #3
Examiner: "You told the bartender that you would follow Jim home, didn't you?"
Opposing attorney: "Did you say anything to the bartender before you left that night?"
Follow up: "What did you say to the bartender?"
Can you see the difference?
The leading question assumes an answer that the examiner is hoping to confirm. Whereas, the non-leading form allows the witness to offer a range of responses from their own recollection of the events — and is therefore non-leading.
In surveys, leading questions can create an enormous bias in the results. Not only would any "findings" be unethical to claim, but the research is not useful to you as a UX professional. It is not revealing a truth; only confirming the answer you were hoping for.
Institutional Review Boards
Institutional Review Boards (IRBs) are essentially formal ethics committees. They can help to prevent bias and unethical testing – in some regulated fields, their approval is required.
The purpose of IRB review is to assure, both in advance and by periodic review, that appropriate steps are taken to protect the rights and welfare of humans participating as subjects in the research. To accomplish this purpose, IRBs use a group process to review research protocols and related materials (e.g., informed consent documents and investigator brochures) to ensure protection of the rights and welfare of human subjects of research.
— U.S. Food and Drug Administration, "FDA Guidance for Institutional Review Boards and Clinical Investigators"
For more information, watch the video below: