Are You a Question Writing Criminal?
Henry Phillips, Unicorn Technical Director and winner of 2012 eLearning industry award for outstanding achievement – individual, looks at the top 10 crimes of question writing. Well, almost…
Although we live in a brave new world of super-fast broadband, 4G, iPads, retina displays, and learning ecosystems, the humble multiple-choice question still has an important role to play in the world of assessment.
There is considerable skill involved in writing good MCQs and lots of ways of writing bad ones too. No matter how well the graphic designers wrap them up in creative and elegant clothing, those old familiar flaws still emerge blinking into the daylight with remarkable frequency.
At Unicorn, we’ve been writing assessment questions for over 20 years and have learnt a lot of simple but important lessons along the way.
Using the recently run Great Duck Race as a template, here’s a list of common question-writing ‘crimes’.
1. Obvious questions: You may not know much about duck racing, but you can probably pick the right answer to this question:
You work in the anti-doping team of the Duck Racing Federation. The latest test results have just come in and a very famous and popular racing duck is showing a positive test. What should you do?
A. Destroy the results as they are clearly incorrect
B. Pass the results on to a junior member of the team to deal with in case they later prove to be incorrect
C. Speak to the duck concerned to tip him off.
D. Follow the DRF standard procedures. Report the positive results to the anti-doping board, complete from DRB6b in triplicate, keep the pink copy for your file and return the other two by recorded delivery to DRF HQ.
2. Labelling questions: I still see too many of them and they usually indicate that the writer doesn’t really understand the subject or is being paid by the question! Take this example. Is this really testing a key learning point that’s going to help me do my job better or is it just lazy question writing?
Which clause in the Duck Racing Code of Conduct is concerned with the allowed length of tail feathers?
A. Clause 3
B. Clause 4
C. Clause 7
D. Clause 9
Other examples include: ‘In what year…?’ ‘What do the initials FSA stand for …?’’Which Act introduced…?’ ‘How many Data Protection principles are there …?’ and so on
3. Arithmetical questions where the distracters are just ‘placeholders’ and not the result of logical misunderstanding / miscalculation.
If six ducks set out through the rapids and three are eaten by crocodiles, how many are left?
This ‘crime’ is often instigated during the updating process when, say, a tax rate changes. The writer updates the correct answer but ignores the distracters, which then become arithmetically improbable.
On a related point, remember wherever possible to keep the arithmetic simple (unless you are testing mathematical skills). Focus on the principle not the maths.
4. Negative stems. Often compounded with negative answer options, these questions are unnecessarily confusing for the student.
Which of the following is NOT a drawback of having extra large flippers?
A. They are not effective on courses with rapids
B. They get tangled in the weeds
C. They don’t attract crocodiles
D. Not getting caught by overhanging branches
5. Use of “All of (None of) the above” as an answer option. Normally the correct answer, it becomes even more inappropriate when the order of the answers is randomised!
Who is responsible for reporting breaches of the duck federation rules?
B. The referee
C. The federation inspectors
D. All of the above
6. Lack of (or inappropriate) alignment between the stem and answer options. If only one answer (or in extreme cases no answer) aligns with the stem, the question is flawed.
Fast ducks will usually defeat strong ducks because they …
A. have bigger flippers
B. strong ducks slow down in rapids
C. the size of the wings
D. fast ducks can outswim crocodiles
On a related point, a well written assessment question normally has a ‘focused’ stem – it should be possible to formulate a response before looking at the answer options.
7. Answer options aren’t discrete. In the example below if option D is correct then options B and C which are components of this answer, are also valid responses.
To enter the Great Duck Race 2013, what, if anything, do you need to do?
A. Nothing, all award attendees will be entered in the race automatically
B. Book a table at the 2013 Awards dinner
C. Design your duck on the Unicorn web site
D. Book a table at the Awards dinner and design your duck on the Unicorn web site
8. Subjective question stem: ‘What would you do?’ (‘What do you think?’ How will you respond?) This may be suitable in exercises within an e-learning module but is not appropriate for objective testing where you are seeking a correct response rather than an opinion.
While competing in the Great Duck Race, you spot a duck in difficulties in the rapids. What will you do?
A. Swim on, as you are there to win
B. Laugh and point him out to other competitors
C. Stop and try to help
D. Look for the nearest lifebuoy.
Depending on the character of the respondent, any of the answers is possible and all of them might be valid. “What should you do?” is marginally better but still too subjective.
9. Answer options that contain ‘never’ or ‘always’ – giving a clue that these are distracters, especially when included with a ‘usually’ option. Which is true of the Drake’s Gulch racecourse?
A. Strong ducks always win
B. Fast ducks usually win
C. Intelligent ducks never win
D. Stable ducks always get eaten by crocodiles
10. If you can’t think of a tenth question, what should you do?
A. don’t write one
B. desperately try to write just one more even though you’ve already run out of learning points
C. rename your article as the ‘Top Nine’ question writing crimes.