Tutorial information for student authors
Best practices for writing good multiple-choice questions
There are some simple guidelines for constructing robust multiple-choice questions. What follows is a brief synopsis of the ‘do’s and don’ts’ that will assist you in authoring unambiguous questions that will serve as good learning and review tools by you and your peers. You should be aware that there is a large body of educational literature aimed at pedagogy, the study of effective teaching and learning methods. In 1956, Benjamin Bloom and colleagues published a framework to categorize educational objectives, and this became known as Bloom’s Taxonomy. The levels range from the simple recall of facts, to the ability to use information to create new knowledge. Each level of this hierarchy builds upon the lower levels. A summary of Bloom’s Taxonomy is included below the basic rules, and some suggested verbs to use for your questions will help you to categorize the questions and the level of knowledge being tested. There are some hyperlinks and references at the end which you may wish to consult for more information and examples.
First, a note about Quizzical and its optional elements. Your instructor may require you to provide either an image (generally from the images from your textbook that your instructor has uploaded to Quizzical) to associate with your question, a reference that might be useful to the test taker in exploring the question in more detail (e.g. a figure number or page from your textbook) and/or the instructor may require that you justify your answers and distractors. All of these options ARE ONLY VISIBLE after the quiz taker has attempted the question. They are displayed such that the quiz taker can see the image and associate it with the question to help them envision the subject/process being tested, and the justifications provide them with the logic of why an answer is correct and the distractors are incorrect. Providing strong justifications is an important aspect of your authorship grade, and they enable your question to become a learning tool. You’ve solidified your knowledge of the subject of your question, and now your hard work will help your peers to understand the subject. If you are required to provide justifications for your answer/distractors, keep in mind that your goal is to help someone understand why the correct answer is correct and EVERY distractor is incorrect. Thus, doing something simple like ‘Answer D is incorrect because answer A is correct.’ is useless and will result in your grade being penalized. Provide a rich (but not too wordy) description of why the answer is correct and each of the distractors is incorrect.
Lastly, you have undoubtedly attempted multiple choice questions that you felt were ‘tricky’ or ‘picky’. This may have been because they were indeed poorly written. If you adhere to the best practices outlined below, that will ensure that your question is a good one. UNLESS…..there can be an enormous number of discipline specific terms that you will need to master in most courses. Knowing the terminology is key to understanding the language and complexity of the subject. Thus, your question could be ambiguous if you don’t learn the ‘vocabulary words’ and employ them correctly. Your goal should be to construct a rigorous question, and the resources you provide (the image, reference and justifications) should allow a student that does not know the answer to learn something.
Answers and Distractors
Finally, and importantly, I suggest that you compose your question, answers and justifications in a word processing program and then cut and paste the relevant information into the form filler boxes on Quizzical. BEFORE you do that, run a spelling/grammar check to be sure that your text is flawless.
Bloom’s Taxonomy of Educational Objectives
Bloom categorized different abilities/outcomes of our cognitive processes to inform our thinking about course/learning objectives. These categories are summarized below, along with some information about learning objectives that are associated with them. To facilitate question writing, some appropriate question verbs are given. Note that for multiple choice questions, some of the more advanced categories are difficult/impossible to test, but it is possible to construct ‘recall’ (knowledge/comprehension) questions and more difficult ‘analytical’ (application/analysis) questions.
Knowledge – Recognizes students’ ability to use rote memorization and recall certain facts.
- Test questions focus on identification and recall of information.
- Learning objectives at this level: know common terms, know specific facts, know methods and procedures, know basic concepts, know principles.
- Question verbs: Define, select, list, state, identify, label, name, who? when? where? what?
Comprehension – Involves students’ ability to read course content, extrapolate and interpret important information and put other’s ideas into their own words.
- Test questions focus on use of facts, rules and principles.
- Learning objectives at this level: understand facts and principles, interpret verbal material, interpret charts and graphs, translate verbal material to mathematical formulae, estimate the future consequences implied in data, justify methods and procedures.
- Question verbs: Explain, distinguish between, classify, predict, interpret, infer, summarize, convert, translate, give example, account for, paraphrase x?
Application – Students take new concepts and apply them to another situation.
- Test questions focus on applying facts or principles.
- Learning objectives at this level: apply concepts and principles to new situations, apply laws and theories to practical situations, solve mathematical problems, construct graphs and charts, demonstrate the correct usage of a method or procedure.
- Question verbs: Arrange the order, How could x be used to y? How would you show, make use of, modify, demonstrate, solve, or apply x to conditions y?
Analysis – Students have the ability to take new information and break it down into parts to differentiate between them.
- Test questions focus on separation of a whole into component parts.
- Learning objectives at this level: recognize unstated assumptions, recognizes logical fallacies in reasoning, distinguish between facts and inferences, evaluate the relevancy of data, analyze the organizational structure of a work (art, music, writing).
- Question verbs: Diagram, determine, order, estimate, differentiate, compare / contrast, distinguish x from y, how does x affect or relate to y? why? how? What piece of x is missing / needed?
Synthesis – Students are able to take various pieces of information and form a whole creating a pattern where one did not previously exist.
- Test questions focus on combining ideas to form a new whole.
- Learning objectives at this level: write a well organized paper, give a well organized speech, write a creative short story (or poem or music), propose a plan for an experiment, integrate learning from different areas into a plan for solving a problem, formulate a new scheme for classifying objects (or events, or ideas)
- Question verbs: Design, construct, develop, formulate, imagine, create, change, write a short story and label the following elements:
Evaluation – Involves students’ ability to look at someone else’s ideas or principles and see the worth of the work and the value of the conclusions.
- Test questions focus on developing opinions, judgments or decisions.
- Learning objectives at this level: judge the logical consistency of written material, judge the adequacy with which conclusions are supported by data, judge the value of a work (art, music, writing) by the use of internal criteria, judge the value of a work (art, music, writing) by use of external standards of excellence.
- Question verbs: Justify, appraise, evaluate, judge x according to given criteria. Contrast, criticize, compare.
Some web resources and references
- Burton, Steven J., Sudweeks, Richard R., Merrill, Paul F., and Wood, Bud. How to Prepare Better Multiple Choice Test Items: Guidelines for University Faculty, 1991.
- Cheung, Derek and Bucat, Robert. How can we construct good multiple-choice items? Presented at the Science and Technology Education Conference, Hong Kong, June 20-21, 2002.
- Haladyna, Thomas M. Developing and validating multiple-choice test items, 2nd edition. Lawrence Erlbaum Associates, 1999.
- Haladyna, Thomas M. and Downing, S. M.. Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 51-78, 1989.
- Morrison, Susan and Free, Kathleen. Writing multiple-choice test items that promote and measure critical thinking. Journal of Nursing Education 40: 17-24, 2001.
- Zimaro, D.M. Writing Good Multiple-Choice Question Exams. Faculty Innovation Center, Univeristy of Texas. 2016.