TUTORIAL

Tutorial information for student authors

Best practices for writing good multiple-choice questions

There are some simple guidelines for constructing robust multiple-choice questions.  What follows is a brief synopsis of the ‘do’s and don’ts’ that will assist you in authoring unambiguous questions that will serve as good learning and review tools by you and your peers.  You should be aware that there is a large body of educational literature aimed at pedagogy, the study of effective teaching and learning methods.  In 1956, Benjamin Bloom and colleagues published a framework to categorize educational objectives, and this became known as Bloom’s Taxonomy. The levels range from the simple recall of facts, to the ability to use information to create new knowledge.  Each level of this hierarchy builds upon the lower levels.  A summary of Bloom’s Taxonomy is included below the basic rules, and some suggested verbs to use for your questions will help you to categorize the questions and the level of knowledge being tested.  There are some hyperlinks and references at the end which you may wish to consult for more information and examples.

First, a note about Quizzical and its optional elements.  Your instructor may require you to provide either an image (generally from the images from your textbook that your instructor has uploaded to Quizzical) to associate with your question, a reference that might be useful to the test taker in exploring the question in more detail (e.g. a figure number or page from your textbook) and/or the instructor may require that you justify your answers and distractors.   All of these options ARE ONLY VISIBLE after the quiz taker has attempted the question.  They are displayed such that the quiz taker can see the image and associate it with the question to help them envision the subject/process being tested, and the justifications provide them with the logic of why an answer is correct and the distractors are incorrect.  Providing strong justifications is an important aspect of your authorship grade, and they enable your question to become a learning tool.  You’ve solidified your knowledge of the subject of your question, and now your hard work will help your peers to understand the subject. If you are required to provide justifications for your answer/distractors, keep in mind that your goal is to help someone understand why the correct answer is correct and EVERY distractor is incorrect.  Thus, doing something simple like ‘Answer D is incorrect because answer A is correct.’ is useless and will result in your grade being penalized.  Provide a rich (but not too wordy) description of why the answer is correct and each of the distractors is incorrect.

Lastly, you have undoubtedly attempted multiple choice questions that you felt were ‘tricky’ or ‘picky’.  This may have been because they were indeed poorly written.  If you adhere to the best practices outlined below, that will ensure that your question is a good one. UNLESS…..there can be an enormous number of discipline specific terms that you will need to master in most courses.  Knowing the terminology is key to understanding the language and complexity of the subject.  Thus, your question could be ambiguous if you don’t learn the ‘vocabulary words’ and employ them correctly.  Your goal should be to construct a rigorous question, and the resources you provide (the image, reference and justifications) should allow a student that does not know the answer to learn something.

The Question:

  • It is suggested that you begin with the idea of ‘what do I want to test?’. Maybe it is one of the learning objectives of the class? With this in mind, write the question and then the answer, followed by the distractors.

  • The stem (question) should be unambiguous and should take the form of a question.

  • Try to avoid negative phrasing (e.g. “Which of the following are not components of the ribosome?). If you must use negative phrasing, it is recommended that you underline, bold, and/or capitalize the ‘NOT’. If you must use NOT, do NOT have negatives in the answer/distractors (e.g. ‘none of the above’ constitutes a double negative that is confusing).

  • Do not refer to lectures, figures, tables or other materials that are not available to the test taker (e.g. “In figure 8.23a, which mechanism of enzyme action….). Note that the figure you specify for your question is only revealed after the quiz has been attempted. This will help the quiz taker to understand your question, the answers, and justifications (see below).

  • Avoid the use of terms such as ‘typically, often, usually’. These words imply that the question relates to a specific situation or exception and may be ‘tricky’.

  • You will be required to categorize your question as either a ‘recall’ question (explores information that you can memorize) or an ‘application’ question (focusses on the use of knowledge to interpret data, make predictions, differentiate between choices, etc.).  On the Quizzical question form filler, there are some suggestions for an appropriate verb to use when constructing each type of question.

Answers and Distractors

  • Avoid the use of ‘Type K’ questions. These are questions for which there is more than one correct answer. For example, some questions have three potential answers, followed by choices such as ‘both a and b’ and perhaps ‘both b and c’. These are effective for testing some topics and can be very discriminating, but keep their use to a minimum.

  • Following from tip 1 above, be sure that there is only one correct answer. Contemplate whether there are circumstances that would make one of the distractors a correct answer. If this is the case, you may wish to reconstruct the stem, or construct a different question.

  • Distractors are potential answers that do not satisfy the question. These must be legitimate. Good distractors are often true statements but they do not address the question.

  • Avoid weak distractors. An example: you might be asked to associate the name of a scientist with a discovery. Having Drake or Shakespeare as one of the choices immediately eliminates that option for most people and makes ‘guessing’ the correct answer easier.

  • Make the distractors of approximately equal length and avoid wordy (long) answers.

  • Use distractors that are similar to the correct answer in language, grammar, and form.

  • Avoid the use of ‘all of the above’ and ‘none of the above’.  Certainly do not employ both in the same question!

Finally, and importantly, I suggest that you compose your question, answers and justifications in a word processing program and then cut and paste the relevant information into the form filler boxes on Quizzical.  BEFORE you do that, run a spelling/grammar check to be sure that your text is flawless.

Bloom’s Taxonomy of Educational Objectives

Bloom categorized different abilities/outcomes of our cognitive processes to inform our thinking about course/learning objectives. These categories are summarized below, along with some information about learning objectives that are associated with them. To facilitate question writing, some appropriate question verbs are given. Note that for multiple choice questions, some of the more advanced categories are difficult/impossible to test, but it is possible to construct ‘recall’ (knowledge/comprehension) questions and more difficult ‘analytical’ (application/analysis) questions.

Knowledge – Recognizes students’ ability to use rote memorization and recall certain facts.

  • Test questions focus on identification and recall of information.
  • Learning objectives at this level: know common terms, know specific facts, know methods and procedures, know basic concepts, know principles.
  • Question verbs: Define, select, list, state, identify, label, name, who? when? where? what?

Comprehension – Involves students’ ability to read course content, extrapolate and interpret important information and put other’s ideas into their own words.

  • Test questions focus on use of facts, rules and principles.
  • Learning objectives at this level: understand facts and principles, interpret verbal material, interpret charts and graphs, translate verbal material to mathematical formulae, estimate the future consequences implied in data, justify methods and procedures.
  • Question verbs: Explain, distinguish between, classify, predict, interpret, infer, summarize, convert, translate, give example, account for, paraphrase x?

Application – Students take new concepts and apply them to another situation.

  • Test questions focus on applying facts or principles.
  • Learning objectives at this level: apply concepts and principles to new situations, apply laws and theories to practical situations, solve mathematical problems, construct graphs and charts, demonstrate the correct usage of a method or procedure.
  • Question verbs: Arrange the order, How could x be used to y? How would you show, make use of, modify, demonstrate, solve, or apply x to conditions y?

Analysis – Students have the ability to take new information and break it down into parts to differentiate between them.

  • Test questions focus on separation of a whole into component parts.
  • Learning objectives at this level: recognize unstated assumptions, recognizes logical fallacies in reasoning, distinguish between facts and inferences, evaluate the relevancy of data, analyze the organizational structure of a work (art, music, writing).
  • Question verbs: Diagram, determine, order, estimate, differentiate, compare / contrast, distinguish x from y, how does x affect or relate to y? why? how? What piece of x is missing / needed?

Synthesis – Students are able to take various pieces of information and form a whole creating a pattern where one did not previously exist.

  • Test questions focus on combining ideas to form a new whole.
  • Learning objectives at this level: write a well organized paper, give a well organized speech, write a creative short story (or poem or music), propose a plan for an experiment, integrate learning from different areas into a plan for solving a problem, formulate a new scheme for classifying objects (or events, or ideas)
  • Question verbs: Design, construct, develop, formulate, imagine, create, change, write a short story and label the following elements:

Evaluation – Involves students’ ability to look at someone else’s ideas or principles and see the worth of the work and the value of the conclusions.

  • Test questions focus on developing opinions, judgments or decisions.
  • Learning objectives at this level: judge the logical consistency of written material, judge the adequacy with which conclusions are supported by data, judge the value of a work (art, music, writing) by the use of internal criteria, judge the value of a work (art, music, writing) by use of external standards of excellence.
  • Question verbs: Justify, appraise, evaluate, judge x according to given criteria. Contrast, criticize, compare.

Some web resources and references

http://cft.vanderbilt.edu/teaching-guides/assessment/writing-good-multiple-choice-test-questions/#stem

 

https://facultyinnovate.utexas.edu/sites/default/files/writing-good-multiple-choice-exams-fic-120116.pdf

  • Burton, Steven J., Sudweeks, Richard R., Merrill, Paul F., and Wood, Bud. How to Prepare Better Multiple Choice Test Items: Guidelines for University Faculty, 1991.
  • Cheung, Derek and Bucat, Robert. How can we construct good multiple-choice items?  Presented at the Science and Technology Education Conference, Hong Kong, June 20-21, 2002.
  • Haladyna, Thomas M. Developing and validating multiple-choice test items, 2nd edition. Lawrence Erlbaum Associates, 1999.
  • Haladyna, Thomas M. and Downing, S. M.. Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 51-78, 1989.
  • Morrison, Susan and Free, Kathleen. Writing multiple-choice test items that promote and measure critical thinking. Journal of Nursing Education 40: 17-24, 2001.
  • Zimaro, D.M. Writing Good Multiple-Choice Question Exams.  Faculty Innovation Center, Univeristy of Texas. 2016.