1. SME does not equal a good assessment writer 

SMEs are often contracted to develop assessments, especially to write questions. Just because someone is a subject matter expert in a given area does not mean that they can develop good assessments. The same is true of instructors, instructional designers, and anyone else who is in the education business. 

Assessment design requires a very specific knowledge base and skill set that is not always included in the professional development of professional educators, and even when included, it is not always consistent. 

A Ph.D. in assessment isn’t required most of the time, but some training and mentoring can go a long way in preparing SMEs or anyone else to develop good assessments. 

2. Constructive alignment through Bloom’s 

You really can’t do assessments properly without understanding constructive alignment, as illustrated below.

The basic idea here is that learning objectives, learning activities, and assessments all have to support each other. The learners should not be doing things for a course that they are not expected to do and will not be assessed on. 

It generally makes the most sense to begin by nailing down the learning objectives as these should drive the instructional content and assessment. Make sure that each assessment item matches at least one learning objective and that all learning objectives have assessment items. Use Bloom’s revised taxonomy to construct learning objectives to ensure all of the instructional content is covered and assessed both in scope and level of difficulty.

3. Tagging, mapping, and blueprinting

Constructive alignment can be further optimized by tagging and mapping all learning assets and then creating a blueprint that maps out how these learning assets will (or can, if there’s flexibility) be utilized. For example, each level of Bloom’s revised taxonomy can be a tag. Let’s say the learning objective is “Describe the different types of clouds.” This is at the Bloom’s level of understanding, and so one of the tags for this learning objective could be “Bloom’s = understanding” or something similar. The associated assessment items can then be tagged to this learning objective and its Bloom’s level along with key words such as “clouds, weather, atmosphere” etc. 

You can then create a blueprint that shows not only a course outline but also which objectives are covered where and at what Bloom’s level, to which you could work backward from an assessment item to view corresponding instructional content. The blueprint can serve as an inventory of learning assets, including assessments, that will help ensure that everything is covered and in the right way.

4. Formative and summative

From a learning theory perspective, it is important to develop both formative and summative assessments. Learners can only process so much new knowledge before taking a quick pause to ensure that they retained what was just covered through a formative assessment. At the end of a larger section, the learners should be able to synthesize across the new knowledge just covered, in a summative assessment. 

So, the summative assessments are more comprehensive than the formative assessments. A common poor practice is to merely put all the formative assessments together as the summative assessment. This, however, is not effective.

5. Variety

Even though multiple-choice questions are the most feasible means of assessing online learning, by no means does this have to preclude variety, which is essential in keeping learners engaged. Also, answering questions often is not representative of the learning that has taken place. Technology is more and more supportive of other forms of assessments, and this should not be overlooked. But even a pool of questions can be varied. 

Limited use of true/false questions is fine. Multiple-choice questions can have just a single correct answer or multiple correct answers. Any of these types of questions can include media, such as a short video or audio file, or even an image. Matching, drag-and-drop, labeling, and hotspot questions enable more variety and promote learner engagement. Of course, you must be sure that whatever software you are using supports these features, especially your LMS.

6. Structure of the assessment

This refers to the individual parts and overall structure of an assessment. For questions, the stem, distractors, and rationales (feedback) must follow guidelines and best practices in order to properly assess learning. A major problem in assessment is the extent to which flawed assessments skew learner performance data and detrimentally affect learner outcomes, especially grades. 

Assessment items that are confusing, misleading, or just do not make sense, set up learners to perform poorly when they might actually have a strong command of the knowledge. It is essential that assessment developers understand how to create good assessments and avoid creating flawed ones.

7. Link to remediation

One of the advantages to online learning that includes assets that are tagged and mapped is that this makes better remediation possible. For example, every individual question can be designed to direct the learner back to associated online course content, specific e-textbook content, or any other online assets. This makes it easier for learners to review content for missed questions than if they had to look up references (such as “textbook p. 271”) or, worse, were left to their own devices to look for the content, which they most likely will not do.

8. Learner performance data

Online assessments, through tagging and mapping, can yield a lot of valuable learner performance data. Beyond a basic dashboard that shows each individual learner’s total score, it’s possible to see which specific questions each student missed, see how many students missed each question, how many students selected each answer choice, whether or not students played associated media files, how many attempts a learner took before getting a question right (if allowed on formative assessments), which areas of the content were missed the most, how cohorts compare with each other, how individual learners perform over time, how overall learner performance trends over time, and more. 

Among other things this information can be used to remediate select learners or an entire cohort as well as identify flawed assessment items (or entire assessments) and make more targeted and impactful revisions to assessments, course activities, and course assets. This brings us to the next and final point.

9. Iterative process

Any assessment strategy should run on an iterative process that ensures not only that the assessments are still relevant according to the learning objectives and course activities but also are continuously improved based on learner performance data and feedback as well as identified gaps.