Programmatic assessment to measure learning outcomes

The role of assessment in higher education

There has been a major shift in the role of assessment in higher education, from assessment of learning to assessment for learning. That is, assessment should provide rich information that guides and fosters the learning process, instead of labeling students as competent or incompetent. To generate and process information, assessment for learning relies on formative assessment approaches such as feedback. 

However, Lorrie Shepard in her article, The role of assessment in a learning culture, proposes that assessment for learning does not suffice to measure student achievement: 

“Only after successful completion of a module can students progress to the next.”

This means that summative assessment is still essential to the evaluation process. 

Programmatic assessment then emerges as an ideal way to harmonize both traditional and modern assessment approaches. The ultimate goal of programmatic assessment is to provide the whole picture of a student’s “competency” via a combination of assessment methods.

This novel approach to assessment seems to be promising, and appealing. But can it be applied widely at institutions, or is it just another hype in education?

The ideal assessment approach for Competency-Based Education

“The idea of programmatic assessment emerges from medical studies, because it mimics so much the process when the doctor diagnoses the illness.”

Dr. Schuwirth summarized the idea behind programmatic assessment with an analogy of medical education. 

To generate accurate diagnosis, the doctor has to rely on different channels like blood tests, physical check-ups, conversations with patients, and so on. Similarly, triangulation is the key to assessing students’ competencies. By triangulating assessment formats across a longitudinal basis, we can generate rich information to make “high-stakes decisions” regarding the student’s competency.

As we keep talking about competency, what is it exactly? Competency is defined as the ability to complete a task by integrating knowledge, skills, attitudes, and professionalism. How competency is divided and measured varies according to institutions. Competency-based education (CBE) is then developed to help students master the core competencies required for their chosen fields of study, while allowing faculties to evaluate how well their learners accumulate the desired skills. 

According to Katie Larsen McClarty and Matthew N. Gaertner, author of the article ‘Measuring Mastery: Best Practices for Assessment in Competency-Based Education”: 

“A CBE model is workable only insofar as its measures of learning yield trustworthy data about students’ prospects for future success.”

A major challenge facing competency-based education (CBE) is to ensure the effective use of robust assessment methodologies that result in useful performance-relevant information (PRI) by which to provide feedback on learning and make decisions about learner progression.

Programmatic assessment; therefore, is the perfect evaluation methodology for the CBE model, due to its reliance on abundant learning data and methods to collect these data. 

Programmatic assessment vs Competency-based assessment

The Competency-Based Medical Education (CBME) Core Components Framework highlights programmatic assessment as one of the 5 essential components for competency-based teaching. In essence, programmatic assessment must ensure the following elements: 

• There are multiple points and methods for data collection.
• Methods for data collection match the quality of the competency being assessed.
• Emphasis is on workplace observation.
• Emphasis is on providing personalized, timely, meaningful feedback.
• Progression is based on documentation of achievement. There is a robust system for decision-making.

The full description of the Core Components Framework can be found here. 

Traditional versus Programmatic assessment 

To further understand the idea behind programmatic assessment, Dan raised the question: “What makes programmatic assessment different from traditional approaches?”

The main difference, according to Dr. Schuwirth, lies in how each approach generates and makes use of the information. While the traditional approach is inherently reductionist, producing rich information from tests then reducing them to a score; programmatic assessment aims to retain the complexity of information. 

‍“Learning is a complex interactive process, you can’t apply a reductionist approach. In Dutch, we usually say – If you disassemble a bike, then reassemble it, you will have a bike. But if you dissect a frog, and glue it back together, it is not a frog anymore. So it is the interaction that counts.” – Dr. Schuwirth commented.

Learning is indeed interactive and complex, and it requires a combination of different methods to accurately measure and gauge how far students have progressed over the course of learning. 

How much is programmatic assessment an assessment for learning?

Student learning and decision-making functions are the focal points of programmatic assessment. 

Instead of assigning grades as incentives to motivate students, meaningful and continuous feedback is provided to help students reflect, improve and become better learners. As stated by Dr. Schuwirth: 

“What we want to instill on our students is that they continue to learn, and have the agency and ability to assess, collect information, self-assess, reflect, and to set goals for future learning after graduation.”

Rich information is then needed to activate lifelong learning. Instead of connecting pass/fail decisions from one single assessment moment, multiple assessment approaches should take place over a longitudinal basis. Eventually, the information generated from this assessment combination becomes sufficient to reach summative decisions. 

The main focus is the student’s learning. Instead of ‘jumping through summative assessment hoops’, assessments should promote learning. This is done by disconnecting the pass/fail decision from one single assessment moment. Instead, the student is provided with meaningful and rich feedback to help them develop themselves. Ultimately, multiple moments of assessment over a longer period of time lead to a pass/fail decision. This decision is made to decide if the student has met the desired learning outcomes. As such, there is no decision moment based on a single assessment; the decision is made based on rich information on the student’s learning trajectory.

The role of technological affordances

The conversation then switched to the question on the role of modern technology in adopting programmatic assessment with Dan asking, “Our students today have all sorts of affordances. Do the Internet, and digital and social infrastructures necessitate programmatic assessment more than traditional assessment?” 

“I would say absolutely! Our students have affordances that are unprecedented.”

And what are those affordances? Students can now be part of multiple communities, have access to different channels and modes to communicate and create knowledge. Most importantly, modern technology exposes students to endless sources of information, and knowledge. This then led to a shift in the imbalanced relationship between teachers and students, as now students also become the knowledge creators. And pedagogical technology has quite a major role to play in this shift in education. 

“What relationship do you think educational technology should have with the development of programmatic assessment at institutions?”  

Edtech can play an enormous role in optimizing the educational processes, as it can help reduce the manual work of administrative tasks, survey forms, logistics, and learning management systems. When integrating edtech into institutions, it is important to make sure there are opportunities for co-construction and co-design. Most importantly, pedagogical technology should be utilized to enhance, augment, and strengthen interactions and engagement throughout the learning process. 

Mentorship is key to programmatic assessment

In programmatic assessment, students are responsible for their own learning trajectory as they are challenged and encouraged to be aware of their strengths and weaknesses. However, they are not alone in this process. A coach, or mentor should be available to assist learners in formulating objectives and facilitate their performance. Mentoring is believed to be a key element in programmatic assessment, as it helps generate safe, entrusted spaces for continuous feedback, self-reflection, and remediation activities.

“The coach is like a sports coach who helps you to become the best version you can be.” – Dr. Schuwirth

Doesn’t having more data points make it more difficult and time consuming for faculties to assess, or to judge students’ competencies?

Programmatic assessment prioritizes the generation of information, or data, since: “More data points are like more pixels, more resolution to the big picture.”

A concern is then raised over the abundance of information and data. 

“Yeah, it does.”, replied Lambert without hesitation. He then continued: “I think it also takes longer to create a Mercedes Benz than to create a cheap car. It’s about quality.”‍

‍Implementing programmatic assessment requires an investment in both the quantity and quality of staff. Training is then the key to develop an expert team that can design, facilitate effective assessment. 

“You need to handpick your coaches, you need to train them and keep them trained, build communities of practice where people can exchange difficult situations and learn from each other.”

This is the advice Lambert gave regarding coach training. 

Modern education is about co-creation of knowledge

Knowledge used to be considered something that only the experts possess, while the non-experts don’t. Education used to rely on this linear transmission model of knowledge, in which instructors provide information, and students receive them. These perceptions have shifted tremendously over the past decades. Knowledge should be co-created, and modern education needs to be the space for such interactions to happen. 

It is easy to implement the linear transmission model, instructors simply record their lectures for students to listen and understand the subject. However, an effective learning process requires more than just one-way interaction. It needs to engage, motivate students in shaping their own knowledge. As Lambert concludes: 

“Education is a complex adaptive process, and assessment should be constructively aligned with that.” 

Programmatic assessment promises to be a critical approach to achieve the higher education deserve, where students are the knowledge owners, instructors become the facilitators, and faculties remain agile and flexible to any changes needed. 

Further resources on Programmatic Assessment 

Below you can find articles that provide more background to programmatic assessment, mostly written by Cees van der Leuten and Lambert Schuwirth, two key advocates of this approach. 

Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles | A consensus statement on the 12 principles underlying programmatic assessment.  

Programmatic assessment: From assessment of learning to assessment for learning | An introduction to programmatic assessment, its definition, principles, and benefits. 

Twelve tips for programmatic assessment | 12 tips on how to implement programmatic assessment.