Join our Webinar on December 17th: From Regulation to Innovation: What the EU AI Act Means for EdTech
chevron_right

Programmatic assessment to measure learning outcomes

Nhi Nguyen
Rebecca LeBoeuf
Rebecca LeBoeuf
|
October 18, 2022
Table of Contents

At inspirED 2022, we had the opportunity to speak with Professor Lambert Schuwirth, one of the pioneers in introducing and implementing programmatic assessment. Our Content Specialist, Dan Hasan and Dr. Schuwirth had a witty and insightful conversation on how programmatic assessment can provide holistic, data-grounded ways to evaluate students, while stimulating lifelong learning. Below are the highlights of this conversation. 

And you can watch the full conversation here

A selection of use cases and rubric templates for self, peer, and group assessment, designed and successfully adopted by institutions worldwide.
DOWNLOAD NOW
Join Cole Groom of FeedbackFruits and Patricia Luna of TAMU for an in-depth webinar exploring authentic assessment and how it can transform your approach to student evaluations
REGISTER NOW

The role of assessment in higher education

There has been a major shift in the role of assessment in higher education, from assessment of learning to assessment for learning. That is, assessment should provide rich information that guides and fosters the learning process, instead of labeling students as competent or incompetent. To generate and process information, assessment for learning relies on formative assessment approaches such as feedback. 

However, Lorrie Shepard in her article, The role of assessment in a learning culture, proposes that assessment for learning does not suffice to measure student achievement: 

“Only after successful completion of a module can students progress to the next.”

This means that summative assessment is still essential to the evaluation process. 

Programmatic assessment then emerges as an ideal way to harmonize both traditional and modern assessment approaches. The ultimate goal of programmatic assessment is to provide the whole picture of a student’s “competency” via a combination of assessment methods.

This novel approach to assessment seems to be promising, and appealing. But can it be applied widely at institutions, or is it just another hype in education?

The ideal assessment approach for Competency-Based Education

“The idea of programmatic assessment emerges from medical studies, because it mimics so much the process when the doctor diagnoses the illness.”

Dr. Schuwirth summarized the idea behind programmatic assessment with an analogy of medical education. 

To generate accurate diagnosis, the doctor has to rely on different channels like blood tests, physical check-ups, conversations with patients, and so on. Similarly, triangulation is the key to assessing students’ competencies. By triangulating assessment formats across a longitudinal basis, we can generate rich information to make “high-stakes decisions” regarding the student’s competency.

As we keep talking about competency, what is it exactly? Competency is defined as the ability to complete a task by integrating knowledge, skills, attitudes, and professionalism. How competency is divided and measured varies according to institutions. Competency-based education (CBE) is then developed to help students master the core competencies required for their chosen fields of study, while allowing faculties to evaluate how well their learners accumulate the desired skills. 

According to Katie Larsen McClarty and Matthew N. Gaertner, author of the article ‘Measuring Mastery: Best Practices for Assessment in Competency-Based Education”: 

“A CBE model is workable only insofar as its measures of learning yield trustworthy data about students’ prospects for future success.”

A major challenge facing competency-based education (CBE) is to ensure the effective use of robust assessment methodologies that result in useful performance-relevant information (PRI) by which to provide feedback on learning and make decisions about learner progression.

Programmatic assessment; therefore, is the perfect evaluation methodology for the CBE model, due to its reliance on abundant learning data and methods to collect these data. 

Programmatic assessment vs Competency-based assessment

The Competency-Based Medical Education (CBME) Core Components Framework highlights programmatic assessment as one of the 5 essential components for competency-based teaching. In essence, programmatic assessment must ensure the following elements: 

  • There are multiple points and methods for data collection.
  • Methods for data collection match the quality of the competency being assessed.
  • Emphasis is on workplace observation.
  • Emphasis is on providing personalized, timely, meaningful feedback.
  • Progression is based on documentation of achievement. There is a robust system for decision-making.

The full description of the Core Components Framework can be found here

Traditional versus Programmatic assessment 

To further understand the idea behind programmatic assessment, Dan raised the question: “What makes programmatic assessment different from traditional approaches?”

The main difference, according to Dr. Schuwirth, lies in how each approach generates and makes use of the information. While the traditional approach is inherently reductionist, producing rich information from tests then reducing them to a score; programmatic assessment aims to retain the complexity of information. 

“Learning is a complex interactive process, you can’t apply a reductionist approach. In Dutch, we usually say – If you disassemble a bike, then reassemble it, you will have a bike. But if you dissect a frog, and glue it back together, it is not a frog anymore. So it is the interaction that counts.” – Dr. Schuwirth commented.

Learning is indeed interactive and complex, and it requires a combination of different methods to accurately measure and gauge how far students have progressed over the course of learning. 

How much is programmatic assessment an assessment for learning?

Student learning and decision-making functions are the focal points of programmatic assessment. 

Instead of assigning grades as incentives to motivate students, meaningful and continuous feedback is provided to help students reflect, improve and become better learners. As stated by Dr. Schuwirth: 

“What we want to instill on our students is that they continue to learn, and have the agency and ability to assess, collect information, self-assess, reflect, and to set goals for future learning after graduation.”

Rich information is then needed to activate lifelong learning. Instead of connecting pass/fail decisions from one single assessment moment, multiple assessment approaches should take place over a longitudinal basis. Eventually, the information generated from this assessment combination becomes sufficient to reach summative decisions. 

The main focus is the student’s learning. Instead of ‘jumping through summative assessment hoops’, assessments should promote learning. This is done by disconnecting the pass/fail decision from one single assessment moment. Instead, the student is provided with meaningful and rich feedback to help them develop themselves. Ultimately, multiple moments of assessment over a longer period of time lead to a pass/fail decision. This decision is made to decide if the student has met the desired learning outcomes. As such, there is no decision moment based on a single assessment; the decision is made based on rich information on the student’s learning trajectory.

The role of technological affordances

The conversation then switched to the question on the role of modern technology in adopting programmatic assessment with Dan asking, “Our students today have all sorts of affordances. Do the Internet, and digital and social infrastructures necessitate programmatic assessment more than traditional assessment?” 

“I would say absolutely! Our students have affordances that are unprecedented.”

And what are those affordances? Students can now be part of multiple communities, have access to different channels and modes to communicate and create knowledge. Most importantly, modern technology exposes students to endless sources of information, and knowledge. This then led to a shift in the imbalanced relationship between teachers and students, as now students also become the knowledge creators. And pedagogical technology has quite a major role to play in this shift in education. 

“What relationship do you think educational technology should have with the development of programmatic assessment at institutions?”  

Edtech can play an enormous role in optimizing the educational processes, as it can help reduce the manual work of administrative tasks, survey forms, logistics, and learning management systems. When integrating edtech into institutions, it is important to make sure there are opportunities for co-construction and co-design. Most importantly, pedagogical technology should be utilized to enhance, augment, and strengthen interactions and engagement throughout the learning process. 

Further resources on Programmatic Assessment 

Below you can find articles that provide more background to programmatic assessment, mostly written by Cees van der Leuten and Lambert Schuwirth, two key advocates of this approach. 

Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles | A consensus statement on the 12 principles underlying programmatic assessment.  

Programmatic assessment: From assessment of learning to assessment for learning | An introduction to programmatic assessment, its definition, principles, and benefits. 

Twelve tips for programmatic assessment | 12 tips on how to implement programmatic assessment.

Share on social media

More from FeedbackFruits tips series

Technology Tips
|
23/3/2023

5 ways to stimulate collaboration with FeedbackFruits Team Based Learning

5 ways optimize the team-based learning process with the support of technology

Technology Tips
|
28/10/2022

Develop critical thinking with Discussion on Work | FeedbackFruits tips

Check out 4 ways to design asynchronous discussions that stimulate critical thinking skills

Technology Tips
|
28/10/2022

Stimulate continuous engagement with Interactive Document | FeedbackFruits tips

Discover how to maintain continuous student engagement at every stage of your course using Interactive Document

Most popular blog posts

student success
|
Nov 16, 2023

Promote active learning in different instructional modes

Explore how to best implement active learning strategies with deep understanding of different modalities

Industry News
|
Sep 11, 2023

Meet the new FeedbackFruits partners | Summer 2023 Highlights

FeedbackFruits announces partnerships with many institutions worldwide over the past 4 months

student success
|
Aug 23, 2023

Competency-based education (CBE) in higher education: A landscape overview

An overview of the state of competency-based education (CBE) in higher education around the world

We gather the most valuable resources and learnings on pedagogy and email them to you every month.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Filling out this form means you agree to our Privacy Policy.