Automated Feedback - Opening the way to an iterative feedback system

Reviewing written work - whether between peers or from teacher to learner - can be an arduous process, exponentially more so with larger student cohorts. And with that large potential time-sink, even less time is available for teaching. What’s more, it turns out that a lot of time spent reviewing is mainly focussed on correcting spelling, grammar, style, and semantics, rather than content and argumentation. We call these ‘lower-order’ skills: ones that don’t necessarily constitute good argumentation or understanding of a topic, but nonetheless are the first things many teachers and students notice when reviewing written work, whether intended or not.

FeedbackFruits has been trying to reduce the amount of time that teachers and students sink into reviewing each other’s work by using AI to identify common errors and make suggestions on written work so that human reviewers can focus on ‘higher-order’ feedback. But how does this process work, and why not rely on existing programs to check our work? I spoke to Joost Verdoorn, R&D Lead at FeedbackFruits, to try to find out...

Interview with FeedbackFruits R&D Lead Joost Verdoorn

Hi Joost. Firstly, where did the idea for Automated Feedback come from?

Well we noticed that one of the main time-sinks for teachers was giving students a lot of in-depth feedback. And of course, this is very important but not every piece of feedback is as valuable for the teacher to give. We started working with Wilco Te Winkel from the Erasmus University, and  noticed that there might be an area where AI can really help the learning process. ‍

And can you summarize briefly what Automated Feedback is about?

It's about giving lower-order feedback on the writing of students to facilitate teachers to be able to give that higher-order feedback on their writing. So in essence, it just unburdens a part of the teachers responsibility, so that they can focus on the aspects in which they're an expert.

"[Automated Feedback] unburdens a part of the teachers responsibility, so that they can focus on the aspects in which they're an expert."

So how does it work, both from a teacher and a student's perspective?

So from a teacher's point of view, in essence, it’s not that different from how you usually design your writing assignment. When you refer to the writing assignment, you have to summarize for your students what is expected of them; what's the word count; what kind of type of work; what kind of sections should be in there; is there any specific referencing style - those kinds of things. Instead of just writing them down, the teacher configures it in the automated feedback system. They can select from a number of options to essentially design their writing assignments.

When that’s set up, it’s handed over to the students who will be able to read these criteria just how the teacher designed it. When they hand in their work, in a couple of minutes - just enough to get a cup of coffee - they will be able to get their feedback. So it's relatively instant. We really tried to help students work iteratively here: to get their feedback really quickly in a way that they're also stimulated to resubmit their work to ensure this iterative process and continuous improvement.

And this, if I may, I think is also one of the key things that we do here - this really quick feedback process. Usually, when you hand in something as a student, it might take a couple of weeks before you get your feedback back - at least that was the case for my university. And by the time you get your feedback, you've forgotten what the writing assignment was about and you don't really care anymore. Whereas if you get your feedback within a couple of minutes, it's still fresh in your memory and you can really improve and you also have more of a drive to improve. So I really hope this sparks that kind of iterative learning.

How did development take place? Was that process more internal or external?

It was immediately really clear that we couldn't develop this in a vacuum. It couldn't just be an internal process so we really had to involve as many teachers from different domains as possible. So that's how we started, we found some literature online, which proposed a rubric, a set of criteria on which the author thought was a comprehensive list of things to look for in writing. And then we took that list, we mailed that around to a couple dozen teachers and tried to get feedback. So which of these criteria do you think are more or less valuable for you? Where do you spend your time mostly? We really tried to get this picture from the teaching community in writing styles. To make sure that we spend our efforts in the right place to really focus on those criteria that benefit the teacher most.

And by using this tool, how is it making life for teachers easier?

It is really about unburdening them. So what we try to do here is develop a tool that, in the hands of the right teacher, can save them a lot of time with lower-order skills. So these are things like spelling and grammar, plus things like page numbers and references. Are you actually using them correctly in your text? - Things in which the teacher usually is not necessarily an expert, but also is not necessarily interested in giving feedback on. And by taking that away we essentially converted them to focus on things that do matter for them.

"We often hear from teachers that they get distracted by minor mistakes, and then it becomes harder to focus on the actual content. And by taking that away, we feel like we can really help that process."

Could you add anything about its integration into either the LMS or existing tools and activities?

Yeah, so we've tried to make this experience as seamless as possible. So Automated Feedback is integrated within most of the tools in which students can upload some work. So we have tools around Peer Review where peers review one another's work and generate feedback. Essentially, Automated Feedback can be added as a step just before that - before your work is sent back to the teachers, the students already get a first load of feedback, so to say.

And now we try to make sure that it integrates really well. And that’s actually part of the reasoning that FeedbackFruits has with its entire tool suite - to just work in existing workflows. So we integrate with the LMS, we're now working really close together with Microsoft, for their Microsoft Teams product to make sure we integrate well there, because we see all these types of workflows, and we want to be as unobtrusive as possible, really fitting into what already exists. And the same is occurring for Automated Feedback within the rest of the feedback suite - it's almost like a Russian nesting doll in that sense.

Word processors can also check spelling and grammar. So what added value or features does Automated Feedback have?

“Every text-processing program these days, and rightfully so, has features for checking your grammar and your spelling. And something like Grammarly goes even more in depth also on your writing style. And all those things I think are really useful if you want your end-product to be as good as possible, but they're not necessarily conducive to a learning experience. Maybe even the contrary, if you can always accept the suggestion that is offered to you with one click of a button, you will not actually learn to do it right yourself. We're trying to flip it around and also focus on some of the more formal aspects.”

So essentially, the teacher has pictured in their mind what they want, “what I want the students to learn”, right? So for some assignments you should not use first-person pronouns, for example, because it’s not considered formal or that's not generally not used in an academic writing style.
And they can't really just click a button and rewrite their sentence to a different form. They really have to do the work and do it themselves. But we do give them the practice and the handles that they need to be able to learn that.

Maybe you could go a bit more in depth about those suggestions. How does it look for a student when they're being corrected on their work? And what resources are they presented with to be able to correct it?

This is actually a continuous process for us. So when we just got started, we didn't know a whole lot on how to give good feedback. But we have learned over time that there are certain feedback models, and what we are trying to adhere to right now is “feed-back feed-forward feed-up”, which in essence says, in a particular situation, you would usually apply passive voice for example, and your situation looks like that. And that's why we suggest you use passive voice. So, we try to give more context and even though the student may not in this particular case may not get all the benefit from it, by also sketching the context in which this is usually the case or in which this should apply, they are able to apply it to the broader picture. Then, of course that is already in the application. So essentially we annotate the work of the student by just trying to put feedback in the document.

That happens just like a teacher would mark their student's work, but when we give feedback on, for example, passive voice, to continue that example, we do provide a link to one of our articles and provide more explanation on what passive voice is, and what are the cases in which you should use it - and also in which cases is it not recommended, showing them that they are not alone in maybe misusing the passive voice, because it is a common mistake for beginning academic writers to really misusing that kind of form. And we have tried to do that for every kind of criterion that we have - tried to sketch the context so that students are able to build up this framework for writing.

In Dutch we have this word 'ambacht', meaning something like artisanal - so it’s, I think, that you learn by practicing a lot and making mistakes - that process, but then giving feedback on that. That is what happens here.

And how would you motivate students as well as teachers to use Automated Feedback? If students use a tool like Grammarly, it's faster and they can fix their sentences right away.

Right? I'm pretty sure that students already know, of course, that they have a spellcheck in their processor, and also that there are various tools like Grammarly that can help them. Yet we still see that teachers are still giving a lot of mixed feedback on these formal formatting kinds of things. So in essence, that just means that they might not be doing their job as a learning tool. So that's where we have to pick up the slack.

How do we motivate them? Well, for teachers the use case is really simple. In essence, they get a load off their work. And they can focus more on other things, hopefully, going more in-depth on the feedback that they give.

Then for students, it should result in a more holistic writing experience, because it is just a less annoying process. If you pour your heart and soul into this piece of paper that you write, and then you hand it in and three weeks later, you get something back and you're so disconnected from that thing. That's just not a nice learning experience. With instant feedback, it’s almost gamified, right? You get this - we'll we're not there yet, but what we really want to get into - that we when we see when you upload an improvement, then we see what you did with it. We're able to correct some of these issues and learn from the mistakes that you make. We really want to commend the student, tell them “this is nice, you pick this up well, and now let's get on to the next step”. So this is an ongoing process for us - how to motivate the students who we work with the feedback that we give them. But I think I think we're getting better at that.

“One end is really trying to almost make this a gamified experience where it becomes fun to produce higher and higher quality work.”

Are there other things in the future roadmap or pipeline for development that you'd like to share?

Yeah, so Automated Feedback doesn't have to stop at giving feedback on academic writing. Of course, that’s an obvious route that this kind of development could take. We see students use language in many more facets of their learning process than just academic writing. So something we're really looking into is how to make the peer review process a more holistic experience for students, where we really try to guide them on how to give good feedback, etc, so their peers learn as much as possible.

So that was one one route we wanted to go into at some point. And another end is really trying to almost make this a gamified experience where it becomes fun to produce higher and higher quality work. I think that would be really nice. And for that, Automated Feedback is definitely not the only thing - it should be integrated into the entire feedback process. So you get a couple of rounds of automated feedback, then you get feedback from your peers, you may do some corrections and the end or may even go to the teacher's desk. So that holistic feedback process from beginning to end. We’re FeedbackFruits after all, so I think that's the direction you should take. But almost all of this is subject to change, where we're always thinking about where to apply the lessons and the technology that we've learned. There may be some surprising things coming up in the near future. Just keep checking the regular channels and you'll find out!

Conclusion

The future of AI in education is difficult to predict, but easy to imagine as becoming more integrated and relied-upon. As the technology catches up with teaching needs, more and more course designs will be able to benefit from this time-saving and focus-assisting tool. To keep up to date with how FeedbackFruits is developing Automated Feedback and how you can start using it today in your feedback setups, please see the following links:

• This use case describes how an instructor from Wageningen University used Automated Feedback in their thesis class.
• This tool page outlines Automated Feedback’s features and even lets you try it out for free!
• In this video, Dr Cali Ellis shares her experience using Automated Feedback with her adult learners at Evergreen State College.

If you're interested in using Automated Feedback for your course, schedule a meeting with us.

‍