The power of Automated Feedback

Hello and welcome to what will be the last episode of this first season of the Learning Experience Lab. I'm your host Dan Hasan and after today we're going to take a short break and be back up and running around the start of september. But for this episode I'm looking forward to sharing with you our first interview with a colleague at FeedbackFruits, R&D lead Joost Verdoorn, about the power of Automated Feedback. To start though, I just want to reflect briefly on the journey so far, since we began back in January.

And if you're only eager to hear about how AI technology is going to shape the future of education, feel free to skip to 4:45 where the interview starts.

Throughout this podcast we've tried to cover some of the fundamentals of feedback and online course design, seeing to what extent we can universalise principles and practices in higher education. Of course, this exploration and discovery was only possible through the sharing of each of our guest speakers. It's been a great experience to learn so much purely from conversations, especially for someone who's more of a book-learner. I suppose that speaks to the power of multimodal learning.

So as a newcomer to the world of instructional design and educational technology, I was lucky to have a chance early-on to speak to John McCormick of Brandeis University, who helped me put parameters and perspective to these domains. And just as important was my following conversation with Lilian Nave of Appalachian State University, who introduced me to the Universal Design for Learning framework, which I've since come across everywhere I look. Lilian's ThinkUDL podcast remains one of my favourite listens, so do make sure to check it out if you can.

More food for thought came from Puteri Sofia of Taylor's University, the only person I'd ever heard of using augmented reality in law education - and with highly acclaimed success at that. As my own masters thesis was on AR in education I'm still curious to follow the journey that this and other novel technologies will take towards becoming the mainstream rather than the outliers. As for more mainstream technologies, i-coach Jeroen Mulder was also a much-needed voice encouraging the integration and training of teaching technologies - for our dutch audiences interested in online course design, Jeroen's Modern Onderwijs youtube channel has tonnes of advice, tips, and tutorials which are highly recommended.

The next guest who opened my eyes to a whole new way of looking at the world was UX veteran Darren Hood, and it's really been the case that since our conversation I can't help but see every user and learning experience through a design lens - have a listen to Darren's World of UX podcast for a seasoned perspective on the urgency of professional design thinking.

More open discussions followed in the next two episodes. We had Business school professors and instructional technologists Jeff Webb and Mustafa Elsawy in a roundtable discussion to discuss group work, the challenges and solutions to its facilitation. What I took away is that even these heroes of hybrid education had encountered the same old problems of free-riding and group grade assignment which is so common in the field.

Then, we came back to the Netherlands with Wageningen University's Researchers and Professors Kazem Banihashem and Omid Noroozi to discuss their research into peer review, which I believe is entering its final stage now. We'll definitely be following up on their results.

After that last one, FeedbackFruits hosted our annual inspiration days event, inspirED 2021, which attracted almost 1000 professors, instructional designers, and educational innovators from around the world. Of the 15-odd sessions we hosted, most of them were under 20 minutes, so I was grateful to one of the speakers, University of Adelaide's, Glen Wheatley, for coming back to talk to us in long-form about feedback for episode 11.

With one more look at the history and future of learning design and technology, computer science veteran Maria Zajac was next to share her experiences and perspectives on the field, giving a critical look at the use of IT in education.

And lastly, a second inspirED speaker, Professor Richard Powers came back for a discussion about inclusivity in teaching and learning, as well as sharing his 10 steps for online course design, which was a great way to round off the topics we'd touched on previously.

So now it's time to dive into the interview with Joost about Automated Feedback, which I won't say anything more about right now, so you can hear it from the expert himself.

‍

---

‍

[Joost] I'm Joost Verdoorn. And I've been at FeedbackFruits already for eight years, and I've been doing quite a number of things actually started as a student programmer, as most of the developers started back in the day. And after my graduation from TU Delft, definitely for technology, I joined the company full time as the Research and Development lead, really trying to gauge which new technologies can help us help students essentially, and help facilitate the learning process.  

[Dan] Okay. So why did you start researching and developing automated feedback?

[Joost] Yeah, that's a good question. We had been trying a number of different projects where we were working on AI technologies to facilitate the learning process. But we noticed that one of the main time sinks, so to say for teachers was giving students a lot of in depth feedback. And of course, this is a very important aspect, but not every feedback is as valuable for the teacher to give. And that's where we noticed together actually with Wilco Te Winkel from the Erasmus University, we started talking, and we noticed that there might be some area there, which we would focus on where AI can really help the learning process.

[Dan] Okay, can you summarize in a sentence or two, what automated feedback is what it's about?

[Joost] Right? It's about giving lower order feedback, the writing of students to facilitate teachers being able to give the higher order feedback on that writing. So in essence, it just unburdens a part of the teachers responsibility, so that they can focus on the aspects in which they're an expert.

[Dan] Okay, I want to ask you, briefly how it works, both from a teacher and a student's perspective.

[Joost] Right. So from a teacher's point of view, in essence, is not that different from how you usually design your writing assignment. So when you refer to the writing assignment, you have to summarize for your students, what is expected of them? How many? What's the word count, right, what kind of type of work, what kind of section should be in there is there any specific referencing style for those kinds of things. And now, instead of just writing them down, the teacher configures it in the Automated Feedback system. And they can select from a number of options, to design their writing assignments, essentially. And then the hand over to the students and the students will indeed, be able to read these criteria, these now that the way the teacher designed it, and when they hand in their work, like a couple minutes after, just enough to get a cup of coffee, they will be able to get the feedback. So it's relatively instantaneous. And yeah we really tried to help students work iteratively there, so to get their feedback really quickly, but they're also stimulated to resubmit their work to ensure this iterative process, continuous improvement. And this, if I may, your I think this is also one of the key things that we do here is this really quick feedback process. Usually, when you add in something, as a student, it might take a couple of weeks before you get your feedback back at least that wasn't the case for my university. And by the time we get your feedback, you've definitely forgot what the writing assignment was about and you don't really care anymore. Whereas if you get your feedback within a couple of minutes, it's still fresh in memory and you can really still improve and you're also have more of a drive to improve. So I really hope this sparks this kind of learning.

[Dan] Could we go into development a little bit? Like how is the development taken place? And what kind of ways to have you developed it with external or internal forces?

[Joost] Yeah, so from the get go, it was immediate, it was really clear for us that we couldn't develop this in a vacuum, as in we couldn't just be, it couldn't just be an internal process that really had to involve as many teachers from different domains as possible. So that's how we started, we found some literature online, which proposed a rubric, a set of criteria on which the author thought that book was a comprehensive list of things to look for in, in writing assignment. And then we took that list, we mailed that around to a couple dozen teachers and got tried to get feedback. So which of these criteria do you think are more or less valuable for you? Where do you spend your time mostly? We really tried to get this kind of picture from, from the community, from the teaching community in writing styles. To make sure that we spend our efforts in the right place to really focus on those criteria that benefit the teacher most.

[Dan] By using this tool, how is technology adding to how is it making life for teachers easier?

[Joost] Right, right. Yeah. So the main point here, and it is really about making the life of teachers easier, right? It is really about unburdening them. So what we try to do here is develop a tool that in the hands of the right teacher can save them a lot of time with, as we already mentioned, the lower order skills. So these are things from spelling and grammar. And do you actually, did you actually include your page numbers and the references? Are you actually using them correctly in your text, things that the teacher usually is not necessarily an expert, but also not necessarily interested in giving feedback on. And by taking that away. And trying to solve that for the teachers by giving feedback on that aspect. We essentially converted them to focus on things that do matter for them. We hope that is giving more in depth feedback on the verb, but it might be Microsoft knows that that's definitely the goal that they are able to focus on the content, right, not just a form of the word. And we often do hear that actually, from teachers we've talked to, that they get distracted by minor mistakes in there, and then it becomes harder to focus on the actual content. And by taking that away, we feel like we can really help that process.

[Dan] Absolutely. And that's very recognizable for myself when I'm yeah, marking my peers working reports as well. Right? It's a distraction. And if you can get rid of it all the better.

[Joost] Sorry, go on.

[Dan] Yeah, it's almost a very annoying fact of life, right? That when there's something just like someone's accent, at some point when someone's accent sounds excellent. It's like being off. You can't help but notice it more and more and comes harder to focus on on that actual conversation. And I think the same I think around become a good writer, and we need to bring you there's enough oddities you weren't able to see the bigger picture. Now with accents, you can't really say there's a right or wrong way criteria checks for voice but yeah, with writing, we can at least. I wanted to ask you something else about the technological value. And that's whether you had anything to say about and this is a bit general to FeedbackFruits tools. But is there anything you can say about the integration of Automated Feedback into either the LMS or existing activities that you find?

[Joost] Yeah, so we've tried to make this experience as seamless as possible. So Automated Feedback is integrated within most of the tools in which students can upload some work. So we have tools around peer review where peers review one another's work or work teachers reviewed, work with students, and get feedback on that. And essentially, automated feedback can be added as a step just before that, before your work is sent out to your peers, with your teachers, the students already get a first load of feedback, so to say. And now we try to make sure that it integrates really well. And then it's actually part of the of the decision that feedback foods has with its entire tool suite. That to just work in existing workflows. So be that an LMS, we integrate with the LMS, we're now working really close together with Microsoft, for their Microsoft Teams product to make sure we integrate well there, because we see all these types of workflows, and we want to be as unobtrusive as possible, really fitting into what already exists. And that the same is occurring for automated feedback within the rest of the feedback store. So it's almost like the Russian nesting doll in that sense

[Dan] Very nice. Now moving on to the competition. What processes can also check spelling and grammar? So what added value or features does Automated Feedback have?

[Joost] Right? Yeah, of course, every text processing program these days, and rightfully so, has features for checking your grammar and your spelling. None, of course, as much as Grammarly was going more in depth also on your writing style. And all those things I think are really useful if you want your end product to be as good as possible, but they're not necessarily conducive to a learning experience. Maybe even the contrary, if you can always accept the suggestion that is offered to you with one click of a button, you will not actually learn to do it right yourself. We're trying to flip it around and also focus on some of the more formal aspects.

So essentially, the teacher has pictured in their mind what they want, what I want the students to learn, right. And then there's all these aspects of writing that go into that. And we really give feedback. So we try to say to the students who are in this particular situation, you should not use first person pronouns, or something like that, because that is not considered formal or that's not generally not used in academic writing style.

And they can't really just then click a button and rewrite their sentence to a different form. And they really have to do the work and do it themselves. But we do give them all the practice, all the handles that they need to be able to learn that. So we have quite in depth articles on all these processes, we tried to process to really help them with this process.

[Dan] Okay, maybe you could go a bit more in depth about those suggestions. How does it look for a student when they're being corrected on their work? And what resources are they presented with? To be able to correct it?

[Joost] Right, and this is actually a continuous process for us. So we when we just got started, we didn't know a whole lot on how to give good feedback. And even though we call feedbackfruits, once you have to start doing it yourself it's different. But yeah, we have learned over time that there are certain feedback models, when we are trying to adhere to right now is feedback feed forward feed up, which in essence says Well, this in this particular situation, you would usually apply passive voice for example, and your situation looks like that. And that's why we suggest you use passive voice. So, he tried to give more context and even though the student may not in this particular case may not get all the benefit from it by also sketching the context in which this is usually the case or in which this should apply, they are able to apply it to the broader picture. Then, of course that is already in the application. So essentially we      annotate the work of the student by just trying to put feedback in the document

That's happening just like a teacher would mark their student's work but then we, when we give feedback on, for example, passive voice to continue that example, we do provide a link to one of our articles and providing a more explanation what a specifier is, what are the cases in which you should use it? And in which cases, is it not recommended, and also show them that they are not alone in maybe abusing the best voice, because it is a common mistake for beginning academic writers to really overuse that kind of form. And we have tried to do that for every kind of criterion that we have really tried to sketch the context so that students are able to build up this framework for writing.

Because it's, like in Dutch we have this word 'ambacht', so like artisanal or something like that? It's something like that, like authentic buchla. Yeah, yeah. So it's, it's, I think that you learn by doing a lot and making mistakes, that process, but then giving feedback on that. That is, that's the thing that happens here.

[Dan] I think I'm starting to really get a very clear picture to that first thematic question of how is it different to tools like Grammarly? They are, and Microsoft Word, Google Docs, providing corrections on your work. We're providing feedback, annotations, actionable things, which you can move forward on. And it's not just this is wrong, this is wrong. This is how it should be. But why should it be like that?

[Nhi] I also have more questions, like, because, um, but then we also, we all see, like, the pedagogical values of automated feedback, yeah, how it's different from like, spelling check. thing, but how would you motivate the students, especially students and teachers to use automated feedback? Because, yeah, how? Like, how would you persuade them to do so? Because they see, if they use grammarly, it's, it's faster? And yeah, they can just like, fix their sentence right away.

[Joost] Right? I'm pretty sure that students already know, of course, that they have a spellcheck in their, their word, and also that there are various grammarly that can help them yet we still see that teachers are still giving a lot of feedback on these formal formatting kinds of things, right.

[Dan] So in essence, that just means that they are not doing their job as a learning tool. So that's where we have to pick up the slack. And how do we motivate them? Well, we have for teachers that that use case is really simple. In essence, they get a lot of the work. And they can focus more on our things, hopefully, going more in depth on the feedback that they give.

[Joost] Then for students, it should result in a more holistic writing experience, because it is just an annoying process. If you pour your heart and soul into this piece of paper that you write, and then you hand it in and three weeks later, you get something back and you're so disconnected from that thing. That's just not a nice learning experience. I feel so I think the instant feedback and the way, that's almost gamified, right? You get this, we'll we're not there yet, but what we really want to get into is that we when we see when you upload an improvement, then we see that you you did with we're able to correct some of these issues and learn from the mistakes that you make. We really, really also want to accommodate the student, commend?. I'm not sure whatever it is for that, and really tell them that this is nice, you pick this up well, and now let's get on to the next step. So this is an ongoing process for us to motivate the students who we work with the feedback that we give them. But I think I think we're getting better at that.

Yeah, good question. And it's a nice segue to I think this will be my last question. But when I'm thinking about the future of automated feedback development, I have a bit of the mindset that more criteria equals better. But obviously, there's an upper limit to how many checks and how many different things that you can look for. So now you

[Dan] You can talk about the criteria. But are there other things in the future roadmap or pipeline for development that you'd like to share?

[Joost] Yeah, so Automated Feedback doesn't have to stop at giving feedback on academic writing. Of course it is. That's an obvious route that this kind of development could take. We see students use language in many more facets of their learning process than just academic writing. So something we're really looking into is how to make the peer review process a more holistic experience for students, where we really tried to guide them on how to give good feedback, etc, so their peers learn as much as possible. So that was one one route we wanted to go into at some point. And another one is really end, that's actually just reiterating that thing I just told Nhi, really trying to almost make this a gamified experience where it becomes fun to produce higher and higher quality work. I think that would be really nice. And for that, automated feedback is definitely not the only thing it should be integrated into the entire feedback process, right. So you, you get around a couple of rounds for automated feedback, then you get feedback from your peers, you may do some corrections and the end of may even go to the teacher's desk. So that holistic feedback process from beginning to end. I think we're feedback fruits after all, so I think that's, that is the direction you should take. But almost all of this is subject to change, where we're always thinking about where to apply the lessons and the technology that we've learned. That may be some surprising or things coming up in the near future. Just keep checking the regular channels and you'll you'll find out

‍

---

‍

Thanks once again to Joost and all of our guest speakers in the previous episodes. Again, we'll be taking a short break after this one and coming back around the start of September with new guests, topics, tips, and insights into optimising your teaching and learning experience in the academic year to come.

‍
Thank you so much for joining me in the Learning Experience Lab so far, and remember that we're always available for questions, queries and feedback. You probably know it by now but you can follow FeedbackFruits on LinkedIn, Instagram, Twitter and Facebook to stay up to date with the exciting things we've got in the works. I hope you'll join us next season for more - till then!