Business School Webinar Special

Hello, everyone and welcome to a special episode of the Learning Experience Lab. As you can tell from the different intro music, we're going in another direction today with this session covering feedback for its business schools webinar from the 5th of May. So last week, I had the pleasure of hosting two wonderful guest speakers for a guided discussion on how to facilitate effective group work and business goals. Those guests were Mustafa Elsawy, Learning Technologist from Georgia State University, and Jeff Webb, Associate Professor from David Eccles School of Business in Utah. So in this episode, I'm pleased to be able to share with you the contents of that conversation, which we hope to resume in a couple of months to discuss what's been happening in both schools since I started off by asking what got our guests interested in education in the first place. So let's get straight to it.

‍

---

‍

[Mustafa] I mean, I started my career as a teacher right now, I'm an educational technologist. But I started my career as a teacher myself. Then, while I was teaching, I just realized that I was more, more interested in educational technology, or in being in that area, educational technology area. So that's what I do for a living now, I work with a lot of faculty from a lot of colleges. I mean, we are one of the largest universities in the United States, here we have about 55,000 students. We used to be two institutions, and then we'll combine together. So this diversity and the number of faculty I work with, gives me the opportunity to explore a lot of technology tools, also see a lot of issues and problems that we need to find solutions for, or maybe see instruction objectives, and that can be best met by certain tools or certain strategies. So that's the my favorite part about my job, you know, solving problems and finding solutions, since the purpose of education is to prepare students for the real world. So group work and teamwork and collaboration activities become very important. And a lot of our faculty like to include that into their teaching, or to incorporate that into the teaching. You know, one of the challenges is like when you get these team members to evaluate each other, and, you know, you know, finding a tool that will do that and save you time, so you don't have to look at all these sheets and documents and, you know, especially if you want to do it online, can be very time consuming. So that definitely sparked the interest. And we started looking for an efficient tool that can do that and save time. So that's why, that's why I'm here today.

[Dan] Thanks for your answer. I'd like to give the floor over to Jeff, please tell us what sparked your interest in education, and especially group work?

[Jeff] My interest in education goes back so many years, I think I can barely remember it. Really, I've been in education for my entire career, I actually have a PhD in English, although now I teach Statistics and Data Science. So in the interim there I I worked in, I made kind of a career change and worked in industry as a data scientist and statistician, and then, you know, could never somehow get away from education. And I came back and I'm teaching in the Business School at the University of Utah, teaching business analytics, master's students, statistics and data science and, you know, group work. Well, I mean, honestly, there's a certain efficiency and group work if students are writing papers, and you've got four students per group. To be perfectly honest, it's more appealing in most cases to read 10 papers rather than 40 papers. So there's that element. However, you know, I do think, to, you know, underscore Mustafa's point in industry, all of my students will be working collaboratively on everything. So they need to, they need to learn how to do that. And so I see these group projects as essential, really essential curriculum and data science, and I think it's rarely treated that way. And so, I got interested in trying to figure out how to make these group projects part of the curriculum and, you know, make those learning opportunities as rich as possible. I would also add that I have done this very badly in the past. And so as recently as last semester, I was, you know, confronted with some really difficult free rider type situations in group projects. And one of the students who'd shouldered most of the burden in the project came in and told me that he regarded it as an ethical problem that one student does all the work. And the others are the other students in the project, think it's great and don't have any complaints about the group process. So that's where I started looking at. And maybe I'm, I may be getting into your other topics here. Again, I apologize. But I worry. That's where I started trying to get serious about thinking about, you know, how to do this fairly and effectively.

[Dan] Okay. In fact, that's a lovely segue for me to open our first poll question, which is going to be about the challenges. So to the audience, I'd like to pose this question about ‘What challenges you've encountered in your various roles?’. So thanks for your answers both. So you said, Jeff, that you had been quite bad in the past. And that's wonderful, because it's through mistakes and failures that you learn the most, is it not? So could we talk maybe a little bit about a specific use case or a specific instance, where you really realized that free writing was a problem? They mentioned this one student? Were there other instances which brought that home for you?

[Jeff] Yeah, if I could just expand on that one case, my course, about it's a 10 week course, five weeks of which roughly are dedicated to a really involved complicated data analysis project. And, you know, I don't think that students, rarely can students be successful. If they do that, by themselves, they really need everybody to be working on it, more more minds dedicated to solving the problem. And yet, in some cases, and the one I'm thinking of, in particular, this was a group of three, and the two students really saw the one very motivated student as an opportunity and kind of sat back and let him do all of the work in the problem in it. So I did have last semester, I did have, you know, a way of adjusting the overall group grade, individually based on individual contribution, the problem with my approach then, and one of the reasons that I really appreciate FeedbackFruits is, you know, software, when I was doing then was looking for a consensus around free ridership. But if you have in a group of three, two students who are free riding, and one who's doing all of the work, the two students who are free riding might well think that, hey, things are going great. It couldn't be better, democratically. Yeah, yeah, I, you know, it's a, this is a wonderful situation. So I don't get a consensus around that. And then I don't feel comfortable adjusting the grades, the individual grades of the two students downwards, because I don't, I don't, I just don't know how to interpret that. I suspect that, you know, there's a problem there. But I don't have a kind of consensus in the group that would support lowering those grades.

[Dan] Indeed, and I'd like to just ask one more detail about this instance. And that's about the summative assessment of collaboration skills. Now, in some courses, learning objectives specify that you will be developing teamwork and interpersonal communication skills, and you will be graded on those. And so if a group project is graded on the, you know, the balance of work, the distribution, the participation, that can sometimes incentivize students to pull their weight a little bit, but in your situation where you had one person pulling the others behind, how was the assessment and the grading weighted in that case? Do you remember?

[Jeff] Yeah, so I think you're absolutely right, that setting this up in advance in the syllabus, for example, or with some kind of discussion is essential for setting student expectations around grading practices. In this case, I did say that there would be individualization of grade based on contribution, but because I didn't really have much in the way of a structure much much in the way of a structure around grading. I didn't, I didn't really get great, you know, great response from students in that respect. So you know, just as I said, I have not been very good at this. And I also just to add a detail to that, I don't think many people are very good at this. Some are and I give them, you know, congratulations for that. But I'm also the Capstone manager for the MsBA program, and I talk informally to students a lot. And I had been quite interested to learn what, what the practices are in our program around group work. Almost all of our classes include group work, but from what I can gather and talking from students, the grading practices are all over the map. It's really quite, quite, quite shocking, really. So I think, if I have to identify low hanging fruit in our program, I would put group member evaluation at the top.

[Dan] Okay, I definitely want to go deeper into this. But I'll ask Mustafa, are you good at this?

[Mustafa] No, I'm definitely not. I don't know many people who are to be honest with you. And I think back to what Jeff was saying, it's difficult to be objective. There has to be a level of subjectivity when you do it this way. But with a tool, like feedback for us, for example, because the way it's so organized and systematic, it takes this subjectivity out of the picture. And it provides solid, clear grades, and, you know, it saves a lot of those awkward conversations that some students want to have after death. Like, why did you give me that grade? or Why did you score me? Give me a low score on that assignment? You have something to go back to and say, okay, well, you worked with these three or four other people. And that's how they evaluated you and evaluated your contribution to the team. It's not some, it's not much I can do about this. It's not me, it's, you know, it's us. So it makes it easier for the faculty to have great group project.

[Dan] Indeed. And have there been any particular problems, surprising or exceptional examples that you've encountered.

[Mustafa] I mean, it happens all the time, that you have faculty saying that we have students that tried to get a free ride, I mean, it happens to every group. But for you to be able to pinpoint that, and targeted specifically in a clear manner with enough data to support your judgment is difficult. When you're trying to do that, by having the students fill out spreadsheets or, you know, a Word document or something, you know, you have to have enough justification, and enough data to be able to say, you know what, you know, you can get the same grade like everybody else, because you did not contribute as much as everybody else on the team.  

[Dan] Okay, particular to and I'm sure you've been talking about, especially the COVID situation, but particular to the move to online. Most of us talked a little bit about this already, that some aspects of group work, such as equal accessibility to comment on things online, some aspects of group work has become easier and more inclusive, with the move to online. But some, like not knowing what's going on with groups, because it's all going on behind the screen, and you don't have a physical observation possible. Some have been made more difficult. Have there been particular challenges from the move to online that you've encountered?

[Mustafa] Okay, well, as bad as COVID is, the global pandemic is a disaster. So it's been a very bad thing. But in the area of instruction, technology, I hate to say it was a blessing. Because we were able to get a lot of faculty to move online and to use strategies and tools that they have not used before. And they realize that oh my god, I can save a lot of time by doing this, or I can maximize my class time by doing that. So, it also exposed some of the weaknesses in the strategies that we've used before. And it also posed new challenges for things that faculty used to be able to do easily in a face to face situation. Now they're having to do that online. So one of those things is group work and team member evaluation. So, you know, when that happened, we started looking for solutions. So we started looking at our current inventory. We did not have a tool that was designed specifically for that. So our first course of action was trying to tweak some of the existing tools to get them to do something they were not designed to do. So we tried so many different approaches from using the quiz tool in the learning management system. We happen to have due to Georgia State that kind of produced some reports, but they were all over the place. And it took a lot of work to go through these reports. Go ahead.

[Dan] I wanted to ask you that, obviously, it wasn't just your experience, but probably faculty wide. What was the feeling? What, was there frustration among faculty about the lack of tooling available? What did you sense on campus or on the digital campus?

[Mustafa] It was definitely a level of frustration, they did not necessarily express it as frustration, but were like, Okay, what am I going to do now, I have to evaluate these people. And the tools that we're using right now are very time consuming. For example, if you have them, if you have, like, you know, a bunch of questions or scale on a Word document, and you have each student do that for each team member, and then the, the instructor will download all of that, and go through them and try to come up with a score for each student. That is, in my opinion, that is brutal. I mean, really, in addition to grading and other things the faculty have to do, that's the last thing they're gonna want to do.

[Dan] You know, and most of you are learning technologists. So investigating and being creative with tooling and solutions is, I assume, towards second nature for you. And Jeff, as a professor, what kind of feelings were you and your colleagues encountering when it came to this fast transition online?

[Jeff] Uhm, it is difficult. Especially for in person classes, where I'm, you know, actually at the University of Utah, we continued in person classes in some form, but it put a huge, we, you know, hugely increased reliance on technology.

[Dan] And did you have particular new hires, or people step up to new roles to help with that by coaching, sometimes it's called or the technology training.

[Jeff] There was some of that available, I personally ended up doing it basically all myself, just because I then I could eliminate any, you know, any, any unforeseen problems. So, you know, about about, you asked about how we responded, and I would save that from for the students. It was extremely stressful. And I really saw, you know, more struggling than usual, just because of the context, the pandemic context. On the other hand, with respect to group work, I think that people felt so isolated. Students felt so isolated, that they really welcomed and treasured, working, you know, intimately, even if facilitated by zoom, working intimately with group members on a project.

[Dan] I think we've covered some of the problems, we've gone a little bit into depth. And I see that we've already started mentioning a little bit of the tools and solutions. So perhaps we can gently move over to the second phase, the discussion, and look a little bit more about the lessons that we've learned from being bad at implementing group work from encountering problems from having dealt with these challenges for now, well over a year. Round Two, gentlemen. And let's have a look now, at examples of where things have gone. Well, a strategy that was employed where you saw results, students expressing satisfaction showing equal participation. Jeff, I'd like to start with you. When we think about this one student pulling the others along, and the problem of free riding, has started to be dealt with, in what you're seeing and what you're doing. And if so, how?

[Jeff] Yeah, absolutely. And I would just refer to a chat from nagla. Who mentioned, you know, I think a very sensible approach to doing assessment. When I was doing that, trying to get a handle on who is free writing in order to do just the overall grade, individually, not only did it not work very well, but that was for me. And the students didn't get much in the way of feedback for themselves to self correct or to, you know, think about their role in a group. And so I really see that as being the essential element here. In their most recent course, I had FeedbackFruits, the group member evaluation tool very generously set that up for me this semester. So, I, at the last minute, put in two, group member evaluation assignments, one, a formative assignment, midway through the project and the other more of a summative assignment at the very end of the project, and what I really liked about the way that was working, is that the students got to see what other students on their teams were saying about their contribution, and, you know, various dimensions of their contribution to the group project. And then I had them reflect on that. And it was really wonderful to see them reflecting on what their peers were saying about their contribution. And in and, and it was that, the formative assessment midway through the project, and I could see them coming up with ideas about how they would self correct for the remainder of the project. So to me, that just seems like exactly what i what i want to see happening on on a group project, that kind of learning.

[Dan] Yeah. And let's make this a concrete link. How does that then lead to developing better career skills and readiness for the working world afterwards?

[Jeff] So I think that, you know, I can, I can get better at framing that and contextualizing that, for that transfer to the work world. But I think honestly, for my students, that's, that's a pretty obvious translation, just because they know that's the work context, they're all going to be working, you know, with, with, with engineers, on the one hand, and with, you know, business executives on the on the other hand, and they're kind of in this in between role, and so, collaboration and communication is just, you know, utterly essential for their, for their success. So I don't really have to do much to make that case and to make the case for the, the importance of that translation.

[Dan] Uhm, becomes self evident, if the course is set up in that way. Mustafa, what are your thoughts on this?

[Mustafa] I have to agree with that, especially with the first about students getting feedback from their peers, you know, hearing what their peers have to say about their performance and about their contribution to the group, and also having the opportunity to reply to these comments. You know, if I think somebody says something about me, that's not fair. Or maybe that requires a clarification, I can clarify, you know, why things happened, or, you know, what was happening that caused me to, you know, act that way. I can also learn from that, definitely, if it's designed as a formative assessment. My favorite part also is the reflection part at the end. So this kind of thing completes the process here. So you have the students work together, to simulate the experience of a real workplace, you know, work with a group, with a team, then get to hear feedback from other team members about their contribution. You know, read these comments, you react to them. And then finally, reflect on the overall experience, the project, the feedback that you got, and then write your own reflection on the whole process, on the one hand, and on what you learned from going through the project. And from the feedback you received from your friends or your peers.

[Dan] I mean, reflection, feedback, crucial components of all learning from especially in a group, we can only give so much feedback and reflection to ourselves. And when we see something from someone else's perspective, that's consolidation. Absolutely. How can you encourage authentic, critical and honest feedback because to some, here in the Netherlands we have a stereotype of being very direct, someone thinks something, they will say it, generally speaking, of course, but of course that doesn't come naturally to everyone. So are there rubrics, strategies or methods which you use to ensure this authentic, constructive and comprehensive feedback in such a group work activity?

[Mustafa] Well, basically, the survey questions themselves need to be well designed. Also the anonymity of the reviewers, you know, I don't know, saying What about me? Or, or I can say I honest, I mean, I can make an honest comment about a team member, without being, you know, worried about the team member coming back and talking to me, I was like, Why did you say that about me, you know, that makes the students feel more comfortable evaluating each other in, in an objective way. Of course, once in a while, you get students who are not being objective. And that's why I love this outliers feature. You know, it uses artificial intelligence to tell you if somebody is being overconfident or if somebody is, you know, overrating themselves, and you know, down, down rating, everybody else to make themselves look like they did most of the work or something. So that also is a very nice feature to have, because it helps you detect patterns like that. It's something that is impossible to do using other methods, like, you know, word documents, spreadsheets, and the D2L quizzes or, or, you know, the normal, what we call the conventional survey tools. So, I think so basically, anonymity, well designed questions, and also using artificial intelligence to detect outliers. All that helps.

[Dan] And, Jeff, what are your thoughts on this?

[Jeff] Yeah, I think I think getting, you know, constructive, or in some cases, hard feedback. Getting students to give that is a challenge. Everybody likes to be nice. But, you know, I think it's, it seems to help to set it up in such a way that, you know, you're really encouraging that kind of constructive feedback, with the aim being, learning and growth. So so there's that, you know, kind of the way, the way that you would, you would frame it. And it was really, sort of, as I began using this tool, it was also really eye opening to me, in the way that the tool was designed, I wouldn't have thought to do this, but the students are rewarded and not penalized. So they're rewarded for giving feedback. And I think the worry about, and you can adjust all of these, all of these parameters in the tool to you know, weigh different things differently. But if you're, if you're rewarding, students are just giving the feedback, then when they're giving the feedback, they're not worried that it's going to really negatively impact the student's grade. Now, in my summative assessment, I did include the feedback from other students more heavily. But in this formative assessment, I really put the emphasis on giving the feedback and, and so I see that students are quite sensitive to, you know, the notion that they might be hurting another student's grade. So they kind of pull back. On the other hand, if a student is angry about free riding, that student is more likely to be quite honest, and give feedback.  

[Dan] So and in terms of setting up, how much preparation and instruction to students need, in order to be able to learn this. Well, the skills and also the attitude towards giving constructive feedback. Jeff, please.

[Jeff] Um, yeah, that's a great, a great point. And I think I can do better than what I've done. You know, so far, it's just been setting up some text at the beginning of the assignment. But I think it needs to go in the syllabus, I think it needs to be foregrounded. Within the class, I, you know, as a preface to some of these assignments, that needs to be justified and, and, and motivated. And I think that large context for the assignments in setting them up, will really help. And I didn't do that. I didn't do that enough, or well enough in this first iteration.

[Dan] While the realization is probably the first point to getting it, and I have definitely seen some teachers going all the way and having a whole session or a whole week about why feedback is important, how to give it and what we intend and what we want the outcomes of this feedback to be. Mustafa, thoughts on this?

[Mustafa]  Okay? Yes, absolutely. I definitely agree with what Jeff said here. But I would add educating the students on the process explaining to them how to give feedback, how to provide positive, effective feedback, while providing negative cognitive feedback at the same time, I mean, you can criticize somebody, but you can still be nice about it, and you can be constructive, you know, I think explicit teaching in this area, goes a long way, can help them do that. Also modeling, you know, showing them how you, you know, provide feedback yourself, you know, you could do that you could improve by doing this or that, also, like we do, what other activities do like, especially when the students are new to a certain technology, we use a low stick assignment. First, to kind of get them through the process, you know, get them used, learn this new tool or new approach of giving feedback, then we can move into things that have more, like higher grades assigned to them. So, and I will also, I like Jeff's idea of adding this to the syllabus, providing very clear and detailed instructions for each assignment. This is what I expect you to do, not something like, Oh, please rate your group members, or teammates on a scale from one to five, you need to provide more detailed and explicit instructions on what you expect them to do and what constitutes constructive feedback. What do you know what constructive feedback looks like?

[Abdulla] We had a couple questions here in the in the Q&A, I thought it would be a good opportunity to

[Dan] Now's the time. Yeah, absolutely.

[Abdulla] So from an anonymous attendee question is, and of course, some stuff on Jeff, feel free to one, if you pick that up?How much guidance do professors receive in developing group projects?

[Mustafa] The project should simulate collaborative projects in the real world. Jeff, you want to share your perspective? Sure.

[Jeff] Yeah, I would, I would just ask guidance from whom we're, we're pretty much on our own. So, you know, certainly, there's not much guidance from, you know, the larger business school where I work. And so I really rely on kind of the thinking that's gone into the construction of this, this tool for group member assessment, and, you know, taking my experience, having done group projects with students for a long time, put that together with the way that the folks that feedback fruits have constructed the tool and kind of come up with an approach, and it's going to be ever evolving. So, you know, just try to keep improving the approach.

[Mustafa] Okay. Well, the other side to this, from an educational technology perspective here is, we would love to provide guidance, we would love to provide help. As a matter of fact, we have an entire instructional design team, who's sitting there and waiting to help our faculty, and provide all the guidance they need. But the question is, is our faculty ready to receive this guide, because, you know, I'm not, sometimes it's like, they are too busy. And I don't blame them, you know, they have to work on the classes they have to bring in, they have to do a lot of stuff, they have to do research, they have to go to conferences. I mean, they got too much on the plate, so I don't blame them. But there is also another side to it's like, I know what I'm doing, you know, just teach me how to use the tools. You know, already, like you guys are just the technology people here. Just you know, let's talk about the tools. Don't tell me how to design. Don't talk to me about designing the research. I mean, the designing the, the questions, and you know, all this guidance. So it's both in you know, we do have a lot of faculty that come in and ask for guidance. But the majority, unfortunately, are either busy, or they're just satisfied with what they know. You know, they think, you know, I got it just, you know, I can do it.

[Dan] I see one more question in the chat. And thanks for your answer both. I think we just have time for this from Carol Emmons. In the real world. There's very little free riding because teams have leaders who evaluate each member's effort and contribution in annual performance evaluations. Is there a way to simulate that process in student group projects? Two very good questions. thoughts. Jeff, I see you nodding.

[Jeff] Yeah, that that is a good point. I think the key difference there is what's at stake. If you're in the real world, you're getting paid for the work that you're doing, and you're highly motivated to remain employed. So you're not, you know, in most cases, you're not going to get a free ride. And I think that's what a key, a key element of the adjustments that I'm going to make is to, in the, for the summative assessment actually have used the tool to adjust the individual grade from the overall group grade. And I think that's the next step for me, is to add that kind of the add in the stakes, that there's a real, you know, there's a real risk for an individual free rider in terms of the grade at the very end of the course. So I think that given sovereignty,

[Dan] Sorry, their transparency, that that's going to happen, depending on their contribution that's announced. Sorry to interrupt.

[Jeff] No, that's exactly right. So I think that gets us closer to the real world, there's a real stake involved here.

[Dan] Thanks very much. Any final thoughts on that one was tougher?

[Mustafa] Well, I would have to agree with that, if the students know that they are not just getting a grade from the entire group, and they will get an individual grade, based on their contribution, they are going to be more motivated to contribute to the group, I find it helpful sometimes to if possible, of course, that doesn't work all the case, to assign certain well defined roles for the different group members. So for example, this one, and I'm gonna try, I'm not a data person here, but I'm gonna try to use an example for data science, like if somebody is going to gather the data, for example, somebody else is going to analyze it, somebody is gonna, you know else is going to report it. So it does help to have well defined roles for the group members in advance. Absolutely. And that, again, comes down to structure and making sure and clear what is expected. People agreed to,

[Dan] I want to ask one more quick question before we wrap up. And most of our you already talked about the relationship between faculty and instructional technology and professors adoption of tools, would there be one tip or one advice, or one paradigm attitude shift that you would give to a teacher who wants to set up a group evaluation activity for the first time, scared to do it?

[Mustafa] For the first time, sit down with either somebody from your instructional technology team, if you have such a team at your institution, or sit down with another faculty member who has done it before, on more than once? and get their perspective on it? See what worked for them? What didn't work for them? what they will do differently the next time and just learn from experience, instead of trying to reinvent the wheel and start from scratch.

[Dan] Jeff, would you agree with that?

[Jeff] Yeah, absolutely. Um, I am not afraid to borrow, to steal. And I think that's a good way to use a template. If it's proven to work, then just you know, borrow it and test it and then iterate from there.

[Dan] Wonderful.

---

Thank you so much. These things always flashed by and already, we're almost an hour in. Thanks once again to our guests for sharing their experiences and insights. Even our listeners not in business schools or teaching positions will surely be interested to learn about how costs and activity designs are set up with these career ended goals in mind, why we opt for group and team based activities and some of the things that we all struggle with whether teachers, students or designers in group work. And I'd like to reiterate that we hope to do this again before long, adding this discussion to our regular podcast publication schedule. In the meantime, you can of course, check out the feedback fruits website for more resources and use cases relating to the themes of this session. And before signing off.

Today, I want to remind our listeners of the much anticipated feedback for its annual event inspirED 2021, which you can find more information about below. We're nearing max capacity for registrations in the last two weeks before the event, but you can still sign up and hear more from many confirmed guest speakers, which you can find on the event page. And on that second day, I'll be speaking to one of our pioneering users of Automated Feedback, looking at how AI and automation has saved students and teachers a ton of time and opened up the arena for even better feedback processes in higher education. I hope to see you there. But until then, thanks for joining us and once more in the Learning Experience Lab.