Should Students Defend Their Work Out Loud?

00;00;00;00 - 00;00;35;02
Welcome to AI, a podcast on the future of AI in Canadian education. My name is Ryan Oliver, the CEO of Ampere and Canada Learning Code. And I'm Melissa Safadi, the co-founder of Canada Learning Code. On this episode, we talk with two professors at the University of Toronto, Steve Jordans, who's a professor of psychology, and Brian Harrington, who teaches computer and mathematical sciences.

00;00;35;04 - 00;00;53;29
And I am someone that has since high school, always thought I had a future in computer science. I ended up taking the back door there and becoming, becoming someone who leads a teaching organization. But increasingly all we hear about is how computer science is dead. Learning computer science is not worth it. No one's getting jobs in computer science anymore.

00;00;54;01 - 00;01;07;02
And so I wanted to start this whole conversation just with asking the computer science teacher is that true?

00;01;07;05 - 00;01;25;18
Well, thanks. Steve and Brian both for joining us. Do you want to talk to us? Maybe first introduce yourselves and then. And then talk to us about talk labs, specifically? Yes, sure. Okay. Yeah. So I'm Steve Jordan. I teach, psychology at the University of Toronto Scarborough, and I teach really big classes, first year classes.

00;01;25;18 - 00;01;44;00
So I get to see young people, regularly. And just as a maybe have a little bit of pre context, I've noticed their social anxiety, has been getting really high and we've actually been measuring it and thinking about, you know, what could we do to to help. Is there a role for AI. And that's where some of the original idea for Talk Lab began.

00;01;44;05 - 00;02;08;17
But I'll, I'll leave it as a teaser there and hand off to my, colleague Brian. Thank you. So Brian Harrington, I teach computer science also at University of Toronto Scarborough. I also teach the big classes. Not as big as Steve's, but we are talking, you know, 506 hundred students. Computer scientists have always had social anxiety.

00;02;08;20 - 00;02;31;09
But it definitely is getting worse. And the other part of this is something I've said for years of computer science. Don't go into computer science expecting that you are going to be that stereotypical computer nerd who will sit alone in a room and write code all day. If you can't explain your code in the real world, it's no good.

00;02;31;13 - 00;02;57;29
You need to be able to pitch, explain, talk through work. So I've for years said, you know, you should be explaining your code. You should be having code reviews. You should be assessing students on how well they can talk about things, not just how well they can do them. That's, Okay. You jump right into what I've been thinking about a lot lately, which is I grew up, the generation told take computer Science and you will work forever.

00;02;58;01 - 00;03;16;17
You know, you'll have that job. It's waiting for you. All we need is computer engineers. As recently as ten years ago. You know, our organizations were founded on that idea that we could bring opportunity across Canada. And then in the last year or two, all you hear is computer science grads aren't getting jobs. And so, Brian, I'm curious, like, is it worth taking a computer science degree?

00;03;16;17 - 00;03;39;29
And if so, like, how is it changing, to meet this moment? And I so this is the hot topic. I have, people coming to me all the time asking these questions. And one of the things I say is I talk about, you know, I came in the age when people were saying, oh, don't go into computer science because this internet thing is going to ruin it.

00;03;40;01 - 00;03;59;18
I'm dating myself a little bit, but, you know, why would anyone hire a computer scientist? People post their code online. I can just download it. And in the end, this did not kill computer science. It made it a lot more because now every company had an app that they wanted to build a website. They needed someone to support, you know, systems.

00;03;59;18 - 00;04;22;26
They needed people to work with. I absolutely think in a next generation, in a decade or so, it will be similar. It will be it's gone from why do we need to hire computer scientists? Because our CEO can vibe code this to oh man, our CEO vibe coded this. We need to go to your site and just you actually make it work and not be security and privacy nightmares.

00;04;23;00 - 00;04;46;29
I heard someone described that I could think the other day for CEOs as like what CEOs love is the startup moments, right? Like we loved that moment. We started this company and we were doing everything and all we want with five coding is now to get back to that moment. And it's like, I could do it again. I could make the app anyway, I apologize, it's, it's it's very true that that vibe coding as a solution is, is, is very popular among the CEO class right now.

00;04;47;02 - 00;05;04;15
So the next place we take this interview is, is to really dive into what Talk Labs is and what it can do. Mel, you are an instructor in a post-secondary education. I am I teach, yeah, I teach leveraging information technology at the Ivy Business School. And I always find it really interesting because I teach a 78 students class based discussion.

00;05;04;22 - 00;05;25;13
But when I hear from other post-secondary student professors like we are in this podcast episode, when you're 500 students and you're like, left with multiple choice or essays, how do you how do you handle that? Especially with AI, right? Either multiple choice doesn't assess learning. There's lots of students, as we know, who may not be able to write multiple choice test.

00;05;25;13 - 00;05;43;25
It's not a reflection of what they know. And then you've got essays that now people are either writing with AI or, you know, professors or Tas or grading, and it's it's not perfect. So I really liked how we get into Talk Lab and just this opportunity to kind of go back to basics when it comes to assessment. So we'll dive into that part.

00;05;43;25 - 00;05;50;05
Now.

00;05;50;07 - 00;06;08;18
Yeah. So so I'll jump in here a little bit. So Brian and I have run into each other in the hallways for, for years we've known of each other because we are two professors who have always been passionate about education, trying to find ways of enhancing the student experience. So I've known of the research Brian was doing. He's known of stuff that I've been doing.

00;06;08;21 - 00;06;29;02
And then really, you know, we we talked about social anxiety a little bit, but I think I want to take a right angle for that because we originally built Talk Lab to really help students learn how to speak to one another. Which seems kind of crazy, but in a world where they've grown up text messaging so much, they've become very uncomfortable with face to face, real time interactions.

00;06;29;02 - 00;07;07;08
And so we saw AI as an opportunity for them to practice some skills without feeling the worry of judgment, because I wouldn't judge. So that God is building Talk Lab, which is a conversational AI platform. But somewhere along the way, I think Brian and I's pedagogical and things kind of kicked in. And in a world where all the other teachers were saying equals cheating, we started seeing other potential and we started saying, hey, now that we've been playing with conversational AI, we see how it could impact student support in ways we've never been able to do before, how it could impact things like teamwork and support students to have effective teamwork, which leads to all

00;07;07;08 - 00;07;30;09
sorts of other positives. But, but, but the main sort of horse we're riding these days is can we finally get assessment right? Multiple choice has been, a deal with the devil. We've accepted multiple choice assessment because it gives us scores, at scale. Nice objective measures of learning at scale. But what is it actually measuring? And how valuable is that?

00;07;30;10 - 00;07;54;09
You know, we've known for years of primarily measures memorization and low level. The better way to assess learning was the way we used to back in the day, oral assessments, having an expert interrogate a student. You know, Brian's alluded to this already. Can they talk about their work? Do they understand what they're doing? Could we use AI to bring aural assessments of learning to scale?

00;07;54;15 - 00;08;15;11
I teach 1800 students. Can we assess 1800 students and have them go through an oral assessment, kind of experience? What do they think of that? Is this the future? And we kind of think it is we think we're in an opportunity to get assessment right and to kind of reform education. As a result of that.

00;08;15;13 - 00;08;38;12
No, I was going to say, like, I read the paper you had published at the C 21, it kind of goes through some examples, but maybe for those who haven't, all those a plug to take a look at it. But those that are listening, could you maybe look up and walk through what that look like in, you know, with the students in your classes and going to give us, you know, maybe more concrete examples of what, you know, assessing orally versus this multiple choice landscape looks like.

00;08;38;15 - 00;08;54;24
Yeah. So I'll give you a taste of, of of that study. And we are being careful. I want to say we're not just kind of going in there and saying, throw away all the multiple choice test. We're doing it this way. We have to get it right for this to work. So the first steps last fall, I had students going through another assignment.

00;08;54;26 - 00;09;13;04
That was kind of complex about finding a research article and then deciding what they would do next as a scientist. But that was one of these things that they could have used I just to do for them. Or they could use a I, you know, good way to really help them do a real quality piece of work that that engage their mind as they were going.

00;09;13;04 - 00;09;28;26
And of course, I was encouraging the latter, but we never know. Right. And when you get to the end, what a lot of, people are doing now in small classes is they are doing oral assessments. They're bringing the student in and saying, you know, do you actually can you explain to me what you handed in? Do you understand that?

00;09;28;28 - 00;09;45;16
That's the part we had the oral of the I do. So after they went through the whole assessment, they had to book a meeting with the I, the I interrogated them about what they did, how they did it, how they used AI, as a as they went through the process. And in so doing, it produced a transcript.

00;09;45;19 - 00;10;02;12
And that's really the extent of the oral assessment part for that project. That transcript was then graded by Tas who were looking for evidence that, yeah, they actually knew what they were talking about or not. So that's what they were kind of looking through. But we were able to do that with 1800 students, have them go through it.

00;10;02;12 - 00;10;23;02
So, you know, one question was logistically, can we make this work at scale? And the answer to that was yes. And then the other question was how do students react? And in that paper there's a word cloud. When they came to class the next day, I said, if you could give me just one word to kind of, express your experience with that assessment, what would it be?

00;10;23;05 - 00;10;39;25
And there were a few little words. If you look closely, you'll see some anxiety or stress or, you know, a few of the negative word. But a lot of the bigger words are like innovative, interesting. They're, they're generally. But as we say in the paper, at the very least, the students are not revolting. You know, they're not saying, oh, this is absolutely horrible.

00;10;39;25 - 00;10;59;09
They seem intrigued. They seem to find this interesting. So that bodes well. Now we've got a bunch of other research projects where we're going to expand it to more classes, and we're going to start looking at, you know, could we pre digest some of that output for the greater so that they have an easier time grading. Could we actually get to a point where we trust AI grades.

00;10;59;09 - 00;11;22;05
If there was always a potential for students to petition. We we always want them to have that ability to petition. But could we get to a point where largely the AI is assessing and providing a grade? And so that's, that's sort of a step into the future. But we're sort of slowly working on that slope, trying to very carefully assess the potential of this approach.

00;11;22;08 - 00;12;00;08
And then so is taking off that. So what we're starting to look towards now is, you know, I don't think we're at the point where you're going to fully trust, okay, the AI gave me a 73% versus a 74%. I'd argue that we probably aren't shouldn't be trusting humans necessarily at that scale. I'm building to alternative grading. But one of the things we are doing right now is, we are looking at, okay, can it do these like, large high level values and sort of go, did you or did you not do the thing we said you do?

00;12;00;11 - 00;12;21;03
So one example we're doing right now is having students who have completed some sort of work. We're working with the co-op department who says, hey, these students went on a four month or an eight month or 12 month co-op session. They're coming back and they would always have to write this nine page report. Students hated doing it.

00;12;21;03 - 00;12;48;19
Faculty hated grading it. You know, honestly, since AI has sort of taken over, students don't do it and faculty don't grade it. But it's sort of this process that everyone steps through. So we said, okay, well, could we have the students talk to and give sort of an exit interview and say, what did you learn? Did you evaluate, have you done some metacognition on the value of these things?

00;12;48;21 - 00;13;11;01
And a lot of talking to, the people doing it. They said one of the things is it's a pass fail assignment, and with a pass fail assignment that we don't want any students to pass because it's like the real thing is, did they do the co-op? We set the bar very low, but students now know the bar is low, so they aim low, which means we then have to set it lower, which means they aim lower.

00;13;11;04 - 00;13;32;06
So can we think a better way through that. So we've come up with this model and we call it try till You fly. It's sort of moving pass fail to pass, try again and so students are never told no you failed that assessment. They're told you didn't prepare like we asked you to prepare. You didn't do the thing we told you to do.

00;13;32;08 - 00;13;48;23
Go away and try it again and we can leave the bar high. Right? We can set the bar. I know this is what we think the bar should be. The bar doesn't need to move. The real penalty you're paying for not meeting the bar is not. You fail a course or you have to go back and write a new assignment.

00;13;48;26 - 00;14;14;07
Okay? You have to close the window. Open the window and try again or try again tomorrow or whatever. But what that does is that gives us this sort of again, you'd always want the students to be able to petition the students to be able to go, hey, I don't think the AI gave me reasonable feedback with it, but if you have the sort of very rough assessments of like, did they do the thing?

00;14;14;07 - 00;14;39;02
Have they gone above that bar? What we found in our preliminary testing is students are very happy with it. Faculty are very happy with the responses it's given and all that worry about hallucinations or errors or bias kind of goes away when the penalty for being wrong is much, much lower.

00;14;39;05 - 00;15;00;18
In this last section, I really want to dive into what the application is for this beyond post-secondary. It feels to me. So I am someone and I think I mentioned in this, I mentioned it to the guys after the conversation when I worked for the government and we had, developed a system for oral reporting, in many cultures and oral, oral culture.

00;15;00;20 - 00;15;22;00
And when I was on the government side and we were expecting handwritten, you know, essays on on how government funding was spent, it becomes very, counterintuitive to the community where ultimately serving. And what I love about this is, is the possibility of being able to use oral, oral assessment, as a different way of assessing learning.

00;15;22;00 - 00;15;34;22
And where we dive next is the practicality of this in K-12 education. And some of the skepticism, it could be around it.

00;15;34;24 - 00;15;59;16
Okay. It's interesting, when we started this podcast and started these conversations, I, you know, in retrospect, I guess it's fairly obvious, but I did not expect the core theme of so much of our conversations around AI and their education to to be around assessment and the way assessment has to ultimately change it just it comes up again and again that if AI is a is a matter of trust, then like we just we have to change the way we assess because we have to change the way we understand knowledge.

00;15;59;16 - 00;16;17;09
Knowledge retention. Yeah. Go ahead. Yeah. I was gonna say, if I can jump on that. I mean, I think Brian and I are both aligned in that we, we kind of see this as a Cooney and paradigm shift moment where AI has come in, and especially in terms of assessment, it's kind of blown up the old way of doing things.

00;16;17;09 - 00;16;34;22
And there's a whole lot of hand-wringing over that. You know, we can't have written assignments now. We can. And anything they can do with AI, we can't have them do and they can do almost anything with AI. So there's a whole lot of, you know, worry about what we've lost. But I think both of us are saying, yeah, but there's new potential to there's new opportunity.

00;16;34;25 - 00;16;51;27
And this is all part of a system kind of in transition. We're going to have to get rid of some old ways of doing things and some of those are our, you know, things we want to more in like written assignments. That's how we taught them how to structure their thinking and whatever. And is that just is writing from scratch dead?

00;16;52;00 - 00;17;11;13
You know, maybe maybe it is. But that doesn't mean we have to give up on the process of teaching them how to structure their thoughts and do stuff, you know, thing like an oral assessment can push them to do that as well. So, I mean, it really is a time I think of AI messing everything up at the same time is providing all these other potential opportunity, these.

00;17;11;13 - 00;17;32;10
And I think if we negotiate this well, it's going to be uncomfortable. It is uncomfortable and it's going to continue to be uncomfortable for a couple of years. But I think we could get to a really nice spot where we're measuring, learning more holistically, where we're measuring things that matter, like critical thinking, communication skills, you know, stuff that students are going to need in an age of AI.

00;17;32;13 - 00;17;49;14
So overall, we think we're on a good path. It's just a bumpy path. And it's one that's not most comfortable to right on, but to just to jump off of that. One of the things that I keep coming back to is, you know, people are saying, oh, there are these problems. And, you know, here's all these issues.

00;17;49;14 - 00;18;15;01
But when you interrogate, for the most part, the status quo, before AI, people were not actually all that happy with, right. Like, why do you have one, maybe two big tests in, in a course? Well, because we only have one big room where we could do it, or because the logistics of doing it are tricky. Why do you.

00;18;15;03 - 00;18;38;06
You know, one of the things I love, love, love group work. I think it's so important we talk to industry and they say communication and teamwork skills basically trump everything else. But then when you talk to faculty, do you do. Oh yeah, we give them teamwork, but we don't assess the teamwork because we can't we don't have a good way of doing it.

00;18;38;08 - 00;19;10;00
Okay. So I has broken we don't really know who did the work in the team, but we actually kind of didn't before. We just all kind of, you know, put our blinders on and pretended this was happening, like dealing with the plagiarism issues. Right? The, the stuff we talk about in computer science, we've known for years that percentage of our students, you could go to Chegg, or you could go to your friend, or you could go and get them to do the assignment for you.

00;19;10;03 - 00;19;34;28
And we had tools and trying to fix some of these things to, to make it. But realistically, we just went, well, we're just going to kind of pretend that's not a thing I do. I mean, it's not actually that different. It's just a scale difference. So a lot of the issues we're doing right now to try to actually say, hey, did you actually understand the work you submitted?

00;19;34;28 - 00;20;02;05
Did you actually, you know, are you able to explain what you did? We probably should have been doing already anyway, we just sort of stuck our heads in the sand and pretended it wasn't a problem. Yeah. It's so interesting, this idea that, like we you said at the start there, like this idea that we built our assessment approach and the way we teach around the available infrastructure and the assets we had, both in terms of available to teachers in the the actual physical infrastructure, we could house those people.

00;20;02;05 - 00;20;19;27
And I can hopefully like, can hopefully change that. In a sense. I'm curious, like, you know, you're working with this out of, you know, if I'm if I'm right on the numbers, correct me if I'm wrong, but around 2000 students are experimenting on this with, do you see a practical application for this, at an elementary like, K to 12 level?

00;20;19;29 - 00;20;51;26
For me, the first thing that comes to mind is, is kids with disabilities. My daughter has dyslexia. Oral assessments has become such as, element of her success. But yeah, it's to. Yeah. No, no, no, that would be interesting. Yeah. Well, we certainly do see, a lot of general use. In fact, we're we're happy to say that we've been working now with, both Dell technology and C21 Canada, to, to kind of think of the K to 12 mark a little bit more, maybe 7 to 12, but maybe K to 12 to, and we actually have an offer for the audience.

00;20;51;26 - 00;21;13;24
So if anyone wants this, we're forming a community of practice, of other educators that are maybe interested in trying this approach out. And we're just asking them, hey, take something you're already doing the way you already do it. But in addition, have some students experience this new way, and we're interested in their reaction to it. We're interested in your reaction to it, and we're interested in together.

00;21;13;24 - 00;21;31;08
There's so many challenges in K-12 like you brought up, you know, one challenge, but there's there's so many in terms of it being inclusive, making sure we're measuring the things that they really care about. And those boards, you know, being curriculum aligned, etc.. So we would love to say, okay, Brian and I have have this thing started.

00;21;31;08 - 00;21;53;07
We would love to see this use widely as the approach to assessment. That's nothing we can accomplish on our own. So we're trying to build out a much bigger team working with not for profits, for profits, but especially teachers. And the nice thing here is we can now offer this to them without any cost. So Dell is picking up all the AI costs for a teacher that wants to try this out.

00;21;53;09 - 00;22;09;17
And the C 21 is is supporting as well. So it's a it's we're trying to now build this community and see if we can get the snowball rolling a little, a little quicker and a little more broadly. And if anybody's interested, they can reach O'Brien and I are both very easy to find. U of t, you can reach out to us, and we would love to.

00;22;09;19 - 00;22;29;01
We'd love to invite you to the to the community. That's great. Yeah. That's amazing for anyone that's skeptical. Like, I mean, know you mentioned the students. No one revolted when you were even to use this. Have you do you anticipate or what would you say to a teacher or professor that might be a bit skeptical about using this, or do you anticipate that?

00;22;29;01 - 00;22;55;11
I'm curious. So so definitely, we've had pushback from people like there are certain people who just go, AI is bad, don't use it anywhere in education. Stop. And to those people, you know, a lot of the the messaging has been kind of we talked about a are you convinced that what you were doing already is actually all that good?

00;22;55;13 - 00;23;17;13
And are you convinced that we can't make it better? And if so, let's have a conversation. But to get those ones who are sort of in the middle, more of the skeptical around again, we are trying to be very careful. Right? Like we are not saying, hey, have the AI do all the grading and set a grade, and it goes into the gradebook and you walk away.

00;23;17;15 - 00;23;41;20
There's obviously problems with that. The other thing that we get is, yeah, but hold on, what about that human touch? Right. We want to grade students because we want the contact with them. And you know, my response to is usually, how much of that are you really getting? Right? Like, obviously in the university context, you have a thousand plus students.

00;23;41;23 - 00;24;07;15
You don't have that contact time with each individual student to really assess them anyway. But even in the K to 12 right now, with the pressures we're having, the increased class sizes, these sorts of things, getting that contact time is, you know, is not. And then think to these teachers about, okay, what are the things you want I to take off your plate.

00;24;07;17 - 00;24;31;04
Right. So I don't want I to do the teaching for me so I can spend my time filling out forms. I want I doing the paperwork for me so I can spend my time teaching. So one of the reasons we went to assessment is that's the thing that like it's a huge time sink for our faculty to do assessment right or just isn't possible.

00;24;31;06 - 00;24;50;14
So if we can take that off my plate so I can spend more time contacting the students as opposed to thinking of it as a replacement for me or for the faculty or the teacher. There's an interesting way to to take that a little bit. So sometimes kind of as Brian says, we'll talk about when people say, hang on, this is what we do, right?

00;24;50;14 - 00;25;07;23
We as teachers, we teach and we assess learning. And that's and now you're bringing the AI and our. And so first there's almost like a you're stealing part of my job away kind of vibe. But then when they think about it a little more and you say a little more, do you enjoy assessing learning? Do you enjoy being the one that assesses learning?

00;25;07;25 - 00;25;25;03
What do you think of a future where it's like a good cop, bad cop, where the where the teacher can be the good teacher and the AI can be the bad AI. And so it's kind of like what I'm doing, my job is to prepare you for that aural assessment that you're going to have. And so I get to be the good guy.

00;25;25;03 - 00;25;51;29
I get to be on your side. Totally. I get to support you. And I'm not the one. Then that's going to put you through the, the test, literally. And there's a dynamic to that. That's kind of nice. It allows the teachers to be teachers. And especially if you have the sort of try to fly. I mean, this is if Brian and I had our way, everything would be like every chapter you take in the textbook would be like, okay, now keep going through the test until you've shown us you've mastered that topic and then go on to the next one.

00;25;52;00 - 00;26;14;05
I, like Brian, said there could be 20 tests in the course, not a midterm and a final. There could be a little assessments all the way along, and everyone. We give the student the opportunity to keep learning until they show us they've learned it. You know, there's dynamics of that. And that's partly when we have to kind of get out of the traditional way of doing things, think about this future and go, is it that or is that kind of cool?

00;26;14;07 - 00;26;31;03
And I mean, Brian and I lean on the kind of cool side, obviously, but but that's our role, I think, in this ecosystem it's great. Try to fly. I love that I'm always like, fake it till you make it. That's that kind of more my no object. No. It's great. I think that's I think it's great. I'm curious.

00;26;31;03 - 00;26;51;10
I know you mentioned a bit about the future, so maybe we can put you in the, in the, you know, crystal ball, see for a minute to think about like, what would a post-secondary exam room or even K-12 like what what is that going to look like? I know you talked a bit about assessment, but and in ten years time I, you know, are we are we even in those big room like what is what does it look like.

00;26;51;13 - 00;27;18;23
So, so one of the things I really love. So I give a talk where I show, the on the slides, the examination schools at Oxford Divinity schools, Oxford University and say this was set up with faculty would sit in a circle and the student would come and present themselves for their viva voce for their oral exam.

00;27;18;26 - 00;27;41;18
And two things. One is they don't know ahead of time exactly what they're being evaluated on. That's sort of you have to be ready for a little bit of everything. And number two, it was when you were ready for it, not on you to September 12th, because that was the date that we have a room booked. It was yeah.

00;27;41;18 - 00;28;13;00
You know. No, I think I'm going to think okay. Now, I think with the help of my tutor and the help of my college or whoever I decide, I think I'm ready for this. Okay. Let's go. So I really like this idea of the assessment being more continuous. Right. It's not sort of like, hey, you throw everything at students for four months or 12 months or four years and then have one day where you go, you know, is everything ready?

00;28;13;00 - 00;28;29;24
Is everything working well? Did you have a good sleep last night? Are you feeling ready for it? Or, you know, do you have a headache today? And. Oops, sorry. There goes all that work. You know, again, part of that is because we have the infrastructure for it. We built it that way. I think most faculties like. Yeah.

00;28;29;24 - 00;28;52;03
If you could have a test every week. Sure. You. Why not what I can because it just the resource explosion is not there. But also this idea of it being sort of individualized to the students, right. That it's not I don't have to write one test that every single student does the same, you know? Yeah. Mel walks into me.

00;28;52;03 - 00;29;11;25
I'm like, okay, I know Mel is very, very good. On topic A, B and C told me many, many times, but I've never actually assessed on D and E, whereas Ryan like well actually he really struggled with B last time. So let's have something where we go with B. And again a lot of people freak out about that.

00;29;11;25 - 00;29;29;23
So but but everyone has to have the same test. Everyone has to be equal. It's a does it or does it have to be equitable. Does it have to be fair. Does it have to be assessing what you do? Logistically, I think, you know, there there will be issues, right? Because, hey, can you allow people to do it at home?

00;29;29;23 - 00;29;52;29
Can you allow it to be online? We know we you know, Covid taught us there's all sorts of logistical problems with this. But actually the oral assessment does solve some of those problems. Right? It's you know, it's not easy. And we've we've we're doing this experiment now. We actually my students love I say hey try to cheat. You are allowed to where you're allowed to have a little earbud in.

00;29;52;29 - 00;30;23;03
You're allowed to have something off the screen. It's a lot different to do that when it's an authentic assessment. When it's a conversation, it's not as easy as you think to be reading something off a screen or listening to ChatGPT in your ear as it is with these more strict, more standardized modes of assessment. Hey, if it can give me a multiple choice answer I can feel and be, but it's not as easy to kind of have that back and forth and get away with that now.

00;30;23;05 - 00;30;42;05
And I think when I think of the future too, what I think is so we've been talking a lot about assessment, but if this assessment also assesses things like critical thinking, creative thinking, communication skills, then I think the education when we say, well, what's education look like? What we're going to see some teaching to the test. We always do.

00;30;42;05 - 00;30;59;15
That's natural. But teaching to the test is good. If the test is measuring what you want, it means we're going to have more group work probably, and more things where students have to use their critical thinking, have to use their creative thinking. And so that's why we're really exciting. It's sort of a reverse design. We think if we're going to get one thing right first let's get assessment.

00;30;59;15 - 00;31;24;13
And if we get assessment right first, that can drive a whole bunch of other innovations in the education system that align with that assessment. And so that's sort of our devious plan. That's incredible. I love this conversation. This has been so interesting. Guys. Thank you so much for joining us on this. If people do want to reach out, we'll maybe put the the ear contact in the or at least a, an easier way to find you in the show notes.

00;31;24;15 - 00;31;32;12
But yeah. Thank you so much, guys, for taking the time for this. Thank you. I was really sure. Thank you.

00;31;32;14 - 00;31;51;17
I is a proud Canadian podcast of the Pinnguaq Foundation and Canada Learning Code. I am your host, Ryan Oliver, joined by Melissa Sariffodeen. The show is produced by Zac Miller and edited by Zac Miller and Kyle Gordon. If you want to support, I support the work we are doing, then please consider a donation to the Pinnguaq Foundation.

00;31;51;24 - 00;31;54;00
You can find us on Canada Helps.