This protocol is ambitious, Jason. I think I would classify it as a curriculum embedded constructed response task. Generally, portfolios ask students to select artifacts from their ongoing work and write about them to point to features of their work that provide evidence of learning in their eyes. What if you did this. Give the students free access to the bot—they decide if, when, and how they use it. The task? Think about the six stories read in class. Rank order them according to how much you “like” the story. Focusing on the top two and the bottom two stories, write an essay pointing out the things you appreciate about the top two and the specific details that made you place the bottom two stories at the bottom. Note: This doesn’t have to mean you “dislike” them. Suggestion: Use the bot to help you generate some criteria you can apply to short stories help you talk those you like a lot and those you like not so much. You will have 40 minutes to complete this task. For a third of your grade, write a post task analysis responding to each question: 1. What problems did you face as you worked on this task? What did you do to solve them? 2) How did you use AI to support your work? How do you feel about these uses? 3) What did you learn about judging short stories? What questions remain for you?
I definitely want to think about how to make this more portfolio-like. With this first attempt, I really wanted to just bring the stages together (whereas our LMS separates them) and finding a way to focus on process.
I called it a portfolio because each stage builds off of the other ones. But I see your point: it's not the same as going back to distinct learning artifacts that student had already created!
Yes. It’s sort of a “portfolio in an instant.” You are asking students to create an “artifact” on the spot and show you how they do it with the intention of grading “process.” One traditional intent of a folio is to let students reveal their own processes during the creation of a significant project or series of projects. Folios afford students the opportunity to look back for themselves and discover their growth in their own work. Situated choice is at the heart of the notion. In my suggestion of ranking the stories, implied are choices, a high level of cognitive activity, and student control and ownership of their work. Let me know how you ultimately work out your idea. Btw Nick asked me to write a guest post for his newsletter on AI and portfolios. Good luck!
This is meant to be 3-4 weeks or work. My students do it piecemeal, and ideally treat each Stage as an artifact to reflect on and connect to other artifacts.
Then, at the end, they reflect on their processes and the possible large-scale meaning of those processes.
There's also a ton of student choice here. I am very open to how students want to approach things.
You and I have different conceptual frameworks regarding choice and its authentic meaning, Jason. I understand where you are coming from and I think why.
Agency. Autonomy is “on your own, sink or swim.” Agency is the right to make decisions in a network (community) and enjoy/suffer the consequences—opportunities to learn by self-induced and intentional behavior. The key then becomes reflective analysis, self and peer feedback, teacher feedback, goal setting and action plans to improve. Lean into difficulty, challenge, risk of failure. It’s ok to fail. It’s not ok to be a bump on a log after failure. Admit it, learn from it, get evaluated on how much you learned, not how badly you performed.
Thanks for sharing this! I'm a writing professor, so I can't entirely junk paper assignments (or else I'll be violating our program policy), but I like what you're doing here in terms of process over product. One question, though: how do you handle students who want to opt out of using AI at all? At my institution we're not allowed to require the use of AI for any assignment (and I agree with this approach, in terms of my own ethics), and so have to have an alternate but equivalent assignment for those who don't use AI (so in essence, I have to create two versions of every assignment and activity, whee!). I didn't see any alternate options in your Google Doc so I'm curious if you actually do require your students to interact with the AI? Do you get any pushback?
And honestly, I don't junk them either. I just count the larger stage as the paper, for right now!
The only reason I don't have an alternate here is that I've cleared it with my students with past assignments. (They had an option to do basic peer review with a bot, as well as a person.)
They totally have the chance to opt out. I imagine, most semesters, that will happen!
Would you like to work together to create an alternate assignment? I'd be happy to work on that!
this is awesome, jason! One question - in the use of AI part of the rubric, how are you going to grade that part? They will write something that describes their use?
This is admirable work! What I particularly like about this is, you've worked on customizing an LLM for your class and you've come up with a systematic process to engage your students in a course of structured thinking, where each stage is guided for them. Like others responding, I'm not sure I'd call the result of a this a "portfolio," but I suppose it's a multi-stage collection of process artifacts that lead to the conceptualization of piece of writing—and in that sense it's not the traditional "paper," but something akin to what we used to do (in my world back in the 70's and 80's when our English teachers used to make us turn in notecards and outlines on the way to the paper that we literally typed up on electric typewriters and later printed on ragged-edge jam-prone dot-matrix printers, when "paper" was really what made the paper!). I don't know if I have an alternative term for your intricately staged assignment...but "paper" certainly isn't it (of course)! Staged, structured-debate, technologically augmented componential essay might be moving into the territory...? I think you're inventing some new form of writing here. :-)
Your work with customizing an AI specifically for engaging your students is something I'd like to try as well! I'm wondering how you found the process of making your Contrarian bot? Is it something you recommend for others to try making for themselves?
Full disclosure: I was thinking working with one of the Llama-LM models, b/c I'd also like to teach my students how to customize their own LLMs in a digital humanities context. (Was thinking of Llama b/c it seems to be something one can develop in one's own local environment and the development promises to be cross-platform compatible--can make it on a Mac, Linux, Windows, etc.--at least I hope so.) I'm still figuring out processes and thinking about methods, but it seems like customizing one's own bot from a giant model is something accessible to lots of us humanities people without comp sci degrees, yes?
Alternatively, I wonder if people teaching writing courses will increasingly find pre-customized bots like your Contrarian to be useful for their classes. Are we going to find ourselves increasingly coordinating together gathering around a handful of AI bots that a few of us have customized for the task of teaching writing and critical thinking skills?
This protocol is ambitious, Jason. I think I would classify it as a curriculum embedded constructed response task. Generally, portfolios ask students to select artifacts from their ongoing work and write about them to point to features of their work that provide evidence of learning in their eyes. What if you did this. Give the students free access to the bot—they decide if, when, and how they use it. The task? Think about the six stories read in class. Rank order them according to how much you “like” the story. Focusing on the top two and the bottom two stories, write an essay pointing out the things you appreciate about the top two and the specific details that made you place the bottom two stories at the bottom. Note: This doesn’t have to mean you “dislike” them. Suggestion: Use the bot to help you generate some criteria you can apply to short stories help you talk those you like a lot and those you like not so much. You will have 40 minutes to complete this task. For a third of your grade, write a post task analysis responding to each question: 1. What problems did you face as you worked on this task? What did you do to solve them? 2) How did you use AI to support your work? How do you feel about these uses? 3) What did you learn about judging short stories? What questions remain for you?
This is a great idea, Terry!
I definitely want to think about how to make this more portfolio-like. With this first attempt, I really wanted to just bring the stages together (whereas our LMS separates them) and finding a way to focus on process.
I called it a portfolio because each stage builds off of the other ones. But I see your point: it's not the same as going back to distinct learning artifacts that student had already created!
Yes. It’s sort of a “portfolio in an instant.” You are asking students to create an “artifact” on the spot and show you how they do it with the intention of grading “process.” One traditional intent of a folio is to let students reveal their own processes during the creation of a significant project or series of projects. Folios afford students the opportunity to look back for themselves and discover their growth in their own work. Situated choice is at the heart of the notion. In my suggestion of ranking the stories, implied are choices, a high level of cognitive activity, and student control and ownership of their work. Let me know how you ultimately work out your idea. Btw Nick asked me to write a guest post for his newsletter on AI and portfolios. Good luck!
That's a great point!
This is meant to be 3-4 weeks or work. My students do it piecemeal, and ideally treat each Stage as an artifact to reflect on and connect to other artifacts.
Then, at the end, they reflect on their processes and the possible large-scale meaning of those processes.
There's also a ton of student choice here. I am very open to how students want to approach things.
You and I have different conceptual frameworks regarding choice and its authentic meaning, Jason. I understand where you are coming from and I think why.
Ok! I definitely want to lean into student autonomy for the next version.
Agency. Autonomy is “on your own, sink or swim.” Agency is the right to make decisions in a network (community) and enjoy/suffer the consequences—opportunities to learn by self-induced and intentional behavior. The key then becomes reflective analysis, self and peer feedback, teacher feedback, goal setting and action plans to improve. Lean into difficulty, challenge, risk of failure. It’s ok to fail. It’s not ok to be a bump on a log after failure. Admit it, learn from it, get evaluated on how much you learned, not how badly you performed.
Thanks for sharing this! I'm a writing professor, so I can't entirely junk paper assignments (or else I'll be violating our program policy), but I like what you're doing here in terms of process over product. One question, though: how do you handle students who want to opt out of using AI at all? At my institution we're not allowed to require the use of AI for any assignment (and I agree with this approach, in terms of my own ethics), and so have to have an alternate but equivalent assignment for those who don't use AI (so in essence, I have to create two versions of every assignment and activity, whee!). I didn't see any alternate options in your Google Doc so I'm curious if you actually do require your students to interact with the AI? Do you get any pushback?
Those are great points!
And honestly, I don't junk them either. I just count the larger stage as the paper, for right now!
The only reason I don't have an alternate here is that I've cleared it with my students with past assignments. (They had an option to do basic peer review with a bot, as well as a person.)
They totally have the chance to opt out. I imagine, most semesters, that will happen!
Would you like to work together to create an alternate assignment? I'd be happy to work on that!
this is awesome, jason! One question - in the use of AI part of the rubric, how are you going to grade that part? They will write something that describes their use?
Great question!
I actually want to tweak the reflections a little bit, to make that more specific! Certainly, that second reflection will help!
I'll also be able to see how that chat with the Contrarian Bot goes, and how it connects to everything else surrounding it!
Amazing - such a robust assessment framework. Great work and thanks for mentioning me!
Of course! Your worked helped me design this!
I'll be curious to hear how this goes.
Thanks! I'll definitely report back!
This is admirable work! What I particularly like about this is, you've worked on customizing an LLM for your class and you've come up with a systematic process to engage your students in a course of structured thinking, where each stage is guided for them. Like others responding, I'm not sure I'd call the result of a this a "portfolio," but I suppose it's a multi-stage collection of process artifacts that lead to the conceptualization of piece of writing—and in that sense it's not the traditional "paper," but something akin to what we used to do (in my world back in the 70's and 80's when our English teachers used to make us turn in notecards and outlines on the way to the paper that we literally typed up on electric typewriters and later printed on ragged-edge jam-prone dot-matrix printers, when "paper" was really what made the paper!). I don't know if I have an alternative term for your intricately staged assignment...but "paper" certainly isn't it (of course)! Staged, structured-debate, technologically augmented componential essay might be moving into the territory...? I think you're inventing some new form of writing here. :-)
Your work with customizing an AI specifically for engaging your students is something I'd like to try as well! I'm wondering how you found the process of making your Contrarian bot? Is it something you recommend for others to try making for themselves?
Full disclosure: I was thinking working with one of the Llama-LM models, b/c I'd also like to teach my students how to customize their own LLMs in a digital humanities context. (Was thinking of Llama b/c it seems to be something one can develop in one's own local environment and the development promises to be cross-platform compatible--can make it on a Mac, Linux, Windows, etc.--at least I hope so.) I'm still figuring out processes and thinking about methods, but it seems like customizing one's own bot from a giant model is something accessible to lots of us humanities people without comp sci degrees, yes?
Alternatively, I wonder if people teaching writing courses will increasingly find pre-customized bots like your Contrarian to be useful for their classes. Are we going to find ourselves increasingly coordinating together gathering around a handful of AI bots that a few of us have customized for the task of teaching writing and critical thinking skills?