Turing Test

Assignment Stages

Stage 1: In-Class Preparation

Before students begin on their group papers, we need some preparation for successful collaboration.  Because the instructor plays a key role at this stage, students cannot know which group will be the AI or Turing Group.

• First, split the students into two groups.  It's unknown at this stage which group will be AI and which will be Turing.

• Next, work out collaboration plans.  With a worksheet and instructor meetings, each group decides the roles each student will play.  Students need to plan for how to keep in touch with each other, how to create their documents, and how they'll track the evidence of their work.

To ensure all students have an opportunity to contribute, a worksheet should lay out specific tasks to be assigned during the first class period.  During this time, the instructor needs to walk from group to group to discuss their collaborative plans and individual roles.  Remember: this will take place before the AI and Turing randomization occurs.

Stage 2: AI and Turing Groups

In the homework component of this project, students work together in AI and Turing groups.  The groups will submit an AI-generated paper and a human-written paper to meet a specific rubric from the instructor, and then try to "trick" the instructor into picking the wrong paper that's AI versus human.

It's very important that the instructor not know which group is which.  At the end of class, the instructor should provide the groups a way to randomly choose, and then leave the room so students can do the process in secret.

Engagement Grading via Worksheet and Group Meetings: in this part of the activity, students are individually graded based on their engagement with their group.  The goal is collaboration — students should be able to choose a direction, delegate responsibilities, and come together to produce a final product.

In this activity, students are randomly assigned into two groups:

• AI Group feeds prompts to the AI software of their choice, and then tries to make that AI material sound as human as possible to trick the instructor into thinking it's human-generated.  The goal is to get the highest possible grade on the rubric.  This group must keep a record of the AI discussions that produced their paper.  They are not allowed to modify the final text from the AI before submitting it, but they are encouraged to have the AI produce multiple versions until they get one that sounds the "most human."

• Turing Group writes the best non-AI material they can in an effort to get the best possible grade on the rubric — and they get bonus points if they can trick the instructor into thinking the paper was composed by AI.  This group must do all their composing in a single document with a verifiable version history.  It is not required that each student write in the document — only that every student contributes to the process of that writing.

Depending on how you want to run the activity, it could be that each group composes a single paper, or that each student writes a paper based on their group.  Another variation is to have each group submit both a human paper and an AI paper.

Stage 3: Instructor Workshop

In this stage, the instructor workshops both papers in person while students watch.  The instructor will go through each paragraph of the paper, and then describe how that section relates to the rubric.  The instructor will also point out any flags for AI, such as hallucinated sources or writing style tells.  For both papers, the instructor will decide if the paper would be graded as "AI" or "human" — and to what degree.  Whether the instructor is "right" or "wrong" is part of the activity.

Attendance Grading via Worksheet: Because this section of the activity is primarily the instructor's show, students cannot be penalized based on "participation."  The most important thing is that each student is paying attention, and a worksheet can provide focused questions for each student to answer as the instructor describes their observations.

Stage 4: Student Reflections

After the submissions have been evaluated, students write individual reflections on the composition and review processes.  These reflections can be either handwritten or typed with a version history.  They should include discussion of how the groups engaged in the composition process, how the instructor's workshop reflected those composition decisions, and whether there were any surprises.

Graded Writing: This part of the activity will be graded based on the quality of the individual student's observations, and it's best if this can be handwritten in-class immediately following the workshop.  In particular, the reflections should have detailed examples of the composition process and the instructor's feedback.  This is also a place for students to talk about interactions within their groups and raise any concerns they had about the group work.

This can be either in-class writing or homework.  As long as students are independently sharing their individual thoughts, that's what matters.

Instructional Notes

All group projects have innate strengths and flaws:

Group Projects Should Foster Collaboration . . . 

Collaboration is a key academic and occupational skill.  Personally, I think it's ideal if every class can offer some degree of collaborative work, but that's not always feasible.  This activity could take an entire week of class time, so it might not be possible to fit this in around other content material.  Also, we have to make sure that students are actually working together rather than simply doing their own individual things before throwing the project together at the very end.

Individual Effort Might Be Obscured

Some individuals may avoid work while others carry the grade.  That's why I feel that every group project must include individual grades.  In this project, the individual grades come from individual reflections that take place outside the collaboration process.  It's also important that the instructor talk to each of the groups and then observe them working in order to get a sense of how the work is being balanced.

Some individuals take over their groups, and others aren't sure how to contribute.  And this is a serious problem — a highly motiviated student may inadvertently (or even purposely) exclude other group members from the work in order to control the quality of the final submission.  Or students who are less assertive might not feel comfortable sharing their ideas.  For this assignment, the Stage 1 worksheet should guide students in dividing up the work equitably, but instructors must allow students the flexibility to change how work is balanced.  When meeting with each group, the instructor needs to individually ask each student for a description of their role.  Otherwise, students who aren't comfortable engaging may be overshadowed by those who are happy to continue talking.