ChatGPT presents a serious risk to the integrity of university qualifications, as continuous-prose assignments are instantly produced by this online tool. ChatGPT constructs unique responses, so any output cannot be detected through traditional means such as Turnitin. As such, it represents a real threat to academic honesty. As I have been experiencing the tool, I have been pondering what we are valuing: the process or the product. This leads to discussion of what students need to be able to demonstrate where technology provides a shortcut to a product. It is fair to refer to ChatGPT as a disruptive technology, especially considering the changes to assessment that may result.

There are some ‘red flags’ I can identify as characteristics of the tool’s output. However, these are not failsafe, and you should build a convincing body of evidence before accusing students of misconduct. It is worth noting that you can use these insights alongside developing AI-detection tools to build your case. Very recent detectors such as huggingface.co, are said to be 60-70% individually effective at identifying ChatGPT assignments. The score is higher if you utilise more than one of these detectors. For example, you may choose to employ all these tools, and gauge their consensus on the submission if you need to develop an evidence-base.

The first characteristic of ChatGPT output is its propensity to construct run-on sentences. These sentences will also tend to combine multiple, and sometimes contradictory, ideas. Long-winded introductory sentences are constructed to provide broad context and may look particularly repetitive. A dot-point list of main points is generated. Finally, another broad summary, which usually mirrors the first sentence. The final summary has some notable absences and presents as a lack of critique, extension, or application. At the moment the tool is unable to construct assignments longer than 1000 words. It is possible for a student to ask ChatGPT to write sections, but as these will typically involve that formulaic structure.

You could ask students to only engage with sources from the last twelve months. ChatGPT will offer nonsensical or outdated responses, as it is not yet able to access information post-2021. Eventually though, ChatGPT will be able to access the web in real-time.

You might think that asking for specific referencing or formatting requirements would negate ChatGPT’s use but ChatGPT can follow referencing instructions. However, it will do this to a fault. It will generate parts of a reference list that it does not have access to and DOIs might be broken or false. Each time it is asked to edit a document, it can shift or replace in-text citations, so a citation may no longer attribute the correct source. If you ask it to include in-text citations it may simply make some up. You may notice a strange pattern of article and journal names in the reference list. These could be ‘fake’ sources that piggyback off a genuine source.

On the positive side, ChatGPT has the potential to assist in the generation of teaching resources and as a learning tool. For example, to develop an essay plan, provide feedback on an assignment, and prompts for discussion. Students may input their assignment questions and analyse the output against marking criteria. They could identify what ChatGPT does not answer, or where there are limitations in critical thinking or extension and upload the original ChatGPT output with their redrafted submission.

My initial engagement with ChatGPT left me convinced that this is the most disruptive tool for education in modern history. However, I am reminded that the calculator was initially treated with suspicion and even panic. After spending time with the tool, exposing the boundaries of possibility, I have come to view it more like the ‘homework machine’. We can try to avoid ChatGPT, or, as some brave innovators are doing, utilise it as a pedagogical tool. Our response might involve a shift to making the learning process assessable, not just the final product.