Like many educators, my first instinct after encountering ChatGPT in late 2022 was to see how it performed against my own assessments. I asked it to answer my quiz questions and watched it generate confident, articulate responses. Some were wrong, but many were disturbingly accurate. As a researcher with an interest in academic integrity and risk management, I recognised this as a vulnerability problem that goes beyond whether AI could easily, confidently, and consistently answer my questions correctly, but what to do about it.

This recognition strongly connected with the TEQSA's 2025 statement on alternative approaches to complementing academic integrity processes. Specifically, the notion that assessments should be designed where generative AI use becomes irrelevant to demonstrating learning outcomes. But how do we operationalize this guidance when AI capabilities evolve faster than our assessment redesign cycles?

The challenge facing educators is fundamentally one of pace. Students gain access to increasingly capable AI tools overnight, while institutional assessment review cycles take semesters or years. Traditional immediate responses of increased surveillance or reversion to face-to-face exams have considerable scale implications, while ignoring some of the questions about quiz design. The question became: could we leverage these tools to help craft less vulnerable assessment designs?

QVAF: Decision-Support, Not Prescription

The Quiz Vulnerability Assessment Framework (QVAF) emerged from this question. The QVAF is a tool that tests online quizzes against AI models. The tool (available as open source on GitHub), runs locally so the data never leaves educator control. QVAF aims to assist educators in assessing and addressing online quiz vulnerability by providing objective measurements for professional judgment, not prescriptive recommendations alongside recommendations for reducing the vulnerability of individual quiz questions.

The QVAF allows users to assess and address the vulnerability of quiz questions one at a time. At scale, the tool can link into your learning management system (currently limited to Moodle) and assess an entire quiz against an AI that relies only on general knowledge. Additional tests can be run against an AI which uses uploaded course materials to answer questions, with the additional functionality to compare responses with and without the uploaded course materials.

As an analytical tool, QVAF measures how correctly AI answers, how confident it is in that answer, and how consistently it responds across multiple attempts. The tool then classifies questions by cognitive demand level and generates a dashboard showing vulnerability patterns. It tells educators which questions AI finds easy or difficult, leaving the decision about what to do with that information squarely with the educator.

A question AI answers correctly without course materials suggests testing general knowledge AI already possesses. A question AI only answers correctly with course materials indicates a different vulnerability profile, one where students uploading course materials to the AI increases the effectiveness of using AI to cheat. A question AI struggles with regardless of resource access may be naturally resistant to this form of assistance.

Building Through Vibe Coding

I built QVAF with minimal formal coding knowledge, primarily through natural language prompts to AI coding assistants. This approach, sometimes called "vibe coding," relies on describing what you want the software to do rather than writing traditional code. Most of the code emerged through conversations with Claude, where I described functionality and the AI generated implementation. I reviewed, tested, and iterated, but the barrier between concept and working prototype collapsed from what would traditionally require months to what took days. The democratization of software development through AI means that the same capabilities students’ access can empower educators to create solutions matching the pace of technological change. This matters because it demonstrates that educators don't need to become programmers to build decision-support tools. If I can build QVAF, others can build tools specific to their contexts and needs.

This project is not just about the QVAF itself. It is also a demonstration of what becomes possible when educators harness AI and build with it. When we can build and iterate quickly, the distance between an idea and a functioning prototype collapse. This shifts our relationship with technology from passive users to active designers of our own teaching infrastructure. In a landscape where the pace of AI adoption outstrips curriculum review cycles, this ability matters. I have no doubt there are many other tools waiting to be created that uncover previously invisible patterns.

In this example, we can explore assessment vulnerabilities before they surface as misconduct cases. Rather than waiting for patterns to appear in student behaviour, we can anticipate those patterns and redesign with intention. In essence, operationalising TEQSA’s guidance becomes less about restriction and more about capability. We can create assessments that are less vulnerable because we can evaluate vulnerability in advance. And we can do so without abandoning online assessment or defaulting to surveillance-heavy approaches. The same tools that enable students to circumvent poorly designed quizzes can also help us design better ones. Try QVAF with your own quiz questions and see what patterns emerge. More importantly, consider what decision‑support tools your discipline needs, and experiment with creating them. Share what you develop so others can adapt, iterate, and extend your ideas.

Banner Image source: ChatGPT