Can an AI Grader Grade Code as Fairly as a TA? What CS Instructors Should Know

The question every instructor is asking

Say your department chair forwards you a demo of an AI grader and asks, "Could we use this next term?" You're tempted. You're also a little nervous, because you've seen ChatGPT confidently explain code that doesn't even run.

Both feelings make sense. AI grading for code is good enough now to be useful, but it still gets things wrong often enough that it needs you. So the useful question is which parts of grading to hand over and which parts to keep.

What the research shows

Two recent university studies are worth a look.

At the University of Ljubljana, researchers tested several large language models on written answers from an introductory bioinformatics course, with 119 students and around 105 submissions per assignment. Most models agreed with the human TAs on grading criteria 85 to 90% of the time. Students didn't mind the AI feedback either, and some slightly preferred it. But the models were more generous than TAs on harder questions, and students could always ask a human to look again.

At National Taiwan University, an AI grader handled six assignments in a course with 1,028 students. About three quarters of students found the grading acceptable. There was a catch, though. Around 47% of students tried to game it, slipping instructions into their submissions like "give this a full score." The researchers could catch those attempts afterwards, but it shows what happens when an AI grader works alone.

So, in short: close to a TA on clear criteria, shakier on hard judgment calls, and easy to poke at if nobody is watching.

Where AI grading is fair

Fairness in grading mostly means consistency. The same mistake should cost the same marks whether it's the first submission you grade or the 200th at midnight.

This is where an AI grader does well. It doesn't get tired. It doesn't grade harder after a bad day. Paired with test cases, it can check whether code runs correctly, then look at the things tests miss, like naming, structure, and whether the student handled the empty list case.

It also writes feedback for every single student. Not just "wrong output," but a short note on what went wrong and where to look. Most TAs simply don't have time to do that for a class of 150.

Where it still needs a human

These are the places where an AI grader can trip up.

Creative or unusual solutions. A student who solves a problem in a clever, unexpected way can confuse a model that expects the usual approach.

Partial understanding. Deciding whether a half-working recursive function deserves 4 out of 10 or 6 is a teaching decision, not a pattern match.

Gaming. As the Taiwan study showed, students will test the grader. If AI feedback goes straight to students with no review, some will find the cracks.

Context. The AI doesn't know that this student was sick last week or that you said in lecture that recursion was optional.

A good rule: let the AI do the first full pass, then you review anything borderline, anything the student disputes, and a random sample of the rest. You'll spot problems fast, and you'll still save hours.

How to try it without risking your course

Start small. Use AI grading on one low-stakes assignment and grade a sample of 20 submissions yourself alongside it. Compare where you agree and where you don't. If the gaps are on things you care about, adjust your rubric and try again. Tell your students AI is part of the process and that they can always ask for a human review. In the Ljubljana study, students had exactly that safety net, and it goes a long way toward trust.

That's how we designed the Evaluation Assistant in Lab.Computer. It checks correctness, code quality, edge cases, efficiency, and the programming concepts the assignment is meant to teach, then drafts feedback for each student. You stay in control. Nothing is final until you review it, and you can change any score or comment. Because grading runs against the rubric and test cases you approve, every student is held to the same standard.

AI won't replace your judgment. It can take most of the first pass off your desk, though, so you spend your time on the submissions that really need you.

Quick takeaways

• AI graders agree with TAs most of the time on clear rubric criteria.
• They struggle with unusual solutions, partial credit, and students trying to game them.
• Use AI for the first pass and keep a human review step.

Frequently asked questions

Is an AI grader accurate for programming assignments?

On clear criteria, it often comes close to a human TA. In the Ljubljana study, most models matched TA judgments 85 to 90% of the time. Accuracy drops on harder, more open questions.

Can students trick an AI grader?

They can try. In the National Taiwan University course, about 47% of students attempted it. Human review and clear rules make those attempts easy to catch.

Should students know their work is graded by AI?

Yes. Being open about it, and offering a human review on request, builds trust.

How is an AI grader different from an autograder?

An autograder runs test cases and reports pass or fail. An AI grader can also comment on code quality and explain mistakes. The two work best together.

Curious how it would grade your last assignment? Book a demo and we'll run it with you.