Linear algebra homework page
PDF page render / handwritten matrix solution / image input to model
Grade Arena Homework Benchmark
How well can LLMs grade university math?
Can an LLM replace a teaching-assistant grader? Grade Arena is finding out. We give models real student homework PDFs, then compare every score they assign—0, 0.5, or 1—with the score a PhD math grader gave the same work. The current comparison tests gemini-3.1-pro-preview and GPT-5.6-sol.
01 / Current Run
How closely does each model grade like a human?
We compare 447 task scores from each model with the PhD grader's score for the same student work. Every result falls into one of three simple groups: the same score, 0.5 points away, or 1 point away.
Green: same score as the grader
Yellow: 0.5 points away
Red: 1 point away
| Model | Same score | 0.5 away | 1 away | Same or close |
|---|---|---|---|---|
gemini-3.1-pro-preview 447 task comparisons |
87.2%exactly like the grader | 10.3%half a point different | 2.5%one point different | 97.5%same or 0.5 away |
GPT-5.6-sol 447 task comparisons |
74.9%exactly like the grader | 18.2%half a point different | 6.9%one point different | 93.1%same or 0.5 away |
More models coming same tasks, same comparison |
Coming soon | |||
02 / Subjects
Results by subject.
The percentage shows how often each model gave the same score as the PhD grader or was only 0.5 points away. Scroll sideways on smaller screens. Discrete mathematics has only 3 comparable submissions, so treat that result cautiously.
Subject
Works
Gemini: same or close
GPT: same or close
Intro Analysis
ВА / proof-heavy foundations
14
97.3%
82.7%
Linear Algebra
ЛА / matrices, bases, maps
19
100.0%
95.8%
Mathematical Analysis
МА / calculus and proofs
33
96.8%
95.6%
Elementary Mathematics
ЭМ / algebra and short proofs
15
96.8%
93.5%
Probability Theory
ТВ / distributions and expectation
14
100.0%
97.7%
Discrete Mathematics
ДМ / small sample
3
83.3%
83.3%
03 / Model Input
Input example.
The model receives rendered pages from student PDFs. These pages contain handwritten solutions, equations, corrections, and partial work.
Mathematical analysis homework page
PDF page render / handwritten limit solution / image input to model
Prompt excerpt
short version
What the model gets
The model gets rendered pages from the student's PDF and a grading instruction. It does not get the official solution. It does not get the original problem statement unless it is visible in the student's pages.
There is no detailed rubric for each task. The model assigns a score using the same coarse scale as the dataset.
You are grading a math homework submission. Input: - rendered pages from the student's PDF - no official solution - no separate problem statement - no task-specific rubric For each visible task, assign: 1 = solution is correct 0.5 = solution is partially correct 0 = solution is wrong or missing Return task scores and short comments.
04 / Problem View
Dataset coverage.
The dataset is mostly introductory university mathematics: early undergraduate courses that mix calculations with short proofs and require students to show their reasoning. Representative problems include proving that a sequence converges, finding the rank of a matrix or a basis for a vector space, and calculating a conditional probability or expected value. The table shows how many homework sets, student submissions, and individually graded tasks are available in each subject.
Topic
Homework sets
Submissions
Task labels
Range
Mathematical Analysis / Calculus
HW 1-9
41
331
limits, derivatives, sequences, series, integration, proof checks
Linear Algebra
HW 1-5
25
186
matrices, inverse matrices, rank, bases, linear maps, algebraic reasoning
Introduction to Analysis
HW 1-3
17
303
foundational proof-writing, inequalities, functions, sequences, epsilon-style arguments
Probability Theory
HW 1-5
17
103
events, conditioning, random variables, distributions, expectation
Elementary Mathematics
HW 1-4
21
146
algebraic manipulation, inequalities, functions, combinatorial-style exercises
Discrete Mathematics
HW 1-2
3
24
logic, counting, graph/discrete structures, short proof tasks
About
Grade Arena
A benchmark for comparing model grades with PhD math grader grades on math homework PDFs.