AI answers or AI hints? What a 1,000-student trial found
Students who practised maths with a plain chatbot did much better in practice and then worse on the exam: 17% lower than classmates who never had the chatbot. Students given a tutor version of the same AI, set up to give hints and not answers, largely avoided that drop. The study is peer reviewed and was published in PNAS in 2025.
The study in one table
| Title | Generative AI without guardrails can harm learning: Evidence from high school mathematics |
|---|---|
| Who | Hamsa Bastani and colleagues |
| Status | Peer reviewed. PNAS, June 2025 |
| People | Nearly 1,000 high school maths students in Turkey |
| Design | A randomised field trial in real classrooms, with three groups |
| Source | doi.org/10.1073/pnas.2422633122 |
The three groups
- Plain chatbot. An ordinary chat window on GPT-4. Ask anything, get the answer.
- Tutor version. The same AI with instructions to give hints and guide the student, not hand over the answer.
- No AI. Practice with the usual course materials.
All three groups practised, then sat an exam with no AI.
What happened
| Group | Practice scores | Exam, with no AI |
|---|---|---|
| Plain chatbot | 48% higher than no AI | 17% lower than no AI |
| Tutor version (hints) | 127% higher than no AI | The drop was largely avoided |
| No AI | Baseline | Baseline |
The authors' own summary is that unrestricted access to GPT-4 "can harm educational outcomes", and that students used the plain chatbot as a "crutch". The practice scores looked like learning. The exam showed it was not.
Why this result matters more than most
- It is an experiment, not a survey. Students were assigned to groups, so the difference can be put down to the tool and not to who chose to use it.
- It is large for this field. About 1,000 students, where many studies have under 100.
- It measured the thing that counts. Not how the practice felt, but what students could do alone afterwards.
- It tested the fix. The same AI, told to hold back the answer, did not show the exam drop.
Its limits
One school in one country, one subject, one short programme. The tutor's instructions were written for this study, so another hint-style tool might not do as well. A correction published in August 2025 fixed one author's affiliation. No results changed.
It also does not show that all AI use hurts learning. In another trial, a carefully built AI tutor for physics students at Harvard more than doubled learning gains (Kestin and others, Scientific Reports, 2025). Design is the difference.
A second study that points the same way
In three experiments with 1,222 adults (Liu, Christian and others, COLM 2026, peer reviewed), 10 to 15 minutes of AI help was enough to change what people did next. Those who received direct answers solved fewer problems on their own afterwards and gave up sooner. Those who used the AI for hints did not show this.
A fair one-line summary. "In a peer-reviewed field trial, AI that gave answers lowered exam scores by 17%. AI that gave hints largely did not."
What to do with this
You do not need a special tutor product to get the hint version. You can ask for it. Tell the AI what you think first, then ask it to guide you with questions. Here is the exact wording, or set it once so it applies to every chat.
Questions
Does using ChatGPT lower exam scores?
In this trial, practising with a plain chatbot that gave answers did: exam scores were 17% lower than for students with no AI. A version that gave hints largely avoided it. So it depends on how the AI is used.
Why did practice scores go up if learning went down?
Because the chatbot was doing the practice problems. A high practice score showed what the student and the AI could do together, not what the student could do alone.
Should schools ban AI because of this?
The study does not say that. It says an AI that withholds the answer and makes the student work avoided the harm. That is an argument for how AI is set up, not for whether it is allowed.