back
729 comments
In the AI era, tests are going to have to be in person and hand written.

I have written an article on how I have adjusted my classes to the situation:

https://htmx.org/essays/universities-and-ai/

Ironically, I think the AI era may make university degrees a better signal of the intellectual abilities of students due to the presence of pre-computer infrastructure like large lecture halls, industrial-scale copiers, etc.

Using a special computer works too. I do my exams with our institution's Computer-Based Testing Facility, a bank of computers with fixed software and firewall rules that only permit connections to the exam site.

As a result, I've been able to challenge students to solve interactive software security challenges on the midterm and final with automatic grading - something that would have been impossible with pen and paper.

Scalability is really the major challenge. We're rolling out more CBTF rooms and rolling out access to other departments due to demand, but it's definitely more resource-intensive than pen-and-paper. One possibility is to treat CBTFs as computer labs when not actively administering exams (or maybe even vice-versa), something we're looking into doing.

> In the AI era, tests are going to have to be in person and hand written.

Forcing hand written should really not be necessary. It would be very cheap in terms of old computing hardware to set up a test room with old desktop PCs that have wired only NICs (with a network connection that goes to a switch in the same room with no uplink, connected to a decent size laser printer only), running something like lubuntu and libreoffice writer as a basic word processor.

Let people at least type their essay with the standard features of a word processor as usable as MS Word 2000 or better.

I'm currently getting my Bachelor's in Computer Science, while in-person is absolutely necessary, it doesn't necessarily need to be hand-written. We now have the exam questions printed out, and we respond on the online exam platform, just like we would before. It is explicitly stated that no aids, including AI, are permitted. There's usually the prof and at least one more TA walking around the room, if anyone is seen with something other than the exam platform open they fail the exam.

Some profs also started requiring special software for doing the exams, which works fine but is pretty annoying to use since it requires Windows, but that application basically hijacks the OS, making it impossible to navigate to any different webpage/ application.

In 2024, we had some coding exams, and AI was explicitly NOT forbidden, which lead to them making the exam more difficult (because with AI, you can solve it so much more quickly, so gotta make it harder), which was really annoying, because it made it practically impossible to solve it yourself because of the time crunch. I hope they realized how wrong that was.

Thanks for the very interesting write up, I am teaching management studies programming Python (people who want to be quants, dana analysts, etc.) and I am struggling with the same problem - how the hell do grading to make it reliable.

LLMs has become good enough so whatever homework task I give, it can be solved easily by any LLM (and, in fact, this becomes unfair to those who are not able to pay for good model). I was asking to add comments & interpretation of results, this slightly helped, but LLMs are increasingly good in all this as well.

So, quite seriously, I am considering some on paper tests & quizes, because what else can be done?

Making people aware about importance of writing code is indeed a good hint to convince, at least part of the people to do this. Another thing is: they will leave university one day, they will search for employment and the employer might be much more hostile towards cheating during job interview and can easily make cheating with AI impossible... Question is, will it still matter?

As a student, I have no issue doing oral exams or written exams without notes. I mean I'm there to learn and out of curiosity so I like that challenge...

I truly don't understand how people can sign up for a degree and then have to cheat? Must feel torturous to endure a class you're not even interested in.

There are limits to what you can assess on timed assessments, and there are students whose performance on such assessments is not a good signal of their intellectual ability.

In addition, at many institutions such tests are given infrequently and can be worth a significant component of a student's overall grade, increasing both student stress levels and the tendency for such assessments to measure short term knowledge students have obtained by cramming, not more meaningful longer-term knowledge gains.

I see you're giving quizzes every three weeks, which is better than twice a semester, but still not what I would consider an ideal cadence. In my course weekly computer-based quizzes comprise 70% of a student's grade, but that's supported by a significant institutional investment in high-frequency computer-based testing: https://cbtf.illinois.edu/.

During uni I only took a Java class. During the exam I had to turn in hand written code. I guess that would work today.
I agree that that is probably the lowest stake solution. Alternatively there are solutions like the safe exam browser which locks down the device quite well during the exam session.

—- Disclosure: I run a small start up which offers teachers a platform to create and conduct digital exams and interfaces with the safe exam browser precisely because cheating is the number one complaint teachers I have spoken to have when it comes to digital exams.

I think most teachers have adapted pretty well. I'm really surprised to find teachers that haven't reckoned with possibility of AI cheating in 2026.

In person writing, etc. But also for anything take home students have to verbally discuss their work in some fashion. Seems to mostly work. Students still us ChatGPT as a search engine, which seems fine.

(Source: married to a professor. And my son is in high school)

I was born in 1970. My teenage years were in the 1980s, in coincidence with the emergence of the home computer. I had a Timex-Sinclair 1000, then a C64, then a C128. One of my main forces that drove myself to learn computers was (besides games) the ability to produce text that was clearly written and readable, thanks to word processors and printers, because my handwriting was abysmal, and I got tired very quickly while writing by hand. In the computer, instead, I got to type exactly what I wanted, I could edit and correct my spelling and grammar mistakes, and then produce very readable and clean output. A win in my book.

Now, I require of students to submit written assignments done by hand. That way, I can at least be certain that there's some learning involved, even if they resort to AI to produce the relevant written part, because evidence points out that writing by hand reinforces learning.

I read that other professors resorted to requiring manual typewriters, which I also hate with a passion.

That is, AI is negating decades of enablement achieved by technology.

Honestly, between these circumstances and the fact that we are "enjoying" (?) brave new prices for RAM and SSDs, I'm finding AI increasingly harder to like.

I have seen it firsthand in the CS department here at Dartmouth. It is bad.

We're currently designing a new intro systems curriculum, and we're thinking of it as an adversarial problem. That is, we're designing the course to ensure that a student optimizing for the best grade per unit work still meets our learning objectives. That means, as everyone else is saying, paper exams, but also 1-on-1 interviews to check that students understand each assignment they turn in. These interviews feature both factual questions ("You're using this macro from that library. What does it do?", "Please describe what this function does and how it works.") and conceptual questions ("Why is this code structured this way instead of $whatever?", "How else did you try solving this?", etc.) This doesn't stop students from generating code, but at least they have to understand that code in detail.

This is not as good as writing the code yourself, but how much worse is it? For math classes, this gap is gigantic. Obviously, understanding someone else's proof is much easier than writing your own. For programming classes, I think (without evidence) that the gap is somewhat smaller.

My experience from the past is that when this kind of evaluation is made clear up front, the students know what to expect and either do fine or drop the class in the first week. If you start with take-home exams and then spring paper exams on them halfway through the course, then half the class is cheating and won't be able to recover, as we read in the article.

In general, our students are somewhat motivated by an abstract desire to learn, but are much more motivated by grades. If there exists a straightforward path through your course that leads to a good grade without doing much work, most students will take it. (Our undergrads' course review website is literally called "Layup List." They are actually this shameless.) It's our job as instructors to ensure that all paths leading to a good grade either require learning the material or are more difficult to pull off than just learning the material.

It's best not to blame the students. They are good at optimizing metrics; that's how they ended up here in the first place. We just need to better align the evaluation metrics with the outcomes that we're looking for.

> When he reported the case to high-ranking officials at Brown, he got a cold reaction. The response from the president, he said, was absolute silence. The dean did not comment either until Serrano took the case before the Academic Code Committee. At that point, he received a note acknowledging that what had happened in his classroom was “a wake-up call.” Serrano, a Madrid-born economist who has been at Brown for 34 years, believes this is not enough. “That cannot be the university’s position before an incident of this magnitude. Academic integrity is a value worth defending. The faculty cannot be left on its own

Is anyone surprised that the academics are essentially being left on their own about is? University management has become increasingly disconnected from the academics doing the work on the ground at least during the last 20 years that I've been in academia. Covid was really an eye-opener in this respect, at our university academics were told with very short notice to make all lectures remote. There was essentially zero support from central management, there was not even budget to buy headsets or cameras and academics were expected to use their research funds.

At the end of the pandemic academics were thanked with two years of 0 and 1% salary increase, due to a tough economic situation. During the same time the president received a 8% raise. The tough economic situation was entirely due to mismanagement, i.e. the university had invested pension funds into fixed interest investments which did not keep up with pension growth (during a time when stock markets were going from strength to strength).

When you're a student in a competitive program at a top university, graded on a curve, and you know your fellow classmates are cheating with AI, you have little choice but to do the same. Especially when jobs for new grads are harder to come by and there's more pressure to also go above and beyond with internships and side projects during your time in school. There's no way to compete without cheating.
His research is in Game Theory. He should have realized that, in a situation where all competitors are (possibly) using LLMs, the game theoretic optimal choice is to use LLMs.
One thing I would say from the student perspective is, when you know for a fact that your fellow students are using LLMs, and you feel like you can do it honestly and earn a B, or use LLMs and get an A, it msut be a tough decision. Grades are often adjusted on a bell curve and honest students are disadvantaged when others cheat. In my undergrad I was leading in nearly all of my classes for 7/8 semesters, but the final semester had online exams due to covid, and I was suddenly a barely above average student in the last semester.
The last take-home test I did was for EE364a: Convex Optimization. It was a 24 hour test, and I had a cold. I booked a hotel room as my apartment didn't have air conditioning. It was brutal. I got most of the programming questions correct, but only a few of the proofs. The average of the class on this test (and most every other assignment) was 80%+. I got an A- overall in that class.

Perhaps this class isn't too hard for Stanford students, but I have to wonder about cheating when the averages on nearly every single assignment is that high. One clue was office-hours: Whenever I dropped by office hours, there was always a line (didn't matter the class). I quickly learned that TAs would often drop a hint that would make particularly hard problems easy to complete. It provided an unfair advantage to students who could attend office hours.

It also reminds me of the huge scam of cheating on the USMLE amongst Nepali medical students: https://www.medpagetoday.com/special-reports/features/113627

I've met many wonderful international medical graduates. Many have shockingly high USMLE scores. It is true that there's no time limit on their preparation (U.S. students have under 2 years to prepare during medical school, international students may wait years after graduation before taking the exam). Before that scandal, I never would've thought someone could cheat on the USMLE. Prometric test centers are crazy locked-down. But that's not how they did it. They did the long-game. Prior test takers would remember a handful of questions and just add them to a secret database. After many years, that database contained nearly every question on the exam. Test takers would work extremely hard - memorizing every single question. The reward of a U.S. residency is life-changing, I get why cheating was(is?) so rampant.

take-home, closed-book type

What an oxymoron. I agree with the others here that AI isn't the problem.

I think one obvious challenge when it comes to Ivy League, and other prestigious schools, is that they attract very ambitious students - likely over average intelligent, too.

If you're the type that applies (and is accepted) to those schools, you are likely very informed on careers out there. So you also know what is at stake.

Good grades, or at least above a certain cut-off point, will open doors to prestigious jobs, as well as further studies. Finance, Law, Tech, you name it.

For these students, the stakes are high - and more akin to professional sports and draft season. I'm not saying this as an excuse to their cheating, but rather what the reality is for them. Again, not only is your competition smart and hardworking to begin with, but this sort of cheating is basically equivalent to academic and intellectual PEDs.

You could be studying English at Brown, with the intent to land a job at some management consulting firm or bulge bracket investment bank (1 out of 4 students at Brown end up in finance and consulting). Work that is miles away from your major, and where you're being provided the necessary training when you join the firms.

It is stupid, but understandable. And if you know others are doing it, it really only impacts you negatively if you don't. When so many of your opportunities come down to a two decimal number, people start taking risks.

As a university professor, I honestly don't understand the point of grading. Who will look at and care about grades? Likely company HR. But then why should we (professors) do the screening for companies for free? Also, grades have long been inflated to a point we might as well just give everyone an A and let companies figure out how to select people.
These articles consistently fail to acknowledge students were cheating in large numbers prior to these AI tools being available.

It was certainly not difficult to cheat at a "closed book" take home exam before.

> In the AI era...

Back in my day, you could also just Google the problems and find the solutions. What mitigated cheating at UVA was the honor code and each professor's faith and trust in our integrity. That culture was enough to not cheat.

Imo, the fix should be to work on culture. Cheating should always be a tempting choice, so that the student may challenge their integrity, which is a muscle that can atrophy.

Damn that's crazy. Guess the take home test is dead now.

I never understood this behavior from undergrads though, you're paying so much for an education and then you just skip the education part? Why bother?

While I am in no way a supporter of AI cheating, or whatever we want to call it, I can tell you from experience that there is nothing more tedious or soul destroying than invigilating a written multi-hour exam. It put me off teaching in higher education.

IMHO to solve many problems we should go with Ivan Illich's ideas: https://en.wikipedia.org/wiki/Deschooling_Society and make education about education, not testing and certification.

I was home schooled - so my perspective of education is at best, unique. I finished a dual major undergrad at 18, got an MBA at 21.

Based on my own experiences and those of people I’ve watched, I have three points. Sample size of 1 and all that; but it’s a lowkey passion topic for me.

- The argument that AI is “the next calculator” and education testing and overall methods need to adapt, is true.

- The distance between learning concepts to passing memorization tests(or worse, non-reality test specific logic) has grown significantly in my lifetime, and the AI education problems are really a mix of the cascade impacts of this issue + the AI impacts on conventional education measures.

- The problem with students who use AI to get past (often arbitrary) difficult courses or testing scenarios may look like one problem compensating aka solving for another (and I love AI for it!) but the parallel problem of AI (and our education system baseline) enabling students to normalize non-cognitive drone like approaches to any problem is super problematic.

The last point is an admittedly recent eye opening moment for me. Working with younger students recently, they are shocked at anything less than a fully clearly defined problem (education system training) and anytime thinking is required, they go to AI (which is fine!) and they can’t think beyond the AI output (which is super not fine!) The latter has been astonishing for myself and coworkers and really has us reconsidering our young talent programs.

This is not surprising. While cheating has always been around, it seems to be more prevalent now with high pressure and easy access.

I’ve talked to a bunch of teachers and school leaders, and see three main ways schools are handling AI use in assessments:

1. Punish it: Detect AI use on homework and take home exams; treat it as cheating.

2. Prevent it: Move to live assessments – oral or offline – that are hard to cheat on.

3. Embrace it: Assess the process, not the output.

The second one seems to be the only real answer for foundational subjects. And the third one can also work for more creative or project-based work.

let people use as much ai as possible. encourage it. and as an educator, you have to learn to leverage it oneself or not (depending on the subject). and be better at using it than students if it does make one 'better'.

if ai doesn't help, then it won't help. if it does help, then you should use it. the metric is your output of whatever is being tested. writing an essay well and clearly understanding the material. solving a pset. whatever.

if you give access all the time for that, and then you test on a hard problem that could be done with or without ai, then it's fair. e.g. "clearly explain these four sentences of Y." obviously ai researching loosely and blathering isn't useful. won't be high signal / dense and correct and worthy of an 'a'. but someone who can harness ai and someone who knows the material well in the end will be rewarded the same by society. what you are testing is correctness and information density in a response. so you have to start now in accepting the reality that those who use ai to get there should be rewarded just the same as those who don't.

the burden is on educators to be as good as they can with ai if it is relevant or not if it is not relevant (and schools to fund them and ai companies to fund them if they have excess capital and are humanitarian).

and note the hard part even for us engineers at tech companies is in the correctness. it is very hard. but the sooner we start teaching how to do things correctly with ai, the more prepared the next generations will be.

"The course, which he has been teaching for years, is not an easy one: it typically attracts few students, but very good ones. He has never had more than 30 students enrolled at a time, and on some occasions he had only eight. This semester, probably because of the new evaluation system, 86 students signed up for the class. The results of the midterm exam, which was administered on March 5, were extraordinary, with an average score of 96 out of 100. Forty students scored a perfect 100. The people who corrected the exams warned him about several irregularities. "Some answers contained unusual passages that coincided with results obtained after running the questions through ChatGPT," he says.

Serrano did not void the midterm exam, but warned students that the final one, which counted for 50% of the final grade, would be held in-person. He also said that if the grade distribution was not similar to the midterm, only the final exam would be taken into account. The average score dropped to 48 out of 100. Of the 89 students who did the midterm exam, only 59 showed up for the final one. And of the 27 who did not show up, 22 had scored a perfect 100 in the midterm exam."

When I visited Yale recently, a professor who taught comp sci complained to me that most of his students were using AI to do the work. I asked him if he knew which students wanted to learn. He said yes. I suggested he teach to them, and to heck with the cheaters.
>He has conclusive evidence that at least 50 students cheated on the March midterm exam

Running all your students exams through an AI checker is one of those Temptations of AI I am sure.

I'm from Hungary and the majority of the exams here are oral one-on-one interviews (depends on the course of course but still). I've never ever had take home exams and or even quiz like tests were very uncommon.
Most people here seem to thank that the only realistic alternative for submitting code / written reports in the age of AI is handwritten invigilated exams.

So just mentioning my own anecdata here; I've had reasonable success converting a "code + written report" assignment into a "code + structured video report", where the code is assessed "exclusively" through the video report, and where the student needs to demonstrate ownership and understanding of the code, explain and defend design decisions, etc.

Yes, in principle it is possible for the student to generate a "script" and narrate it during the video presentation, but this is fairly easy to detect, and in general, given enough flexibility / subjectivity in the marking scheme, you can always downmark such students on the basis that their presentation demonstrated lack of understanding and ownership of the codebase, without having to go into too much specifics.

Obviously one implication of the above is that you no longer care if their code was "manually" written vs AI "assisted"; only if they have a good grasp / ownership of it, and can defend the broader architecture and design decisions. But I feel that this is realistic in the age of AI, since this is largely what will be expected from their future employers too.

"Take home closed book" exam is asking for mass cheating. It was probably happening way before AI too. AI is just an excuse now.
He is fighting a losing battle. That's why nobody in the administration cared. Academic integrity already doesn't exist anymore. As Nassim Taleb would say; there are plenty of highly educated idiots with PhDs.

Even the idea that MIT is somehow better than some other universities is itself a fiction. People are conflating financial success with academic ability. The former is mostly the result of social connections which are formed within the academic institutions and have very little to do with actual capabilities.

Universities should just sell degrees for a high price without requiring the students to attend. If they're rich enough, their skills aren't going to matter anyway; they'll succeed in their careers regardless so the university will still look good. 'Academic integrity' will be intact. Especially true for business, economic degrees or other humanities.

Law schools are entirely essay driven. They’ve had armored word processors for testing for at least twenty years. I’m sure those companies would license a version for the rest of campus (most universities have a law school already so they have a contract they can simply extend to the whole campus)

I seem to recall they submit the exam via usb drive.

now they may have a closed exam mode where they can finish the exam without network and the. Submit over the network.

Of course this type of exam must be conducted in person or else the examinee could simply use a phone and type the ai provided answer.

Also is the professor living under a rock? Everyone is going to use AI if they can. So if you want them to not, asking them nicely isn’t going to cut it.

I took an EE course last year that had a pretty effective way to assess, in my opinion.

Exams are definitely in person. Homework is not graded, except that once a week, one problem would be randomly selected from the homework for an in class quiz. The numbers would be changed up.

From my perspective, you didn't really have a choice but to be able to solve all of the problems on the homework unless you wanted to gamble failing that week's quiz. The answer key and AI are always available, but the learning got to happen on my terms. It was a good balance. And grading for the professor was relatively light since it was only one problem per student.

I had a couple of courses that had open-book exams, where you could bring _any_ material you wanted. Any books, notes, equipment, really anything you wanted. The exam was still hard, because it focused on defining problems. Knowing what question to ask. This is still the hardest problem, even with LLMs. My guess is that most teachers don't like open-book exams, because they are hard to make. But maybe LLMs could help with this too. There is no putting the genie back in the bottle anymore, and restrictions are only going to be effective for dumb cheaters. You have to create exams which make the smart cheaters do actual work.
> This year, the economist decided that both the midterm and the final exams for his course would be of the take-home, closed-book type (there is a certain tradition of this at Ivy League schools). “It’s a very nice kind of exam, because as you’re giving students practically unlimited time to complete it, it lets you make it harder than normal, to see how far they can go.”

I have no doubt that AI was used for this, but I am shocked that this is even a thing. Before AI, any rich or connected student (which I assume is essentially all ivy league students) could just get someone to take the test for them?

"take home, closed book"

This is a trap. I understand they've done this in the past, but profs are paranoid now.

I don't believe he's 100% correct on each incident of fraud and he's going to ruin students [academic] lives because of it.

"This year, the economist decided that both the midterm and the final exams for his course would be of the take-home, closed-book type (there is a certain tradition of this at Ivy League schools). “It’s a very nice kind of exam, because as you’re giving students practically unlimited time to complete it, it lets you make it harder than normal, to see how far they can go.”

...

"But it also hurts him that the one time in 34 years that he decided to offer a take-home exam, for highly justified reasons, the response was wide-scale fraud."

-----------------

Not to in any way defend or condone academic misconduct, the fact that this was his teaching-career-first take-home exam is probably relevant. Take home exams can be fiendish. I remember having one in grad school where we were given a very insufficient 36 hours to complete it, and many people just didn't sleep. That was from a prof who knew what he was doing. This guy may have accidentally made his exam absolutely sadistic.

Couple this with the fact that students often have other exams they need to be studying for in the same time window. The pressure can be immense. The temptation to use AI to help is going to be hard for many to resist unless the penalties are severe and strictly enforced.

AI cheating is probably going to be a problem going forward in all situations, but open-book, take-home tests are going to bring it out more strongly than other test formats.

All these over-complicated "solutions" are incredibly funny. At my uni in Germany education worked in a very straight forward way. There is no graded homework (you're in a university not an elementary school), at the end of the semester there's a three hour exam, on pen and paper (our CS profs deducted a point per syntax error btw so mind your parentheses) and if you don't do any homework or don't show up that's on you because you're an adult, but good luck making it through the test.

Of course 70% or so usually crashed out in particular in Calculus and I suspect given that US education is paid for daycare that's exactly the thing that can't happen which is why they're never fixing it

In the 2010s all of my exams were still hand written and I went to Brown.. has everything really been computerized that much?

I think the tricky part here is essays, but maybe they should be replaced with an oral exam + essays? A lot of the humanities and classics courses I took were pretty small.

Overall I don’t think these problems are that hard, I just don’t know if the incentives of universities align with fixing them. If their reputation is all that it’s for, grades don’t really matter as long as the $$$ is coming in.

I get why AI might be detrimental to the students especially in competing fields or classes, but they have a high chance to fail in life because they can't rely on ChatGPT or other AIs their entire lives and be successful.

But being upset that people use available technology to solve problems is quite an exaggeration and makes the guy close to being a luddite. He can just say "Hey, we do exams on paper and in class next time" and be done with it if he does not like technology.

The number of people here defending AI cheating as an alternative/clever form of "learning" is eye-opening. Also the people saying it's not the students fault they cheated, it's the systems fault.

All I know is I wouldn't want one of these cheaters performing open heart surgery on me, designing the airplane I'm flying in, or managing my financial affairs. Or marrying my daughter.

I would have thought the defense of cheating to be a much more marginal position.

Administration needs to eschew "technology" and demand analog solutions: hand written exams in proctored rooms, no devices out in the classroom, no take home work, etc.
> He has conclusive evidence that at least 50 students cheated on the March midterm exam, making it the biggest known scandal at Brown and in the entire Ivy League

I'd like a citation for this being the "biggest known scandal" in the "entire Ivy League". Frequently such situations are kept somewhat quiet, for a variety of reasons. But fifty students is not a large number in courses that can enroll hundreds or up to a thousand students.

Nobody would hire a chess coach and then use Stockfish to cheat on the problems. Whatever the students are paying for here, it certainly isn't an education.
Another amazing benefit of AI, the increased expense of training and teaching.

I’ve studied in classes of 40 to 50 students to one teacher. The challenge of grading students at that ratio, means you need automation. It also reduces the kind of teaching possible. You really can’t provide one on one attention when you have multiple classes of 40+ students to teach.

I think the irony here is that LLMs are the ultimate tool for auto-didacts and people who love to learn independently, bar none. Using them to cheat is such a profound waste to me. I've beeen able to accelerate my learning thanks to LLMs, and it saddens me that the potential for this kind of personal enrichment is lost on most.