back
25 comments
Worrying that AI might make exams obsolete is kind of odd, I mean, it is a symptom I guess but only at the very end of a long stupid cascading failure.

Students cheat because they want the degree but don’t care to learn the material. Or maybe they want to learn the material, but see employment at the end as requiring better grades than they can get naturally. Either one is the result of bullshit credentialism. (Bullshit credentialism probably comes in part as a result of bullshit jobs where work-product can’t be evaluated because it’s all useless).

Hopefully students manage to cheat on so many tests that grades can become completely useless for employers. Then, they can become something useful for the students, a way to evaluate their progress and get feedback.

> Then, they can become something useful for the students, a way to evaluate their progress and get feedback.

That's exactly why it'll become useful to employers. The degree to which student evaluations are frank assessments is the degree to which employers will value them even more. The more honest schools are the higher stakes they become.

If you want to lower the stakes then increase the number of nice seats in the economy.

Grades are something useful for students as a way to evaluate their progress and get feedback.

Anytime you implement a way of demonstrating someone's competence on a scale (GPA etc) you will have some companies that target the higher end of that scale.

I don't think cheating will result in grades becoming something useful for students. If they're useful for students in showing how they demonstrate performance, employers will use that to set a baseline for expected competency.

The only way to remove that is to not have a GPA type metric but more of a pass/fail metric that puts all graduates on an equal level.

Credentialism isn't really bullshit. For any decent entry-level job there will be an enormous surplus of applicants (elite overproduction). It isn't practical to evaluate them all in detail, the hiring managers simply don't have time. So they fall back on simple, somewhat arbitrary factors like educational degrees, school rankings, and GPA. It's imperfect but what else can work at scale?
Sounds like a "just-so story."

Credentialism is still around due to inertia, not any conscious thought or decision-making. Tell your conservative, F500 director you're going to get rid of degree requirements and watch your career evaporate.

Nah. Conservative F500 directors are quick to eliminate degree requirements as soon as they have trouble filling a critical req at an acceptable cost. Despite what you might think of those managers they are not financially illiterate.

https://www.cbsnews.com/news/college-degree-job-requirement/

There's so much wrong here; from including a propaganda mill as a source, to that source not affirming your assertions, to not lining up at all with what I've experienced.

Yes, small and medium sized business are slowly removing the degree requirement, but there is no evidence for this in ultra-large international businesses. I'm curious to know how many directors you know at F500, because from my experience they get much less latitude for creativity and risk than their smaller and nimbler competitors.

N.B. No one hires "critical reqs" through normal channels.

My experience is different. Know a bunch of F500 directors. I think you're simply wrong in the general case, although you can certainly find some organizations or individual hiring managers where that fits.
Very well, then I'll have to be more conscious of the mental model I've built, and test it again.
This says more about the exams than the AI or the students…
For a time teachers resisted calculators because they learned the hard way. But learning with a calculator doesn’t mean we suck at math, we can still learn what’s behind it. AI have to be a tool for learning better and faster and expand our reasoning abilities. It’s of no sense to be stuck in the past and resist it. We have to develop a more critical thinking, use AI in schools and criticize the output and make something better out of it.

And we have to change the way knowledge is assessed. Teachers have to work hard to make better exams.

I hope it will destroy the stupid tests where students have to learn like parrots for a set date then forget everything after. It’s so stupid.

I agree 100%. This AI wave hopefully if will force universities to rethink the way students are assessed. When I was faculty I had the luxury to teach small classes, and I could always be extremely flexible with assessments so I could assess students as the course progresses, or I could make it project-based, or a 1-on-1 oral exam. But I had colleagues that were teaching a class of 300 students, which meant most of the exams were looked at only by teaching assistants, or they were so heavily automated (e.g. multiple choice) that these days they could totally be fooled by an LLM. Something needs to change and it will (fingers crossed).
Not really. The point of an exam is to assess whether or not the student has learned the material.

That an LLM can answer all the questions in 2024 is not really all that much different from how it was circa 1980 when probably every integral on my calculus exam was in a table of integrals that I could buy at the college bookstore.

It's just one more thing to add to the list of things that you need to prevent students from using during the exam. In 1980 that included that table of integrals on calculus exam. In 2024 it needs to include LLMs.

This fits well with what we know about AI and Bloom's Taxonomy of Learning, which goes from 'remembering' on the lowest step to 'creating' on the top step (remember, understand, apply, analyse, evaluate, create).

Undergrad exams are usually somewhere around 'remembering', simple fact or definition regurgitation. Most of these facts should be in chatGPT's training data. As the degree proceeds things get harder and we move up the taxonomy, and that's where we know LLMs fail: there's nothing in there that can really 'understand', let alone 'create'.

Lesson learned, don't go into psychology.
This is just a way more reactionary way to communicate model benchmarks.

“A computer algorithm performed better than humans on a task it was designed for” sounds like the last forty years in a nutshell.

The big question remains - "Is there any benchmark at all on which algorithms can't pass human performance given sufficient resources?". If I were a being man, I would put my money on 'no'
"Given sufficient resources" is the key phrase here.

Humans are ridiculously energy efficient. We are capable of extremely complex thought — as well as a range of physical movement! — while using less energy than most household appliances. I am skeptical that we will ever create non-biological AI that can compete with that.

Yes, the current crop of world knowledge AIs are smarter than any human who ever lived.

And big names are calling them useless.

This is proof the human race is not generally capable to solving novel problems, so I hope people will stop expecting AIs to solve every novel problem.

Next you will tell me that a calculator can beat students at addition and subtraction.
Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many?

Can AI love, yet?

To what degree are we just avoiding dealing with existential threats by churning through resources to play god and make robots in our image? (Albeit a subset of humanity, and not without bias)

I'm not yet convinced this AI work isn't a waste of time and other resources. I'd far rather we put our efforts into land/water stewardship and a "new" vision for human existence based on many of the old ways that got us this far, so that we might go another several hundred thousand years.

In an unbroken oral tradition, what stories might those future people tell about this time?

>Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many? Can AI love, yet?

I'm sorry to say that to me all these questions sound as semantically meaningful as "can submarines swim?"

Though I am admittedly an AI pessimist, as regards your penultimate paragraph, I think AI evangelists would argue that the aforementioned technology may offer a pathway to the new paradigm that you allude to.

What consistently goes unremarked upon, however, is the massive political “realignment” (a polite term for revolution) that the proliferation of AI may lead to in the not-so distant future. Disruption and likely violence that may, at best, lead to such a “new vision” for human existence.

I don't know what AI can or can not do, but it's getting good enough that I can't tell if this is written by ChatGPT.

If you can't pass the Turing test, setting higher bars for AI doesn't make sense.

Ohno.