▲ 154 points
back
163 comments
I have the extremely fortunate scenario of watching my wife just begin a computer science degree now, with no prior experience programming whatsoever. This is not hyperbole: ChatGPT is an absolute game-changer when it comes to getting through the basics.
It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight.
I thought she would have to ask me, someone who has been slinging Python in some form since 2008, a question at least once a week. In reality it's been about 3 times over the last 9 months.
ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
> It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight.
That's because it is easier. Multiple choice is easier than open questions.
It's also a good way to not really grasp anything deeply.
The good thing about ChatGPT is that you can keep asking it to explain something in a different way until it explains it in a way that “clicks” with you.
Of course you then have to compare this explanation against all others and see if it fits or if it’s not a valid explanation.
It’s also great for exploring topics which have polluted namespaces on search engines.
Overall though I think the majour benefit of ChatGPT is it just teaches people to clearly define problems as natural language questions. Developing this ability helps the subconscious mind solve problems when the user is away from screens.
> ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies.
Interesting, my impression was that ChatGPT helps experienced SWEs by filling out large amounts of boilerplate-ish code. It sounds like that's not your experience, though?
Experienced SWE here. ChatGPT is awesome for me. It's great for learning new technologies and it's great for writing programs that I'm too lazy to write myself.
Also as a learner - what i've found incredible is the ability to get AN ANSWER...without context switching. 1 window, 1 result location, no fishing...its trivial to test that given solution...as a beginner...my problems are fairly one dimensional.
Not to mention...things like "cool want to learn python"..."wait - wtf..how do i setup a venv in python!!"
Graying SWE here. ChatGPT is quite helpful, and so is Copilot. I mostly use ChatGPT for well defined functions that I need to implement that don’t rely on too much external context. I love it for writing unittests. It frequently makes mistakes, but it’s easier than starting from zero.
> ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
Yes and no. It doesn't help me that much with technologies that I'm senior within, but it has allowed me to work with a wider tech-stack. For example, I can now confidently write any SQL querys I want (so far anyway) although my prior knowledge in SQL isn't that deep.
If you are an experienced SWE and you aren’t getting something out of it, you are holding it wrong.
I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not seem like a majority of the students are using ChatGPT at all, even though I did encourage the use of it at the beginning of the course.
For my own course, I think several factors contributed to students not utilizing ChatGPT as much:
- The assignments are not in English, and performance of ChatGPT in languages other than English is subpar.
- The programming language that I'm teaching is C, I'd imagine Python/Javascript and other more popular languages might lead to different outcomes
- I did specifically design the assignments so that copy/pasting the assignment to ChatGPT does not lead to a usable answer (by restricting use of certain standard library functions, making the assignment more complicated)
- The course is not introductory, i.e. a previous course already taught the basic syntax of C and basics of programming, so I can make my assignments much more advanced
I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door".
I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
I've been using ChatGPT and Bard with C++ and it is quite helpful for boilerplate and reference (replacing Google/Stackoverflow).
Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.
> from what I can see right now, LLMs still have trouble solving novel problems
Have you been using 3.5 or 4?
I knew a few people who thought they could cheat their way through a CS degree. They all dropped out or changed majors within a few semesters.
It’s generally a problem that solves itself, in my experience. Perhaps a benefit of these tools is that we stop the obsession over cheating, which inconveniences honest students in many ways. Cheating has always occurred, but now we can’t even pretend that it’s preventable.
I think LLMs are wonderful study aids and can bridge gaps in understanding by providing collaborative discovery with a “copilot,” but absolutely, if you cheat off of it, things catch up quickly. First, the sort of stuff ChatGPT can do out of the box is introductory course material. More advanced material requires a deep semantic understanding of the domain. Even if you use ChatGPT to produce boilerplate and syntax, or ask it for semantic advice, it won’t solve the crucial nuggets for you. Second, there is always a “test” component where you are required to remember what you learned impromptu without aids. Third, a lot of cs programs require collaborative work in projects. It’ll become very obvious who doesn’t know anything. Finally, even if you manage to graduate cheating through the entire program, when you interview for work, it’ll be transparent you learned nothing. A degree is an end, it’s a means to an end - be it research of industry, and experts will always be able to sniff out dilettantes.
Would you be able to elaborate on why they dropped out? did they hit a point where cheating wasn't possible? or maybe realized it was pointless and unhelpful once they need to do a job?
I have no idea what the "age of chatgpt" is supposed to be, it changes nothing in regards to cheating. What my uni professor told us when I did my CS degree was. "You're not in kindergarten, if you want to copy your exercises you can copy them, but you have to do the exam at the end of the semester by hand without assistance and if you don't understand the exercises and do them yourself you're screwed". It's not like profs even bother to invent entirely new exercises every year so you could always just go through old course notes if you really wanted it easy.
Doesn't matter if people Google, Stackoverflow, use ChatGpt, ask their wizkid neighbor or what have you. You don't need to resist cheaters, especially not when we're talking about adults, it's their responsibility. Cheaters don't learn so when they're tested they sooner or later flunk out.
>CTRL+F GPT4, GPT 4, GPT-4
Disappointing. I cannot evaluate claims about GPT when people say ChatGPT instead of GPT4, if they actually mean the latter.
And if someone really is discussing the usefulness of AI while only having used ChatGPT (GPT3.5), then they’re missing out on a major improvement and their input is less valuable than those discussing GPT 4.
Of course GPT-4 has plenty of limitations but people just need to be clear that they’re familiar with the state of the art.
Also, statements to the effect of “it just predicts the next word” do not appreciate the major difference in capacity for learned abstractions between GPT 3.5 vs GPT 4. So to me it’s just not a useful way of thinking about LLMs. It may technically be true, but at some level that can also be said of human beings. In other news, an airplane “just” flies.
By the time the kids are done their CS degree (4 years from now), are the LLMs going to be writing all the code anyways?
If I were 18 again, I am not sure what the answer is - would it even be a good idea to pursue this field of study if there aren't going to be many jobs post graduation...
I hope I am wrong...
Building an entire application is way beyond what any LLM can do today and probably for years, if ever. Writing a method is basically all they can do, and even that is often unreliable. Maybe some companies will try to replace junior engineers with AI, but will learn really quickly that it doesn't work. The bad thing is that companies will do it anyway, meaning the juniors being replaced will never learn all the skills necessary to build applications or lead teams, so in the future getting anything built will become less and less likely.
I've always said that if AI had to deal with some of the executives and product teams I have, with constant insane changes and shipping demands, the AI will more likely figure out how to eliminate people than deal with them.
We are talking AGI. It is funny people think programmers will go first because AI can generate code. It can also generate words. Why do you need Lawyers? Politicians? Doctors? etc. when you have a knowledgable word predictor.
I think humans need to move up the levels of abstraction and master specific domain knowledge. Barring AGI, there will still be need for humans to elicit and learn the requirements from customers and other stakeholders, then translate those into more concrete specifications.
Future LFMs (Large Foundation Models, since they will be multimodal) can help automate some or much work from specifications onward. Humans will need to validate and revise the results by working with an LFM-powered system (Ref: the spiral model in software engineering).
I think there will be more jobs making software 4 years from now. But whether most CS degree programs will be teaching the most useful skills to do those jobs, I really have no idea.
People thought the same thing many times, for many different reason, over the last decades. I'm sure they will continue to think the same thing for many more.
Compare that to the whole genAI art thing and people here sneering that being an "Artist" is dead as a career. I wouldn't feel a lot of sympathy if such CSbros found themselves out of a job.
> are the LLMs going to be writing all the code anyways?
No, because that's not an expectation based in reality.
I see ChatGPT as the next “layer” on top of google search in the greater scheme of information. Before google, one had to spend a lot of time perusing the internet manually. And before that, people had to read physical books or call someone up and have a conversation over the phone. Google drastically sped up that process by giving you the information in 2 seconds, for what would normally take a few hours.
ChatGPT is a game changer for this same process. It speeds up the googling that would normally take a few hours and gives me my information in a few queries.
There have been so many times where I’m reading a book and need to research a bunch of technical terms it throws at me; which then consists of wading through blog posts or documentation, with varying levels of difficulty and quality.
This process is sped up 100x because of chat GPT, because I can followup with questions and customize them to my specific application. Sometimes I need things explained like I’m five; others, a deep technical deep dive.
Point is, it allows for dynamic interaction with content; it helps when I’m struggling with a bug, documentation, a book I’m reading, or pieces of code myself or someone else has written. It’s been an absolute game changer.
Not just programming. If you are interested in any kind of topic, be it scientific, or humanistic, GPT is an amazing tool to slash that learning curve.
Just as an example, try using ChatGPT to explain an article in a foreign language. Can even go as low as a letter-by-letter break down of each word. No private tutor will ever have the patience to teach people at this level.
Not using chatGPT to code is like not using antibiotics to treat infection. Sure, it can be done but why go back to the old ways? How many people know how to treat infections without antibiotics? How many people know how the antibiotics they use actually work ?
Coding will become like the liturgical exercises of monks toiling in isolated monasteries while the rest of the world will move on.
I cant wait for a chatGPT decompiler. IDA, watch out!
That analogy does open you up to the natural comparison of overuse of antibiotics leading to resistant superbugs doesn't it though.
Hopefully our bugs aren't evolving though.
Drivel.
A much more interesting consideration would be "Teaching Programming in the Age of Stack Overflow", which involves a well-established, gigantic resource for in-the-trenches programmers that generally provides functional, peer-reviewed and expansively commented solutions to the most common and sometimes most interesting problems.
What programming will be like in an age where LLM's can correctly code is something we won't have any idea about for some time yet (if ever).
Huh? I have personally committed production code that was 100% generated by an LLM. Simply because the code was correct and written more quickly than I could have physically typed it out. Does that not count as correct code to you?
DSl)program-teacher52344g
Perusing the comments here, it's interesting how short-sighted HN can be when it comes to LLMs. Generative "AI" is only going to get better, and the question of whether or not ChatGPT can write FizzBuzz, or avionics firmware, or a CRUD app, seems to be missing the forest for the trees.
Even if LLMs or some successor technology never ends up supplanting programmers entirely, it is guaranteed we will see a mixture of deskilling (certain skills no longer being required, like spelling in the age of autocorrect) and massive productivity gains (meaning n-m workers can now do the job of n). These tools aren't going away, and they're only going to improve.
Thus the market for programmers will shrink henceforth. There are no doubts about it. Maybe slowly, maybe quickly, but shrink it will, for the same reason that the market for radiologists is shrinking, or the reason that engineering firms no longer employ whole floors of draftsmen.
> the market for radiologists is shrinking
On the contrary, it's expanding and there's a global shortage of them [1].
AI has been changing the role of radiologists but not replacing it [2] and radiologists are needed more than ever, but the industry has been struggling to communicate this to students worried by ill-informed claims the profession is under threat from AI [3].
[1] https://www.rsna.org/news/2022/may/global-radiologist-shorta...
[2] https://www.hcinnovationgroup.com/imaging/radiology/article/...
Assuming LLM improve productivity why do you think the market will shrink instead of getting even more stuff done? Say if some actually smart LLM could improve my team productivity with 25% then
1 we could fire a person
2 we could handle 25% more tasks, features and bugs
So I think 2 will happen.
My latest example of LLM issues,| I asked ChatGPT and competition to convert a line of jQuery into native JS, they all got it wrong because jQuery has some selectors like ":header" that is native to JS but the LLM used it anyway though somewhere in it's big memory it has the information that his is not native and if you prompt it right it will fix the issue. So seems to me the LLM are focusing too much on the prompt and failing to use it's full memory on the problem, so it can fix small individual micro tasks but is a complete waste of time for something a bit more advanced , my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
Assuming that HumanEval is a good benchmark (it's not) and assuming you can naively scale under fat-tails (you can't) then according to OpenAI's own gpt4 report (someone did the math on hn/reddit where they reproduced the curve but i can't find the link) at 100x the training cost of gpt4 you will still have a 15% error rate on medium difficulty tasks.
"Programmers" is ill defined. Once it meant translating maths and engineering specs for assemblers. Now it can mean anything from someone writing verified and audited device drivers for robotics to designers a/b testing retail workflows. In the last century, 'programming' became a major economic and technological field of activity.
To think some new 'programming' tool will make it shrink is not at all warranted. Tools that make computer programming more accessible, more flexible and more powerful will increase demand for people who can reason about it, work on it, teach it, evaluate it etc etc.
It may get better but I'm not convinced it will get good. Currently I am not impressed. By which I mean I'm really impressed, it's pretty cool stuff but also hugely overrated.
I'm not even slightly worried about my job security. I tried Copilot and it sucked.
At the end of the day, LLMs are fakers. They do not possess real intelligence, they merely fake intelligence using statistics and training data. There comes a point where you can't fake it any more.
> meaning n-m workers can now do the job of n... Thus the market for programmers will shrink henceforth.
Maybe in the far future when we have programmed nearly everything there is to program... Until then companies will just produce n*(n/(n-m)) more output.
Actually, the reduction of the cost of programming output will logically lead to more demand as solutions that were previously deemed too expensive to produce will become viable.
The assumption that these models can only improve is a pretty big one.
duck I guess I really should be worried.
GPT is going to change everything, it’s not even a year in and most code is already written by AI.
It save a lot of time but write a lot of local problem while ignoring the bigger picture. (for now)
The only thing valuable is deep understanding of architecture and of the limitation of hardware.
Know both end of the system, let the AI create all the glue in between.
Whose code? I simply don't understand this. Are you writing code with GPT? I have simply been unable to get it to solve a technical problem in a production level application AT ALL. The only solutions that work for me are the type of solutions that might already have a stack overflow answer. Being generous, it MIGHT improve my productivity by a percentage point in a good day.
I would be skeptical that "most" code is written by AI. Some code, certainly - but major enterprises are still terrified of the copyright outcome for existing non-local GPT implementations.
People I know that use ChatGPT for coding ask the occasional question to simplify boilerplate or similar to stack overflow. Sometimes they'll "see if it can do X" (e.g. build a crud app) and generally it's horribly painful.
I have seen no evidence of "most code [being] written by AI" and believe this to be completely false.