I worry our Copilot is leaving some passengers behind

I think the problem is that any tool like this (even one theoretically much more powerful) is most beneficial to those that need it the least and least beneficial to those that need it the most. If you're an expert you can identify the mistakes and they are not generally a roadblock. But if you're a novice you can't and you'll simply be unaware of any hallucinations. The benefit SO has over this is just the extra friction of needing to copy paste or retype because it slows you down and forces an opportunity to think.

My worry is that we become too reliant on tools and outsource our thinking to them before they are ready to take on that task. This will only accelerate the shitification of things we have. More apps that use far too many resources. Things that are security nightmares. Interfaces with more friction. All of it.

The problem is pareto efficiency. 80% of your code is written in 20% of your time but 80% of your time is is required for 20% of your code. The problem is that the devil is in the details. So even a 95% or 99% accurate code generator is going to make for hard work. That's 1 in every hundred lines of code. I hope the compilers people are writing good error messages.

Copilot is a competent coder. If I tell it to generate a function with certain parameters, a class that follows a Gang of Four pattern, mass rename variables, or refactor loops into maps, then it does a pretty good job.

Copilot is a bad engineer. If I tell it to build something, unless exceedingly simple, it usually fails. The ability for it to create something seems correlated to how many 5 minute tutorials for that exist on the internet. Which given its training, makes perfect sense. So if I think I could find an answer on Stack Overflow, then I will just ask Copilot instead.

10 Years ago everyone was afraid of the Stack Overflow developer, now its the GPT developer. I think its a combination of actual worry and hurt pride that your job can be accomplished by someone copy/pasting. But like usual, good engineers will learn to think for themselves when leveraging tools. And an exceedingly amount of code produced works but is bad by some arbitrary metric.

I think the footer example is hilarious, because its exactly inline with web development trends of the last decade. Why use native elements when I can script my own behavior in Javascript on a div? And in a rush to "not use tables for formatting," I bet there are some 25 nested div websites out there. Even on Google sites, I have see grids built using absolutely positioned boxes with Javascript layout logic. The web is a wild place once you start looking past the tutorials and best practices.

The vacuum cleaner analogy didn't land for me. I've bought that vacuum cleaner, and I didn't return it.

I'm referring to one of the countless models of robot vacuum, of course. They clean the floor, most of it, most of the time, but they miss spots, they get stuck on things, and they don't have the suction of a full-size vacuum.

I wish none of those things were true, but it saves me labor nonetheless, so I kept it. I can detail corners and pull the thing off the corner of the rug, and still get a mostly-clean floor, automatically. Sure, it doesn't get all the schmutz out of carpets, but it gets enough that I can go over them monthly instead of weekly.

Yes, I'm talking about LLM code assistants. They have embarrassing failure modes, but experienced developers get a sense of what they can and can't do, and the result is something which saves time. I've found they're particularly good at "dumb debugging", where there's some fat-fingered error in the code and I can't spot it just by looking. I can copypasta the function into ChatGPT in seconds, and it gives a step-by-step description of what the code does, which routinely points out exactly where the bug is.

I have my concerns about what these tools will do to the up-and-coming generation of developers, it's easy to imagine them as a crutch, training wheels which never come off. But that's a separate matter, and I trust that the more natively talented juniors will recognize the hazard there, and understand that a chatbot can't substitute for becoming a skilled programmer.

Copilot is a decent tool for experienced developers (though, hasn't replaced Google-foo by any stretch) and a trap for inexperienced ones. Sure it may be able to speed things up in the beginning but it's a crutch for long-term sustainability in the industry. You inevitably have to understand the paradigms and patterns that LLMs regurgitate; taking them at face value (which I suspect is what most LLM users do), is a recipe for disaster and unfounded confidence.

I work with a guy who is absolutely dedicated to using LLMs to generate C++ code. If I ask him for a specific small thing I'll get back a PR with hundreds of lines of irrelevant crap and when I ask why it has this move constructor or whatever, they won't have a good reason. Even though my colleague is an industry veteran, their new habit has made it feel like they are delegating all their work to the stupidest teammate I've ever had.

I feel like we are going to need to work out some norms and customs in this industry for using code-generating systems in a way that respects the time and attention of coworkers.

One concern that I have is that copilot is inherently additive in nature. It is unable to suggest that blocks of code be deleted which creates a bias that adding more code is always the solution and a lot of code that shouldn't be written ends up in the codebase.

I believe this is a problem works against less experienced engineers because more senior engineers are better at recognizing that problem. In my experience the most senior engineers respond to that by just turning the tool off.

I'll use a locally hosted Llama 2 or CodeLlama instance as a 'consultant', via a chat window. These models can be great for that! A well-formulated question often elicits a precise and accurate answer, even from the unspecialized model.

I won't use Copilot or anything else that integrates that tightly into my workflow, even though it is now possible to do so without losing the incremental-cost and customizability benefits of selfhosting.

The context switch is important. To a very good first approximation, our task as engineers is to think before we assume, and I have found Copilot recklessly encourages the latter at the expense of the former.

Personally I think of LLM code helpers as a warning smell.

If I’m working on something where I’m tempted to generate a bunch of boilerplate from a bot that knows very little about the context of the project, am I really spending my time on the right thing? Either I should be working on something higher level, or the amount of boilerplate should be so low that I can write it myself. Anything else suggests that there’s a problem and the LLM bloat band-aid isn’t the solution.

I agree with most of the stuff in this article but I'm a bit puzzled by some of the attitudes of the author. They seem to care more about the LLM delivering code with poor accessibility than they care about the LLM delivering completely wrong answers.

The "any good developer would realize this is bad code" rings strongly as "no true developer would think the LLM's bad answer was correct". Seems like a short sighted opinion to me.

I also think you can replace "accessibility" with any number of programming meta concepts and find problems too.

How about "internationalization"? Are LLMs any good at producing code that is nicely internationalized?

Or more importantly "security". Are LLMs going to produce millions of lines of poorly secured code that people never double check? Almost assuredly.

The fact is that LLMs are prediction engines. They run off of probabilities based on the prompt and the training model. Thus, unless the training model is weighted towards cherry picked examples of excellent code, it's going to follow the masses.

And the masses write bad-to-average code mostly.

> Copilot loves suggesting about 25 nested divs as a starting point.

> I assume this is because of a flaw in how LLMs work.

I know with some LLM implementations, you can configure the sampling to penalise repetitions – this is making me wonder if Copilot might benefit from that?

> What does it say about Copilot’s knowledge of accessibility when it will hand us code even basic checking tools would flag?

Maybe it could do with some fine-tuning based on those checking tools? e.g. sample many answers to same prompt, run them through checking tool, and then fine-tune it to prefer the answers which caused the least warnings?

Or: run the suggestion through checking tools, and if it triggers warnings, sample a new suggestion, and see if the new one doesn't. This could be done on the client side in a loop – run suggestion through checks, if it fails, ask the LLM for a new and different suggestion, repeat until we get one which passes checks, or we give up.

The biggest problem with Copilot/LLMs is that they effectively operate against anything that programming languages were designed for. What makes programming languages special is that they're well defined, semantically and syntactically rigorous and intended for machine execution. They give us the capacity to formally reason. Instead what we've got now is tools that literally argue with us, rather than anything that actually augments my capacity to reason about, inspect and understand the real performance and hardware of a system my code runs on. What I need is more Coq and less of something that just makes natural language suggestions.

What makes a good engineering tool is something that can look at the code right there as it is, use the formal guarantees that programming languages were designed for and give me some verifiably correct suggestions. Not average out 90% of Stackoverflow answers and then hallucinate up some statistical response.

Contrast Copilot with tree-sitter. What makes tree-sitter so good as a tool is that it leverages the regularity of programming languages. It can parse and correctly reason about code, instead of relying on some random regex collections and prayers. We've had so many good advances in recent years like the borrow checker in Rust. Why are we going back now and introducing tools that are by design incapable of ensuring correctness? Just to type a little bit faster?

I think a lot of these are actually solvable problems today, Copilot just hasn't prioritized actually improving the product (don't need to improve the product for breakneck growth when you have github.com as a distribution channel!)

It feels like there are a lot of well-intended AI coding products that just don't pay attention to getting the details right. I actually started building my own extension recently, with an emphasis on getting all the little things right, because I got so frustrated at Copilot. Things like closing brackets properly, not interrupting me and destroying my train of thought when writing comments, not suggesting imports unless it's highly certain (or verified with intellisense), etc. Like why am I wasting my precious time talking to copilot chat with gpt3 when gpt4 exists?

It's still a pretty early version, and ultimately we're using the same underlying model for completion, but I think getting these details right make a huge difference (at least to my biased self).

If you want to try it: https://marketplace.visualstudio.com/items?itemName=doublebo...

You'll need to install the pre-release version for auto-complete.

> Copilot is encouraging us to block users unnecessarily, by suggesting obviously flawed code, which is wrong on every level: wrong ethically, wrong legally, and the wrong way to build software.

I share many of the same worries as the author. This is why I think teams need to build and run their own Copilot-like systems, so that they can guide the suggestions they receive. Each developer and team has their own way of building software, and they need to be able to shape and evolve the suggestions they receive to fit their definition of the "right" way: https://blog.continue.dev/its-time-to-collect-data-on-how-yo...

I don't know if its mindset or my owner ignorance, but I find myself using Copilot and other language model tools as a teacher, a debugger, a reviewer, and and idea brainstormer. I find each use case to enhance my ability to think more deeply about my code and helps keep me more engaged in problem solving.

For some reason I find a inference from a compressed model which contains almost every notable open source program written in the history of humanity to be a decent sidekick.

My experience tells me no software engineer is an expert at everything. Having a tool which allows us to try new things faster is a good thing.

I've come to largely agree with the author. These days, I keep it off by default, but is a Cmd+' away from being flipped on and filling in what I _know_ to be boilerplate that's well suited. If I was younger with less money, I probably couldn't justify the price, but these days if it can save me a half hour of busywork per month on my personal projects the $10 is more than worth it.

Leaving it on while doing any thoughtful or challenging coding is super distracting for me.

It’s appalling that most website/app developers even have to deal with those kinds of low-level considerations, after decades of web-tech evolution, instead of using a UI builder tool (or a better UI modeling language) that provides all the building blocks for the most common 97% of use cases, and where you would need to go out of your way to create a non-accessible link.

TFA is right about LLMs, but it’s also an indictment of the web UI stack.

It's only a cause for concern if its capabilities are going to plateau.

More likely, advances in the field will mean that we end up in a more accessible world, where developers who don't normally think about accessibility have a generation engine doing a pass over their work adding appropriate labeling, fixing elements to work with screen readers, etc.

We just had a big paper about using genAI to improve test coverage.

And we haven't even really hooked LLM code generators up to linters and test suites broadly yet.

I can foresee a future where language specific Copilot features might include running suggested generations for HTML though an ARIA checker while running Python generations through a linter, etc. Especially when costs decrease and speed increases such that we see multiple passes of generation, this stuff is going to be really neat.

I still mostly consider the tech (despite its branding) in the "technical preview" stage moreso than a "finished product," and given the capabilities at this stage plus the recent research trends and the pace of acceleration, it's a very promising future even if there's valid and significant present shortcomings.

It is clear that despite these tools having flaws on the whole they save a lot of time. It is not clear what the tradeoff with introducing poorly understood or faulty code will bring, but given the utility we're never going back.

Copilot is bad at accessibility because web engineers are bad at accessibility. All of the bad habits in this post were learned from its training data.

That's not to say this can't be fixed: a recurring lesson of LLMs is that the quality of the training data is /everything/. OpenAI made their models better at chess by feeding in higher quality chess data - they could absolutely make it better at accessible frontend code by curating and boosting better code examples.

I doubt they'll do that any time soon, purely because there are so many other training data projects they could take on.

Thankfully we aren't nearly as dependent on a few closed research labs as we used to be.

It would be very exciting to see fine-tuned openly licensed models that target exactly this kind of improvement.

Does anyone who has found a good workflow with copilot have a good resource to share that demonstrates how to get the most out of it?

I really want it to be more useful, but rarely find that it's helpful for completing more than a single line or two.

Do you write out comments for everything you're going to do and then just write it yourself if the suggestion isn't useful?

Is there a trick to getting it to read your code itself across files?

> Shouldn’t the results I get from a paid service at least be better than a bad StackOverflow suggestion that got down-voted to the bottom of the page (and which would probably come with additional comments and suggestions letting me know why it was ranked lower)?

I don't know why you would expect this, when the model is likely trained on StackOverflow material (or similar publicly available code examples).

Redmonk says Kotlin is the 17th most popular programming language ( https://redmonk.com/sogrady/2023/05/16/language-rankings-1-2... ). So can any of these LLMs and whatnot, even the ones supposedly geared toward programming do something like this:

"Write a function in Kotlin that take a Long as a parameter, and sends back a List containing Long types. The parameter is a number, and the return is a list of prime numbers less than that number. All in one function."

It seems it should be pretty simple, in fact I have written this program a number of times. If you think a list of prime numbers might take up too much memory, I have also done prompts only asking it to just give the largest prime under the input parameter.

It is not a difficult task, and Kotlin is between Objective-C and Rust in popularity. Have any neural network programming tools been able to complete this? No. Some can, if the number input is 18L or the like. None have been able to handle 600851475143L (taken from the third Project Euler). If the program runs at all I get "java.lang.OutOfMemoryError: Java heap space". Even if I warn it to watch heap memory, it still is the same result.

As I said, this is a prompt for a list, but even if I ask for only the largest prime number before 600851475143L, or any long such as that number, I have not seen any LLM or the like that can write that function. Especially ChatGPT 4, which I have tried it on extensively.

I'm not saying LLMs will not get there, but this part of the third question on the Project Euler site, from a fairly popular language. It's a pretty simple question - a straightforward function to write. They can't do it yet.

I see people worrying about AI being on the verge of taking programmers jobs. Until it can do something incredibly specified and simple as this, I am not worried at all.

These code generation systems should probably prepend a hidden "Generate accessible, secure, maintainable etc code" prompt

Of course that doesn't provide any guarantee, and no developer should rely on it, but the average results would probably be a little better

What we will see is that llms become so good in writing code that LLM first will emerge.

LLM first means we will test it against our libraries, best practices and potentially even create a new language for it.

Then programming in the classical sense won't exist anymore.

The ara of code will end when we will deploy the first code written with LLM to write new code.

Javallm or #llm.

It might be full of examples for a LLM, it might focus on analyzing logic and fixing it on a higher level, until the AI is good enough to self write, evaluate and deploy it.

After that it will become no longer understandable by us and researchers will start analyzing it after it was written.

Historians will start tracking when ai started to create more efficient abstractions etc.

This is one of the more thoughtful, nuanced criticisms of the current LLM fad that I've read, and I'm delighted to see it make it show up on HN. The author starts off with a series of well-thought experiments that show Github Copilot generating _pretty valid_ frontend code, code that works and fulfills the prompt: but code that ignores every web accessibility rule of thumb in the most egregious ways. Sure, yes, bad web devs write bad code, and Copilot is -- on its best day -- a perfectly cromulent bad developer. Yawn, news at 11, etc.

But where he takes those examples and where his thoughts end up is where this essay really hit home for me:

> As more and more of the internet is generated by LLMs, more and more of it will reinforce biases. Then more and more LLMs will consume that biased content, use it for their own training, and the cycle will accelerate exponentially.

And 'biases' here isn't the usual "models are woke-lobotomized!" yammering, but rather a thoughtful take on how the use of LLMs for code generation may, at least for the current state of LLMs, slowly normalize _writing worse code_.

> Why do we accept a product that not only misfires regularly, but sometimes catastrophically?

RIGHT?!!

ChatGPT went completely bonkers today, and people are basically ":shruggie: it happens"

THAT IS WILD TO ME.

People are using ChatGPT for medical advice. People are using ChatGPT for activities that have financial implications. How is this okay?!

> Copilot loves suggesting about 25 nested divs as a starting point.

To be fair it costs a huge amount of money to hire a React/Tailwind person to create 25 nested divs as a starting point.

> In a lot of ways, in fact, “AI” is just the newest iteration of a very old form of colonial capitalism; build a wall around something you didn’t create, call it yours, and charge for access. (And when the natives complain, call them primitive and argue they’re blocking inevitable progress.)

What a wonderful analogy. LLMs also feel very pythagoran, where a secret cult (of capital owners; the bourgeoisie) guards the secrets of forbidden math, using it for their own benefits, and denying it to the masses. The amount of data and computing power needed to train a good model means it is pretty much inaccessible to the masses, the public can only ever hope to use an already trained model which is provided to us by this secret cult.

No mention of testing in the article. It seems odd how often accessibility advocates talk about following rules rather than testing. Shouldn’t we be testing with screen readers or something?

If a website doesn’t work in Firefox, we fault the developer for not testing it in Firefox. Similarly for mobile browsers.

If testing is in place, LLM’s are much safer to use. You’ll notice when they give you code that doesn’t work.

It's a very powerful autocomplete. "It doesn't generate all the code I need in full and if it does I have to poke at it" is just poor criticism. You don't have to press tab and insert everything it suggests. It will usually generate me half a line after typing the first half - that's pretty awesome in my opinion.

If you stick to using it to merely speed-spell out what you were in fact already in the process of writing, and ignore 90% of the terrible crap it proposes, it's a nice productivity boost and has no way to make code worse by itself.

Basically, instead of writing a big comment and then a function signature and expect it to do the rest, just start writing out the function, tab when it gets it, don't when it doesn't, or (most of the time) tab then delete half of it and keep the lines you intended, likely with some small tweak.

Surely LLMs will be able to go so much more and without constant supervision in the future, but we're not there. That doesn't mean they're bad. Especially copilot since it's just there with its suggestions and doesn't require breaking flow to start spelling out in regular text what you're doing.

Half the time copilot doesn’t even return a solution

I wonder though if Copilot had fared better here had it been told to pay attention to accessibility.

I mean, maybe it should do it by default (and maybe it could be part of its system prompt or otherwise in its material), but it's still a tool that needs some expertise for using, even if it's trying its best to trick people into believing otherwise. Ultimately I don't think there's a solution to people misusing tools.

Paraphrasing sentiment I don't quite recall exactly: "If anyone can do it, then anyone will."

> In a lot of ways, in fact, “AI” is just the newest iteration of a very old form of colonial capitalism; build a wall around something you didn’t create, call it yours, and charge for access. (And when the natives complain, call them primitive and argue they’re blocking inevitable progress.)

This is pithy, but the dynamic between OSS devs and Microsoft/OpenAI is not exactly comparable to the dynamic between a colonial government and an indigenous population. I don’t think it really needs to be said, but open-source maintainers are not colonized natives.

Even overlooking the very questionable metaphor, they’re not building a wall around existing repositories of code and selling them back to us. They spent a lot of money training an AI model on that code, and now they’re selling access to that model. You don’t need to pay Microsoft for access to the GitHub repos or Stack Overflow answers that they trained on.

Is it just me or has copilot gotten progressively worse lately? It used to feel like it was making well informed guesses, now they feel like literal guesses with no context at all. For example in my phoenix live view (elixir) app it guesses “xxx@xxxxx” for _any_ attribute I pass in to a component.

I am somewhat amused by all of the "copeelot bad" articles, and I dearly hope they keep proliferating, so that those of us who enjoy its frankly insane productivity boost get to stay ahead of the competition. I perceive no quality/reliability drawbacks in my own code. If anything, the ability to iterate more quickly makes my code better than ever.

It's a skill issue. (You had it coming.)