Open source projects drown in bad bug reports penned by AI
Open source projects drown in bad bug reports penned by AI
Python security developer-in-residence decries use of bots that 'cannot understand code'
One thing I've also noticed is people doing code reviews using ai to pad their stats or think they are helping out. At best it's stating the obvious, wasting resources to point out what doesn't need pointing out. At worst it's a giant waste of time based on total bullshit the ai made up.
I kinda understand why people would think LLMs are able to generate and evaluate code. Because they throw simple example problems at them and they solve them without much issue. Sometimes they make obvious mistakes, but these are easily corrected. This makes people think LLMs are basically able to code, if it can solve even some harder example problems, surely they are at least as good as beginner programmers right? No, wrong actually. The reason the LLM can solve the example problem, is because that example (or a variation) was contained within its training data. It knows the answer not by deduction or by reason, it knows the answer by memorization. Once you start actually programming in the real world, it's nothing like the examples. You need to account for an existing code base, with existing rules, standards and limitations. You need to evaluate which solution out of your toolbox to apply. Need to consider the big picture as well as small details. You need to think of the next guy working with the code, because more often than not, that next guy is you. LLMs crumble in a situation like this, they don't know about all the unspoken things, they haven't trained on the code base you are working with.
There's a book I'm fond of called Patterns of Enterprise Application Architecture by Martin Fowler. I always used to joke it contained the answer to any problem a software engineer ever comes across. The only trick is to choose the correct answer. LLMs are like this, they have all these patterns memorized and choose which answer best fits the question. But it doesn't understand why, what the upsides and downsides are for your specific situation. What the implications of the selected answer are going forward. Or why this pattern over another. When the LLM answers you can often prompt it to produce an answer with a completely different pattern applied. In my opinion it's barely more useful than the book and in many ways much worse.
I use LLM-type AI every day as a software developer. It's incredibly helpful in many contexts, but you have to understand what it's designed to do and what its limitations are.
I went back and forth with Claude and ChatGPT today about its logic being incorrect and it telling me "You're right," then outputting the same/similar erroneous code it output before, until I needed to just slow down and fix some fundamental issues with its output myself. It’s certainly a force multiplier, but not at any kind of scale without guidance.
I'm not convinced AI, in its current incarnation, can be used to write code at a reasonable scale without human intervention. Though I hope we get there so I can retire.
One thing you gotta remember when dealing with that kind of situation is that Claude and Chat etc. are often misaligned with what your goals are.
They aren't really chat bots, they're just pretending to be. LLMs are fundamentally completion engines. So it's not really a chat with an ai that can help solve your problem, instead, the LLM is given the equivalent of "here is a chat log between a helpful ai assistant and a user. What do you think the assistant would say next?"
That means that context is everything and if you tell the ai that it's wrong, it might correct itself the first couple of times but, after a few mistakes, the most likely response will be another wrong answer that needs another correction. Not because the ai doesn't know the correct answer or how to write good code, but because it's completing a chat log between a user and a foolish ai that makes mistakes.
It's easy to get into a degenerate state where the code gets progressively dumber as the conversation goes on. The best solution is to rewrite the assistant's answers directly but chat doesn't let you do that for safety reasons. It's too easy to jailbreak if you can control the full context.
The next best thing is to kill the context and ask about the same thing again in a fresh one. When the ai gets it right, praise it and tell it that it's an excellent professional programmer that is doing a great job. It'll then be more likely to give correct answers because now it's completing a conversation with a pro.
There's a kind of weird art to prompt engineering because open ai and the like have sunk billions of dollars into trying to make them act as much like a "helpful ai assistant" as they can. So sometimes you have to sorta lean into that to get the best results.
It's really easy to get tricked into treating like a normal conversation with a person when it's actually really... not normal.
Well said!
Also, we monitor beginners heavily because the smallest unsignificant error (in their eyes) can have long lasting downsides and cause strange problems further down the road...
Managers usually love to say they, too, coded back in the day, but they didn't, they wrote some small scripts and thinks everything is easy like that so why not use AI, and why is it taking long to fix that bug?!
Managers usually love to say they, too, coded back in the day, but they didn't, they wrote some small scripts and thinks everything is easy like that so why not use AI, and why is it taking long to fix that bug?!
To be fair, some of us were real developers with real experience; you just don’t tend to hear us making claims about how easy dev work is and how AI is going to take over all the coding.
My project has merge requests by AI that have conflicts and don't make sense.
So...ye
One wonders who would have the time, interest and money to setup and control AI to do all this... One wonders 🤔 and the one remembers -just as a random example- Microsoft funding SCO with tens of millions of dollars right after which it attacked Linux with fake copyright claims for years, after which Microsoft extorted large corporations into switching to Microsoft platforms. Also, why controls GitHub now? Anyway, I digress.
OS will deal with this, I imagine it won't be too hard to setup tools that will deal with this shit, but I'm so sick and tired of continuously having to deal with this shit. Can we just formonce have something nice?
More than some nefarious corpo, I think this is more an evolution of the same problem that existed before AI was popular.
Some people realised that their credibility as a job candidate was tied on a very surface level to their GitHub profile, so they sought to optimise it. They started going to cool projects and proposing absolutely stupid merge requests, like “replace single quotes with double quotes in README.md” or “improved spacing in this sentence” in the hopes that the developers would go “well why not”, so they could show that they contributed to tensorflow or redis or what have you. Already years ago, a lot of FLOSS projects were plagued by spam PRs.
Now coming up with absolutely stupid reasons to issue a PR is a tedious job and you have a very fierce competition of people doing the same thing as you, so… why not gain the edge with AI?
No, this is definitely big corporations. It has Microsoft written all over it.
Microsoft has now gone "all in on Open Source" (except for their own code, of course).
They rely on OSS for most of their revenue (Azure). And they force their employees to use Copilot for everything.
It would only make sense for them to flood the devs of OSS they use with Copilot-generated bug reports and feature requests.
Another step in the slow death of the open WWW... Are we all gonna retreat to smaller, more controlled environments?
I work in customer support and tech support. I can see it now, people will start using AI assistants to order things and contact companies with problems. That will probably be frustrating, it will be like an office assistant ordering something for their boss but they don't understand what or why they're ordering.
I can see a company implementing this through their shitty chatbots paired with an IT layoff tbh. No malicious/negligent user necessary.
Just keep the bug-reporting bots pre-occupied by having them talk to the CI Bot. Problem solved b