I remember so little from my studies I do tend to wonder if it would really have cheating to… er… cheat. Higher education was like this horrendous ordeal where I had to perform insane memorisation tasks between binge drinking, and all so I could get my foot in the door as a dev and then start learning real skills on the job (e.g. “agile” didn’t even exist yet then, only XP. Build servers and source control were in their infancy. Unit tests the distant dreams of a madman.)
The bullshit is that anon wouldn't be fsked at all.
If anon actually used ChatGPT to generate some code, memorize it, understand it well enough to explain it to a professor, and get a 90%, congratulations, that's called "studying".
Yeah, if you memorized the code and it's functionality well enough to explain it in a way that successfully bullshit someone who can sight-read it... You know how that code works. You might need a linter, but you know how that code works and can probably at least fumble your way through a shitty 0.5v of it
Yeah fake. No way you can get 90%+ using chatGPT without understanding code. LLMs barf out so much nonsense when it comes to code. You have to correct it frequently to make it spit out working code.
If we're talking about freshman CS 101, where every assignment is the same year-over-year and it's all machine graded, yes, 90% is definitely possible because an LLM can essentially act as a database of all problems and all solutions. A grad student TA can probably see through his "explanations", but they're probably tired from their endless stack of work, so why bother?
If we're talking about a 400 level CS class, this kid's screwed and even someone who's mastered the fundamentals will struggle through advanced algorithms and reconciling math ideas with hands-on-keyboard software.
Are you guys just generating insanely difficult code? I feel like 90% of all my code generation with o1 works first time? And if it doesn't, I just let GPT know and it fixes it right then and there?
the problem is more complex than initially thought, for a few reasons.
One, the user is not very good at prompting, and will often fight with the prompt to get what they want.
Two, often times the user has a very specific vision in mind, which the AI obviously doesn't know, so the user ends up fighting that.
Three, the AI is not omnisicient, and just fucks shit up, makes goofy mistakes sometimes. Version assumptions, code compat errors, just weird implementations of shit, the kind of stuff you would expect AI to do that's going to make it harder to manage code after the fact.
unless you're using AI strictly to write isolated scripts in one particular language, ai is going to fight you at least some of the time.
My first attempt at coding with chatGPT was asking about saving information to a file with python. I wanted to know what libraries were available and the syntax to use them.
It gave me a three page write up about how to write a library myself, in python. Only it had an error on damn near every line, so I still had to go Google the actual libraries and their syntax and slosh through documentation
I just generated an entire angular component (table with filters, data services, using in house software patterns and components, based off of existing work) using copilot for work yesterday. It didn't work at first, but I'm a good enough software engineer that I iterated on the issues, discarding bad edits and referencing specific examples from the extant codebase and got copilot to fix it. 3-4 days of work (if you were already familiar with the existing way of doing things) done in about 3-4 hours. But if you didn't know what was going on and how to fix it you'd end up with an unmaintainable non functional mess, full of bugs we have specific fixes in place to avoid but copilot doesn't care about because it doesn't have an idea of how software actually works, just what it should look like. So for anything novel or complex you have to feed it an example, then verify it didn't skip steps or forget to include something it didn't understand/predict, or make up a library/function call. So you have to know enough about the software you're making to point that stuff out, because just feeding whatever error pops out of your compiler may get you to working code, but it won't ensure quality code, maintainability, or intelligibility.
You mean o3 mini? Wasn't it on the level of o1, just much faster and cheaper? I noticed no increase in code quality, perhaps even a decrease. For example it does not remember things far more often, like variables that have a different name. It also easily ignores a bunch of my very specific and enumerated requests.
isn't it kinda dumb to have coding exams that aren't open book? if you don't understand the material, on a well-designed test you'll run out of time even with access to the entire internet
when in the hell would you ever be coding IRL without access to language documentation and the internet? isn't the point of a class to prepare you for actual coding you'll be doing in the future?
disclaimer did not major in CS. but did have a lot of open book tests—failed when I should have failed because I didn't study enough, and passed when I should have passed because the familiarity with the material is what allows you to find your references fast enough to complete the test
Assignments involved actual coding but exams were generally pen and paper when I got my degree. If a question involved coding, they were just looking for a general understanding and didn't nitpick syntax. The "language" used was more of a c++-like pseudocode than any real specific language.
ChatGPT could probably do well on such exams because making up functions is fair game, as long as it doesn't trivialize the question and demonstrates an overall understanding.
I mean, I don't know how to code but I imagine it's the same as with other subjects. like not being able to use a calculator during some math tests. The point of the examination is for you to demonstrate you know and understand the concepts. It's not for you to be tested in the same way you would be in the real world.
I know people that used to work in programming with zero internet connection... this was ~10 years ago... never underestimate the idiocy of companies. P.s. it wasnt even a high security job, the owners were just paranoid boomers.
With that said, with a decent IDE with autocomplete, you can get by a lot without documentation. Its ussually the niche stuff that you need to look up on how to do it.
Giving me flashbacks to a college instructor that marked my entire functioning code block, written on paper, as wrong because I did not clearly make a ; on one line of about 100 lines. I argued that a compiler would mark that in the real world, but he countered with "It still won't run without that ; " That made me rethink my career path in CS. Fuck that guy.
You would think eventually some of it would sink in. I mean I use LLMs to write code all the time but it’s very rarely 100% correct, often with syntax errors or logic problems. Having to fix that stuff is an excellent way to at least learn the syntax.
Every time we let AI solve a problem we could’ve solved ourselves, we’re trading long-term understanding for short-term productivity. We’re optimizing for today’s commit at the cost of tomorrow’s ability.
I like the sentiment of the article; however this quote really rubs me the wrong way:
I’m not suggesting we abandon AI tools—that ship has sailed.
Why would that ship have sailed? No one is forcing you to use an LLM. If, as the article supposes, using an LLM is detrimental, and it's possible to start having days where you don't use an LLM, then what's stopping you from increasing the frequency of those days until you're not using an LLM at all?
I personally don't interact with any LLMs, neither at work or at home, and I don't have any issue getting work done. Yeah there was a decently long ramp-up period — maybe about 6 months — when I started on ny current project at work where it was more learning than doing; but now I feel like I know the codebase well enough to approach any problem I come up against. I've even debugged USB driver stuff, and, while it took a lot of research and reading USB specs, I was able to figure it out without any input from an LLM.
Maybe it's just because I've never bought into the hype; I just don't see how people have such a high respect for LLMs. I'm of the opinion that using an LLM has potential only as a truly last resort — and even then will likely not be useful.
Not even. Every time someone lets AI run wild on a problem, they're trading all trust I ever had in them for complete garbage that they're not even personally invested enough in to defend it when I criticize their absolute shit code. Don't submit it for review if you haven't reviewed it yourself, Darren.
My company doesn't even allow AI use, and the amount of times I've tried to help a junior diagnose an issue with a simple script they made, only to be told that they don't actually know what their code does to even begin troubleshooting...
"Why do you have this line here? Isn't that redundant?"
"Well it was in the example I found."
"Ok, what does the example do? What is this line for?"
Crickets.
I'm not trying to call them out, I'm just hoping that I won't need to familiarize myself with their whole project and every fucking line in their script to help them, because at that point it'd be easier to just write it myself than try to guide them.
"Every time we use a lever to lift a stone, we're trading long term strength for short term productivity. We're optimizing for today's pyramid at the cost of tomorrow's ability."
LLMs are absolutely not able to create wonders on par with the pyramids. They’re at best as capable as a junior engineer who has read all of Stack Overflow but doesn’t really understand any of it.
Precisely. If you train by lifting stones you can still use the lever later, but you'll be able to lift even heavier things by using both your new strength AND the leaver's mechanical advantage.
By analogy, if you're using LLMs to do the easy bits in order to spend more time with harder problems fuckin a. But the idea you can just replace actual coding work with copy paste is a shitty one. Again by analogy with rock lifting: now you have noodle arms and can't lift shit if your lever breaks or doesn't fit under a particular rock or whatever.
Actually... Yes? People's health did deteriorate due to over-reliance on technology over the generations. At least, the health of those who have access to that technology.
Unless they're being physically watched or had their phone sequestered away, they could just pull it up on a phone browser and type it out into the computer. But if they want to be a programmer they really should learn how to code.
I work in a dept. at a university that does all the proctored exams. None of that technology is allowed in the exam rooms. They have to put their watch, phone, headphones, etc in a locker beforehand. And not only are they being watched individually, the computer is locked down to not allow other applications to open and there are outgoing firewalls in place to block most everything network wise. I’m not saying it’s impossible to cheat, but it’s really really hard.
Some instructors still do in class exams, which would make it easier, but most opted for the proctored type exams especially during Covid.
Yeah that's object oriented programming and interfaces. It's shit to teach people without a practical example but it's a completely passable way to do OOP in industry, you start by writing interfaces to structure your program and fill in the implementation later.
Now, is it a good practice? Probably not, imo software design is impossible to get right without iteration, but people still use this method... good to understand why it sucks
Mine were actually useful, gotta respect my uni for that. The only bits we didn't manually program ourselves were the driver and the tomcat server, near the end of the semester we were writing our own Reflections to properly guess the object type from a database query.
A lot of kids fresh out of highschool are pressured into going to college right away. Its the societal norm for some fucking reason.
Give these kids a break and let them go when they're really ready. Personally I sat around for a year and a half before I felt like "fuck, this is boring lets go learn something now". If i had gone to college straight from highschool I would've flunked out and just wasted all that money for nothing.
It's super easy to learn how algorithms and what not work without knowing the syntax of a language. I can tell you how a binary search tree works, but I have no clue how to code it in Java because I've never used Java.
And similarly, i could read code in a language I dont know, understand what it does and how it works even if I don't know the syntax well enough to write it myself
I'm a full stack polyglot and tbh I couldn't program in some languages without reference docs / LLM even though I ship production code in those language all the time. Memorizing all of the function and method names and all of the syntax/design pattern stuff is pretty hard especially when it's not really needed in contemporary dev.
If it's the first course where they use Java, then one could easily learn it in 21 hours, with time for a full night's sleep. Unless there's no code completion and you have to write imports by hand. Then, you're fucked.
If there's no code completion, I can tell you even people who's been doing coding as a job for years aren't going to write it correctly from memory. Because we're not being paid to memorize this shit, we're being paid to solve problems optimally.
My undergrad program had us write Java code by hand for some beginning assignments and exams. The TAs would then type whatever we wrote into Eclipse and see if it ran. They usually graded pretty leniently, though.
It was the same for the class I took in high school. I remember the teacher saying that its to make sure we actually understand the code we write, since the IDE does some of the work for you.
He should be grateful. I hear programming interviews are pretty similar, as in the employer provides the code, and will pretty much watch you work it in some cases. Rather be embarrassed now than interview time. I'm honestly impressed he went the entire time memorizing the code enough to be able to explain it, and picked up nada.
Been a TA when chatGPT was released. Most students shot their own foot this way before we figured what was happening.
Grades went from bell shaped to U shaped. A few students got 85+, the rest failed, it was brutal.
Thought I failed my students horribly before I found out it was happening in all classes.
If you actually stuck in such a situation, solve as many problems as you can. An approach that will work for most people:
Try to solve
Fail
Take a peek, understand your failure. If the peek didn't include full solution, go back to step 1. Else continue to step 4.
Move to the next question and go back to step 1.
Make sure to skip questions if they are too easy.
Evey 4~ hours take a 20 minutes nap (not longer than 25 minutes).
If you actually manage to solve enough problems to pass, go to sleep, 4.5 hours or a longer multiplier of 1.5 hours.
After the exam go back and solve all homework yourself. DO NOT cram it, spread it or you will retain nothing long term.
Idk I knew people who got filtered by the intro Java class. It was because they both had no prior experience and also always cheated to avoid writing code themselves for assignments.
Yup. I had a classmate I did HW with. About 1/3 of the way through our study sessions me and person C realized. Classmate never helped but pretended too. Dude failed that class so hard when we got to the final and it was hand written. 🤣