I finally watched the talk today and that wasn't what I thought he meant. What I thought he was getting at was that the rust parts of the kernel interact with lots of other modules written by people who don't know rust. When those C modules change their semantics in ways that break the rust code, they can't go fix it because they don't know rust. In fact, whenever they make a change, they don't even know if they broke some rust module, because they don't understand the rust code well enough. And this is something that everyone is going to have to live with for the foreseeable future, because you can't force all those other kernel hackers to learn rust.
If you are that good in C(pp), I guess understanding rust code of a module is not sooo hard.. I mean, I learned what I know about C from reading stuff in the Kernel that made my embedded Linux project not working.
It basically now comes down to the current devs depending on new Rust devs for anything that interacts with Rust code.
They could just work together with Rust devs to solve any issues (API for example).
But their ego doesn't allow for it. They want to do everything by themselves because that's how it always was (up until now).
Sure, you could say it's more efficient to work on things alone for some people, and I'd agree here, but realistically that's not going to matter because the most interactivity that exists (at the moment) between Rust and C in Linux is... the API. Something that they touch up on once in a while. Once it's solid enough, they don't have to touch it anymore at all.
This is a completely new challenge that the Linux devs are facing now after a new language has been introduced. It was tried before, but now it's been approved. The only person they should be mad at is Linus, not the Rust devs.
Switching everything from C to Rust because it has better memory safety is more akin to changing languages from English to Esperanto because it has gender neutral pronouns and other cool features. Maybe it's a good idea, but it's understandable that some people are reluctant.
Maybe it’s a good idea, but it’s understandable that some people are reluctant.
I understand that position. I also understand how the words and phrases that the C community has used to communicate with the Rust community seems to be completely dismissive, not just reluctant.
I quoted what I did explicitly because of how a statement like that comes off to the person it's aimed at. It doesn't make them feel like they're on an even footing working on the same project with the overall goal of it becoming better.
memory safety is more akin to changing languages from English to Esperanto because it has gender neutral pronouns.
I mean... not at all? Memory safety is huge for cybersecurity, buffer overflows and the like are common attack surfaces. C requires you to have deep knowledge of safe memory management practices and even then you can end up with memory issues. Rust was developed to avoid such issues entirely. I understand the reluctance but it feels to me like arguing "we should just stick with COBOL because it works."
Vast majority of the cybersecurity community: "an absolute ton of exploits come from memory safety issues with C/C++, we should move to memory safe languages like Rust to greatly reduce security risk and make everyone safer"
You: "Ehh Rust has a couple features, but it's totally not worth switching from my precious precious C"
😂i wish my country switched from german to English because of how difficult it is to talk genderless in that language. Like, every fucking word seems to be gendered here.
Except in this case, it was a bunch of old C devs who aren't just resistant but openly hostile to change, and they'd rather bully people into silence than try to progress.
If I go to any of the teams I interact with who program their components in C++ and proposed Rust or anything else, I'd get a similar reaction. They're very good at C++ and they very rarely have memory and threading issues. 😂
Ironically the majority of the rust memory management ruleset is called ownership, and they are unwilling to release any of it, and claiming all of it, so there's an out of memory error.
I knew this ageist bullshit would pop up. I know we lost our mentors and are kinda feeling in the dark, but the moment people pop out the ageist slurs I know they've got nothing to say.
Rust is sufficiently different that you cannot expect C developers to learn rust to the level they have mastered C in order to be working at the kernel level. It's not going to happen.
I don't really know too much about rust. Maybe one day I'll actually mess around with it. But the one time I looked at a rust git repo I couldn't even find where the code to do a thing was. It's just different enough to be problematic that way.
So I think probably, the best way IS to go the way linus did. Just go ahead and write a very basic working kernel in rust. If the project is popular it will gain momentum.
Trying to slowly adapt parts of the kernel to rust and then complain when long term C developers don't want to learn a new language in order to help isn't going to make many friends on that team.
But that's the thing where you are wrong. They clearly state they don't want C developers to learn Rust. In the particular video posted he was saying "I want you to explain to me how this particular API works so that I can do it"
The concerns about who fixes what on a merge when the C code breaks Rust code are valid, but that's easily fixed by gathering with the Rust developers, explaining the changes and letting them fix it.
I've inserted myself into your C project because only idiots write C. Rust is the one true god, mUh MeMoRy sAfteY! Now please explain to me how C works.
But the one time I looked at a rust git repo I couldn't even find where the code to do a thing was.
IMO that tells more about how the project was organized and names things than the language used.
So I think probably, the best way IS to go the way linus did. Just go ahead and write a very basic working kernel in rust. If the project is popular it will gain momentum.
As the other commenter pointed out, there's Redox. The issue is that this completly disregards an incremental approach: you have to rewrite everything before it comes usable, you can't do it piece by piece. Currently the approach of Rust for Linux is not even to rewrite things, but to allow writing new drivers in Rust.
Trying to slowly adapt parts of the kernel to rust and then complain when long term C developers don't want to learn a new language in order to help isn't going to make many friends on that team.
Have you seen the conference video? That's not just refusal to learn a new language, it's open hostility. And it's not the only instance, for example Asahi Lina also reported unreasonable behaviour by some maintainers just because she wrote Rust code, even when Rust was not involved.
I think the point of redox is more than just rewriting Linux in Rust. Architecturally they are very different. Redox uses the more modern microkernel approach, whereas Linux is a modular monolith. There are advantages and disadvantages to both designs. They are actually polar opposites in fact. The compromise is something called a hybrid kernel which is used by Windows NT.
RedoxOS! There's been solid progress too, beyond just having a functional microkernel, they have many of the userspace tools/their version of coreutils, even a desktop environment already mostly implemented!
My understanding is that it shouldn't be too bad to port some other things over as well. The main issue I had was just the lack of drivers, especially since it's still tricky even on Linux, and the microkernel architecture (though more secure) also means there's no way to reuse any of those from Linux
I think this overall is a better idea. I'm going to say this because, I thought I'd look into rust today. So I installed it, setup vscode to work with it etc. And it's all up and running. I thought I would port over a "fairly simple" C# project I wrote recently as a bit of a test.
While I've generally had success (albeit with 30+ tabs open to solve questions I had about how to do certain things, and only making it about 20% into the task) I'm going to say that it's different enough from C, C++ and C# (all of which I can work with) that I really don't think it is fair to expect C developers that have day jobs and work on the kernel in their spare time to learn this. It's fundamentally different in my opinion.
Now, I don't condone any bad attitude and pushing away of rust developers from the project. But there's no way they're going to want to do anything to help which involves learning a new language. It's just not going to happen.
Likewise, C is not a language most new developers are learning. So, I feel like over time there won't be so much of an influx of new kernel developers and any Rust based kernel could find itself with more contributors over time and taking over as the de-facto kernel.
In terms of Redox (not looked into it yet). So long as there's a different team working on the userspace tools. I would say the main task should be getting a solid kernel with drivers for most popular hardware etc in place. The existing GNU tools will do until there's a kernel that is able to compete with the C one. But that's just my opinion.
Hey you guys with 20-30 years of experience doing a single thing very well. Let's nullify most of that skillset and replace it with a thing we're good at.
Don't worry, we will teach you.
They're not technically wrong about Rust being a better choice for a kernel, of course. They're just incredibly misinformed about the social hurdles they need to climb over for it to happen.
Honestly, if anyone has become a master in C, they can become a rust master in short order. It's different, but not THAT different. The roots are the same.
Just go ahead and write a very basic working kernel in rust.
I don't get this stance, really. If I want to write a driver in Rust I should start by creating a completely new Kernel and see if it gains momentum? The idea of allowing Rust in kernel drivers is to attract new blood to the project, not to intentionally divert it to a dummy project.
Rust is sufficiently different that you cannot expect C developers to learn rust to the level they have mastered C
If you watch the video, no one asked anything from the C developers other than documentation. They just want to know how to correctly make the Rust bindings.
Note that Rust is not replacing C code in the Kernel, just an added option to writing drivers.
Has a much more C-style syntax, except much more refined from the years of hindsight. The catch? No corporate backing, didn't jump on the "immutable by default" trend when functional programming evangelists said for loops are a bad practice and instead we should just write recursive functions as a workaround, memory safety is opt-in (although "safe by default" can be done by starting your files with @safe:), some of the lead devs are "naive centrists" who want to "give everyone a chance at coding even if they're bad people (nazis)", implementing new changes to the lang has slowed down significantly up until the departure of Adam D Ruppe and the drama surrounding it, etc.
"safe by default" can be done by starting your files with @safe:
Last time I heard about that it was much more limited than Rust, for example it even disallowed taking references to local variables. Has something changed since then?
part of the problem is that old-time kernel developers are used to C and don't know Rust," Torvalds said. "They're not exactly excited about having to learn a new language that is, in some respects, very different. So there's been some pushback on Rust."
Linus hit the nail on the head. If you've been a Kernel dev for a decade or more, and have spent decades learning the ins and outs of C, why would you want to switch to something that is similar, but different in a lot of ways, just because a small subset of devs think it's the best way forward? Let them handle Rust and the majority of devs will keep using C, even though Rust is objectively better.
As one of the other quotes suggested: fork the kernel project and rewrite it entirely in Rust, that way there isn't any push back from the C devs. Replacing C with Rust in the upstream kernel is akin to replacing the engine in a car while it's running or being used every day.
This specific talk was about defining shared common interfaces so these different groups could work together and the guy who actually talked him into stepping down essentially said "I'm gonna keep writing C and if that breaks your rust stuff that's not my problem". This isn't about convincing the c devs to write rust it's about convincing them to work together when some of them seem to have made up their mind to sabotage rust support (either through indifference or willful interface regressions). Personally I'm more ashamed what this points to for someone new wanting to come in contribute to Linux.
Ah, but I still agree with the C devs, it creates unnecessary headaches for them. Also, old habits die hard.
I view it as the same way ZFS is supported: Linus and Greg KH are like "you can maintain it, but we don't give a shit about it, and if what we do breaks ZFS support, well too bad."
I think all the Rust devs should remove their code and leave. And when in the future the Linux devs change their tune and ask for their help, they should refuse.
The kernel is probably too large to rewrite the whole thing at once. This could lead to a future without any new C kernel devs, leading to stagnation, while the Rust kernel could be many years away from being finished. (Assuming we actually move away from C.)
At that point you might as well just start an entirely new kernel and hope it is good enough to eventually replace the Linux one once all devs are gone. Kinda the X11 and wayland thing.
the Rust kernel could be many years away from being finished.
the number I saw floating around was 3 years to production useful. regardless, C's end days as the go-to, large systems level language are drawing nigh.
I agree. C isn't going anywhere anytime soon, but if we don't start modernizing the kernel now we could end up with a future like the US government is in where all critical systems run on COBOL code and no one wants to touch it for the fear of breaking everything.
I'm not sure if it was in my above post or not, but the article said we should start modernizing the kernel now before someone does to Linux what Linux did to Unix.
Redox OS already exists and is functional (meaning it boots and has a GUI, but it's lacking in various aspects), from what I understand it's pretty much Linux/Unix rewritten entirely in Rust and looks pretty promising. In 5 or so years it could be a competitor with BSD and then overtake Linux once it has a proven track record.
As one of the other quotes suggested: fork the kernel project and rewrite it entirely in Rust
That's not practically possible given the scale of the kernel. And doing a total rewrite is almost always a recipe for getting stuck and, if you ever create anything, creating something worse.
Replacing C with Rust in the upstream kernel is akin to replacing the engine in a car while it's running or being used every day.
Almost all real-world software development is like this. That's what we do.
Yeah it is a monumental task, but it's also the one with the least push back. I don't mean start from scratch, but convert the C code to Rust in a dev branch or something and release a Linux-Rust kernel image.
Almost all real-world software development is like this. That's what we do.
I'm aware, I've written my own software even though I'm a SysEng, all I'm saying is that it's not an easy process with a potential for disaster. Just look at CrowdStrike (not saying that they were attempting to switch languages but just the scale of the fuck up and the fallout that it caused), we don't want that to happen with Linux.
I'm not rejecting it, I'm just saying that it's very difficult to completely change the code of a critical piece of software. The long-term goal is for Rust to overtake C in the kernel (from what I understand, I'm a System Engineer, not a software dev. I know Go, not Rust) due it being memory-safe and about 30 years newer. Critical code gets left untouched (a lot of the time) because no one wants to be the one that breaks shit (and get bitched out by Linus 😂) so I'm sure there is tons of code from the early 90s that could be made better with a newer language like Rust, but it's not as mature as C right now so that's not going to happen for a while, if at all.
I'm a C/C++ dude but I heard it being called the "Karen compiler". It doesn't look that scary based on samples I've seen, but there's way more to it I am assuming.
I'm not a software dev (I'm a SysEng), and have never touched Rust (I've looked at the Rust source code and it scares me haha), but I know a fair amount of Go, and even the Go compiler is a pain in the ass occasionally. I've "mother fucked" it so many times.
I'm not a software dev, but I'd imagine that the codebase could definitely be reduced once most things are converted to Rust. From what I've heard, the kernel is a huge mess of spaghetti code that most people don't want to touch, for the fear of going insane in the process 😂
One detail about Rust in the kernel that often gets overlooked: the Linux kernel supports arches to which Rust has never been ported. Most of these are marginal (hppa, alpha, m68k—itanium was also on this list), but there are people out there who still use them and may be concerned about their future. As long as Rust remains in device drivers only this isn't a major issue, but if it penetrates further into the kernel, these arches will have to be desupported.
(Gentoo has a special profile "feature" called "wd40" for these arches, which is how I was aware of their lack of Rust support. It's interesting to look at the number and types of packages it masks. Lotta python there, and it looks like gnome is effectively a no-go.)
Assuming that it works out, yes, this might fix the problem. On the other hand, I remember gcj, which kind of quietly vanished after a while, so I prefer to reserve judgement until gcc's Rust implementation is ready for production use.
Or wait for rust to support the extra languages. With LLVM adding new architectures or projects like gccrs. But all of these options are a way out and rust will remain device driver only for a long time I suspect - it is still experimental after all. I would hope that as rust in the kernel matures so do the available architectures that rust supports.
Linus is the leader of the kernel project. As a leader, it's his job to get the maintainers to agree. It's not Rust's job to make the C devs stop bullying them.
If Linus thinks Rust is a good direction, he should show it by actually standing up to Ted and developers like him and making them behave.
If he doesn't think it's a good direction, he should say that too, so the remaining Rust devs can stop wasting time on the project.
When someone in a niche part of the project steps down like this, that's a problem with the top-level leadership. Linus' record on leadership is.. mixed. Trending in a good direction the last few years, but this makes me wonder. He can still save this, but he has to want to.
Adding rust to a massive mature C project that targets lots of architectures and has many contributors is a difficult process. If it succeeds it is going to take a lot more time and patience.
Especially when Rust has limited support for less common architectures. This has been forcing distros like gentoo to drop support for more niche arches since many common packages like python-cryptography are now pulling in rust as a mandatory dep.
Not an expert in both the languages but I heard that C developers are trained to use memory smartly, sometimes even reuse a range of allocated memory for completely different purpose to save cycles freeing and reallocating. But for Rust developers, everything is about making sure when one should get the hand away from the memory, and whose memory is allowed to be touched.
Sounds to me like sharing rides that maximise economically but we may have some oops moments sitting on someone's laps vs absolute private rides to make sure no one in your family will be harmed but we have to make sure everyone gets a car only when needed.
It is quite interesting to see how it will work out eventually...
I heard that C developers are trained to use memory smartly
Kernel coders are an entirely different breed, and when I worked with a few of them they were just stunning. The smartest man I know on the planet so far coded on the Unix kernel -- the one that IBM forced back to Novell who'd already fired their staff after selling it, and thus shelved it and killed Unix. He is and was amazing.
So yes, I can confirm that Kernel devs know how to manage their memory -- they use very little, they allocate and free it, and they build very small, tight, optimized kernels by knowing how the optimizer will do things and how to hint it to do what they know needs to happen.
Yeah, it's a skill. Yeah, it takes skilled people. I'd like to one day find out that really big training wheels will let anyone build code that well, but I've seen the goal and I don't expect we're there yet.
They are amazing but at the end of the day they are still humans and they can make mistakes. In the YouTube video referenced one of the C devs is heavily against rust.
Decided to go look for CVEs from code the guy manages (Ted Ts'o) I found these
CVE-2024-42304 — crash from undocumented function parameter invariants
CVE-2024-40955 — out of bounds read
CVE-2024-0775 — use-after-free
CVE-2023-2513 — use-after-free
CVE-2023-1252 — use-after-free
CVE-2022-1184 — use-after-free
CVE-2020-14314 — out of bounds read
CVE-2019-19447 — use-after-free
CVE-2018-10879 — use-after-free
CVE-2018-10878 — out of bounds write
CVE-2018-10881 — out of bounds read
CVE-2015-8324 — null pointer dereference
CVE-2014-8086 — race condition
CVE-2011-2493 — call function pointer in uninitialized struct
CVE-2009-0748 — null pointer dereference
Do you see a pattern in the type of error here? It's pretty much entirely memory related and right in the wheelhouse of something rust would just outright not allow short of just slapping everything into unsafe blocks.
The Old Guard is not perfect, and they are acting as a barrier to new talent coming in. Sometimes change is good and I'm heavily in the camp that rust one of those times. Linus seems to agree as he allowed the code into the kernel which he would never do lightly or just because it's fomo
But on the other hand you can’t expect some smaller and smaller subset of the population to primarily just learn C and meet the criteria of a kernel dev.
I absolutely agree with all your points, and most rust devs would agree, but the general idea is that over time that energy (which would have been spent tweaking malloc and such) should be spent on the rust compiler and memory management systems, which is already magic as someone who as written a lot of c, c++, and spent the better part of a year learning rust. (I’m no expert of course, but I have a pretty decent grasp on the low level memory management of both the Linux kernel and the rust compiler).
So that over time the effort that would be spent on memory management and kernel functionality can be properly divided. Rust not being efficient somewhere in catching memory faults or managing memory? Fix it. Someone writing unsafe rust code? Fix it.
I think at the end of the day everyone wants the same thing which is a memory safe kernel, and I think that rust Is being shoehorned into kernel projects too early in places where it shouldn’t be, but I also think there is unnatural resistance to it just because it’s different elsewhere to "how it's always been done."
I guess the question is, what happens to the kernel when all the people who learned on C are gone? The majority of even the brightest new devs aren't going to cut their teeth on C, and will feel the same resistance to learning a new language when they think that there are diminishing returns to be had compared to what's new and modern and, most importantly, familiar.
I honestly get the hostility, the fast pace of technology has left a lot of older devs being seen as undesirable because the don't know the new stuff, even if their fundamental understanding of low level languages could be a huge asset. Their knowledge of C is vast and valuable, and they're working on a project that thrives because of it. To have new people come to the project and say "Yeah, we could do this without having to worry about all that stuff" feels like throwing away a lot of the skill they've built. I'm not sure what the solution is, I really don't think there are enough new C developers in the world to keep the project going strong into the future though. Maybe a fork is just the way to go; time will tell which is more sustainable.
Unfortunately there are a lot of problems created by using C in the kernel, and having all of this done manually. Many kernel vulnerabilities including several severe ones have been due to issues with memory management. Even the whitehouse has spoken on these issues related to C. Rust has been proven to be comparable to C in terms of performance, sometimes even faster. So it doesn't make a great deal of sense to keep using C for new projects.
That all being said Rust has had its own issues. There was a recent vulnerability in older versions of cargo the Rust package manager for instance. It's a somewhat new language so obviously teething issues are to be expected, and it might be too soon to use Rust for mission critical systems. It's also a harder language to learn and understand, so that makes adopting it more difficult especially for very experienced C developers like those who work on the Linux kernel. It might be better to wait and see what other languages like Zig and Carbon manage to do, but those are even newer and will take more time to actually be production ready.
Expecting C programmers to like a compiler-based approach to memory safety is like expecting petrolheads to like a car purely because it's electric. They have always viewed compiler based memory safety techniques as guard rails for novices. In their view, good bowlers don't need guard rails at the bowling alley. It's a massive massive clash of cultures and the rust folks come into the discussion with an assumption that C devs would leap with joy at the chance to automate memory management. Rust and C are complete opposites, but rust programmers seem to assume that just because rust is fast C programmers will love it.
That is honestly a decent analogy. So, on what rides is it ok if something goes wrong and a young family member is killed? Rust says, it is never ok so we won’t let you do it.
To use your analogy though, the issue is the driver feeling quite confident in their skills and rating the risk as low. Then a tire blows on a corner. Or somebody else runs a red light. Or, there is just that one day when an otherwise good driver makes a mistake. History tells us, the risk is higher than the overconfident “good” drivers think it is.
In particular, history shows that 70% of the real world injuries and fatalities come from passengers without seat belts. So, instead of each driver deciding if it is safe, we as a society decide that seat belt use is mandatory because it will prevent those 70% of injuries and fatalities ( without worrying about which individual drivers are responsible )
Rust is the seat belt law that demonstrably saves lives regardless of how safe each individual driver thinks they are. It is a hard transition with many critics but the generation that grows up with seat belts will never go back. Eventually, we will all realize just how crazy it was that they were not always used.
Is there an email thread where c and rust Linux kernel devs actually discuss what's about? Because so far I see some Linux drama and I have no slightest clue what's about
Rust guys want to make the kernel safer, more expressive, and easier to maintain. To do that they need to know how the kenrnel talks between its parts to ensure they are creating matching behavior. The C guys don't really care about the Rust guys and say that they can't be bothered to guarantee interoperability because they like to change how things work on the C side to make things better in the C code.
The rust community is its own worst enemy. The political infighting and constant compulsion to shit on other languages is a turn off to many, and there are plenty of applications where memory safety is not the highest priority.
In this case, the issue is really the C kernel devs, not the Rust devs. Some are not open to new ideas at all. Take a look at the conference video he linked to for example: https://youtu.be/WiPp9YEBV0Q?t=1529. He clearly states that he's not trying to make the C devs learn Rust.
I've seen the video and I agree with Ted. Anyone with experience understands creep, and although Wedson denies it it's exactly where they're headed. Ted and others are right to voice these concerns and attempt to set very clear expectations for the rust developers.
They took on the task knowing it was experimental, would be difficult, and that they would be second class citizens - you don't get to agree to the terms and then complain about them later.