eBPF programs traditionally run in the kernel and are written in C. I know enough C to get by, but eBPF demanded more of it than I was willing to invest - the esoteric constraints, the manual memory discipline, the lack of any safety net. I wanted something with C-level power but more forgiving. I wanted Rust.

The problem was I didn't know Rust.

Why not just write the C

The kernel-side of an eBPF program has real constraints. No standard library. No heap allocation. A verifier that will reject your program if it can't statically prove safety. Writing C under those conditions means careful manual management of everything the language normally leaves to you.

Most complex eBPF programs have at least 2 parts:

  • The eBPF Code (C - runs in the kernel space)
  • Userspace Code that loads the program into the kernel
  • (Optional if not rolled into the above) another userspace program that interacts with eBPF maps

The canonical answer to writing a loader was to use libbpf in C. The effect on your codebase was that you had to use libbpf as a submodule (and I hate submodules with a passion) and to start from libbpf-bootstrap which attempted to make this process easy - another lesson I found out the hard way.

But I already had a userspace language I was good at (Go), a sense of what I was giving up (memory safety, modern ergonomics, a decent type system), and a strong preference not to go backwards or maintain any more Makefile magic. If I was going to learn a new systems language, I wanted it to be one I'd actually want to use again.

Rust was the obvious candidate. Memory safe by construction, no garbage collector, strong type system, a robust build system, increasingly solid ecosystem for systems work. The question was whether I'd actually learn it or just bounce off it like I had every other time I'd tried to pick it up.

The stubborn path

The thing about using a language to solve a real problem is that you can't quit without giving up on the problem too. Every time the borrow checker rejected my code, or a trait bound I didn't understand blocked compilation, or I struggled to express something that felt simple in Go - I had to push through, because I actually needed the program to work.

That's the forcing function. Not a course, not a tutorial, not a toy project. A real thing I needed to ship, in a language I barely knew, with a verifier that would also reject my code at load time if I got it wrong.

The tool I used was aya - a Rust library for writing eBPF programs. Both the kernel-side code (which runs in the eBPF VM) and the userspace loader and controller are written in Rust. One language, both sides. No context switching back to C for the interesting part.

aya in practice

What made aya specifically tractable as a learning vehicle is that it's well-designed enough that using it correctly and using Rust correctly tend to point in the same direction. The no_std constraint on the kernel side forced me to understand what the standard library actually provides and what I needed to replicate. The proc macros that mark eBPF program entry points introduced me to Rust's macro system by example. I was forced to deal with errors gracefully, and not just unwrap().unwrap().unwrap() my way to success on the eBPF side. The userspace side was more forgiving, and the error messages from clippy and the compiler were easy to understand when I'd gone wrong.

It helped that aya has an active community. When I got stuck - which was often - there were people who could explain not just how to fix the code but why Rust wanted it that way.

From user to contributor

I didn't start contributing code. I started improving documentation - the parts of open source that are always needed and rarely glamorous.

However, it wasn't long before I wanted to do something that wasn't yet supported in the library. I am immensely grateful to Alessandro Decina for both his guidance and his patience with my early contributions. The takeaway is that just because it compiles, it doesn't make it right. I learnt more about idiomatic Rust and API design from working on Aya than I have in any other project, since Aya's design was very deliberate.

I have some very fond memories of the rabbit holes this has led me into:

  • Reading raw bytes of BTF and BPF programs for fun and profit
  • Learning how compilers work, reading LLVM IR and working on a Rust eBPF compiler
  • Tinkering with Parser/Combinators

The borrow checker

The borrow checker is usually the thing that separates those who've failed to learn Rust and those who love it. If you're coming from a language that doesn't have strong opinions on ownership or mutability then buckle up, you're in for quite a ride. The learning curve is steep, but I can assure you it's worth it. Even if you don't love Rust at the end of it, you'll actually come away with your own "internal borrow checker" that lives in your head, rent free, and is portable to other languages. Ask me how I know. Every time I'm writing Go now I constantly ask myself - who owns this? who needs to mutate it?

The thing I eventually realized is that fighting the borrow checker is like fighting the final boss in a video game where the final boss is... your evil counterpart - "shadow you". Yes, you're fighting your own bad design. Either something is being shared when it shouldn't be, or you're trying to hold onto a reference longer than its owner lives, or the structure of your program doesn't match the structure of the problem.

Once I stopped trying to win arguments with the compiler and started treating its errors as design feedback, things got considerably easier.

The honest summary

Learning Rust through eBPF is not the efficient path. It's two hard things at once. The eBPF execution model is unusual, the Rust ownership model is unusual, and when something doesn't work you have to figure out which unusual thing is responsible.

But having a real problem to solve kept me in it when I would otherwise have given up. Stubbornness is underrated as a learning strategy. Don't learn a language in the abstract if you can help it. Find something you actually need to build, pick the language you want to know, and make yourself finish it.