Skip to content

Rook

How Rook Auto-Backports to Each Release Branch Without the Manual Cherry-Picks

Rook
Location Distributed (CNCF Project)
Customer Since January 2021
Team Size 8 maintainers
TN

Travis Nielsen

Senior Principal Software Engineer at Red Hat, Lead Maintainer for Rook

Company

Rook is the CNCF graduated cloud-native storage project, the operator that runs Ceph storage clusters on Kubernetes. The repository has crossed 270 million downloads and counts more than 400 unique contributors, but the day-to-day work happens in a maintainer team of around eight people.

Challenges

  • Backporting fixes to multiple release branches by hand was painful enough that fixes often weren't backported at all
  • A small group of part-time maintainers had to keep review quality high across PRs from a wide pool of outside contributors
  • Patch releases shipping every one to two weeks meant a constant trickle of merges across multiple long-lived branches

Solutions

Rook is the operator that runs Ceph storage clusters on Kubernetes. It graduated from the CNCF in 2020 and now sits behind production storage at hundreds of organizations, with 270 million+ downloads and 400+ unique contributors since 2016. Travis Nielsen is the only one of the original creators still on the project, leading a maintainer team of around eight people, only a handful of whom work on Rook full time.

The volume of contribution that flows into a project like Rook is the kind of thing that quietly destroys small maintainer teams. Every PR opened is a context switch. Every release branch that needs a fix is a manual cherry-pick someone has to remember to do. Rook ships a release every three to four months and a patch release every one to two weeks on top of that, which means there are always two or three branches alive at the same time.

I'd say we backport a lot more now than we used to because it was more painful to backport manually in the past.

TN

Travis Nielsen

Senior Principal Software Engineer at Red Hat, Lead Maintainer for Rook

The pattern Rook landed on is narrow and effective. All PRs target main. Once a PR is approved by a maintainer and CI is green, the maintainer merges it. So far, nothing unusual.

What happens next is the part that pays. When a PR carries a backport label, Mergify automatically opens the matching backport PRs against the active release branches. Each backport PR runs CI, and once it's approved, Mergify merges it on its own. The maintainer who made the original merge decision doesn't need to think about the release branches again. Manual cherry-picks and the "did anyone backport this fix?" Slack messages mostly stop happening.

Travis singles out two smaller features that round out the workflow: notifications when a PR develops a conflict, and a warning when someone opens a PR directly against a release branch instead of against main. Both are the kind of thing nobody asks for in a demo, but that quietly remove a category of PR-review-mistakes once they're in place.

If you have a similar workflow with release branches and merge to the main branch first, configure the backport to open PRs automatically and merge automatically. Get that feature working!

TN

Travis Nielsen

Senior Principal Software Engineer at Red Hat, Lead Maintainer for Rook

Across the past 12 months, Mergify has handled about 360 PRs on the Rook repo from 57 distinct contributors. The numbers aren't enormous compared with a monorepo at a 200-engineer company, but they don't have to be. The point of the case for Rook is that those PRs land without anyone having to babysit them.

The behavioral change Travis points to is the most telling part. The project backports more fixes than it used to, not because the maintainers are working harder, but because the cost of doing it dropped to roughly zero. Patch releases ship more reliably as a result. For a CNCF project run by a few full-time engineers, that's the kind of automation that decides whether release management feels sustainable or whether it slowly grinds the maintainers down.

Other customer stories

Engineering teams we helped merge faster, safer, and cheaper

Move faster. Break less.

2k+ organizations use Mergify to merge 75k+ pull requests a month without breaking main.