### Kali Linux: Delivery of a rolling distro at scale with Mirrorbits Slides available at<br> https://arnaudr.io/slides/ Git repo for the slides at<br> https://salsa.debian.org/arnaudr/debconf25-kali-mirrorbits-slides/

### About me - Arnaud Rebillout \<arnaudr\> - Kali Linux Developer (day job @ OffSec) - Debian Developer since 2021 - [https://arnaudr.io](https://arnaudr.io)

### Talk Summary - Kali Linux Intro - Mirrors and Redirector - From MirrorBrain to Mirrorbits - Mirrorbits for a Rolling Distro - Interlude - 3 Years of Kali Mirrors Management - Mirrorbits for Debian?

### Introducing Kali Linux What is Kali Linux?

https://www.kali.org/

<small>Below is a rough overview of how Kali Linux came to be:</small> | Date | Project Released | Base OS | |----------------|---------------------------|-----------------------------| | 2004-August-30 | Whoppix v2 | Knoppix | | 2005-July-17 | WHAX v3 | Slax | | 2006-May-26 | BackTrack v1 | Slackware Live CD 10.2.0 | | 2007-March-06 | BackTrack v2 | Slackware Live CD 11.0.0 | | 2008-June-19 | BackTrack v3 | Slackware Live CD 12.0.0 | | 2010-January-09| BackTrack v4 (Pwnsauce) | Ubuntu 8.10 (Intrepid Ibex) | | 2011-May-10 | BackTrack v5 (Revolution) | Ubuntu 10.04 (Lucid Lynx) | | 2013-March-13 | Kali Linux v1 (Moto) | Debian 7 (Wheezy) | | 2015-August-11 | Kali Linux v2 (Sana) | Debian 8 (Jessie) | | 2016-January-16| Kali Linux Rolling | Debian Testing | <!-- .element.table: style="font-size:0.6em;" --> <small>Source: https://www.kali.org/docs/introduction/kali-linux-history/</small>

The 2 Kali Linux "archives" (aka. repositories): - packages archive: https://http.kali.org/ - images archive: https://cdimage.kali.org/ Infrastructure to deliver it to the world: - Mirrors from community (~ 70) - Redirector (mirrorbits) - Cloudflare CDN (since 2019)

Kali Community Mirrors: https://http.kali.org/README?mirrorstats

### What's a mirror, what's a redirector SSH, Rsync, HTTP - that's all it takes

### From MirrorBrain to Mirrorbits - Kali Linux used **MirrorBrain** for 10 years - Switched to **Mirrorbits** in December 2023

- **MirrorBrain**: https://mirrorbrain.org/ - born in 2006, last commits in 2015 - written in C, Python and Perl - stuck in the past: - Apache module (no support for NGINX) - Python 2 (EOL Jan 1, 2020) - Legacy MaxMind GeoIP databases (retired Jun 1, 2022)

MirrorBrain development stopped in 2015: Source: https://github.com/poeml/mirrorbrain/graphs/contributors

- **Mirrorbits**: https://github.com/etix/mirrorbits - written in Go, use the Redis database - started in 2014 by Ludovic Fauvet for VideoLAN

Mirrorbits development stopped from 2020 to 2023: Source: https://github.com/etix/mirrorbits/graphs/contributors

- Nov 2023: new project owner: Jean-Baptiste Kempf - Commits get merged again - Most of the Kali Linux patches are upstream - Apr 2025: version 0.6 is released! 🎉 - Mirrorbits is alive and well!

### Mirrorbits for a rolling distro - the `kali/` archive has around 500k files - updated every 6 hours - metadata updated in-place (eg. `Release` file)

Challenge 1: RAM usage - Redis DB is entirely in RAM - Mirrorbits is hungry for RAM! - MirrorBrain: `3.7 GB` -> Mirrorbits: `13 GB` (x 3.5) - Before v0.6: 2 DB, one for HTTP and one for HTTPS

Challenge 1: RAM usage (solution) - Repo cleanup: from ~ 1 M files to ~ 500k - DB size is function of: `n files x n mirrors` - Add support for HTTP+HTTPS mirrors in Mirrorbits v0.6

Challenge 2: Metadata updated in-place - `kali/dists/kali-rolling/Release` - Redirect only if remote file matches (size, modtime) - Mirrorbits scans local repo every 5 minutes, remote mirrors every hour - Otherwise: fallback mirror to the rescue! - Consequence: 2 hours out of 6, `apt update` traffic is redirected to fallback, then gradually back to mirrors

Challenge 2: Metadata updated in-place (solution) - Allow redirections to outdated files in Mirrorbits v0.6

Challenge 3: File deletion - Mirrorbits returns 404 when file missing in local repo - Outdated metadata refer to files that no longer exist in the pool, so `apt upgrade` requests files that don't exist

Challenge 3: File deletion (solution) - Keep deleted packages for a grace period (3 days) - Done at the reprepro level (snapshots)

Challenge 4: Latency spikes in Redis - Redis is single-threaded - Mirror scan result is expensive to commit to DB (500k files) - Results in latency spikes, caused by two things: - Unbounded `MULTI` transactions (up to 5 seconds) - Expensive commands `SINTERSTORE` and `SDIFF` (up to 500ms)

Challenge 4: Latency spikes in Redis (solution) - Break `MULTI` transactions in smaller chunks (patch not upstreamed...) - Expensive commands `SINTERSTORE` and `SDIFF`: no solution in sight, so we live with 500 ms latency spikes

### Interlude - Kali Linux is a rolling distro based on Debian Testing - Delivery: network of ~70 mirrors and a redirector - MirrorBrain did the job for the first decade - Mirrorbits is the successor, in prod for 2 years - v0.6 works very well!

### 3 Years of Kali Mirrors Management - I took over Kali mirrors admin in Jan 2023 - Transitioned to Mirrorbits end of 2023 - Mirrors network: from 48 → 72 - Takeaways? What does it take to operate a network of mirrors and a redirector?

- Mirrorbits upfront cost: - Understand the basics, configure, test, deploy, monitor - Mirrorbits operational cost: - Minimal

Mirrors network, operational cost: - Check monitoring: - https://mirror-traces.kali.org/ - https://cdimage.kali.org/README?mirrorstats - Fix issues with mirrors: - out of space, TLS certificate, firewall, ... - 80/20 rule - Onboard new mirrors - Develop more/better monitoring tools

### Mirrorbits for Debian? If it works for Kali, would that work for Debian?

Debian is significantly "bigger": | | Kali | Debian | Factor | | ---------- | ------: | --------: | ------ | | Files | 390 k | 1,640 k | x 4.2 | | Mirrors | 72 | 375 | x 5.2 | | Redis keys | 28 M | 615 M | x 22 | | RAM usage | 12.4 GB | 272 GB | | <!-- .element.table: style="font-size:0.8em;" --> <small>Beware! Crude approximation!</small> - N files: https://cdn-fastly.deb.debian.org/debian/ls-lR.gz - N mirrors: https://mirror-master.debian.org/status/Mirrors.masterlist <!-- .element.ul: style="font-size:50%;" -->

Solutions? - Only the "best" mirrors in the pool - Keep RAM usage reasonable - Won't fix latency spikes! - Mirrorbits / Redis clustering

Kali Mirrorbits instance in numbers - Busy server: from 20 to 50 millions of requests / day (230 to 580 requests / seconds) - Bare metal @ OVH Canada - RISE « the most affordable » dedicated server - CPU: Intel Xeon-E 2236 - 6c/12t, RAM: 32 GB - RAM usage: 14 GB, CPU idle at 80%

### Thanks: - [OffSec](https://www.offsec.com/) - my employer - [reveal.js](https://revealjs.com/) - the HTML presentation framework - **you** for attending