* AGGREGATE. You can extend single windows past 262K native via YaRN but don’t. If this is your first time not using APIs, hi! 750 TPS is about 500-600 words per second.

Link to previous chapter | Link to next chapter (WIP) | Link to companion GitHub repo.

Your env/hardware should reflect what’s been set up in the series, especially Chapter 5.

Definition: “such a wide margin”

If you told someone in Q4 2025 that a model with the below specs would be open sourced in the next 6 months, they’d laugh at you. If you told them that it would also be under 300B parameters they’d cry for you. More on time machines in later sections.

Qwen 3.6 27B dense TL;DR:

• Destroys the Qwen 3.5 397B-A17B, which was no slouch, on most benchmarks.

• 262k native context with a KV cache memory footprint smaller than your car keys.

• SWE-bench Pro: 53.5 vs Google’s recent “SoTA” Gemma 4’s 35.7

• SWE-bench Verified: 77.2 vs Google’s recent “SoTA” Gemma 4’s 52.0

• Terminal-Bench 2.0: 59.3 vs Google’s recent “SoTA” Gemma 4’s 42.9

• Claw-Eval (lmao) Avg: 72.4 vs Google’s recent “SoTA” Gemma 4’s 48.5

And it’s 12% smaller than Gemma 4 31B, which does matter when you’re VRAM poor.

35B-A3B TL;DR: is a 3B active sparse model that beats Gemma 4 31B dense on coding benchmarks while also being around 6-9x faster for inference on a 5090. The 3.6 27B dense absolutely destroys the Gemma 4 31B dense, which is a fairer comparison.

If you’re on a 5090 and have other models in /hub/, rm -rf that because the 35-A3B beats Opus 4.7 …is what you’d read on LinkedIn 🤡. Okay it does, but A FLAMINGO? Nah. It unironically trades blows with Sonnet 4.5 though. Lots of threads like this:

And we’re about a spin up a cracked version of it on a cracked vLLM image. Strap in.

Prerequisites

Install 596.36+ Nvidia drivers or higher on the host and CUDA 13.2+ in WSL. Run:

# Before anything, kick this off in a separate terminal window because it takes a while:
docker pull yevai/local-inference-qwen36:sm120-cu132-v4
# - Bleeding edge(lord) private fork of torch@2.13.0.dev20260502+cu132 with extra goodies
# - My startup's vllm@0.0.0+sm120.custom.cu132 private fork with a few goodies left out
# - I didn't do anything for this except build from main: llmcompressor@0.10.1.dev127+g76b28ce7
# - flashinfer@0.6.10rc1 with a few bug fixes / perf improvements. Their code is getting good.

# Also kick these off in separate terminal window(s):
hf download sakamakismile/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-NVFP4
hf download sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP
# * PREVIOUS CHAPTERS! hf is aliased to huggingface-cli

# Pull down the latest version of the companion repo and check that all is well:
cd docs/chapter-09
chmod +x wsl-cuda-upgrade.sh
./wsl-cuda-upgrade.sh #Idempotent #FYP

# The explanation to this one is that I love you and you're awesome.
mkdir -p "${HOME}/.cache/vllm"

Configurations

Swarm vs hive use cases have been beaten to death to such an extent that even some of the LinkedIn ArmchairI Experts™ are starting to get it right so we’ll let that one sit.

If the 35B-A3B can’t cut it, swap in the 27B. I decided to post this before I unf*cked my DFlash speculative decoder for the 27B so it’s quality but much slower. My speculative decoder game is rusty. The last time I was good at autoregressive drafters was Sully AI and that got them 1,000+ TPS per user on a 400B model for ~30x cheaper than APIs. Last time I was good at block/parallel drafters was never so it’s a learning experience.

When these start up, try “./docs/chapter-09/fluffer.sh” to get them in the mood. If you don’t have the .env file, see “Getting Started” / setup.sh in Chapter 5. If you noticed vLLM spinning up with, say, ~300K more context capacity than you’d expect a model of this size and KV cache arch to have on 32GB of VRAM, no that is not a coincidence.

# DX QOL utils
cd docs/chapter-09
chmod +x run.sh

# See available options
./run.sh

# Tail inference container logs
./run.sh logs

# Get container status as JSON
./run.sh status

# Open interactive GPU perf monitor
./run.sh perf

# Stop + remove all shared containers
./run.sh purge

Qwen 3.6 A3B Swarm

700+ TPS across five 131k windows for you! Swarms are a crowd favorite for codebase modernization, package upgrades, wide-sweep refactors, and most other daily tasks.

cd docs/chapter-09 && ./run.sh start:swarm-only
# Tail logs
cd docs/chapter-09 && ./run.sh logs
# Inference AND indexing. The only way you'll have the VRAM for this is if you implemented
# the previous chapter and your only video output is from the motherboard's thunderbolt.
cd docs/chapter-09 && ./run.sh start:swarm

0.89 only allows for a KV cache of 566k tokens, which is a maximum of 4.32x requests, but LM code harness swarms are guaranteed to have extensive shared caching (initial system prompts, etc). The odds of all 5 requesting unique 131k contexts are very low.

Qwen 3.6 A3B Hive

450+ TPS across three 262k windows on this one. Most hive use cases are slightly above this model’s weight class so I didn’t spend much time optimizing this one.

cd docs/chapter-09 && ./run.sh start:hive-only
# Tail logs
cd docs/chapter-09 && ./run.sh logs
# Inference AND indexing. The only way you'll have the VRAM for this is if you implemented
# the previous chapter and your only video output is from the motherboard's thunderbolt.
cd docs/chapter-09 && ./run.sh start:hive

0.93 only allows for a KV cache of 723k tokens, which is a maximum of 2.76x requests, but LM code harness hives where 1 hub drives >3 spokes while thread-swapping with one tend to have even higher shared cache guarantees than most swarm topologies.

Qwen 3.6 A3B Assistant

185+ TPS with 90ms TTFT single request. Use this if you downloaded the prerequisites above, went back to 2023 in your time machine, and wiped your memory of the future.

# Not recommended unless you're bringing your own code indexing!
cd docs/chapter-09 && ./run.sh start:assistant-only
# Tail logs
cd docs/chapter-09 && ./run.sh logs
# You have enough VRAM for this and then some! I recommend this one.
cd docs/chapter-09 && ./run.sh start:assistant

Up Next (still)

We’ll be covering the following for a specific (your own) codebase:

1. Step outside and touch grass. Yes, for your codebase.

2. Lay the groundwork for profiling and evals.

3. Evals for a direct model-vs-model comparison.

4. Evals for coding harness comparisons.

5. The delta between, and interaction of, key metrics on #2 and #3.

6. Understand why optimizing for #2 often hurts #3 and how to fix it.

7. Define target performance metrics and improve a harness to reach them.

I don’t know why you’re still scrolling but if you’re this lonely:

1. Refer to item #1 in “Up Next (still)”.

2. Use your time machine better.

3. Last resort: ./run.sh chat

[IMPORTANT] - I’m working on updated images and docker compose for Part 9, which you can expect today or tomorrow. Part 9 implements TurboQuant, Qwen3.5, Gemma4, and three other new papers I found highly useful. The new advances also make this post optional. Feel free to skip it unless:

• You want to free up that last 2GB of VRAM for something else

• You want pixel-perfect, <1ms desktop extension to a MacBook Pro

  • This includes 48kHz sound so you can use it as a speaker for your PC!

• You’re curious, bored, and/or easily amused

This chapter is for anyone with an M4 MacBook Pro. It lays the groundwork for taking full advantage of all of our local compute and treating our PC as a true headless bare metal cloud node. This is useful for the casual crowd because it lets you extend your PC desktop to any number of MacBooks, iPads, or iPhones for free, locally, with little to no visual quality loss and 1-3 ms latency (it forwards audio too!).

Link to previous chapter | Link to next chapter | Link to companion GitHub repo.

What are we doing?

1. Side Quest: Add a PC ←→ MacBook 2.5G Ethernet connection

   1. Why (casual): extend your PC desktop to the MacBook’s 16” XDR display

      …for free, without internet, at full resolution, with 120Hz, and ~3ms latency.

   2. Why (actually): dedicated GPU resource lock and IPMI control like WOL. We’ll use the iGPU to render / encode / stream our screens and free up those USB4/DP ports for direct 40Gbps interconnects with our MacBook so we can:

      1. Remote KV cache offload from our PC to our MBP’s massive 128Gb.

      2. Cross-arch (CUDA/MPS) model + KV cache partitioning with Exo.1

No compute left behind, ladies and gents: we take the PC and MBP we are given.

Side Quest: Casual

We’re gonna do this simple, fast, free (software-wise), and dirty. If you already know networking and distributed GPU compute, I apologize in advance for what follows.

Disclaimer: Amazon Affiliate links. I don’t shill stuff and genuinely recommend them.

Required items: USB-C to 2.5G Ethernet ($30)2 and a Cat 8 RJ45 cable ($10)3. Next, connect your devices: MBP USB-C → TPLink Adapter → Cat 8 → PC 2.5G RJ45 Port.

PC Network Setup

Open up PowerShell as Admin and run ”Get-NetConnectionProfile”:

Find the entry with Ethernet in its InterfaceAlias and the lowest InterfaceIndex. For us, that entry’s Name is “Unidentified Network” and its Alias is “Ethernet 3”. In PowerShell:

# Make the network private
Set-NetConnectionProfile -Name "Unidentified Network" -NetworkCategory Private

# Allow incoming ping requests
New-NetFirewallRule -DisplayName "Allow Ping" -Direction Inbound -Protocol ICMPv4 -IcmpType 8 -Action Allow

# Set your Ethernet IP and subnet mask (hardcoding for simplicity)
New-NetIPAddress -InterfaceAlias "Ethernet 3" -IPAddress 192.168.205.1 -PrefixLength 24

# Set manual DNS - we have to use netsh directly for this to work
netsh interface ip set dns "Ethernet 3" static none

# You'll see a "There are no DNS servers ...blah blah" message - this is expected.

Mac Network Setup

In terminal:

# Get the LAN network device name
networksetup -listallhardwareports | grep " LAN" | awk -F': ' '{print $2}'

# You'll see something like "USB 10/100/1000 LAN" - substitute the below as applicable

# Set IP and subnet (hardcoding to keep it simple)
sudo networksetup -setmanual "USB 10/100/1000 LAN" 192.168.205.2 255.255.255.0 ""

# Set manual DNS
sudo networksetup -setdnsservers "USB 10/100/1000 LAN" "Empty"

# Confirm you can ping your PC, should be sub-millisecond
ping 192.168.205.1

This works since a 255.255.255.0 (/24 CIDR) subnet mask is basically “any IP starting with the same first three numbers as yours is a local neighbor”. CIDR /24 means:

192 (8 bits) . 168 (8 bits) . 205 (8 bits) . [Any 0-255 number here is a local neighbor]

8 + 8 + 8 = 24! We went with 205 because… I’ll spare you the painful stories. Basically, consumer networking hardware hogs the low numbers (.0, .1, .10, etc), VPNs love .50, a lot of management interfaces sit at .100, and enterprise hardware LOVES 210+. The goal here was to not inadvertently break your 💩 if you have a work VPN, router, etc.

PC Software Setup

Virtual Display

First up, we need VDD. Here is a direct download link for the VDC 25.7.23 Beta. Extract the ZIP contents to C:\VirtualDisplayDriver and run the executable. Hit “Install Driver” and “Restart Driver”. Your OS will be unstable until reboot. You should see:

Close the GUI. Open C:\VirtualDisplayDriver\vdd_settings.xml in Notepad or another editor of your choice. We want to add the native MBP 16” Retina display spec (ProMotion is variable 120Hz) to the tail end of “resolutions”, which would be:

    <resolution>
      <width>3456</width>
      <height>2234</height>
      <refresh_rate>120</refresh_rate>
    </resolution>

The end of that XML (cue Java/Maven flashbacks) should look like this:

Restart your PC, right-click on the Desktop, select “Display Settings”. Click on the new virtual display so it’s highlighted in blue, hit the dropdown and make sure it’s set to “Extend desktop to this display”. Hit “Apply”. It should look similar to this:

Scroll down, set the “Display resolution” to 3456 x 2234. Scroll down some more, click “Advanced display” and set the refresh rate to 120Hz. Apply → Keep changes.

Casting Host

Next up, we need Sunshine - and no, not actual sunshine you lunatic. We don’t touch grass out here and our only tan comes from the monitors → direct download link. DL and install it. When everything is up and running, you’ll be able to access the UI here.

Do your first-time login, set the username and password, restart sunshine, re-log in.

Select “Network”, set “Address Family” to IPV4 Only and “Origin Web UI Allowed” to “Only localhost may access Web UI” ← Never use sunshine without this. Like so:

Remember:

• The UI lets you accept device pairing requests.

• Paired devices may view, or control, your PC.

• Maintain security best practices on paired devices.

If all you want is screen extension:

1. Select “Input” in the top menu

2. Uncheck “Enable Gamepad Input”

3. Uncheck “Enable Keyboard Input”

4. Uncheck “Enable Mouse Input”.

5. Click “Save” and “Apply”.

Mac Software Setup

Is simple! Moonlight Is All You Need; here’s the direct download link for what you want.

Download → Install → Grant Permissions → Launch → Click the gear icon in top right:

Use the above settings. Go back to the main screen, click the “+” sign in the top right.

The IP address we want is “192.168.205.1” - this will forces our stream to only use the Ethernet connection we set up earlier. Click on it, you’ll get a “PIN” popup on the MBP.

Go here on your PC, put in the PIN and “Mac” for the Device Name. Like this:

Back on your Mac, open a terminal and run “ping 192.168.205.1” in it for three reasons: (1) keepalive, (2) confirm it’s sub-millisecond, and (3) so you can tab out.

Open this article anywhere but your MBP (Phone, PC, etc) before proceeding.

On the Mac, in Moonlight, click on the PC icon, and click the play button that appears when you mouse over the “Desktop” application. You’re going into the screen share!

To switch which screen you’re casting, use Fn + Ctrl + Option + Shift + F1/2/3/4. Go ahead and switch to the virtual screen now. Next up, toggle Moonshine statistics on by pressing Ctrl + Option + Shift + S. To exit Moonshine, use Ctrl + Option + Shift + Q.

Or use Command + Tab to that terminal to toggle between MacOS and Moonshine!

That’s it. As a filthy casual, you can rinse/repeat this process to extend any number of virtual PC displays to any number of iOS (or other PC) devices, audio included.

Side Quest: Actually

The point is to preserve our convenient daily driver workflow while fully offloading our 5090 and leaving both USB4 ports free for 40Gbps connections. For now, we just need one - but it does have to be a dedicated one. We can’t have other stuff going over it.

And we might as well abuse our useful daily driver solution (extend to MBP) for it!

We’ll lay the groundwork for setting up an enterprise WiFi network in a residential property. I’m installing Ubiquiti at one of my properties (IoT prep) and will detail the wiring, wall runs, terminations, AP positioning, and Ubiquiti wired / wireless setup.

I always do home automation myself because brands like Crestron, Savant, Control4, Lutron, etc are extremely limited in their options / capabilities and the vast majority of installers for them do not know a single f*cking thing about networking security.

Analogy: The Water Pipe

“Throughput" is how many liters or gallons per minute can flow through a water pipe.

“Latency” is the time delay (not speed) for water to go between two ends of the pipe.

“Reliability” is how much water leaks out of the pipe along the way.

“Robustness” is the water pipe’s ability to maintain pressure and flow rate under stress.

“Congestion” your water main struggling to simultaneously supply a garden hose and 3 showers. For us, this relationship will be one to many, not many to many (switches).

Analogy: Mail and Packages

To make this easier, read the real-world example first:

When you walk into FedEx, you give them an address to send something to. Next, you declare if the package has batteries and if you want it shipped via Overnight Air, Ground, etc. Whether you decide to use UPS instead of FedEx, you still choose if the recipient should know details, like box dimensions, about the package.

Tracking numbers let you know where the packages are delivered, lost, returned, etc. If delivery requires a signature the tracking number shows “delivered” slower, but we know they got it. If it doesn’t, the “delivery” is faster but we don’t know if they got it.

Delivery is optimized: Amazon delivers multiple packages at once under the same tracking number. One of the best ways to deliver something is a briefcase with a PIN code, shipped without requesting signature, that fires a signal when it’s been opened.

You have to leave your suite or floor to send a package. Buildings are grouped by zip code or city. Let’s say each company in a building has its own floor. A floor has desks.

In an office with bad security, what someone with a given name can access would be governed by what desk they sit at. We say “given name” because someone can walk up and tell you whatever name they want, like writing your own nametag. This is why office buildings use Employee ID badges that security guards check with HR to make sure they’re valid.

Now let’s add the networking explanation. If you lose track, refer to the plain real-world example above to reinforce how each concept relates to the others:

When you walk into FedEx, you give them an address to send something to. This is OSI Layer 7 (DNS, HTTP, HTTP/3, etc): the human and software friendly “where”.

Next, you declare if the package has batteries and if you want it shipped via Ground, Overnight Air, etc. The “how” details of delivery are Layer 6. They include encryption (SSL / TLS), compression (GZIP, Brotli), formatting (JSON, Protobuf, gRPC), etc.

If you decided to use UPS instead of FedEx, that’s gRPC: a different Layer 7 framework. UPS and Fedex both let you choose whether the recipient should know details, like the dimensions and contents (metadata), about the package (Protobuf). If you’re shipping to high-volume a warehouse, those details help process it faster. Residential? Meh.

Layer 5 is the state and management of repeated package deliveries. Sessions are like tracking numbers: you know whether the packages are delivered, lost, returned, etc.

Layer 4 is shipping “logistics”. If delivery requires a signature, that’s TCP: the tracking number is sometimes slower to show “delivered”, but we know they got it. UDP is dropping it off at an address: it went there, but we don’t know if a recipient got it.

Amazon delivering multiple packages under the same tracking number is SCTP. A briefcase shipped without “signature requested” (UDP) with a PIN code (SSL / TLS) to open it that sends a signal when it’s been opened (ACK) is QUIC (HTTP/3 uses it).

Note: floors and suites is used interchangeably. Assume each floor only has one suite.

You have to leave your suite or floor (Layer 4, logical ports) to send a package, which is more relevant to “did someone get a package” than the building’s physical location (IP address, Layer 3). Buildings are grouped by city or zip code (Layer 3, subnet mask).

The procedures for shipping between zip codes (BGP) are an advanced concept we’ll skip.

Let’s say that each company (Subnet, Layer 3 IP range) in a building has its own floor (VLAN, Layer 2 broadcast domain). Just like sharing a floor between companies is complex and painful, subnets and VLANs usually have a 1:1 relationship in practice.

A floor has desks (Layer 1, physical port - let’s assume this is RJ45 and not SFP+). In an office with bad security, what someone with a given name (Layer 2, MAC address) can access would be governed by what desk they sit at. This is what the vast majority of home automation installers do: assign specific physical ports to specific VLANs.

We say “given name” because someone can walk up and tell you whatever name they want. Apple devices will do this (rotate their MAC addresses) for privacy by default. That’s like if anyone wrote up name tag saying “John Doe” had his permissions.

This is why office buildings use Employee ID badges (802.1X digital certificates) that security guards (switches) check with HR (RADIUS server) to make sure they’re valid.

Additional Terminology

Many of these definitions are not accurate outside of the context of this article.

• “IoT”: Internet of Things (temperature, humidity, occupancy sensors, etc).

• See this concise article for “Airwave” and “Frequency” (WiFi, RF, Zigbee, ZWLR)

• “Interference” (NOT congestion): noisy neighbor with airwaves and frequencies.

• “Smart Home”: a residential property with airwave-based (wireless) IoT devices.

• “Wired Connections”: RJ45 or SFP+ form factor Ethernet (LAN) Connections.

  • Residential RJ45 connections are classified as Cat5e, Cat6, Cat6a, or Cat8.

• “Gateway” / “Router”: physical (wired) device interconnecting our LAN and WAN.

• “AP” / “Access Point”: a physical airwave broadcaster that connects to a Router with a wired connection. We will not be covering wireless mesh networking.

• “PoE”: Power over ethernet. This wired connection also powers a device (AP, etc).

• “Backhaul”: Connection between an AP and a Router. For us, that’s PoE RJ45.

• “Handover” / “Roaming”: a wireless device changing which AP it’s connected to.

• “Band Steering”: a wireless device changing which frequency it uses with an AP.

• “Switch”: for us, a managed switch that interconnects devices in the same subnet.

  • In smart homes, switches are used to inject PoE into RJ45 connections.

  • PoE RJ45 connections out of this can also go over Coaxial Cable with MoCA.

• “OSI”: Open Systems Interconnection. This is an open standard for how devices communicate with each other. It can be implemented in many different ways.

Pics or It Didn’t Happen

RG59/RG6 Cable, RF, Material, Zigbee, Z-Wave, ZWLR, and IoT hardware isn’t here yet :(

Networking Hardware

Media Box We’ll Be Replacing

Tools, Terminations, and Wiring

The downstream articles will be relevant to you if you are curious, easily amused, or:

• Care about WiFi throughput, latency, reliability, and robustness.

• Have a large (4000sqft+) property and neighbors with WiFi.

• Have multiple floors, the flooring is wood, LVP, or carpet.

• The walls are drywall, rock-wool filled or otherwise (not brick).

• The property is pre-wired with a Media Panel with a 120V outlet.

I wrote two previous (not so beginner) networking articles for cloud: Part 1 and Part 2.

WHY? WHY DO WE NEED AI IN A ROUTER? Spoiler: We don’t. But now that it’s here like most of the subpar use cases for AI/ML being shoved down everyone’s throat…

This abomination has two SoCs. The first is a 4-core 2.6GHz Broadcom BCM4916 and this does what you’d expect a router to do. The second is …a Synaptics SL1680 with 4GB RAM, 32GB EMMC, and a 7.9 TOPS Neural Processing Unit. Hilariously enough, Asus put an embedding model on it with HF and python behind the scenes for fully local “LM” setup assistance. This was neat in that they didn’t use a transformer LM at all (SL1680s ASICs are CNN-optimized), but it worked. Here’s where stuff gets weird:

The secondary SoC is …updated by the primary SoC that’s also the admin network interface? Naturally, the first question is: how are these things connected? As I’m digging around these menus, in Settings → Administration → System …enable SSH:

Three minutes later…

Ok, so this is definitely the primary Broadcom SOC (aes, crc32). Why isn’t it updating?

^ That’s what’s in the syslog when the Web UI is busy failing at life. So …what’s rc?

“What?” indeed. Let’s heed this cue and try infil from the other SoC’s side. A few minutes of digging around later, I found the main reason I got this thing: the secondary SoC can run docker containers via Portainer under “AI Board”.

Cool, let’s install that and open it up:

Literally not even a minute later …Asus dun goofed:

Aight, bet. Basic privilege hijacking 101: run root, priv, host net, and elevated mounts:

And SSH in via the Portainer UI. And just like that, boom, we’ve got host OS root:

ls /host_root
chroot /host_root /bin/sh
ls /usr/bin
cat /usr/bin/fwupdate

Here, we find that there is definitely some kind of IPC bridge between the two:

This thing is remotely triggered by the host SOC, and monitored via IPC. Like this:

What. The. Actual. Fuqq? Yep, that’s the other SoC:

We fix the checksum, pull the firmware, update the “AI Board” from within, and follow that up with an RPC trigger from the primary SoC (that weird script up above looped into standby until we wrote to /dev/shm/reboot_flag) on the secondary SoC. Web UI:

It’s not exactly a rescue …or is it? Yep, we can update the firmware on the secondary SoC by triggering factory reset with a new recovery image. It’s written to isolated dual-boot partitions so what’s on the primary EMMC also survives this, uh, “recovery” 🤡.

Surely now I can spin up the containers I want and mount my USB dri— NOPE. The secondary SoC, the one with docker on it, can’t even access the USB ports. 🤦‍♂️

Let’s fix that. Question is, how? Asus managed to strip fuse/samba kernels out of the secondary SoC, and NFS kernels out of the primary. This was likely intentional …but that 700MB+ firmware file downloaded FAAAST. Way too fast for RS232 or anything of the sort. Some iptable and ndig later, we find that the secondary SoC has a local loopback (169.254* namespace) to the primary. Where there’s a will, there’s a way:

Ok, we have a viable pipe. Luckily, there was a BE19000 Merlin release 2 weeks ago so I didn’t have to splice cdc_acm, cp210x, and god knows what other kernels into it.

Entware it is, I suppose. Nuke a drive, format ext 4 + journal + bin, install it. AsusWRT is locked down with a read-only file system …so we’ll just bind-mount on top of it 😂

Note: Substack blocked me posting this as a code snippet - I literally couldn’t save or publish the draft. I’m guessing they think we boutta do bad thangz. They’re right.

Boom! Our secondary SOC can now mount USB SSDs to Docker at native speeds:

And wollah, the router is jailbroken. You can bypass RSA checks on the “factory reset” script and write whatever you want to the secondary SoC’s OS partitions while bind-mounting over the primary SoC’s FS. The best part? both SoCs can use the USB3.2 SSDs simultaneously. So why did I do this (outside of the fact that this is what I do)?

1. Attach a 4TB SSD on here and run Plex. On your router. With zero other hardware.

2. Your router is now also your local security camera recorder and viewer.

3. #2 lets you start using the SL1680 SoC for things it can be useful for, like running real-time local object detection and classification with Frigate - no more sending your home’s camera footage to Google Home, Amazon, or god knows who else.

4. You’re in charge of your data, it doesn’t leave your house. I plan to build on top of this with a PoE-based whole house automation system. 4GB RAM and 4 cores is more than enough to run hundreds of IoT devices, Home Assistant, HomeKit, etc.

5. From here, you can socat one of the ports to make USB devices accessible to the SoC running docker containers. That means we can plug in a hub and get:

   1. 6000ft+ range on ZigBee, ZigBee2MQTT with the SMLIGHT-SLZB-MR1.

   2. Full home RF spectrum remote coverage with the Bond Bridge Pro.

   3. Full home Z-Wave and ZWLR coverage with the HAC ZWA-2.

   4. And bridge that mesh to also control Bluetooth with Shelly Gen4s.

That even covers Matter/Thread protocol, giving you everything you need to build a pleasant and convenient home that makes your life easy with just a router (which you need anyways) and some thingies plugged into it along with a tiny 400-500W UPS. No bulky servers. That $900 price tag, for a router, is partially justifiable …maybe.

Final verdict: Now that it’s fixed I actually quite like this thing, and it’s not necessarily because of the tech. This lets me create automation that makes my life easier and is reasonably transferrable to renters or buyers - I’d just be handing off a secure local enclave in the form of a house with really good, albeit minimal, networking equipment.

Bonus: Updating Portainer

Is a bit tricky (not really, not if you do DevOps) because Portainer is what’s running the admin container that lets us SSH into the host. As always, it’s good practice to keep our admin container stateless and ephemeral (like all containers). Steps:

1. Log into router Web UI → AI Board → Control & Deploy → Open

2. click “>_” sign to open a web SSH session in the jailbreak container via /bin/ash:

Here’s a summary of how many things I’ve seen go sideways over 15 years of DevOps:

# Run this inside your Alpine admin container via the Portainer web console
apk update && apk add docker-cli
export DOCKER_API_VERSION="1.41"
docker ps # Sanity check

# Pick a tag here (never use latest): https://hub.docker.com/r/portainer/portainer-ce/tags
export PORTAINER_TAG="portainer/portainer-ce:2.39.0"
export PORTAINER_NAME="$(docker ps --format '{{.Names}}|{{.Image}}' | grep "portainer" | cut -d '|' -f 1)"
export PORTAINER_MOUNT="$(docker inspect $PORTAINER_NAME -f '{{ range .Mounts }}{{ if eq .Destination "/data" }}{{ .Name }}:{{ .Destination }}{{ end }}{{ end }}')"
export PORTAINER_EXPOSE="$(docker inspect $PORTAINER_NAME --format '{{json .Config.ExposedPorts}}' | grep -oE '[0-9]+' | awk '{print "-p "$1":"$1}' | xargs)"
export DOCKER_SOCKET="$(docker context inspect $(docker context show) --format '{{.Endpoints.docker.Host}}' | sed 's|unix://||' | awk '{print $1":"$1}')"

# Confirm things look right
cat << EOF
Container: $PORTAINER_NAME
New Tag: $PORTAINER_TAG
Mount: $PORTAINER_MOUNT
Ports: $PORTAINER_EXPOSE
Socket: $DOCKER_SOCKET
EOF

# Once again, weird things happen on SOCs. You should see "Image pulled successfully"
docker pull "$PORTAINER_TAG" && docker image ls | grep -q "$PORTAINER_TAG" && echo "Image pulled successfully."

# Update portainer
docker run -d --rm \
  -v $DOCKER_SOCKET \
  alpine:latest \
  sh -c "apk add --no-cache docker-cli && export DOCKER_API_VERSION=$DOCKER_API_VERSION && sleep 15 && docker stop $PORTAINER_NAME && docker rm $PORTAINER_NAME && docker run -d $PORTAINER_EXPOSE --name $PORTAINER_NAME --restart=always -v $DOCKER_SOCKET -v $PORTAINER_MOUNT $PORTAINER_TAG"

After update (might take 1-3 minutes, that’s normal), go back to admin container and:

# Space is precious here (we haven't installed an SSD yet) so do cleanup:
docker system prune -f

# This nukes *ALL* images that aren't attached to a running container. Should be OK:
docker image prune -a -f

# Don't touch volumes for now. That's gonna come in a later article for home automation.

Next up, we’ll learn how to:

• Permanently mount a 1TB WD NVMe SSD with a trendy matching enclosure.

• Then we’ll start connecting the secondary SOC to IoT frequencies/networks.

• Find out what, if any, useful embedded models we can run. We will train a tiny RL model to balance a home’s twin HVAC system (one on each floor) including gable vents, blowoff, UV control, humidifier and dehumidifier cycling. I wonder if I can get efficiency and comfort by using occupancy sensors and tracking sun position.

Disclaimer: Amazon affiliate links - consider using them if this was valuable to you.

In this chapter we take a detour due to the large amount of feedback I got from folks that have a 5000 series (not a 5090) and want codebase indexing but not inference.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo.

Getting Started

This setup is runnable on most Blackwell GPUs, including laptops. The defaults are 100% stable on 16GB+ VRAM. 8GB is a no-go but if you’re at 12GB just add lower:

• MAX_CLIENT_BATCH_SIZE

• MAX_BATCH_TOKENS

To the ./codebase-indexing/.env file generated in the steps below.

I set aside a few hours to polish up the containers and do a better implementation of Jina AI’s listwise re-ranker architecture in the manager. Hardcoded, filthy, but it works!

# Pull down latest repo version
git fetch --all && git pull

# Download the larger 1.5B code embedding model
hf download jinaai/jina-code-embeddings-1.5b

# Navigate from repo root to ./codebase-indexing
cd codebase-indexing

# Delete the previous .env file for regeneration
rm .env

# Run setup with a non-default (for us) embedding model
EMBEDDING_MODEL="model/jinaai/jina-code-embeddings-1.5b" ./setup.sh

# Start the new containers
docker compose --profile dedicated-codebase-indexing up

Going forward in this series, we’ll be using the Roo Code Nightly build. Download and install it, making sure to delete the existing main release branch. Make sure that you have the repo root open in VS Code before proceeding to the rest of the article:

Your new Roo Code indexing settings (database-looking icon in the bottom right) are:

Uncheck indexing, click save, then re-check it, click save, and click Start Indexing.

Direct Test

# In ./codebase-indexing, launch the tester:
chmod +x qdrant.sh
./qdrant.sh

Select the proper collection (should be index 0). Select a number of results (5-15 works best). When you’re at “Enter search query (or quit or exit):“, try these out:

• “Where do we configure the runtime between Node, Deno, and Bun?” should return exactly what you’re expecting to be there, with scores around 0.55 to 0.75.

• “Where is the runtime-agnostic script shell entry point and setup?”: works.

• FS liveliness check: “Where are LRU caches and API proxies implemented?”

  • If this doesn’t return results from manager.py open the file, add a space, and save it. Retry the query and you should see it now. Your FS listeners are funky.

Let’s Blow It Up

We are going to be intentionally dealing with a 2-3GB+ (compressed) Qdrant vector database so fully expect VS Code to crash several times during this indexing process:

Wait for the indexer to finish between each git clone command. Seeing some batch size and concurrency errors is perfectly normal - Roo Code retries these automatically.

# Run in project root - NOT ./codebase-indexing

git clone https://github.com/RooCodeInc/Roo-Code
git clone https://github.com/langchain-ai/langchain
git clone https://github.com/langchain-ai/langgraph
git clone https://github.com/langfuse/langfuse
git clone https://github.com/n8n-io/n8n
git clone https://github.com/pytorch/pytorch
git clone https://github.com/vllm-project/vllm

# And in case any normal people are reading this...
git clone https://github.com/vercel/next.js

Note: I used my daily driver .rooignore template; consider adopting it for your own use.

If you open nvitop you will see 95%+ spikes pulling 550W+. This means the code and optimizations are pipelining efficiently enough to bandwidth cap the GPU (good):

There is a practical reason for why we just did this. Context7 is nice but it’s also slow, remote, and has rate caps. You can use this approach with additional .rooignore magic to maintain a robust and up-to-date index of the tools / libraries your code uses.

Once the indexing is done go to your local Qdrant dashboard:

Trial By Fire

Open up Roo Code Nightly with a smaller model like haiku 4.5 or gemini 3 flash so the model itself can’t brute-force despite a poor environment, close all open files because they’re sent in the context and can skew our checks, switch to “Ask” mode, and try:

“How does the torch dynamo symbolic tracing graph compilation work in this project?”

At some point in the task, you’ll see embedding searches that look like this (it’s a hit):

Next task: “Tell me about the LangGraph agent state graph node edges in this project.”

Next up: “How does the Next.js middleware request rewrite redirect work in this project?”

I encourage you to compare the code search quality of this setup and custom proxy against what you get out of the box with Visual Studio Code, Cursor, and Windsurf.

Next Up

Given the pivot we’re taking, now that we have high quality codebase indexing and embedding search, the next logical step is to get under the hood and see exactly what these LM code assist tools are actually sending to the model and how to improve it.

We’ll be doing that with a LiteLLM (which we’ll use for many other things) proxy:

That ties our IDEs and LM code assist tools into LangFuse’s tracing and observability:

We will cover how to create a data set and use it with LLM-as-a-judge auto-evals to profile and improve Roo Code performance for your specific workflow and codebase.

If you’d like to jump ahead, I made a docker-compose that includes both.

In this chapter we combine inference, codebase indexing, and optimizations to fit both, alongside the host OS, on a 5090 for a fully local baseline SWE environment.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo.

Getting Started

# Pull down the latest version of the companion repo:
git fetch --all && git pull

# Pull down the inference container (may take 15-30+ minutes):
docker pull yevai/local-inference-qwen3:sm120-cu131-v1

If you set up WSL correctly, you should see the below when you run “lscpu -e”:

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE
  0    0      0    0 0:0:0:0          yes
  1    0      0    0 0:0:0:0          yes
...
 27    0      0   13 13:13:13:0       yes

If you see less than 27 vCPUs in WSL, adjust the “cpuset“ in “docker-compose.yaml”.

Next up, make sure that whatever you’re running on the host is consuming less than 2.5GB VRAM by running nvitop. If you don’t have ten billion tabs of 4K videos open because it’s 5am and you ended up in the weird part of YouTube, you should be good.

# Run this in root (requires going through previous chapters):
docker compose --profile dev --env-file codebase-indexing/.env up

Roo Code profile settings for this configuration:

You’re now able to use Roo Code just like in this video:

That’s it! Runs fast, right? We’ll be taking advantage of that. In the next Chapter, we’ll:

• Set up observability for our LM, embedder, and re-ranker.

• Run benchmarks to evaluate the performance of Roo Code.

• Create Roo Code settings for good quality with these models.

You can skip the below info if you’d prefer to wait for the next hands-on chapter.

Docker: inference-model

The yevai/local-inference-qwen3 image adds single-GPU SM120 optimizations and patches stability and performance bugs in the new version of the vLLM engine.

Now let’s go over why we did (most) of what we did in the new docker-compose.yaml!

runtime: nvidia tells docker to use the NVIDIA Container Runtime instead of runc. This allows injection of the proprietary driver files and CUDA libraries from the host.

CPU Settings

cpuset: “0-15” pins the inference container to CCD1 on the 9950X. CCD stands for “Core Complex Die”, a single physical chiplet on the CPU die. This gives vLLM a private 32MB cache, avoids Noisy Neighbor problems, and ensures we don’t grind CPU pre-processing to a halt by sending traffic across the (relatively) slow Infinity Fabric.

If you want to see inference grind to a halt, try removing the cpuset attributes from the first three containers - they will thrash each other and you’ll get ~20-30TPS.

pid: host lets the container use the host’s Process ID namespace. This removes PID translation overhead and lets preprocessing run with proper CPU-GPU signal handling.

ipc: host lets PyTorch access the host’s IPC namespace (shared memory). We want this because PyTorch relies on /dev/shm to move data between CPU worker threads.

memlock: -1 lets NVIDIA drivers pin memory pages to RAM, which lets it use Direct Memory Access because it knows the memory is (1) reserved and (2) not in a page file.

stack: 67108864 gives us a 64MB stack instead of the default 2-8MB, without which Python itself might get lost in the sauce of recursive matrix operations and fail silently.

Environment Variables

TORCH_CUDA_ARCH_LIST=12.0 tells PyTorch to compile JIT kernels for Blackwell.

NCCL_P2P_DISABLE=1 explicitly disables the NCCL bridge since we’re using 1 GPU.

VLLM_USE_V1=1 tells vLLM to use all the fancy stuff we fixed for SM120/Blackwell.

VLLM_NO_USAGE_STATS=1 disables (internal) observability for a performance boost.

VLLM_FLASH_ATTN_VERSION=3 could be its own article. At a high level, it enables:

• WGMMA, which lets thread groups collaborate on matrix math much faster.

• TMA, which is a specialized asynchronous hardware unit on Hopper+ GPUs.

• A bunch of FP4/8 specialized goodies that we want but won’t go into here.

You might be thinking “But Yevgen, we’re using FlashInfer anyways?” Not quite :)

[OPM/MKL]_NUM_THREADS=16 explicitly uses the same number of threads we cpuset.

CUDA_DEVICE_MAX_CONNECTIONS=1 prevents CUDA from creating multiple streams at the hardware level. All software queues are streamed into a single hardware command queue, which prevents thrashing, massive serialization jitter, and other bad things.

TORCH_CUDNN_V8_API_ENABLED=1 enables the cuDNN v8 Graph API, which enables Blackwell-native Kernel Fusion mechanisms. A sequence like Conv → Bias → ReLU, which is usually 3 separate launches and memory round-trips, is fused into one GPU operation. This massively helps throughput by keeping data resident in the L2 cache.

vLLM Launch Args

—quantization modelopt_fp4 tells vLLM to use NVIDIA ModelOpt, TensorRT-LLMs quantization engine. It’s the only reason this model fits on a 5090 with room to spare. Older 4-bit formats required software unpacking. This feeds directly into the Blackwell FP4 Tensor Cores, effectively doubling memory bandwidth and compute throughput.

--enable-prefix-caching should be a default for all LM code assist tools.

—max-num-batched-tokens “2048” is there because we also had to fit the re-ranker and embedding model into VRAM. The CUDA Graph config offsets the performance hit from this while conserving precious VRAM, which we don’t have that much of:

Thanks to that, we can use --max_model_len “131072”: a 131k context window. Our version of Qwen3 has 262k context natively but we’d have to turn off the embedding and re-ranker to afford VRAM for a 262k KV Cache - which you can! That’d put our very own local model context window up there with a good chunk of SoTA models.

--kv-cache-dtype fp8_e4m3 is the other piece of magic that makes this possible by quantizing our KV Cache. A 131k FP16 cache would nuke VRAM. In our very specific case the accuracy loss from quantization is <0.05%. 3 mantissa bits distinguish subtle attention signals from noise by trading dynamic range to buy back inference accuracy.

—max-num-seqs “1” is not great for agent swarms. In fact, it sucks. But it’s what we can afford in VRAM poverty. Combined with —gpu-memory-utilization “0.75”, this gives us a 1.07x token buffer which is enough for PyTorch’s memory shuffling.

These things are what let LM code assist tools like Roo Code properly use the model:

• —enable-auto-tool-choice

• —tool-call-parser qwen3_coder

And that’s it! With a relatively simple setup, we’re off to the races with SoTA codebase indexing capabilities and a capable primary model boasting a 131k context window. I might try to find a GGUF-able embedding and re-ranker and try that out to see if we can flex a full setup with 262k context even though we will almost never need it.

In previous chapters, we set up local inference. Today, we tackle one of the primary hurdles of LLM-assisted coding: context. Most RAG setups fail because they are too eager. They flood the context window with "kinda relevant". Let’s go ahead & fix that.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

This setup passes CoIR, the de-facto benchmark for these things, with flying colors.

The companion repo contains high-performance local codebase indexing & retrieval for Roo Code. This this is (inference) model-agnostic and API-compatible with any LM-based code assistance tool that relies on Qdrant since that’s the API we are overriding.

This setup enforces a “pseudo-CoT” (Chain of Thought) by applying strict reranking filters. If an initial search lacks high relevance, the system returns narrow or empty results. This compels your AI coding assistant of choice to refine its query, ensuring only high-quality context enters the context window. The result is superior steering and a significantly more productive developer experience.

Requirements

Please see Chapter 1 for compatible hardware - will expand the docker images to cover a more broad set of hardware later on if this series resonates with people.

• Unix-like environment

• HuggingFace CLI

• NVIDIA Blackwell GPU with 6GB+ available VRAM

  • The consumer versions are RTX 50 series

  • The workstation version is RTX PRO 6000

• AMD Ryzen 9000 series CPU (LTO / other compiler optimizations)

• 590+ drivers with CUDA 13.1 (check nvidia-smi)

• Docker and a logged in huggingface-cli

Getting Started

# Pull down the companion repo
gh repo clone https://github.com/NotYevvie/OnlyLocals

# Navigate to the module for this article
cd OnlyLocals/codebase-indexing

# Download the 1.0GB embedding model
hf download jinaai/jina-code-embeddings-0.5b

# Download the 1.2GB reranker model
hf download jinaai/jina-reranker-v3

# The below setup script is idempotent (safe to run multiple times)
# This checks a lot; fix the messages 1 by 1 and keep re-running it
chmod +x setup.sh
./setup.sh

# Download the embedding container (these are 9-14GB)
# This may take 15-30min+ depending on your connection
docker pull yevai/codebase-index-embed:sm120-cu131-v1

# Download the retrieval container (these are 9-14GB)
# This may take 15-30min+ depending on your connection
docker pull yevai/codebase-index-rerank:sm120-cu131-v1

# Start it up!
docker compose --profile codebase-indexing up

Connecting it to Roo Code and other IDEs that rely on qdrant:

• Enable Codebase Indexing

• Any string works for API key

• Use the below settings

• Click “Start Indexing”

When you open Roo Code, click on the database-looking icon in the bottom right:

Try It Out

This setup shines primarily in large, complex code-bases where the dependencies are also checked out as git sub-modules so that their contents are available in retrieval.

Negative Test Use Case

Once indexed, try it out with (in “Ask” mode if using Roo Code): “Search your embeddings about this codebase to find what it knows about CoIRS and if it has any interesting innovations about reranking / RAG”

If ran in this repo, little to no results outside of this mention get returned - this means our re-ranker is filtering properly.

Positive Test Use Case

Next up, try: “Search your embeddings about this codebase and give me the highlights.“

This should result in narrow returns. A few queries later you should see a high-quality overview of what this repo is.

Direct Query Utility

# With the docker compose container running
chmod +x qdrant.sh
./qdrant.sh

This is a simple utility similar to Roo Code’s search that lets you run queries* directly against collections locally viewable here. Disclaimer: this is untested and unpolished.

* This is a semantic search query, not an LM prompt or Google search. Some tips:

• Search for behavior, not keywords.

  • “DB connection” → “How does this application connect to the database?”

• Use standard industry terms even if your code or implementation are weird

  • “makeDataGoBrrt() =” → “Data processing pipeline triggers”

TL;DR: Is it a full sentence? Does it describe an action that the code performs? Good.

Embedding Container

This is a version of HuggingFace’s text-embeddings-inference with CUDA 13.1 support and performance enhancements.

• I’ve upgraded the following libraries to support SM120 (blackwell) and CUDA 13.1:

  • huggingface/candle Source Repository | Upgraded Link

  • huggingface/candle-extensions Source Repository | Upgraded Link

  • huggingface/candle-index-select-cu Source Repository | Upgraded Link

• Performance-optimized Flash Attention and LTO as well as CPU optimizations.

• A sprinkle of extra low-level magic with a focus on single-GPU performance.

Reranker Container

This is a Qdrant retrieval proxy built from NGC PyTorch 25.12-py3 that includes:

• Highly customized version of FBGEMM tuned for SM120 performance.

• Flash Attention 2 and TorchAO int4 quantization for Blackwell.

TL;DR: Pull top 100, rerank with extreme prejudice, track latest query for context.

Vector DB Container

Standard Qdrant image. Once up, browse collections: http://localhost:6333/dashboard

Having Trouble?

I put some effort into making the setup script comprehensive and user-friendly but I fully expect this to not work out-of-the-box on all applicable machine types. DM me on LinkedIn if you run into issues!

What’s Next

If this gets any traction, I’ll create containers optimized for:

• ABM (Apple baremetal, the M-series chips)

• Older NVIDIA GPUs (be warned - bad perf)

And test on / expand compatibility for Cursor, Windsurf, etc

Beyond that, I’d like to slowly distill more of my secret sauce into the public docker image of the re-ranker without my latest startup’s lawyers and technical due diligence team having an aneurysm. This includes stuff like temporal context linkage and some rudimentary linear algebra dark magic for the vectors themselves.

This chapter covers optimizing a general-purpose Qwen3 model to run at 150+ TPS.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

The following clip, which you’ll replicate in this post, is in real-time and not sped up:

Benchmarking

Unoptimized

# Install aiperf
pipx install aiperf

# Start with previous unoptimized vLLM in a separate terminal window
"$(git root)/docs/002_inference/run-model.sh"

# Warm up vLLM
curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "nvidia/Qwen3-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 10
  }'

# Run a basic perf test
aiperf profile \
  --url http://localhost:1337 \
  --model "nvidia/Qwen3-30B-A3B-NVFP4" \
  --endpoint-type chat \
  --streaming \
  --concurrency 1 \
  --num-requests 10

If running on similar hardware to that in Chapter 1, you should see something like:

Now stop the vLLM container. 113.51 TPS is already faster than a Vertex, Bedrock, OpenAI or GitHub Copilot API. But why stop here? Let’s do some sketchy stuff.

Let’s Get Weird

Pull down the latest version of the GitHub repo if you’ve been following along.

What’s better than pulling sus nightly builds? Nothing. Nothing could be better. If that wasn’t enough, there’s even a /sed/ patch …in the Dockerfile. We’re living the wild life.

# Build container from sketchy nightly wheels and shady locker room trolls
# chmod +x "$(git root)/docs/003_optimization/build.sh"
# "$(git root)/docs/003_optimization/build.sh"

# Run with newly added optimizations in a separate terminal window
chmod +x "$(git root)/docs/003_optimization/run-model.sh"
"$(git root)/docs/003_optimization/run-model.sh"

# Warm up vLLM
curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "nvidia/Qwen3-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 10
  }'

# Run the same perf test
aiperf profile \
  --url http://localhost:1337 \
  --model "nvidia/Qwen3-30B-A3B-NVFP4" \
  --endpoint-type chat \
  --streaming \
  --concurrency 1 \
  --num-requests 10

You should see ~172TPS, a roughly 51% increase. Spin up the UI and try how fast it is:

"$(git root)/docs/002_inference/run-ui.sh"

Link to local UI. This is where “Effective AI-assisted development using local resources” is going: Modern LMs are plenty capable and this is notably faster than API calls.

Most folks don’t fully appreciate how heavily they’re guilty of model overkill. With the right setup you’ll have LM code assist that works ~90% as well but is 400-700% faster.

Next up: Setting up coding-specialized models and connecting them to VS Code.

This chapter covers running a 30B Qwen3 model at 60-90TPS (we’ll optimize this later).

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

WSL Docker Setup

We should install Docker Engine inside WSL so that nvidia-container-toolkit can interface directly with the WSL Linux kernel via dxgkrnl. This lets WSL sub in for a native host OS by directly managing the GPU resources.

# Install Docker
curl -fsSL https://get.docker.com | sh

# Add user to group
sudo usermod -aG docker $USER && newgrp docker

# Reload
newgrp docker

# Confirm socket (should show CONTAINER ID in red)
docker ps | grep "CONTAINER ID"

# Add Nvidia repositories
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update sources and install
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure NVidia runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart docker
sudo systemctl restart docker

Downloading a 30B Model

The casual favorite. Make an account on HuggingFace and get an access token here.

# Install the HuggingFace CLI
brew install huggingface-cli

# Enable Git credential store
git config --global credential.helper store

# Log In (also store as git credential)
hf auth login

# Confirm you're logged in
hf auth whoami

# Save your auth token - replace YOUR_HF_TOKEN with your actual token.
echo 'export HF_TOKEN="YOUR_HF_TOKEN"' >> "$HOME/.zshrc-pre.sh" && source

# Create a folder for HuggingFace models
mkdir "$HOME/HF_Models"

# Add it to the pre-zshrc hook from Chapter 2
echo 'export HF_HOME="$HOME/HF_Models"' >> "$HOME/.zshrc-pre.sh" && source "$HOME/.zshrc"

# Confirm it's available. Your models will download here.
echo $HF_HOME

Qwen3-30B-A3B-NVFP4 is a solid Blackwell “Hello World” model. While it’s technically a 30B model, it uses only ~3.3B active parameters per token, giving it decent throughput on a 5090. It’s not particularly great (even compared to other single-GPU 5090 options) at anything, but it is one of the rare cases where NVFP4 quantization resulted in higher scores on some of the benchmarks we care about.

hf download nvidia/Qwen3-30B-A3B-NVFP4

Running it with vLLM

Blackwell GPUs support NVFP4 (1 sign, 2 exponent, and 1 mantissa bit) models and we’re going to be heavily abusing this. This groups weights into blocks of 16, each of which shares a high-precision 8-bit (E4M3) scale factor and allows the 4-bit quantized coefficients to handle the finer details.

They also come with the 2nd Gen Transformer Engine, which which has Tensor Core instructions that operate directly on the compressed 4-bit blocks. The result of this is much more efficient use of VRAM that lets us run much larger models with only ~28GB of available VRAM (4GB for host OS).

vLLM recently added experimental support for NVFP4 so we don’t have to build TensorRT-LLM from source and compile our models for TRT-LLM to get reasonable TTFT (time to first token) and TPS (tokens per second).

vLLM also comes with a number of optimizations like PagedAttention for the KV cache.

# Build the vLLM container.
chmod +x "$(git root)/docs/002_inference/build.sh"
"$(git root)/docs/002_inference/build.sh"

# Start the vLLM container. This takes A WHILE (8-10 minutes) for precompute.
chmod +x "$(git root)/docs/002_inference/run-model.sh"
"$(git root)/docs/002_inference/run-model.sh"

Chatting with it

# Start the UI server.
chmod +x "$(git root)/docs/002_inference/run-ui.sh"
"$(git root)/docs/002_inference/run-ui.sh"

Head over to http://localhost:1338

First, you’ll have to “warm up” vLLM with a simple hello world prompt. This will take a while, after which you can expect 60-90TPS. We’ll optimize this later. When you first open localhost in your browser, you may see errors. Click on “Check again” and reload!

Bonus: Resource Monitoring

# Install pipx
sudo apt install -y pipx

# Install nvitop
pipx install nvitop

# Add to path
echo 'export PATH="$HOME/.local/bin:$PATH"' >> "$HOME/.zshrc-pre.sh" && source "$HOME/.zshrc"

# Run nvitop
nvitop

This chapter covers proper setup of a WSL distro for the series.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

Windows Subsystem for Linux (WSL)

We’re setting up a linux-based environment for this guide because most modern cloud deployment targets are ARM or linux-based x64. WSL uses GPU Paravirtualization (GPU-PV), whereas your production B200 and/or NVL72 nodes use PCIe passthrough (DDA) via AWS Nitro, GCP Titan, or Azure Boost. GPU-PV incurs a relative performance penalty but, for the purposes of our guide, it is negligible.

Host Setup

1. Enable WSL on Windows

# PowerShell Admin
wsl --install

2. Install Ubuntu LTS (”Long Term Support”: use in prod unless building from Alpine).

# PowerShell Admin
wsl --install -d Ubuntu-24.04 --name Yevai --web-download

3. Download and install VSC (Visual Studio Code - Link)

4. Use this knowledge to connect to WSL with VSC.

Everything else in this guide, unless otherwise explicitly indicated, will be done in WSL.

WSL Setup

Assume all commands to be ran, unless otherwise explicitly indicated, in $HOME (~) of the active WSL user. You will see “$HOME” used instead of ~. This is best practice for many reasons we’ll skip in this guide. For example, in some target environments like AWS Lambda, you will need to override $HOME to enable libraries to use the /tmp/ ephemeral FS layer cache. $HOME and ~/ do not always interact the way one would expect.

1. Update OS components and compensate for the fact that we are in WSL

sudo apt update
sudo apt upgrade

# Install WSL utilities
sudo apt install wslu

2. I have mixed feelings about Homebrew but it makes this guide much easier.

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Activate it in the current shell
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

# Install the Github CLI, ZSH, PL10K, and JQ.
# Several LM-based coding tools benefit from PL10K!
brew install gh zsh powerlevel10k jq

3. Shell setup - feel free to skip if you have your own preferences. The guide will rely on this specific setup. Checking out the companion repository is non-optional.

# Allow ZSH and set it as default
command -v zsh | sudo tee -a /etc/shells
chsh -s $(which zsh)

# Install OhMyZSH
sh -c "$(wget https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"

# Install and go to ~/Git. $_ gets the argument of the previous command.
mkdir "$HOME/Git" && cd $_

# Log into GitHub ("open" works thanks to wslu). Don't use tokens unless you have no choice.
gh auth login

# Confirm you're logged in
gh auth status

# Set the global git config - doing so from the active GH CLI is best practice.
# Note: this is especially useful when automating in CI with service accounts.
git config --global user.name "$(gh api user --jq '.name')"
git config --global user.email "$(gh api user/emails --jq '.[] | select(.primary) | .email')"

# Confirm your git username and email
echo "$(git config --global --list)" | grep user

# Clone the companion repository and go to it.
gh repo clone https://github.com/yev-ai/personal-yev-substack && cd personal-yev-substack

# Git aliases are useful, especially for rebases / merges / automation.
git config --global alias.root "rev-parse --show-toplevel"

# This now shows the folder of the git repository you're in.
git root

# Mount ZSH and PL10K configs - this also serves to confirm our paths.
chmod +x "$(git root)/docs/001_wsl_setup/setup.sh"
"$(git root)/docs/001_wsl_setup/setup.sh"

# Export the Git Root directory into our pre-zsh hook. This is where you checked out the repo.
echo "export GIT_ROOT=\"$(dirname "$(git root)")\"" >> "$HOME/.zshrc-pre.sh"

# Source the new ZSHRC
source "$HOME/.zshrc"

# Make sure our GIT_ROOT is set correctly - we'll use this heavily in later sections:
echo "$GIT_ROOT"

Confirm GPU Virtualization

If your host and WSL are set up correctly, including the latest drivers, this should be available in WSL:

nvidia-smi

You should see output similar to:

The windows host OS reserves around ~2GB VRAM. This guide will gracefully work around lack of exclusive GPU control which we normally have in cloud nodes.

Confirm Memory Allocation

# Install a basic resource monitor for WSL
brew install htop

# Run it
htop

You should see output similar to this:

Only 50% of our host’s 96GB RAM is available to WSL under “Mem”, along with a 12GB swap file. This will become important later.

This chapter covers the hardware build used for the series. TL;DR RTX 5090 required.

Link to next chapter

Parts

• CPU: AMD Ryzen 9 9950X3D

• Mobo: ASUS ROG STRIX X870-I

• GPU: Gigabyte 5090 Gaming OC

• SSD: WD_Black SN8100 4TB

• RAM: F5-6000J3036F48GX2-FX5

• PSU: CM V1100 SFX Platinum

• Case: Cooler Master NR200P V3

• Cooler: CORSAIR Titan 280 RX

• Riser: LINKUP PCIE 5.0 Riser Cable

• Fans: Noctua NF-A14x25r G2 PWM

OS

1. Install Windows 11 Pro

2. Run Activation https://github.com/massgravel/Microsoft-Activation-Scripts

# PowerShell Admin
irm https://get.activated.win | iex

3. Run Remove AI https://github.com/zoicware/RemoveWindowsAI

# PowerShell Admin
& ([scriptblock]::Create((irm “https://raw.githubusercontent.com/zoicware/RemoveWindowsAI/main/RemoveWindowsAi.ps1”)))

4. Run Debloat https://github.com/Raphire/Win11Debloat

# PowerShell Admin
& ([scriptblock]::Create((irm “https://debloat.raphi.re/”)))

5. Install common VCC redist (Download)

6. Install latest AMD drivers (Link)

7. Install clean NVidia drivers (Link)

8. OC Hardware. TSE 22,478 -> 24,106

Rant

I’m an experienced PC modder/OCer and like to rebuild my work PC every 3-5 years. My last build was the fastest 3960X in the world at 19,745 TSE. Both 3090s were shunted, delidded, and EVC2-modded. The goal for this build was portability and ML performance but seeing the 5090 clock in at 25,506 Graphics Score on TSE was extremely disappointing since my twin 3090s scored 23,012 five years ago.

…so I delidded the GPU, replaced thermals, shunted the VRMs, rebuilt the power delivery circuitry to safely draw up to 850W (the 12VHPWR connector is a travesty), and flashed a 1000W BIOS ...again ¯\(ツ)/¯. A few CNC modifications were made to the case so it could maintain proper temps during the 8-hour stability test ran at 78f ambi.

Pics and Wrap-Up

Or it didn’t happen, of course. The GPU is secured to the case with screws in 3 additional places to keep it safe during air and ground transport.

When your desires - not impulses - your purpose, values, and morals - scream louder than your fears of criticism, hardship, or failure, you gain a clear destination and sense of what you’re willing - and, more importantly, unwilling - to do to get there: an identity.

This tends to be the point at which someone’s impulses change from feelings of being pushed to run away from things to feeling of being pulled to run toward things - it’s also the point at which adversity begins forging people rather than destroying them.

Leveling Up

What you want to do can’t be done alone. You need a team so you start solo grinding the stats necessary to effectively use your spells and abilities in a team environment.

External motivation collapses under adversity. Internal motivation carries you, but it burns out. Intrinsic motivation - from purpose - is a superpower we call obsession.

Purpose also makes it easier for people to decide if they want to play with you. For starters, they can see if you’re playing Rocket League, Mario, WoW, or StarCraft 2.

Matchmaking

Boom, you’re a mage. You know where you want to go, what you have to get there, and what you need. You found a priest (partner) to play 2s, a rogue (best friend) for 3s, and a core group for 5s. Two of them are in the same guild as you (coworkers)!

Your teams overlap, and poison in any one spreads fast. You’re unbothered because your priorities are unfucked and everyone knows where they stand. Your partner, friends, coworkers, and family members are your teammates. They’re also your teams.

The gap between the levels of love, trust, and respect you grant yourself and what you can accept from others is your matchmaking rating band - and it is entirely up to you.

Team Composition

Every team is unique. You can’t run Mage/Rogue like Mage/Priest, or play either of those the same way against Warrior/Paladin and Hunter/Druid. How fast you learn your teammates’ tendencies and cadence - and how well you complement them - determines how well your team plays. Are you playing with a Druid or a Warlock?

Depending on your comp and what you’re up against, someone’s getting ran the fuck over most matches. If you’re the Mage in Rogue/Mage/Priest, it’s you. Your job is to survive getting DP’d by a Warrior and Hunter until your Rogue lines up the burst.

It feels like you’re getting fucked the whole time, but you just pulled off what most people couldn’t even dream of - because your team played well under pressure and you were able to trust someone else to carry when the stakes were extremely high.

Losing

All teams do well when things go well. Top teams also perform when they don’t.

Our team lost in the world semi-finals for 3v3 Arena in 2015. Everyone knew who and what caused the loss. All of us had important things riding on the line. Our teammate who misplayed wasn’t able to house his folks, or send his sister to college. It sucked.

When you care about what someone brings, you’re afraid of losing them. When you care about them, you’re afraid of hurting them or letting them down. Only one of those will keep your team afloat when a key member can’t bring much to the table.

We practiced for another season, couldn’t recover, and disbanded in 2016. I’d do it all over again because going for broke and getting blown the fuck out is easier to live with than the what-ifs of compromise or the guilt of going at it half-hearted.

Winning

Is valuable because it bonds. It’s best served when you have people to celebrate it with, and those people were in the shit with (and especially against) you. That said, people who don’t know the difference between opponents and enemies are unfun.

In 2018, our team took world first and $600K with it. Our guild and other pro teams were staying in AirBnBs near Anaheim. The teams we played against celebrated at the house afterparty. The amount of shit that got talked deserves its own numerical prefix.

Most of my favorite memories exist because I don’t know how - or when - to give up.

There is no point to this post. It’s Halloween, I just worked another 120+ hour week, and wanted to write a bit to help me decide whether I’m going to bed or the bar 🤗

For normies unfamiliar with the acronym: “OP” = original poster.

Quote from OP’s post

“Your AI isn’t lying to you when it tells you that Abraham Lincoln invented the iPhone. It’s doing something way weirder. Hallucinations aren’t bugs. They’re the feature working exactly as designed, just not how you expected. The thing is, large language models don’t “know” anything. They’re pattern-matching machines that learned to predict words by reading the entire internet.”

[Comment 1] - Me

RE: “large language models don’t “know” anything. They’re pattern-matching machines that learned to predict words”. This is only true for pure first-gen decoder-only transformers - NOT language models. I checked your profile and it’s unclear whether you have a background / experience in CS, Statistics, or ML. Before I say anything else - what’s your level of hands-on experience?

[Comment 2] - OP response

Yevgen Reztsov I’m not a CS/ML guy but I’m the cofounder of an AI automation platform, I spend most of my waking hours in this space. What do you think has changed in frontier models that we should cover here?

[Comment 3] - Me, 1/2

[REDACTED TAG] BERT is making a large comeback, and there are a lot of rapid changes in the attention layers. Most of the new LMs are no longer pure decoder-only transformers - the attention layer is often hybridized with RL (linear, state-space, etc). 2025 Examples:

Self-generation, decomposition, and verification of model reasoning: https://arxiv.org/abs/2501.13122

Progressive improvement via task self-generation and self-verification with no external data: https://arxiv.org/abs/2508.05004

Notable “reasoning” self-improvement with self-proposed tasks and self-verified solutions (no external data): https://arxiv.org/abs/2505.03335

Arena-based self-improvement (We did this with adversarial RL for AlphaStar when I worked at Google DeepMind in 2017, after working at OpenAI in 2016): https://arxiv.org/abs/2509.07414

This one is 💩 but it’s still self-bootstrapped “reasoning” with no external data: https://arxiv.org/abs/2203.14465

Most of these trace back to a 2022 paper on distillation from self-generated synthetic data sets: https://arxiv.org/abs/2202.07922

Predictive generation tends to be synonymous with eventual model collapse (a classic transformer trained on its own outputs enough times becomes nonsense). These show the opposite.

[Comment 4] - Me, 2/2

[REDACTED TAG] (Didn’t fit into one comment). Point is: I’m VERY hesitant to say models “reason” because we understand HOW they work, not WHY they work. From a neuro- and cognitive science perspective, we also have no idea how human cognition (important nuance) works. It COULD be an extremely complex form of recurrent next-symbol prediction with other aspects of cognition mixed in for all we know.

We *DO* know that pattern recognition and pattern prediction are FACETS of cognition. Memory is also a facet. So are procedurals. Etc.

That puts a big pickle into “do any of these actually reason”. We don’t even know what that means. In short, we OBJECTIVELY don’t know.

Outside of LI influencer hype, virtually no ML researcher or scientist will say that “Language Models” reason - or vice versa. That’s because it is tantamount to career suicide in the ML scientific community as of now.

No disrespect meant, but if you (or anyone reading this) think about the last time you’ve talked to (or saw an LI post from) a frontier ML scientist / researcher actively publishing papers... RIP. Zero. I am one of those. Personally, I’d say we don’t know ...unless I missed a memo about the next Nobel Prize winner in 3+ cats 🥲

That doesn’t make for a great LI post tho.

Wrap-up

For my subscribers: are fast blurbs like this something you want to see, or would you prefer I stick to more thought-out ones (Substack spams your email when I post) where I take the time to write, edit, and proof-read it out of respect for yours?

Let me know!

Meme credit goes to this LI post from Eduardo Ordax (Gen AI Lead at )!

The Gist

Things got busy! It’s been two weeks since my last post. Paying customers came much sooner than expected. We now have 4 design partners totaling $18K MRR. Ba dun tss.

That’s a good number for continuous real-world feedback, so that phase is full. Not all customer feedback should be actioned - more on core competencies in later posts.

Highlights

• Gathered a round of feedback for desired MVP scope and gave date estimates.

• 70%+ odds of a mid-5 digit ARR AI/ML security contract from the Pentagon.*

* If signed and permissible, I’ll cover it in anonymized, customer-general terms.

Priorities

• Finalized Statement of Work, MSA, EULA, and TOS updates by 10/03.

• Pre-hiring: I need to find my last qualified ML engineer for the project by 10/06.

That’s it! Staying razor-focused and only having 1-3 priorities at a time is important.

Process

• Established POCs/SOPs for Customer Success (me): emails, Slack channels, etc

• Basic sales pipeline and status tracking (also me): just Google Sheets for now.

I had to set up federated SSO (Google Workspace made this a breeze) and will share how to do this for common IT vendors like Linear, Notion, and more in a post soon!

Notes

If you read ML papers, make sure you read this. I’ve been using this approach for years, but it’s good to see it quantified. TL;DR: if you can automatically verify step-by-step problem solving, you can also automatically improve it with RLAIF.

Also: a notable, more accessible, and readily applicable paper on in-context learning.

Total startup cost: ~$1,000–1,500 upfront, and ~$200/month upkeep. This includes putting yourself on W2 payroll (sans wages), which you want in place from day one.

Zen Business

• Great feedback from fellow founders about speed, quality, addons, and support.

• Covers state filings, beneficial ownership reports, and registered agent services.

• The “Filing + Tax Advice” addon at $99/mo is great for getting started.

Stripe Atlas = vendor lock-in; Carta Launch doesn’t cover basic legal compliance. For the uninitiated: always make a Delaware C-corp. Get your SIC and NAICS codes right.

Chase

• Excellent baseline support for things like ACH, treasury, and wire transfers.

• Widespread branch network is useful when things inevitably go wrong.

• Plays well with most vendors, like QuickBooks and Rippling for expenses.

Mercury and Brex are popular alternatives; my knowledge there is dated.

Carta

• Industry-standard cap table management that’s trusted by lawyers and investors.

• Grows with you from founder stock, SAFEs, and options to 409As and Series A+.

• Built-in document templates, data room, and investor relations are excellent.

Stripe

• Their billing portal saves time; their APIs are built for a good developer experience.

• Webhooks, observability tools, audit logs, and first-class test environment support.

Rippling

• Handles onboarding, offboarding, HR compliance, and has great legal templates.

• I’m a big fan of their LMS, Perf Management, and Workflows for HR automation

• Supports most countries for overseas payroll which is otherwise a total nightmare.

• Once you have 2+ FTEs, their ASO and PEO are great: benefits, workers comp, etc.

• Bonus: it provisions the two things Google Workspace doesn’t: 1Pass and devices.

Rippling lets tech founders handle HR, ops, and compliance correctly from day one.

• The Gist - A subjective soapbox rant.

• Highlights - Objective recent events.

• Priorities - What stays and what goes.

• Process - System and process changes.

• Notes - Occasional reflections.

Let’s get into it!

The Gist

Maintaining my own code for 15+ years has taught me that deciding how prescriptive to be about tech stack and code contracts is an unforgiving decision theory problem. My friend Cassie Kozyrkov is a pioneer in the field, and her work is worth following.

Throw in upgrades, infra, cost, durability, uptime, and SLAs into the mix, and you’ve got a high leverage set of choices. The goal is to be minimally prescriptive and solve the problem in front of you without locking into assumptions that may backfire later.

Do this well and you’ll move significantly faster while taking on less tech debt.

Highlights

• Finding initial customers went eerily well; I’m at 3, totaling just under $10k MRR.

• The wedge theory was overkill: people want hands-off PostHog and Langfuse.

• Unexpected upside: minimizing cloud costs is a universally shared value-add.

People want to sign up for an endpoint their engineers can immediately use.

Priorities

• Testing pipeline for minimally prescriptive AKS, EKS, and GKE by 09/06.

• Langfuse setup across three clouds with API + integration tests by 09/10.

• Ingress for AWS, GCP, Azure, and Cloudflare to make it usable by 09/13.

Process

• Non-tech vendors added: ZenBusiness, Chase, Stripe, Rippling, and Carta.

• IT vendors added: Google Workspace, 1-Password, Slack, Linear, and Notion.

I’m relentless about keeping the vendor count low. I’ll share a separate post on what we choose, how we integrate it, and which system becomes the source of truth for each operational domain - a decision you absolutely need to get right the first time.

Notes

Per my last post, this is a “starting my own (again) and will build it in public” deal. That means everything is from scratch. Last time, I made the unfortunate mistake of using Microsoft / AD for SSO federation. We’re going with Google Workspace this time.

About Me

I left the startup I was working for one week ago. I’m starting my own (again) and will build it in public. This is my founder’s diary: gathering customer feedback, finding the wedge product, getting first customers, and hiring. I’ll post Wednesdays & Saturdays.

I’ve run 3 startups and carried DevOps, DX (developer experience), and ML infra from pre-seed to Series C for many others. Making AWS, AZU, GCP, Pulumi, Terraform, and ML boring and reliable is what I’m very good at according to people I’ve worked with.

The Problem

As capabilities for writing code accelerate, the need for observability, environment parity, deployment, and debugging grows proportionally. One day, out of the blue, feature velocity plummets, the app becomes unstable, and fixes (to bugs that you should’ve detected before your biggest customer told you about it) take forever.

There are excellent hosted and open-source solutions for all of those. Deploying and implementing them to enable tracing, telemetry, deployments, monitoring, alerting, and analytics is hard. It’s cross-company code, infra, and tech debt duplication.

The Customer

Series A and beyond startups. They all have a lot in common, including a strong bias toward action. Taking on calculated tech debt to secure ARR is the correct choice. The consequences of that choice catch up as teams scale from ~5 to 10+ engineers.

Past Attempts

Per the recent MIT study, ~95% of AI implementations fail. This is true for most tech purchases that need implementation whereas those that don’t mostly succeed. The lesson is that joint ownership doesn’t work. Single responsible party = success.

UI‑first PaaS blurs ownership and makes edge cases impossible. Well designed GitOps components have clear ownership boundaries, avoid vendor lock-in, and handle edge cases by giving us the ability to fall back to Terraform and Pulumi whenever necessary.

The Solution

My job has always been to make the other engineers’ jobs mostly business logic. I’m productizing that. Pure GitOps - no fancy UIs that blur ownership boundaries, kill the ability to deal with edge cases, and take forever to build while delivering zero value.

I’m building a framework of components that are SOC2/HIPAA/etc ready* and retain the ability to handle edge cases. They just work together, come with app integration code, playbooks for zero-downtime deployment, managed upgrade PRs, SLAs, and built-in observability that covers common use cases from serverless to Kubernetes.

* For transparency: this is being built so we can retrofit compliance. The promise being made is that making the components themselves compliant will not require customer-side code changes. Updating them, testing them, and getting them certified is for later.

The Wedge

Imagine adding session replay that correlates user clicks to every API and database call emitted by them to your React application in 2-5 days. That includes deploying self-hosted OpenTelemetry, PostHog, Langfuse, and code to tie all of them together into your app. Success looks like going from sign-up to production in 2-5 days.

Crystal clarity in ownership means a milestone-based definition of “done”. You don’t pay for access to infrastructure as code - you pay to have it working in your app.

Defensibility

I’m posting the plan because this isn’t a "I’m a VC, I’ll bankroll my friend” problem. As Chainguard (love those guys!) aptly puts it: this shit is hard. Professional services don’t scale and very few people have the knowledge and experience to do this properly.

This isn’t “move fast and break things”. A single mistake brings down production for a customer and destroys all trust. AI-assisted test generation and code upgrades will be needed to maintain this and make sure it works in all 3 cloud providers for K8S-native and cloud-native deployments. In that sense, this post can be considered a pitch deck.

What’s Next

I find the first 1–10 paying design partners. Want to self-host complex, distributed OSS software that normally requires a dedicated DevOps team? Book me on Calendly or reach out via LinkedIn with “Hey, we run [docker] on [aws] and want [langfuse]!”.

The first few customers will be design partners. Your input will be heavily weighted for roadmap prioritization and you will essentially have me as a dedicated DevOps engineer for a fraction of the cost. I greatly value my reputation and won’t be taking on more customers than I can properly serve. This will scale slowly, but it’ll scale right.

When it comes to infrastructure, reliability (aka quality) is everything.

In a sea of clueless bullshitters LinkedIn AI Influencers, the above post stood out to me as I scrolled with my morning coffee. It was to the point, grounded, and genuine. The theme here is knowledge that is useful and easy to leverage - let’s roll with that.

I woke up at 4:45 and will start work at 5:30 so I need to finish this post by then. Let’s make it short, fast, actionable, and hopefully useful to at least one of you. We’ll walk through a simplified version of a setup I recommend to my non-engineer friends.

What Will This Get You?

Use case: “Good morning. What did I miss?” (Lots of red but its all accurate/useful).

Use Case: “What do you remember from the last two days?”

Use Case: someone sends Platform a message on slack about an outage. This:

• Picks it up from Slack automatically and dispatches a Boomerang job.

• Checks Sentry, Prometheus, GCP, K8S (kubectl), and LogRocket.

• Checks recent memories, Google Drive, Linear, JIRA, Calendar, etc.

• Replies to that person in Slack, let you know, and if necessary, creates tickets.

The tickets a setup like this creates are GOOD. Like, actual acceptance criteria an Engineer can pick up and start immediately good. Similar tools govern my daily priorities, what I need to do, help me communicate via email, and other things.

What Are We Doing?

3 personalized contexts, an MCP server, and an A2A implementation Are All You Need. After 15 years of ML research (and applied ML - I do things! sometimes), I understand things well enough to explain them in simple and concise terms.

These 3 things (queue BuzzFeed clickbait) are the AI “80” of 80/20 for non-engineers:

• CONTROL YOUR DATA. A synced OneDrive/iCloud/GDrive folder is a good start.

• Check HF Leaderboards for models relevant to YOUR use cases every 2 weeks.

• Create stateful* contexts for YOUR use cases. I’ll show you how to do this below.

*- Stateful here means with persistent memory and dedicated embedding back-ends.

Let’s Do The Thing

Install Visual Studio Code (ab: VSC). Click on the icon that looks like a lobotomized Rubik’s Cube in the top left sidebar and install the Cline and Roo Code extensions:

Get API keys from OpenAI, Anthropic, and Tavily (web search that doesn’t suck). In the top middle of VSC (Visual Studio Code), there is a text box. We’re going to click on that text box. Then we will move our hands over to our keyboard, type in “>Cline”, and press Enter or put our hands back on the mouse to click “Cline: Open In New Tab”

A wild UI component appears. Don’t panic! We got this. Click on the box that looks like the three stacked pizzas you ate last Saturday before passing out (bottom left):

Okay, my coffee is slowly kicking in so we’re going to go a bit faster now. Go to the marketplace tab on the left and install the Tavily MCP server plugin (pictured below).

Cline will walk you through the entire process, test, and if necessary troubleshoot + fix the MCP server installation. All you have to do is type in (presumably) English.

We’re AI wizards now, Harry. We now have a place where we can use any model with robust web search functionality (please don’t go on LinkedIn and call this an Agent).

Cline In A Nutshell

Think of Cline as a middle manager. It’s good at orchestrating MCP servers and Roo Code, but it doesn’t actually do any work except Rules and Workflows - you can find those by clicking on what the US justice system should’ve been. Bottom left of Cline:

You may have noticed the annoying yellow highlight that both lets us know we are not using an Apple product and sums up the first thing life crushes in adulthood: “Plan”

Cline is good at freeform planning. Ask it to plan something, then prompt it to update that plan. Like the middle manager that it is, Cline is good for doing deep research which we can then ask it to store in our local Mem0 MCP server for Roo Code.

That brings us to the limit of usefulness that is Cline. To reiterate:

• Configuring and troubleshooting MCP Servers

• Rules and Workflows. Read more about those.

• Deep research and local memory management

Roo Code: The Nut In The Shell

Cline and Roo used to be the same person (Roo is a branch of Cline). Then they had a fight, and now they’re frienemies. They still have similar mannerisms. Let’s open the settings for both so we can synchronize our Cline and Roo Code MCP servers:

• Cline: go to Remote Servers (same place as Marketplace above) and click “Edit Configuration” at the bottom. This opens up the Cline settings JSON.

• Roo Code: the three Pizza boxes are located on the top right just because. We click that, then scroll down to the bottom and click “Edit Global MCP”.

A visual depiction of tech debt wanting to be business value when it grows up:

The format is very similar, but we will need to make a few small changes:

Or delegate to one of the 3-8 instances of OpenHands you surely run in parallel.

The orchestrator mode in Roo Code is what we’re after because it can:

• Have its own API and MCP settings (use C4Opus for the Orchestrator itself)

• Automatically select between contexts we define (and is decent at it)

• Each of those contexts have their own model, API, and MCP settings

• They can run in parallel with shared MCP state and memory back-ends

Speaking of those contexts, let me give you a simple example: Secretary

Intermediate example: DevOps

Through pragmatic use of allowed commands (aka NOT kubectl deploy) and local MCP servers, you will get pretty good mileage. I suggest setting up, at the very least:

• File System

• Google Calendar

• Google Drive

• Slack

A Note On Memory

The importance of local MCP memory (I personally prefer mem0) initialization and scheduled updates can not be understated. Do this with Cline. Example prompts:

• "memorize the slack channel ID and name correlations, then check your memories and confirm that you remember them."

• "memorize that the latest Linear engineering requests are added under and checked from the [REDACTED] linear project - make sure to read the project description for instructions and memorize those."

• "look through the latest 20 messages in each slack channel you have access to, create concise channel purpose summaries, and store them alongside your existing slack channel memory mappings to increase your awareness of which slack channels you should look in and post to for specific topics"

Other Protips

• LMs are dumb. Structure projects / *.MDs to mitigate that and tell ‘em to RTFM.

• Use “brew tree” and forced tool calls to let LMs better navigate your projects:

• Make sure your OpenHands instances each have a separate Mem0 MCP store.

  • Hotwire your OpenHands instances to plex via A2A (agent to agent).

• Just like *.MD files, due to training data, LMs do well with Mermaid Diagrams.

• Do not overkill the model used for a task. Larger models are not always better.

  • #1, by far: know when to use other model types (not LMs).

  • #2: Know when to use, or not use, thinking models.

  • #3: Know when to use fast SLMs or nano LMs.

• Not all memory is created equal. Even for a sh*tty local only setup, you want to use a combination of Graph, Semantic, Episodic, and Embedding memory.

• RAG sucks unless you need dynamic in-context chain of thought

  • Forced tool calls via something like Algolia are better 90% of the time.

• Learn your MCPs like you learned your ABCs.

  • Example: if you use LMs for code without basics like Context7 …yeah.

  • Limit ‘em. 10-15 max per context and use RAG to assist with selection.

• Good prompts are as important as they are model-specific.

  • A model-agnostic prompt library will make your life better.

  • Pre-initialized templates are underrated. (ie., you pre-populate the first 1-5 assistant : user message pairs with a focus on what the LM thinks it said)

Effective system prompt example:

ML is a great hammer but not everything is a nail. In general, LMs and RAG are a last resort. They are both slow, expensive, and highly prone to errors. People who are new to ML (aka most of the world) are in a honeymoon phase. Unless you’re an intern, you already know that a system’s observability should be proportional to its autonomy and a system’s reliability should be proportional to its parent ecosystem’s complexity.

In terms of “code paths to cover with unit tests”, ML is a f*cking nightmare.

Conclusion (aka Self-Promotion)

Did you think that today was the day you’d see a post on the internet without self-promotion? Look at the bright side: it’s not “hey bro I’ve got this sick report free(tm) all you need to do is comment on my post or DM me your email” — Sincerely, a “coach” (unemployed) who claims to have scaled startups from 0 to $10M+ / have F500 clients while attempting to sell some “program” for $10-$5000 on LinkedIn.

Unless you want to join our wonderful team over at Sully AI (I’m hiring and build our Platform/AI tools - unlike this 2023 technology, we actually use modern tooling) this should be around 60th-70th percentile in terms of AI tooling you can get for free.

TLDR even if all you do is OpenAI API calls, your users’ TTFT (time to first token) decreases by 395-870+ms, your vendor footprint also becomes smaller, you save money, have better consistency, lower downtime, and your developers love you.

This bridges the gap between companies wanting cross-cloud benefits and those with a Platform team and budget to deploy and maintain K8s clusters in every cloud, with vertically / horizontally scaled Istio sidecars, private SSL certificates, VPC peering, and an Envoy service mesh. Despite its transitional nature, this approach is robust, resilient, maintainable, performant, and scalable. It can and has served 1M+ cross-VPC RPS.

Previous posts: “ML: A Networking Perspective, Pt1/X”, “ML: A Hardware Perspective”.

Benefits (Attempt #2)

1. User Experience

Everything we do is for our users. Seamless UIUX requires sane response times:

• 20-100ms: Feels instant, highly responsive - this is the ideal experience.

• 100-300ms: Noticeable, but still considered near-instant and seamless.

• >300ms: Noticeable delay, mild but acceptable UX degradation.

• >500ms: Perceived lag, sharp decline in UX, and user frustration.

• >1sec: Significant UX degradation and likely user abandonment.

Infra is necessary to build things that are fast. Abstracting that infra is necessary for enabling engineers to build those things fast. On the front-end, that looks like this:

Analysis paralysis caused by free-form text is the silent killer of user experience. A generic, easy-bake recipe for reducing user cognitive load and improving UIUX is:

Superior UIUX for ML products often involves moving away from free-form, text-only interactions as quickly as possible (e.g., Lovelace). This requires a data-driven UI player.

The backend sends a JSON payload representing your users' UI layout (ex: open code canvas, file workspace, query engine, document uploads or downloads, etc.). Like this:

2. Developer Experience

The ideal application developer experience involves at least 70% - ideally over 90% - of what they write being business logic. Less boilerplate equals more customer value.

This series will provide a comprehensive guide for a single engineer to empower your organization with the following advantages in 1-3 days, troubleshooting included:

Front-End Development

• Provide a simple way to use CSR, SSR, SSG, ISR, and SSSR with minimal boilerplate.

• Remove caching, chunking, compression, and asset optimization from app logic.

• Provide a centralized GUI for testing and optimizing front-end performance.

• Remove authorization and authentication from app logic by centralizing it.

• Reliable global <15ms web, mobile, and native front-end performance.

Back-End Development

• Faster TTFT and TPS (tokens per second). OpenAI example:

  • Public API: 600-800ms TTFT / 600ms-1.5sec total latency. Not great UX.

  • Azure API: 200-300ms TTFT / 300-700ms total latency. Much better UX.

  • Azure also offers enterprise security, compliance, and data residency.

  • Services in AWS and GCP will be able to take full advantage of this.

  • Unlike public APIs, API downtime in Azure is virtually unheard of.

• Significantly faster embedding queries.

  • AlloyDB takes 10-20ms to query against millions of vectors.

  • Supabase pgvector takes 30-80ms or more at similar scale.

  • Inter-VPC vs public transit shaves off another 85-200ms+.

  • This makes your RAG queries 95-270ms faster.

  • Your models query their embeddings locally.

• The above takes us from frustrating response times to a great user experience. Every company I've done this for had an immediate increase in user satisfaction.

• Reduce your vendor footprint and allow application developers to call any service or model using unified internal authorization, instead of managing vendor API keys, rotations, etc., on a per-application basis - the definition of insanity.

• Minimize toil by centralizing key management and model / API vendor updates.

• Improve security by never exposing ALBs, NLBs, or services for external access.

  • Explicit and centralized outbound access control and logging.

• Simplify end-user and organization billing and usage tracking by unifying it.

• Centralize vendor failover and adherence to your usage / throttling limits.

There are many other advantages, like enabling access to serverless models on all major clouds (Gemini Flash, for example, has uses in virtually every LLM workflow), but this article's goal is to cover everyday benefits for users and application developers.

A Robust Full-Stack Interface

Enabling your backend and frontend applications to call the same service endpoints will make your full-stack engineers happy and they will say nice things about you.

This also lets you to intercept and centrally authorize any vendor or endpoint. Making a call to Together AI, OpenRouter, Weights & Biases, ScrapingBee, etc.? No problem!

Your service’s SDK called what it thinks is https://scrapingbee/foobar, and your internal DNS routed it 30-100ms+ faster via the proxies and a global backbone while injecting the proper API keys without your back-end service ever knowing about it.

These proxies support DB connections and other cool stuff - they cover HTTP/S/2/3 (QUIC), TCP and UDP, TCP and UDP Streams, GRE, IPIP, IPSec, gRPC and WebSockets.

3. Performance

Fast, predictable infrastructure performance directly improves user satisfaction and reduces infrastructure costs. Here are some real-world benchmarks that show this:

• 800Mbps+ per-node throughput on streams and large RQs.

• 1200-1600 RPS with 3-4ms internal traffic between clouds.

• 350-400 RPS with 13-14ms global front-end performance.

Inter-VPC performance (TLDR, 3-4ms):

This is made possible by physical colocation of the proxy VMs, although it uses public network transit (the internet). I ran these tests for 48 hours and for 11 minutes of that, congestion increased latency to 9-14 ms with zero packet loss. Compared to the VPC peering alternative of $2700-3500 per month, this also gives us higher burst capacity.

Front-end performance (TLDR: 13-14ms):

These were non-cached requests to an NGINX server on the nodes (using more CPU than proxy forwarding), during which none of them exceeded 25% CPU or 630 MB RAM. Speaking of our nodes, this level of throughput costs all of $21 per month:

This also gets us universally accelerated outbound API requests which are normally either unavailable out of the box altogether or cost insane amounts in each cloud.

We will add redundancy and fault tolerance while also making it 60-90% cheaper with auto-healing scaling groups composed of spot instances. This step is also necessary to verify that our design is robust enough for production usage. People talk about resilience and durability, but until you run entirely on spot instances, you have neither.

4. Cost

This setup pays for itself by reducing cloud costs and operational complexity. By the time we’re done, you’ll be running all of it for under $100/mo across all 3 clouds.

Engineering Overhead Savings

• Application development teams move faster because they can self-service infra.

  • No per-service endpoint environment variables and update redeployments.

  • Unified request tracing drastically simplifies debugging of production issues.

  • Unified request logging removes redundant middleware from our services.

• No more per-cloud and per-account SSL certificate rotation (aka Platform KTLO).

  • This also saves us from TLS/SSL termination overhead in all our services.

• Platform is no longer a dependency or blocker and can focus on new features.

Compute Overhead Savings

• Per-account/CDN/ALB AWS Global Accelerators ($18/mo each)

• Per-account/CDN Azure Front Doors ($35/mo standard, $330/mo premium)

  • Cloudflare performance is equivalent to premium Azure Front Doors.

• Per-VPC AWS private DNS ($0.50/mo per zone + $0.40 per 1M requests)

• Per-VPC GCP private DNS ($0.20/mo per zone + $0.40 per 1M queries)

  • GCP NAT Gateway costs (this can get insanely expensive very quickly)

• Per-VPC Azure private DNS ( $0.50/mo per zone + $0.40 per 1M queries)

  • Azure NAT Gateway costs ($32/mo each)

What’s Next

In the next article, Pt3/X, we'll set up a proxy in each cloud that responds to a health-check endpoint and connect it to our public domain via Cloudflared. This entire series will be done in a secure zero trust way that never makes our VMs externally accessible by using neat techniques like UDP hole-punching, reverse tunnels, and relay networks.

After that, the course schedule will be:

• Pt4: Add direct inter-VPC mesh communication to proxies.

• Pt5: Set up automagic DNS routing to proxies in each VPC.

• Pt6: Optimize egress routing through Cloudflare's backbone.

• Pt7: Introduce fault tolerance with spot instance groups.

• Pt8: Expand our mesh a micro-cloud like Lambda Labs.

After that, we’ll do an "ML: A Software Perspective" series, where we write middleware and provide examples using popular frameworks like PyTorch and TensorFlow. We'll also include tools like Autogen, Semantic Kernel, mem0, Composio, LangGraph, AG2, CrewAI, Wetrocloud, Vercel SDK, LlamaIndex, and other easy-bakes for the "AI" crowd.

Let’s make sure the below depicts our competitors’ customers and APIs - not ours.

1. That’s too much. Most companies are LangChain and API bolt-ons.

2. How do we actually set something like this up? Need DevOps help.

This will be a series aimed at addressing both of these things. We’ll set up a global cloud-agnostic backbone with persistent connections between each cloud provider.

This article sets the groundwork for the series with a short action item at the end.

Who This Is For

• Startups working on “AI” - API calls, bolt-ons, frameworks like LangChain.

• Startups working on ML - training, inference, fine-tuning, self-hosted.

• You don’t have a Platform team or their skill ceiling is not the best.

Benefits

• Quickly and safely capitalize on all 3 clouds’ advantages in every cloud:

  • Use GCP’s AlloyDB, Vertex, GAI Studio, Doc AI, BigQuery, etc in AWS/AZU.

  • Use Azure’s OAI, CosmosDB, Dbricks, AML, Cog Serv+Srch, etc in AWS/GCP.

  • Use AWS’s Bedrock, SageMaker, Forecast, Comprehend, etc in GCP/AZU.

• Vastly reduce latency and jitter for most of your external vendor API calls.

• Set up inference endpoints with Lambda Labs’ DGX SuperPods or Scalars.

• Security. Easier IAM/SG/etc management, no more exposing ingress, etc.

• Your future DevOps team will say good things to you AND about you.

• Unified cross-cloud analytics, image optimization, global ingress backbone.

Plus better performance than you’d have with (and save you the monthly charges of) AWS Global Accelerator ($18/mo each), Azure Front Door ($35/mo each), and GCP Premium Tier networking - plus the rather steep bandwidth costs for all of those. A lot more bells and whistles come with this setup but we’ll dig into those in later articles.

Drawbacks

It’s one more place where things can go wrong. This setup is reliable and I’ve used it for many companies, but things CAN go wrong. We will cover DR and redundancy to mitigate that. It’ll cost extra but will still be over 10x cheaper than peering at baseline.

What Are We Doing?

In this series, we’ll implement the diagram below. This involves swapping DNS to Cloudflare and adopting a bit of Pulumi in your infra. I’ve built out MLOps/DevOps infra for dozens of companies, and not a single one has regretted doing either.

I’m not going to bore you with too many of the networking details but here’s the gist:

We’ll go with envoy for management APIs and a few other specific capabilities. Our design goals (ordered): Cost, Security, Maintainability, Performance, and Robustness.

This will be built in a specific way that allows us to easily add Disaster Recovery, Fault Tolerance, Durability, and an Istio control plane for practical management later on.

Alternatives

VPC Peering

The downside to this is cost. Below is one of the most basic peering solutions you can have and the total of the below prices along with Edge Routers ($200-300/mo/cloud) and Monitoring Tools ($100-200/mo) would be at least ~$2,736-3,436 per month.

AWS ↔ GCP

AWS Direct Connect Port: $262/mo 1Gbps, GCP Partner Interconnect: $151/mo 1Gbps, Cross-Connect: $200-300/mo, Subtotal: $613-713/mo

AWS ↔ Azure

AWS Direct Connect Port: $262/mo 1Gbps, Azure ExpressRoute Circuit: $305/mo 1Gbps, Cross-Connect: $200-300/mo , Subtotal: $767-867/mo

GCP ↔ Azure

GCP Partner Interconnect: $151/mo 1Gbps, Azure ExpressRoute Circuit: $305/mo 1Gbps, Cross-Connect: $200-300/mo, Subtotal: $656-756/mo

WireGuard: Advantages

WireGuard is a VPN protocol designed for simplicity and high-performance encrypted networking. Cloudflare Warp (what we’ll be using) is based on the WireGuard but saves us from dealing with things like WG key and configuration management!

• Extreme Performance: Minimal latency and negligible CPU overhead.

• Simplicity and Transparency: Easy-to-understand config, fewer security risks.

• Cost Efficiency: Free and open-source software (FOSS); only server costs.

• Fully Self-Managed: Complete control over security, routing, keys, and privacy.

WireGuard: Disadvantages

• Manual Key & Configuration Management: You have to manually handle cryptographic keys, configurations, updates, deployments, and other stuff.

• Infrastructure Management Required: Self-hosted infra is more complex.

• No Built-in Firewall or Security Policies: it's just a VPN layer.

TailScale: Advantages

Tailscale is built on WireGuard and similar to Cloudflare Warp, but with an automatic mesh that simplifies key distribution and network setup. In our case, TailScale could be 1-3ms faster but it is a direct P2P solution that exposes external IPs, suffers from public network transit jitter, and doesn’t backbone us to external API vendors.

• Zero-Configuration Setup: TailScale is to WireGuard what Istio is to Envoy. It handles WireGuard keys, peer discovery, configuration, and other handy stuff.

• Automatic NAT Traversal and Mesh Routing: Direct peer-to-peer connections even behind NAT are great for warm, low latency, and distributed deployments.

• Strong Security Defaults: Strong auth via Single Sign-On (SSO) integration.

TailScale: Disadvantages

• Dependence on Tailscale Infrastructure: Relies on Tailscale’s control plane.

• Limited Customization: Cannot fully customize encryption, auth, or keys.

• Compliance & Privacy Considerations: Central control plane.

Nebula: Advantages

Nebula is a scalable mesh networking platform developed by Slack. It gives us a secure overlay network and handles dynamic node discovery (kind of like Istio).

• Scalable & Secure by Design: Uses certificate-based authentication (PKI), scales well from 10 to 1000+ nodes. This is extreme overkill in scale and complexity.

• Rich Security Features: Built-in SGs and firewall-like policies give good control.

• Robust NAT Traversal: Traverses firewalls and NAT for hybrid and multi-cloud.

• Open-Source: Highly customizable, transparent, and fully self-hostable.

• Proven in Prod: Heavily battle-tested by Slack in large prod environments.

Nebula: Disadvantages

• Complexity: Initial setup requires PKI certificates and custom configuration files.

• Self-Managed Infra: Similar to WireGuard, we deal with more self-managed infra.

• Operational Overhead of Key Management (PKI): Things can go really wrong.

ZeroTier: Advantages

ZeroTier is software-defined networking (SDN) for cross-cloud virtual Ethernet.

• Simplicity & Rapid Deployment: Very user-friendly with a web GUI.

• Automatic NAT and Firewall Traversal: This saves a lot of headache.

• SDN Capabilities: Routing, bridging, and network segmentation out of the box.

• Hosting Options: ZeroTier’s control plane can be cloud-hosted or self-hosted.

ZeroTier: Disadvantages

• Centralized Control Plane (by Default): they’re not as big as CloudFlare.

• Performance Tradeoffs (Layer-2 Ethernet Emulation): Slightly higher latency due to Layer-2 tunneling overhead versus Layer-3 VPNs (WireGuard/Tailscale).

• Enterprise Scale Costs: can scale quickly and brutally at enterprise levels.

• Complexity at Large Scales: Virtual Ethernet networks are complex in large multi-cloud environments, especially with network loops, multicast, and broadcast.

What Are We Doing In This Post?

In this post we’ll figure out where to put our proxies in each VPC for them to be as physically close to each other as possible. Networking 101 for the Big 3 Clouds:

Region = a geographically wide group of interconnected Availability Zones.
Availability Zone (AZ) = a physical group of data centers in a Region.
Virtual Private Cloud (VPC) = logically isolated network spanning all AZs in a Region.
Subnet = a network segment inside a VPC with its own CIDR IP address range.

The relationship between Subnets and AZs is not the same in AWS, GCP, and Azure.

GCP

A Subnet spans all AZs in a Region by default. Specifying an AZ during creation of a VM guarantees it will be in a specific physical location. Example: us-east4-a, -b, or -c.

Azure

A Subnet also spans all AZs in a Region by default. Specifying an AZ during creation of a VM guarantees it will be in a specific physical location. Example: “eastus” and “1”.

AWS

A Subnet is tied to a specific AZ at creation but the mapping of AZs (“us-east-1a”, “us-east-1b”) to physical locations is randomized per-AWS-account at creation.

In a per-tenant AWS account setup (which you should have, security 101), you need to reverse-map AZs for each account. This is done by spinning up public IP EC2 instances in each account and latency testing between them. Example: if the latency from a VM in “us-east-1a” for AWS Account “Foo” to “us-east-1c” for AWS Account “Bar” is <2ms consistently, you can safely conclude that they are in the same physical location.

Let’s Do The Thing!

We’ll use PBK to find the lowest latency VM combination between the following AZs:

• GCP us-east4-a, us-east4-b, and us-east4-c.

• Azure eastus 1, eastus 2, and eastus 3.

• AWS us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, and us-east-1f

I prepared some code to do this and wrote the guide for it here. In our case, they are:

• AWS us-east-1a

• GCP us-east4-a

• Azure eastus-1

In the next article we’ll set up a proxy in each cloud provider that responds to a simple health check endpoint. We’ll then use WRK and Autocannon to do a few performance tests with and without features like Smart Routing. These will be externally mapped:

• api.[TLD]/api/v5 → Proxy in AWS VPC

• api.[TLD]/api/v6 → Proxy in GCP VPC

• api.[TLD]/api/v7 → Proxy in Azure VPC

VPC-to-VPC communication will be handled later on in a secure and performant way.