WHY? WHY DO WE NEED AI IN A ROUTER? Spoiler: We don’t. But now that it’s here like most of the subpar use cases for AI/ML being shoved down everyone’s throat…

This abomination has two SoCs. The first is a 4-core 2.6GHz Broadcom BCM4916 and this does what you’d expect a router to do. The second is …a Synaptics SL1680 with 4GB RAM, 32GB EMMC, and a 7.9 TOPS Neural Processing Unit. Hilariously enough, Asus put an embedding model on it with HF and python behind the scenes for fully local “LM” setup assistance. This was neat in that they didn’t use a transformer LM at all (SL1680s ASICs are CNN-optimized), but it worked. Here’s where stuff gets weird:

The secondary SoC is …updated by the primary SoC that’s also the admin network interface? Naturally, the first question is: how are these things connected? As I’m digging around these menus, in Settings → Administration → System …enable SSH:

Three minutes later…

Ok, so this is definitely the primary Broadcom SOC (aes, crc32). Why isn’t it updating?

^ That’s what’s in the syslog when the Web UI is busy failing at life. So …what’s rc?

“What?” indeed. Let’s heed this cue and try infil from the other SoC’s side. A few minutes of digging around later, I found the main reason I got this thing: the secondary SoC can run docker containers via Portainer under “AI Board”.

Cool, let’s install that and open it up:

Literally not even a minute later …Asus dun goofed:

Aight, bet. Basic privilege hijacking 101: run root, priv, host net, and elevated mounts:

And SSH in via the Portainer UI. And just like that, boom, we’ve got host OS root:

ls /host_root
chroot /host_root /bin/sh
ls /usr/bin
cat /usr/bin/fwupdate

Here, we find that there is definitely some kind of IPC bridge between the two:

This thing is remotely triggered by the host SOC, and monitored via IPC. Like this:

What. The. Actual. Fuqq? Yep, that’s the other SoC:

We fix the checksum, pull the firmware, update the “AI Board” from within, and follow that up with an RPC trigger from the primary SoC (that weird script up above looped into standby until we wrote to /dev/shm/reboot_flag) on the secondary SoC. Web UI:

It’s not exactly a rescue …or is it? Yep, we can update the firmware on the secondary SoC by triggering factory reset with a new recovery image. It’s written to isolated dual-boot partitions so what’s on the primary EMMC also survives this, uh, “recovery” 🤡.

Surely now I can spin up the containers I want and mount my USB dri— NOPE. The secondary SoC, the one with docker on it, can’t even access the USB ports. 🤦‍♂️

Let’s fix that. Question is, how? Asus managed to strip fuse/samba kernels out of the secondary SoC, and NFS kernels out of the primary. This was likely intentional …but that 700MB+ firmware file downloaded FAAAST. Way too fast for RS232 or anything of the sort. Some iptable and ndig later, we find that the secondary SoC has a local loopback (169.254* namespace) to the primary. Where there’s a will, there’s a way:

Ok, we have a viable pipe. Luckily, there was a BE19000 Merlin release 2 weeks ago so I didn’t have to splice cdc_acm, cp210x, and god knows what other kernels into it.

Entware it is, I suppose. Nuke a drive, format ext 4 + journal + bin, install it. AsusWRT is locked down with a read-only file system …so we’ll just bind-mount on top of it 😂

Note: Substack blocked me posting this as a code snippet - I literally couldn’t save or publish the draft. I’m guessing they think we boutta do bad thangz. They’re right.

Boom! Our secondary SOC can now mount USB SSDs to Docker at native speeds:

And wollah, the router is jailbroken. You can bypass RSA checks on the “factory reset” script and write whatever you want to the secondary SoC’s OS partitions while bind-mounting over the primary SoC’s FS. The best part? both SoCs can use the USB3.2 SSDs simultaneously. So why did I do this (outside of the fact that this is what I do)?

1. Attach a 4TB SSD on here and run Plex. On your router. With zero other hardware.

2. Your router is now also your local security camera recorder and viewer.

3. #2 lets you start using the SL1680 SoC for things it can be useful for, like running real-time local object detection and classification with Frigate - no more sending your home’s camera footage to Google Home, Amazon, or god knows who else.

4. You’re in charge of your data, it doesn’t leave your house. I plan to build on top of this with a PoE-based whole house automation system. 4GB RAM and 4 cores is more than enough to run hundreds of IoT devices, Home Assistant, HomeKit, etc.

5. From here, you can socat one of the ports to make USB devices accessible to the SoC running docker containers. That means we can plug in a hub and get:

   1. 6000ft+ range on ZigBee, ZigBee2MQTT with the SMLIGHT-SLZB-MR1.

   2. Full home RF spectrum remote coverage with the Bond Bridge Pro.

   3. Full home Z-Wave and ZWLR coverage with the HAC ZWA-2.

   4. And bridge that mesh to also control Bluetooth with Shelly Gen4s.

That even covers Matter/Thread protocol, giving you everything you need to build a pleasant and convenient home that makes your life easy with just a router (which you need anyways) and some thingies plugged into it along with a tiny 400-500W UPS. No bulky servers. That $900 price tag, for a router, is partially justifiable …maybe.

Final verdict: Now that it’s fixed I actually quite like this thing, and it’s not necessarily because of the tech. This lets me create automation that makes my life easier and is reasonably transferrable to renters or buyers - I’d just be handing off a secure local enclave in the form of a house with really good, albeit minimal, networking equipment.

Bonus: Updating Portainer

Is a bit tricky (not really, not if you do DevOps) because Portainer is what’s running the admin container that lets us SSH into the host. As always, it’s good practice to keep our admin container stateless and ephemeral (like all containers). Steps:

1. Log into router Web UI → AI Board → Control & Deploy → Open

2. click “>_” sign to open a web SSH session in the jailbreak container via /bin/ash:

Here’s a summary of how many things I’ve seen go sideways over 15 years of DevOps:

# Run this inside your Alpine admin container via the Portainer web console
apk update && apk add docker-cli
export DOCKER_API_VERSION="1.41"
docker ps # Sanity check

# Pick a tag here (never use latest): https://hub.docker.com/r/portainer/portainer-ce/tags
export PORTAINER_TAG="portainer/portainer-ce:2.39.0"
export PORTAINER_NAME="$(docker ps --format '{{.Names}}|{{.Image}}' | grep "portainer" | cut -d '|' -f 1)"
export PORTAINER_MOUNT="$(docker inspect $PORTAINER_NAME -f '{{ range .Mounts }}{{ if eq .Destination "/data" }}{{ .Name }}:{{ .Destination }}{{ end }}{{ end }}')"
export PORTAINER_EXPOSE="$(docker inspect $PORTAINER_NAME --format '{{json .Config.ExposedPorts}}' | grep -oE '[0-9]+' | awk '{print "-p "$1":"$1}' | xargs)"
export DOCKER_SOCKET="$(docker context inspect $(docker context show) --format '{{.Endpoints.docker.Host}}' | sed 's|unix://||' | awk '{print $1":"$1}')"

# Confirm things look right
cat << EOF
Container: $PORTAINER_NAME
New Tag: $PORTAINER_TAG
Mount: $PORTAINER_MOUNT
Ports: $PORTAINER_EXPOSE
Socket: $DOCKER_SOCKET
EOF

# Once again, weird things happen on SOCs. You should see "Image pulled successfully"
docker pull "$PORTAINER_TAG" && docker image ls | grep -q "$PORTAINER_TAG" && echo "Image pulled successfully."

# Update portainer
docker run -d --rm \
  -v $DOCKER_SOCKET \
  alpine:latest \
  sh -c "apk add --no-cache docker-cli && export DOCKER_API_VERSION=$DOCKER_API_VERSION && sleep 15 && docker stop $PORTAINER_NAME && docker rm $PORTAINER_NAME && docker run -d $PORTAINER_EXPOSE --name $PORTAINER_NAME --restart=always -v $DOCKER_SOCKET -v $PORTAINER_MOUNT $PORTAINER_TAG"

After update (might take 1-3 minutes, that’s normal), go back to admin container and:

# Space is precious here (we haven't installed an SSD yet) so do cleanup:
docker system prune -f

# This nukes *ALL* images that aren't attached to a running container. Should be OK:
docker image prune -a -f

# Don't touch volumes for now. That's gonna come in a later article for home automation.

Next up, we’ll learn how to:

• Permanently mount a 1TB WD NVMe SSD with a trendy matching enclosure.

• Then we’ll start connecting the secondary SOC to IoT frequencies/networks.

• Find out what, if any, useful embedded models we can run. We will train a tiny RL model to balance a home’s twin HVAC system (one on each floor) including gable vents, blowoff, UV control, humidifier and dehumidifier cycling. I wonder if I can get efficiency and comfort by using occupancy sensors and tracking sun position.

Disclaimer: Amazon affiliate links - consider using them if this was valuable to you.

In this chapter we take a detour due to the large amount of feedback I got from folks that have a 5000 series (not a 5090) and want codebase indexing but not inference.

Link to previous chapter | Link to next chapter (WIP) | Link to companion GitHub repo.

Getting Started

This setup is runnable on most Blackwell GPUs, including laptops. The defaults are 100% stable on 16GB+ VRAM. 8GB is a no-go but if you’re at 12GB just add lower:

• MAX_CLIENT_BATCH_SIZE

• MAX_BATCH_TOKENS

To the ./codebase-indexing/.env file generated in the steps below.

I set aside a few hours to polish up the containers and do a better implementation of Jina AI’s listwise re-ranker architecture in the manager. Hardcoded, filthy, but it works!

# Pull down latest repo version
git fetch --all && git pull

# Download the larger 1.5B code embedding model
hf download jinaai/jina-code-embeddings-1.5b

# Navigate from repo root to ./codebase-indexing
cd codebase-indexing

# Delete the previous .env file for regeneration
rm .env

# Run setup with a non-default (for us) embedding model
EMBEDDING_MODEL="model/jinaai/jina-code-embeddings-1.5b" ./setup.sh

# Start the new containers
docker compose --profile dedicated-codebase-indexing up

Going forward in this series, we’ll be using the Roo Code Nightly build. Download and install it, making sure to delete the existing main release branch. Make sure that you have the repo root open in VS Code before proceeding to the rest of the article:

Your new Roo Code indexing settings (database-looking icon in the bottom right) are:

Uncheck indexing, click save, then re-check it, click save, and click Start Indexing.

Direct Test

# In ./codebase-indexing, launch the tester:
chmod +x qdrant.sh
./qdrant.sh

Select the proper collection (should be index 0). Select a number of results (5-15 works best). When you’re at “Enter search query (or quit or exit):“, try these out:

• “Where do we configure the runtime between Node, Deno, and Bun?” should return exactly what you’re expecting to be there, with scores around 0.55 to 0.75.

• “Where is the runtime-agnostic script shell entry point and setup?”: works.

• FS liveliness check: “Where are LRU caches and API proxies implemented?”

  • If this doesn’t return results from manager.py open the file, add a space, and save it. Retry the query and you should see it now. Your FS listeners are funky.

Let’s Blow It Up

We are going to be intentionally dealing with a 2-3GB+ (compressed) Qdrant vector database so fully expect VS Code to crash several times during this indexing process:

Wait for the indexer to finish between each git clone command. Seeing some batch size and concurrency errors is perfectly normal - Roo Code retries these automatically.

# Run in project root - NOT ./codebase-indexing

git clone https://github.com/RooCodeInc/Roo-Code
git clone https://github.com/langchain-ai/langchain
git clone https://github.com/langchain-ai/langgraph
git clone https://github.com/langfuse/langfuse
git clone https://github.com/n8n-io/n8n
git clone https://github.com/pytorch/pytorch
git clone https://github.com/vllm-project/vllm

# And in case any normal people are reading this...
git clone https://github.com/vercel/next.js

Note: I used my daily driver .rooignore template; consider adopting it for your own use.

If you open nvitop you will see 95%+ spikes pulling 550W+. This means the code and optimizations are pipelining efficiently enough to bandwidth cap the GPU (good):

There is a practical reason for why we just did this. Context7 is nice but it’s also slow, remote, and has rate caps. You can use this approach with additional .rooignore magic to maintain a robust and up-to-date index of the tools / libraries your code uses.

Once the indexing is done go to your local Qdrant dashboard:

Trial By Fire

Open up Roo Code Nightly with a smaller model like haiku 4.5 or gemini 3 flash so the model itself can’t brute-force despite a poor environment, close all open files because they’re sent in the context and can skew our checks, switch to “Ask” mode, and try:

“How does the torch dynamo symbolic tracing graph compilation work in this project?”

At some point in the task, you’ll see embedding searches that look like this (it’s a hit):

Next task: “Tell me about the LangGraph agent state graph node edges in this project.”

Next up: “How does the Next.js middleware request rewrite redirect work in this project?”

I encourage you to compare the code search quality of this setup and custom proxy against what you get out of the box with Visual Studio Code, Cursor, and Windsurf.

Next Up

Given the pivot we’re taking, now that we have high quality codebase indexing and embedding search, the next logical step is to get under the hood and see exactly what these LM code assist tools are actually sending to the model and how to improve it.

We’ll be doing that with a LiteLLM (which we’ll use for many other things) proxy:

That ties our IDEs and LM code assist tools into LangFuse’s tracing and observability:

We will cover how to create a data set and use it with LLM-as-a-judge auto-evals to profile and improve Roo Code performance for your specific workflow and codebase.

If you’d like to jump ahead, I made a docker-compose that includes both.

In this chapter we combine inference, codebase indexing, and optimizations to fit both, alongside the host OS, on a 5090 for a fully local baseline SWE environment.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo.

Getting Started

# Pull down the latest version of the companion repo:
git fetch --all && git pull

# Pull down the inference container (may take 15-30+ minutes):
docker pull yevai/local-inference-qwen3:sm120-cu131-v1

If you set up WSL correctly, you should see the below when you run “lscpu -e”:

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE
  0    0      0    0 0:0:0:0          yes
  1    0      0    0 0:0:0:0          yes
...
 27    0      0   13 13:13:13:0       yes

If you see less than 27 vCPUs in WSL, adjust the “cpuset“ in “docker-compose.yaml”.

Next up, make sure that whatever you’re running on the host is consuming less than 2.5GB VRAM by running nvitop. If you don’t have ten billion tabs of 4K videos open because it’s 5am and you ended up in the weird part of YouTube, you should be good.

# Run this in root (requires going through previous chapters):
docker compose --profile dev --env-file codebase-indexing/.env up

Roo Code profile settings for this configuration:

You’re now able to use Roo Code just like in this video:

That’s it! Runs fast, right? We’ll be taking advantage of that. In the next Chapter, we’ll:

• Set up observability for our LM, embedder, and re-ranker.

• Run benchmarks to evaluate the performance of Roo Code.

• Create Roo Code settings for good quality with these models.

You can skip the below info if you’d prefer to wait for the next hands-on chapter.

Docker: inference-model

The yevai/local-inference-qwen3 image adds single-GPU SM120 optimizations and patches stability and performance bugs in the new version of the vLLM engine.

Now let’s go over why we did (most) of what we did in the new docker-compose.yaml!

runtime: nvidia tells docker to use the NVIDIA Container Runtime instead of runc. This allows injection of the proprietary driver files and CUDA libraries from the host.

CPU Settings

cpuset: “0-15” pins the inference container to CCD1 on the 9950X. CCD stands for “Core Complex Die”, a single physical chiplet on the CPU die. This gives vLLM a private 32MB cache, avoids Noisy Neighbor problems, and ensures we don’t grind CPU pre-processing to a halt by sending traffic across the (relatively) slow Infinity Fabric.

If you want to see inference grind to a halt, try removing the cpuset attributes from the first three containers - they will thrash each other and you’ll get ~20-30TPS.

pid: host lets the container use the host’s Process ID namespace. This removes PID translation overhead and lets preprocessing run with proper CPU-GPU signal handling.

ipc: host lets PyTorch access the host’s IPC namespace (shared memory). We want this because PyTorch relies on /dev/shm to move data between CPU worker threads.

memlock: -1 lets NVIDIA drivers pin memory pages to RAM, which lets it use Direct Memory Access because it knows the memory is (1) reserved and (2) not in a page file.

stack: 67108864 gives us a 64MB stack instead of the default 2-8MB, without which Python itself might get lost in the sauce of recursive matrix operations and fail silently.

Environment Variables

TORCH_CUDA_ARCH_LIST=12.0 tells PyTorch to compile JIT kernels for Blackwell.

NCCL_P2P_DISABLE=1 explicitly disables the NCCL bridge since we’re using 1 GPU.

VLLM_USE_V1=1 tells vLLM to use all the fancy stuff we fixed for SM120/Blackwell.

VLLM_NO_USAGE_STATS=1 disables (internal) observability for a performance boost.

VLLM_FLASH_ATTN_VERSION=3 could be its own article. At a high level, it enables:

• WGMMA, which lets thread groups collaborate on matrix math much faster.

• TMA, which is a specialized asynchronous hardware unit on Hopper+ GPUs.

• A bunch of FP4/8 specialized goodies that we want but won’t go into here.

You might be thinking “But Yevgen, we’re using FlashInfer anyways?” Not quite :)

[OPM/MKL]_NUM_THREADS=16 explicitly uses the same number of threads we cpuset.

CUDA_DEVICE_MAX_CONNECTIONS=1 prevents CUDA from creating multiple streams at the hardware level. All software queues are streamed into a single hardware command queue, which prevents thrashing, massive serialization jitter, and other bad things.

TORCH_CUDNN_V8_API_ENABLED=1 enables the cuDNN v8 Graph API, which enables Blackwell-native Kernel Fusion mechanisms. A sequence like Conv → Bias → ReLU, which is usually 3 separate launches and memory round-trips, is fused into one GPU operation. This massively helps throughput by keeping data resident in the L2 cache.

vLLM Launch Args

—quantization modelopt_fp4 tells vLLM to use NVIDIA ModelOpt, TensorRT-LLMs quantization engine. It’s the only reason this model fits on a 5090 with room to spare. Older 4-bit formats required software unpacking. This feeds directly into the Blackwell FP4 Tensor Cores, effectively doubling memory bandwidth and compute throughput.

--enable-prefix-caching should be a default for all LM code assist tools.

—max-num-batched-tokens “2048” is there because we also had to fit the re-ranker and embedding model into VRAM. The CUDA Graph config offsets the performance hit from this while conserving precious VRAM, which we don’t have that much of:

Thanks to that, we can use --max_model_len “131072”: a 131k context window. Our version of Qwen3 has 262k context natively but we’d have to turn off the embedding and re-ranker to afford VRAM for a 262k KV Cache - which you can! That’d put our very own local model context window up there with a good chunk of SoTA models.

--kv-cache-dtype fp8_e4m3 is the other piece of magic that makes this possible by quantizing our KV Cache. A 131k FP16 cache would nuke VRAM. In our very specific case the accuracy loss from quantization is <0.05%. 3 mantissa bits distinguish subtle attention signals from noise by trading dynamic range to buy back inference accuracy.

—max-num-seqs “1” is not great for agent swarms. In fact, it sucks. But it’s what we can afford in VRAM poverty. Combined with —gpu-memory-utilization “0.75”, this gives us a 1.07x token buffer which is enough for PyTorch’s memory shuffling.

These things are what let LM code assist tools like Roo Code properly use the model:

• —enable-auto-tool-choice

• —tool-call-parser qwen3_coder

And that’s it! With a relatively simple setup, we’re off to the races with SoTA codebase indexing capabilities and a capable primary model boasting a 131k context window. I might try to find a GGUF-able embedding and re-ranker and try that out to see if we can flex a full setup with 262k context even though we will almost never need it.

In previous chapters, we set up local inference. Today, we tackle one of the primary hurdles of LLM-assisted coding: context. Most RAG setups fail because they are too eager. They flood the context window with "kinda relevant". Let’s go ahead & fix that.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

This setup passes CoIR, the de-facto benchmark for these things, with flying colors.

The companion repo contains high-performance local codebase indexing & retrieval for Roo Code. This this is (inference) model-agnostic and API-compatible with any LM-based code assistance tool that relies on Qdrant since that’s the API we are overriding.

This setup enforces a “pseudo-CoT” (Chain of Thought) by applying strict reranking filters. If an initial search lacks high relevance, the system returns narrow or empty results. This compels your AI coding assistant of choice to refine its query, ensuring only high-quality context enters the context window. The result is superior steering and a significantly more productive developer experience.

Requirements

Please see Chapter 1 for compatible hardware - will expand the docker images to cover a more broad set of hardware later on if this series resonates with people.

• Unix-like environment

• HuggingFace CLI

• NVIDIA Blackwell GPU with 6GB+ available VRAM

  • The consumer versions are RTX 50 series

  • The workstation version is RTX PRO 6000

• AMD Ryzen 9000 series CPU (LTO / other compiler optimizations)

• 590+ drivers with CUDA 13.1 (check nvidia-smi)

• Docker and a logged in huggingface-cli

Getting Started

# Pull down the companion repo
gh repo clone https://github.com/NotYevvie/OnlyLocals

# Navigate to the module for this article
cd OnlyLocals/codebase-indexing

# Download the 1.0GB embedding model
hf download jinaai/jina-code-embeddings-0.5b

# Download the 1.2GB reranker model
hf download jinaai/jina-reranker-v3

# The below setup script is idempotent (safe to run multiple times)
# This checks a lot; fix the messages 1 by 1 and keep re-running it
chmod +x setup.sh
./setup.sh

# Download the embedding container (these are 9-14GB)
# This may take 15-30min+ depending on your connection
docker pull yevai/codebase-index-embed:sm120-cu131-v1

# Download the retrieval container (these are 9-14GB)
# This may take 15-30min+ depending on your connection
docker pull yevai/codebase-index-rerank:sm120-cu131-v1

# Start it up!
docker compose --profile codebase-indexing up

Connecting it to Roo Code and other IDEs that rely on qdrant:

• Enable Codebase Indexing

• Any string works for API key

• Use the below settings

• Click “Start Indexing”

When you open Roo Code, click on the database-looking icon in the bottom right:

Try It Out

This setup shines primarily in large, complex code-bases where the dependencies are also checked out as git sub-modules so that their contents are available in retrieval.

Negative Test Use Case

Once indexed, try it out with (in “Ask” mode if using Roo Code): “Search your embeddings about this codebase to find what it knows about CoIRS and if it has any interesting innovations about reranking / RAG”

If ran in this repo, little to no results outside of this mention get returned - this means our re-ranker is filtering properly.

Positive Test Use Case

Next up, try: “Search your embeddings about this codebase and give me the highlights.“

This should result in narrow returns. A few queries later you should see a high-quality overview of what this repo is.

Direct Query Utility

# With the docker compose container running
chmod +x qdrant.sh
./qdrant.sh

This is a simple utility similar to Roo Code’s search that lets you run queries* directly against collections locally viewable here. Disclaimer: this is untested and unpolished.

* This is a semantic search query, not an LM prompt or Google search. Some tips:

• Search for behavior, not keywords.

  • “DB connection” → “How does this application connect to the database?”

• Use standard industry terms even if your code or implementation are weird

  • “makeDataGoBrrt() =” → “Data processing pipeline triggers”

TL;DR: Is it a full sentence? Does it describe an action that the code performs? Good.

Embedding Container

This is a version of HuggingFace’s text-embeddings-inference with CUDA 13.1 support and performance enhancements.

• I’ve upgraded the following libraries to support SM120 (blackwell) and CUDA 13.1:

  • huggingface/candle Source Repository | Upgraded Link

  • huggingface/candle-extensions Source Repository | Upgraded Link

  • huggingface/candle-index-select-cu Source Repository | Upgraded Link

• Performance-optimized Flash Attention and LTO as well as CPU optimizations.

• A sprinkle of extra low-level magic with a focus on single-GPU performance.

Reranker Container

This is a Qdrant retrieval proxy built from NGC PyTorch 25.12-py3 that includes:

• Highly customized version of FBGEMM tuned for SM120 performance.

• Flash Attention 2 and TorchAO int4 quantization for Blackwell.

TL;DR: Pull top 100, rerank with extreme prejudice, track latest query for context.

Vector DB Container

Standard Qdrant image. Once up, browse collections: http://localhost:6333/dashboard

Having Trouble?

I put some effort into making the setup script comprehensive and user-friendly but I fully expect this to not work out-of-the-box on all applicable machine types. DM me on LinkedIn if you run into issues!

What’s Next

If this gets any traction, I’ll create containers optimized for:

• ABM (Apple baremetal, the M-series chips)

• Older NVIDIA GPUs (be warned - bad perf)

And test on / expand compatibility for Cursor, Windsurf, etc

Beyond that, I’d like to slowly distill more of my secret sauce into the public docker image of the re-ranker without my latest startup’s lawyers and technical due diligence team having an aneurysm. This includes stuff like temporal context linkage and some rudimentary linear algebra dark magic for the vectors themselves.

This chapter covers optimizing a general-purpose Qwen3 model to run at 150+ TPS.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

The following clip, which you’ll replicate in this post, is in real-time and not sped up:

Benchmarking

Unoptimized

# Install aiperf
pipx install aiperf

# Start with previous unoptimized vLLM in a separate terminal window
"$(git root)/docs/002_inference/run-model.sh"

# Warm up vLLM
curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "nvidia/Qwen3-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 10
  }'

# Run a basic perf test
aiperf profile \
  --url http://localhost:1337 \
  --model "nvidia/Qwen3-30B-A3B-NVFP4" \
  --endpoint-type chat \
  --streaming \
  --concurrency 1 \
  --num-requests 10

If running on similar hardware to that in Chapter 1, you should see something like:

Now stop the vLLM container. 113.51 TPS is already faster than a Vertex, Bedrock, OpenAI or GitHub Copilot API. But why stop here? Let’s do some sketchy stuff.

Let’s Get Weird

Pull down the latest version of the GitHub repo if you’ve been following along.

What’s better than pulling sus nightly builds? Nothing. Nothing could be better. If that wasn’t enough, there’s even a /sed/ patch …in the Dockerfile. We’re living the wild life.

# Build container from sketchy nightly wheels and shady locker room trolls
# chmod +x "$(git root)/docs/003_optimization/build.sh"
# "$(git root)/docs/003_optimization/build.sh"

# Run with newly added optimizations in a separate terminal window
chmod +x "$(git root)/docs/003_optimization/run-model.sh"
"$(git root)/docs/003_optimization/run-model.sh"

# Warm up vLLM
curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "nvidia/Qwen3-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 10
  }'

# Run the same perf test
aiperf profile \
  --url http://localhost:1337 \
  --model "nvidia/Qwen3-30B-A3B-NVFP4" \
  --endpoint-type chat \
  --streaming \
  --concurrency 1 \
  --num-requests 10

You should see ~172TPS, a roughly 51% increase. Spin up the UI and try how fast it is:

"$(git root)/docs/002_inference/run-ui.sh"

Link to local UI. This is where “Effective AI-assisted development using local resources” is going: Modern LMs are plenty capable and this is notably faster than API calls.

Most folks don’t fully appreciate how heavily they’re guilty of model overkill. With the right setup you’ll have LM code assist that works ~90% as well but is 400-700% faster.

Next up: Setting up coding-specialized models and connecting them to VS Code.

This chapter covers running a 30B Qwen3 model at 60-90TPS (we’ll optimize this later).

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

WSL Docker Setup

We should install Docker Engine inside WSL so that nvidia-container-toolkit can interface directly with the WSL Linux kernel via dxgkrnl. This lets WSL sub in for a native host OS by directly managing the GPU resources.

# Install Docker
curl -fsSL https://get.docker.com | sh

# Add user to group
sudo usermod -aG docker $USER && newgrp docker

# Reload
newgrp docker

# Confirm socket (should show CONTAINER ID in red)
docker ps | grep "CONTAINER ID"

# Add Nvidia repositories
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update sources and install
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure NVidia runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart docker
sudo systemctl restart docker

Downloading a 30B Model

The casual favorite. Make an account on HuggingFace and get an access token here.

# Install the HuggingFace CLI
brew install huggingface-cli

# Enable Git credential store
git config --global credential.helper store

# Log In (also store as git credential)
hf auth login

# Confirm you're logged in
hf auth whoami

# Save your auth token - replace YOUR_HF_TOKEN with your actual token.
echo 'export HF_TOKEN="YOUR_HF_TOKEN"' >> "$HOME/.zshrc-pre.sh" && source

# Create a folder for HuggingFace models
mkdir "$HOME/HF_Models"

# Add it to the pre-zshrc hook from Chapter 2
echo 'export HF_HOME="$HOME/HF_Models"' >> "$HOME/.zshrc-pre.sh" && source "$HOME/.zshrc"

# Confirm it's available. Your models will download here.
echo $HF_HOME

Qwen3-30B-A3B-NVFP4 is a solid Blackwell “Hello World” model. While it’s technically a 30B model, it uses only ~3.3B active parameters per token, giving it decent throughput on a 5090. It’s not particularly great (even compared to other single-GPU 5090 options) at anything, but it is one of the rare cases where NVFP4 quantization resulted in higher scores on some of the benchmarks we care about.

hf download nvidia/Qwen3-30B-A3B-NVFP4

Running it with vLLM

Blackwell GPUs support NVFP4 (1 sign, 2 exponent, and 1 mantissa bit) models and we’re going to be heavily abusing this. This groups weights into blocks of 16, each of which shares a high-precision 8-bit (E4M3) scale factor and allows the 4-bit quantized coefficients to handle the finer details.

They also come with the 2nd Gen Transformer Engine, which which has Tensor Core instructions that operate directly on the compressed 4-bit blocks. The result of this is much more efficient use of VRAM that lets us run much larger models with only ~28GB of available VRAM (4GB for host OS).

vLLM recently added experimental support for NVFP4 so we don’t have to build TensorRT-LLM from source and compile our models for TRT-LLM to get reasonable TTFT (time to first token) and TPS (tokens per second).

vLLM also comes with a number of optimizations like PagedAttention for the KV cache.

# Build the vLLM container.
chmod +x "$(git root)/docs/002_inference/build.sh"
"$(git root)/docs/002_inference/build.sh"

# Start the vLLM container. This takes A WHILE (8-10 minutes) for precompute.
chmod +x "$(git root)/docs/002_inference/run-model.sh"
"$(git root)/docs/002_inference/run-model.sh"

Chatting with it

# Start the UI server.
chmod +x "$(git root)/docs/002_inference/run-ui.sh"
"$(git root)/docs/002_inference/run-ui.sh"

Head over to http://localhost:1338

First, you’ll have to “warm up” vLLM with a simple hello world prompt. This will take a while, after which you can expect 60-90TPS. We’ll optimize this later. When you first open localhost in your browser, you may see errors. Click on “Check again” and reload!

Bonus: Resource Monitoring

# Install pipx
sudo apt install -y pipx

# Install nvitop
pipx install nvitop

# Add to path
echo 'export PATH="$HOME/.local/bin:$PATH"' >> "$HOME/.zshrc-pre.sh" && source "$HOME/.zshrc"

# Run nvitop
nvitop

This chapter covers proper setup of a WSL distro for the series.

Link to previous chapter | Link to next chapter | Link to companion GitHub repo

Windows Subsystem for Linux (WSL)

We’re setting up a linux-based environment for this guide because most modern cloud deployment targets are ARM or linux-based x64. WSL uses GPU Paravirtualization (GPU-PV), whereas your production B200 and/or NVL72 nodes use PCIe passthrough (DDA) via AWS Nitro, GCP Titan, or Azure Boost. GPU-PV incurs a relative performance penalty but, for the purposes of our guide, it is negligible.

Host Setup

1. Enable WSL on Windows

# PowerShell Admin
wsl --install

2. Install Ubuntu LTS (”Long Term Support”: use in prod unless building from Alpine).

# PowerShell Admin
wsl --install -d Ubuntu-24.04 --name Yevai --web-download

3. Download and install VSC (Visual Studio Code - Link)

4. Use this knowledge to connect to WSL with VSC.

Everything else in this guide, unless otherwise explicitly indicated, will be done in WSL.

WSL Setup

Assume all commands to be ran, unless otherwise explicitly indicated, in $HOME (~) of the active WSL user. You will see “$HOME” used instead of ~. This is best practice for many reasons we’ll skip in this guide. For example, in some target environments like AWS Lambda, you will need to override $HOME to enable libraries to use the /tmp/ ephemeral FS layer cache. $HOME and ~/ do not always interact the way one would expect.

1. Update OS components and compensate for the fact that we are in WSL

sudo apt update
sudo apt upgrade

# Install WSL utilities
sudo apt install wslu

2. I have mixed feelings about Homebrew but it makes this guide much easier.

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Activate it in the current shell
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

# Install the Github CLI, ZSH, PL10K, and JQ.
# Several LM-based coding tools benefit from PL10K!
brew install gh zsh powerlevel10k jq

3. Shell setup - feel free to skip if you have your own preferences. The guide will rely on this specific setup. Checking out the companion repository is non-optional.

# Allow ZSH and set it as default
command -v zsh | sudo tee -a /etc/shells
chsh -s $(which zsh)

# Install OhMyZSH
sh -c "$(wget https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"

# Install and go to ~/Git. $_ gets the argument of the previous command.
mkdir "$HOME/Git" && cd $_

# Log into GitHub ("open" works thanks to wslu). Don't use tokens unless you have no choice.
gh auth login

# Confirm you're logged in
gh auth status

# Set the global git config - doing so from the active GH CLI is best practice.
# Note: this is especially useful when automating in CI with service accounts.
git config --global user.name "$(gh api user --jq '.name')"
git config --global user.email "$(gh api user/emails --jq '.[] | select(.primary) | .email')"

# Confirm your git username and email
echo "$(git config --global --list)" | grep user

# Clone the companion repository and go to it.
gh repo clone https://github.com/yev-ai/personal-yev-substack && cd personal-yev-substack

# Git aliases are useful, especially for rebases / merges / automation.
git config --global alias.root "rev-parse --show-toplevel"

# This now shows the folder of the git repository you're in.
git root

# Mount ZSH and PL10K configs - this also serves to confirm our paths.
chmod +x "$(git root)/docs/001_wsl_setup/setup.sh"
"$(git root)/docs/001_wsl_setup/setup.sh"

# Export the Git Root directory into our pre-zsh hook. This is where you checked out the repo.
echo "export GIT_ROOT=\"$(dirname "$(git root)")\"" >> "$HOME/.zshrc-pre.sh"

# Source the new ZSHRC
source "$HOME/.zshrc"

# Make sure our GIT_ROOT is set correctly - we'll use this heavily in later sections:
echo "$GIT_ROOT"

Confirm GPU Virtualization

If your host and WSL are set up correctly, including the latest drivers, this should be available in WSL:

nvidia-smi

You should see output similar to:

The windows host OS reserves around ~2GB VRAM. This guide will gracefully work around lack of exclusive GPU control which we normally have in cloud nodes.

Confirm Memory Allocation

# Install a basic resource monitor for WSL
brew install htop

# Run it
htop

You should see output similar to this:

Only 50% of our host’s 96GB RAM is available to WSL under “Mem”, along with a 12GB swap file. This will become important later.

This chapter covers the hardware build used for the series. TL;DR RTX 5090 required.

Link to next chapter

Parts

• CPU: AMD Ryzen 9 9950X3D

• Mobo: ASUS ROG STRIX X870-I

• GPU: Gigabyte 5090 Gaming OC

• SSD: WD_Black SN8100 4TB

• RAM: F5-6000J3036F48GX2-FX5

• PSU: CM V1100 SFX Platinum

• Case: Cooler Master NR200P V3

• Cooler: CORSAIR Titan 280 RX

• Riser: LINKUP PCIE 5.0 Riser Cable

• Fans: Noctua NF-A14x25r G2 PWM

OS

1. Install Windows 11 Pro

2. Run Activation https://github.com/massgravel/Microsoft-Activation-Scripts

# PowerShell Admin
irm https://get.activated.win | iex

3. Run Remove AI https://github.com/zoicware/RemoveWindowsAI

# PowerShell Admin
& ([scriptblock]::Create((irm “https://raw.githubusercontent.com/zoicware/RemoveWindowsAI/main/RemoveWindowsAi.ps1”)))

4. Run Debloat https://github.com/Raphire/Win11Debloat

# PowerShell Admin
& ([scriptblock]::Create((irm “https://debloat.raphi.re/”)))

5. Install common VCC redist (Download)

6. Install latest AMD drivers (Link)

7. Install clean NVidia drivers (Link)

8. OC Hardware. TSE 22,478 -> 24,106

Rant

I’m an experienced PC modder/OCer and like to rebuild my work PC every 3-5 years. My last build was the fastest 3960X in the world at 19,745 TSE. Both 3090s were shunted, delidded, and EVC2-modded. The goal for this build was portability and ML performance but seeing the 5090 clock in at 25,506 Graphics Score on TSE was extremely disappointing since my twin 3090s scored 23,012 five years ago.

…so I delidded the GPU, replaced thermals, shunted the VRMs, rebuilt the power delivery circuitry to safely draw up to 850W (the 12VHPWR connector is a travesty), and flashed a 1000W BIOS ...again ¯\(ツ)/¯. A few CNC modifications were made to the case so it could maintain proper temps during the 8-hour stability test ran at 78f ambi.

Pics and Wrap-Up

Or it didn’t happen, of course. The GPU is secured to the case with screws in 3 additional places to keep it safe during air and ground transport.

When your desires - not impulses - your purpose, values, and morals - scream louder than your fears of criticism, hardship, or failure, you gain a clear destination and sense of what you’re willing - and, more importantly, unwilling - to do to get there: an identity.

This tends to be the point at which someone’s impulses change from feelings of being pushed to run away from things to feeling of being pulled to run toward things - it’s also the point at which adversity begins forging people rather than destroying them.

Leveling Up

What you want to do can’t be done alone. You need a team so you start solo grinding the stats necessary to effectively use your spells and abilities in a team environment.

External motivation collapses under adversity. Internal motivation carries you, but it burns out. Intrinsic motivation - from purpose - is a superpower we call obsession.

Purpose also makes it easier for people to decide if they want to play with you. For starters, they can see if you’re playing Rocket League, Mario, WoW, or StarCraft 2.

Matchmaking

Boom, you’re a mage. You know where you want to go, what you have to get there, and what you need. You found a priest (partner) to play 2s, a rogue (best friend) for 3s, and a core group for 5s. Two of them are in the same guild as you (coworkers)!

Your teams overlap, and poison in any one spreads fast. You’re unbothered because your priorities are unfucked and everyone knows where they stand. Your partner, friends, coworkers, and family members are your teammates. They’re also your teams.

The gap between the levels of love, trust, and respect you grant yourself and what you can accept from others is your matchmaking rating band - and it is entirely up to you.

Team Composition

Every team is unique. You can’t run Mage/Rogue like Mage/Priest, or play either of those the same way against Warrior/Paladin and Hunter/Druid. How fast you learn your teammates’ tendencies and cadence - and how well you complement them - determines how well your team plays. Are you playing with a Druid or a Warlock?

Depending on your comp and what you’re up against, someone’s getting ran the fuck over most matches. If you’re the Mage in Rogue/Mage/Priest, it’s you. Your job is to survive getting DP’d by a Warrior and Hunter until your Rogue lines up the burst.

It feels like you’re getting fucked the whole time, but you just pulled off what most people couldn’t even dream of - because your team played well under pressure and you were able to trust someone else to carry when the stakes were extremely high.

Losing

All teams do well when things go well. Top teams also perform when they don’t.

Our team lost in the world semi-finals for 3v3 Arena in 2015. Everyone knew who and what caused the loss. All of us had important things riding on the line. Our teammate who misplayed wasn’t able to house his folks, or send his sister to college. It sucked.

When you care about what someone brings, you’re afraid of losing them. When you care about them, you’re afraid of hurting them or letting them down. Only one of those will keep your team afloat when a key member can’t bring much to the table.

We practiced for another season, couldn’t recover, and disbanded in 2016. I’d do it all over again because going for broke and getting blown the fuck out is easier to live with than the what-ifs of compromise or the guilt of going at it half-hearted.

Winning

Is valuable because it bonds. It’s best served when you have people to celebrate it with, and those people were in the shit with (and especially against) you. That said, people who don’t know the difference between opponents and enemies are unfun.

In 2018, our team took world first and $600K with it. Our guild and other pro teams were staying in AirBnBs near Anaheim. The teams we played against celebrated at the house afterparty. The amount of shit that got talked deserves its own numerical prefix.

Most of my favorite memories exist because I don’t know how - or when - to give up.

There is no point to this post. It’s Halloween, I just worked another 120+ hour week, and wanted to write a bit to help me decide whether I’m going to bed or the bar 🤗

For normies unfamiliar with the acronym: “OP” = original poster.

Quote from OP’s post

“Your AI isn’t lying to you when it tells you that Abraham Lincoln invented the iPhone. It’s doing something way weirder. Hallucinations aren’t bugs. They’re the feature working exactly as designed, just not how you expected. The thing is, large language models don’t “know” anything. They’re pattern-matching machines that learned to predict words by reading the entire internet.”

[Comment 1] - Me

RE: “large language models don’t “know” anything. They’re pattern-matching machines that learned to predict words”. This is only true for pure first-gen decoder-only transformers - NOT language models. I checked your profile and it’s unclear whether you have a background / experience in CS, Statistics, or ML. Before I say anything else - what’s your level of hands-on experience?

[Comment 2] - OP response

Yevgen Reztsov I’m not a CS/ML guy but I’m the cofounder of an AI automation platform, I spend most of my waking hours in this space. What do you think has changed in frontier models that we should cover here?

[Comment 3] - Me, 1/2

[REDACTED TAG] BERT is making a large comeback, and there are a lot of rapid changes in the attention layers. Most of the new LMs are no longer pure decoder-only transformers - the attention layer is often hybridized with RL (linear, state-space, etc). 2025 Examples:

Self-generation, decomposition, and verification of model reasoning: https://arxiv.org/abs/2501.13122

Progressive improvement via task self-generation and self-verification with no external data: https://arxiv.org/abs/2508.05004

Notable “reasoning” self-improvement with self-proposed tasks and self-verified solutions (no external data): https://arxiv.org/abs/2505.03335

Arena-based self-improvement (We did this with adversarial RL for AlphaStar when I worked at Google DeepMind in 2017, after working at OpenAI in 2016): https://arxiv.org/abs/2509.07414

This one is 💩 but it’s still self-bootstrapped “reasoning” with no external data: https://arxiv.org/abs/2203.14465

Most of these trace back to a 2022 paper on distillation from self-generated synthetic data sets: https://arxiv.org/abs/2202.07922

Predictive generation tends to be synonymous with eventual model collapse (a classic transformer trained on its own outputs enough times becomes nonsense). These show the opposite.

[Comment 4] - Me, 2/2

[REDACTED TAG] (Didn’t fit into one comment). Point is: I’m VERY hesitant to say models “reason” because we understand HOW they work, not WHY they work. From a neuro- and cognitive science perspective, we also have no idea how human cognition (important nuance) works. It COULD be an extremely complex form of recurrent next-symbol prediction with other aspects of cognition mixed in for all we know.

We *DO* know that pattern recognition and pattern prediction are FACETS of cognition. Memory is also a facet. So are procedurals. Etc.

That puts a big pickle into “do any of these actually reason”. We don’t even know what that means. In short, we OBJECTIVELY don’t know.

Outside of LI influencer hype, virtually no ML researcher or scientist will say that “Language Models” reason - or vice versa. That’s because it is tantamount to career suicide in the ML scientific community as of now.

No disrespect meant, but if you (or anyone reading this) think about the last time you’ve talked to (or saw an LI post from) a frontier ML scientist / researcher actively publishing papers... RIP. Zero. I am one of those. Personally, I’d say we don’t know ...unless I missed a memo about the next Nobel Prize winner in 3+ cats 🥲

That doesn’t make for a great LI post tho.

Wrap-up

For my subscribers: are fast blurbs like this something you want to see, or would you prefer I stick to more thought-out ones (Substack spams your email when I post) where I take the time to write, edit, and proof-read it out of respect for yours?

Let me know!

Meme credit goes to this LI post from Eduardo Ordax (Gen AI Lead at )!

The Gist

Things got busy! It’s been two weeks since my last post. Paying customers came much sooner than expected. We now have 4 design partners totaling $18K MRR. Ba dun tss.

That’s a good number for continuous real-world feedback, so that phase is full. Not all customer feedback should be actioned - more on core competencies in later posts.

Highlights

• Gathered a round of feedback for desired MVP scope and gave date estimates.

• 70%+ odds of a mid-5 digit ARR AI/ML security contract from the Pentagon.*

* If signed and permissible, I’ll cover it in anonymized, customer-general terms.

Priorities

• Finalized Statement of Work, MSA, EULA, and TOS updates by 10/03.

• Pre-hiring: I need to find my last qualified ML engineer for the project by 10/06.

That’s it! Staying razor-focused and only having 1-3 priorities at a time is important.

Process

• Established POCs/SOPs for Customer Success (me): emails, Slack channels, etc

• Basic sales pipeline and status tracking (also me): just Google Sheets for now.

I had to set up federated SSO (Google Workspace made this a breeze) and will share how to do this for common IT vendors like Linear, Notion, and more in a post soon!

Notes

If you read ML papers, make sure you read this. I’ve been using this approach for years, but it’s good to see it quantified. TL;DR: if you can automatically verify step-by-step problem solving, you can also automatically improve it with RLAIF.

Also: a notable, more accessible, and readily applicable paper on in-context learning.

Total startup cost: ~$1,000–1,500 upfront, and ~$200/month upkeep. This includes putting yourself on W2 payroll (sans wages), which you want in place from day one.

Zen Business

• Great feedback from fellow founders about speed, quality, addons, and support.

• Covers state filings, beneficial ownership reports, and registered agent services.

• The “Filing + Tax Advice” addon at $99/mo is great for getting started.

Stripe Atlas = vendor lock-in; Carta Launch doesn’t cover basic legal compliance. For the uninitiated: always make a Delaware C-corp. Get your SIC and NAICS codes right.

Chase

• Excellent baseline support for things like ACH, treasury, and wire transfers.

• Widespread branch network is useful when things inevitably go wrong.

• Plays well with most vendors, like QuickBooks and Rippling for expenses.

Mercury and Brex are popular alternatives; my knowledge there is dated.

Carta

• Industry-standard cap table management that’s trusted by lawyers and investors.

• Grows with you from founder stock, SAFEs, and options to 409As and Series A+.

• Built-in document templates, data room, and investor relations are excellent.

Stripe

• Their billing portal saves time; their APIs are built for a good developer experience.

• Webhooks, observability tools, audit logs, and first-class test environment support.

Rippling

• Handles onboarding, offboarding, HR compliance, and has great legal templates.

• I’m a big fan of their LMS, Perf Management, and Workflows for HR automation

• Supports most countries for overseas payroll which is otherwise a total nightmare.

• Once you have 2+ FTEs, their ASO and PEO are great: benefits, workers comp, etc.

• Bonus: it provisions the two things Google Workspace doesn’t: 1Pass and devices.

Rippling lets tech founders handle HR, ops, and compliance correctly from day one.

• The Gist - A subjective soapbox rant.

• Highlights - Objective recent events.

• Priorities - What stays and what goes.

• Process - System and process changes.

• Notes - Occasional reflections.

Let’s get into it!

The Gist

Maintaining my own code for 15+ years has taught me that deciding how prescriptive to be about tech stack and code contracts is an unforgiving decision theory problem. My friend Cassie Kozyrkov is a pioneer in the field, and her work is worth following.

Throw in upgrades, infra, cost, durability, uptime, and SLAs into the mix, and you’ve got a high leverage set of choices. The goal is to be minimally prescriptive and solve the problem in front of you without locking into assumptions that may backfire later.

Do this well and you’ll move significantly faster while taking on less tech debt.

Highlights

• Finding initial customers went eerily well; I’m at 3, totaling just under $10k MRR.

• The wedge theory was overkill: people want hands-off PostHog and Langfuse.

• Unexpected upside: minimizing cloud costs is a universally shared value-add.

People want to sign up for an endpoint their engineers can immediately use.

Priorities

• Testing pipeline for minimally prescriptive AKS, EKS, and GKE by 09/06.

• Langfuse setup across three clouds with API + integration tests by 09/10.

• Ingress for AWS, GCP, Azure, and Cloudflare to make it usable by 09/13.

Process

• Non-tech vendors added: ZenBusiness, Chase, Stripe, Rippling, and Carta.

• IT vendors added: Google Workspace, 1-Password, Slack, Linear, and Notion.

I’m relentless about keeping the vendor count low. I’ll share a separate post on what we choose, how we integrate it, and which system becomes the source of truth for each operational domain - a decision you absolutely need to get right the first time.

Notes

Per my last post, this is a “starting my own (again) and will build it in public” deal. That means everything is from scratch. Last time, I made the unfortunate mistake of using Microsoft / AD for SSO federation. We’re going with Google Workspace this time.

About Me

I left the startup I was working for one week ago. I’m starting my own (again) and will build it in public. This is my founder’s diary: gathering customer feedback, finding the wedge product, getting first customers, and hiring. I’ll post Wednesdays & Saturdays.

I’ve run 3 startups and carried DevOps, DX (developer experience), and ML infra from pre-seed to Series C for many others. Making AWS, AZU, GCP, Pulumi, Terraform, and ML boring and reliable is what I’m very good at according to people I’ve worked with.

The Problem

As capabilities for writing code accelerate, the need for observability, environment parity, deployment, and debugging grows proportionally. One day, out of the blue, feature velocity plummets, the app becomes unstable, and fixes (to bugs that you should’ve detected before your biggest customer told you about it) take forever.

There are excellent hosted and open-source solutions for all of those. Deploying and implementing them to enable tracing, telemetry, deployments, monitoring, alerting, and analytics is hard. It’s cross-company code, infra, and tech debt duplication.

The Customer

Series A and beyond startups. They all have a lot in common, including a strong bias toward action. Taking on calculated tech debt to secure ARR is the correct choice. The consequences of that choice catch up as teams scale from ~5 to 10+ engineers.

Past Attempts

Per the recent MIT study, ~95% of AI implementations fail. This is true for most tech purchases that need implementation whereas those that don’t mostly succeed. The lesson is that joint ownership doesn’t work. Single responsible party = success.

UI‑first PaaS blurs ownership and makes edge cases impossible. Well designed GitOps components have clear ownership boundaries, avoid vendor lock-in, and handle edge cases by giving us the ability to fall back to Terraform and Pulumi whenever necessary.

The Solution

My job has always been to make the other engineers’ jobs mostly business logic. I’m productizing that. Pure GitOps - no fancy UIs that blur ownership boundaries, kill the ability to deal with edge cases, and take forever to build while delivering zero value.

I’m building a framework of components that are SOC2/HIPAA/etc ready* and retain the ability to handle edge cases. They just work together, come with app integration code, playbooks for zero-downtime deployment, managed upgrade PRs, SLAs, and built-in observability that covers common use cases from serverless to Kubernetes.

* For transparency: this is being built so we can retrofit compliance. The promise being made is that making the components themselves compliant will not require customer-side code changes. Updating them, testing them, and getting them certified is for later.

The Wedge

Imagine adding session replay that correlates user clicks to every API and database call emitted by them to your React application in 2-5 days. That includes deploying self-hosted OpenTelemetry, PostHog, Langfuse, and code to tie all of them together into your app. Success looks like going from sign-up to production in 2-5 days.

Crystal clarity in ownership means a milestone-based definition of “done”. You don’t pay for access to infrastructure as code - you pay to have it working in your app.

Defensibility

I’m posting the plan because this isn’t a "I’m a VC, I’ll bankroll my friend” problem. As Chainguard (love those guys!) aptly puts it: this shit is hard. Professional services don’t scale and very few people have the knowledge and experience to do this properly.

This isn’t “move fast and break things”. A single mistake brings down production for a customer and destroys all trust. AI-assisted test generation and code upgrades will be needed to maintain this and make sure it works in all 3 cloud providers for K8S-native and cloud-native deployments. In that sense, this post can be considered a pitch deck.

What’s Next

I find the first 1–10 paying design partners. Want to self-host complex, distributed OSS software that normally requires a dedicated DevOps team? Book me on Calendly or reach out via LinkedIn with “Hey, we run [docker] on [aws] and want [langfuse]!”.

The first few customers will be design partners. Your input will be heavily weighted for roadmap prioritization and you will essentially have me as a dedicated DevOps engineer for a fraction of the cost. I greatly value my reputation and won’t be taking on more customers than I can properly serve. This will scale slowly, but it’ll scale right.

When it comes to infrastructure, reliability (aka quality) is everything.

In a sea of clueless bullshitters LinkedIn AI Influencers, the above post stood out to me as I scrolled with my morning coffee. It was to the point, grounded, and genuine. The theme here is knowledge that is useful and easy to leverage - let’s roll with that.

I woke up at 4:45 and will start work at 5:30 so I need to finish this post by then. Let’s make it short, fast, actionable, and hopefully useful to at least one of you. We’ll walk through a simplified version of a setup I recommend to my non-engineer friends.

What Will This Get You?

Use case: “Good morning. What did I miss?” (Lots of red but its all accurate/useful).

Use Case: “What do you remember from the last two days?”

Use Case: someone sends Platform a message on slack about an outage. This:

• Picks it up from Slack automatically and dispatches a Boomerang job.

• Checks Sentry, Prometheus, GCP, K8S (kubectl), and LogRocket.

• Checks recent memories, Google Drive, Linear, JIRA, Calendar, etc.

• Replies to that person in Slack, let you know, and if necessary, creates tickets.

The tickets a setup like this creates are GOOD. Like, actual acceptance criteria an Engineer can pick up and start immediately good. Similar tools govern my daily priorities, what I need to do, help me communicate via email, and other things.

What Are We Doing?

3 personalized contexts, an MCP server, and an A2A implementation Are All You Need. After 15 years of ML research (and applied ML - I do things! sometimes), I understand things well enough to explain them in simple and concise terms.

These 3 things (queue BuzzFeed clickbait) are the AI “80” of 80/20 for non-engineers:

• CONTROL YOUR DATA. A synced OneDrive/iCloud/GDrive folder is a good start.

• Check HF Leaderboards for models relevant to YOUR use cases every 2 weeks.

• Create stateful* contexts for YOUR use cases. I’ll show you how to do this below.

*- Stateful here means with persistent memory and dedicated embedding back-ends.

Let’s Do The Thing

Install Visual Studio Code (ab: VSC). Click on the icon that looks like a lobotomized Rubik’s Cube in the top left sidebar and install the Cline and Roo Code extensions:

Get API keys from OpenAI, Anthropic, and Tavily (web search that doesn’t suck). In the top middle of VSC (Visual Studio Code), there is a text box. We’re going to click on that text box. Then we will move our hands over to our keyboard, type in “>Cline”, and press Enter or put our hands back on the mouse to click “Cline: Open In New Tab”

A wild UI component appears. Don’t panic! We got this. Click on the box that looks like the three stacked pizzas you ate last Saturday before passing out (bottom left):

Okay, my coffee is slowly kicking in so we’re going to go a bit faster now. Go to the marketplace tab on the left and install the Tavily MCP server plugin (pictured below).

Cline will walk you through the entire process, test, and if necessary troubleshoot + fix the MCP server installation. All you have to do is type in (presumably) English.

We’re AI wizards now, Harry. We now have a place where we can use any model with robust web search functionality (please don’t go on LinkedIn and call this an Agent).

Cline In A Nutshell

Think of Cline as a middle manager. It’s good at orchestrating MCP servers and Roo Code, but it doesn’t actually do any work except Rules and Workflows - you can find those by clicking on what the US justice system should’ve been. Bottom left of Cline:

You may have noticed the annoying yellow highlight that both lets us know we are not using an Apple product and sums up the first thing life crushes in adulthood: “Plan”

Cline is good at freeform planning. Ask it to plan something, then prompt it to update that plan. Like the middle manager that it is, Cline is good for doing deep research which we can then ask it to store in our local Mem0 MCP server for Roo Code.

That brings us to the limit of usefulness that is Cline. To reiterate:

• Configuring and troubleshooting MCP Servers

• Rules and Workflows. Read more about those.

• Deep research and local memory management

Roo Code: The Nut In The Shell

Cline and Roo used to be the same person (Roo is a branch of Cline). Then they had a fight, and now they’re frienemies. They still have similar mannerisms. Let’s open the settings for both so we can synchronize our Cline and Roo Code MCP servers:

• Cline: go to Remote Servers (same place as Marketplace above) and click “Edit Configuration” at the bottom. This opens up the Cline settings JSON.

• Roo Code: the three Pizza boxes are located on the top right just because. We click that, then scroll down to the bottom and click “Edit Global MCP”.

A visual depiction of tech debt wanting to be business value when it grows up:

The format is very similar, but we will need to make a few small changes:

Or delegate to one of the 3-8 instances of OpenHands you surely run in parallel.

The orchestrator mode in Roo Code is what we’re after because it can:

• Have its own API and MCP settings (use C4Opus for the Orchestrator itself)

• Automatically select between contexts we define (and is decent at it)

• Each of those contexts have their own model, API, and MCP settings

• They can run in parallel with shared MCP state and memory back-ends

Speaking of those contexts, let me give you a simple example: Secretary

Intermediate example: DevOps

Through pragmatic use of allowed commands (aka NOT kubectl deploy) and local MCP servers, you will get pretty good mileage. I suggest setting up, at the very least:

• File System

• Google Calendar

• Google Drive

• Slack

A Note On Memory

The importance of local MCP memory (I personally prefer mem0) initialization and scheduled updates can not be understated. Do this with Cline. Example prompts:

• "memorize the slack channel ID and name correlations, then check your memories and confirm that you remember them."

• "memorize that the latest Linear engineering requests are added under and checked from the [REDACTED] linear project - make sure to read the project description for instructions and memorize those."

• "look through the latest 20 messages in each slack channel you have access to, create concise channel purpose summaries, and store them alongside your existing slack channel memory mappings to increase your awareness of which slack channels you should look in and post to for specific topics"

Other Protips

• LMs are dumb. Structure projects / *.MDs to mitigate that and tell ‘em to RTFM.

• Use “brew tree” and forced tool calls to let LMs better navigate your projects:

• Make sure your OpenHands instances each have a separate Mem0 MCP store.

  • Hotwire your OpenHands instances to plex via A2A (agent to agent).

• Just like *.MD files, due to training data, LMs do well with Mermaid Diagrams.

• Do not overkill the model used for a task. Larger models are not always better.

  • #1, by far: know when to use other model types (not LMs).

  • #2: Know when to use, or not use, thinking models.

  • #3: Know when to use fast SLMs or nano LMs.

• Not all memory is created equal. Even for a sh*tty local only setup, you want to use a combination of Graph, Semantic, Episodic, and Embedding memory.

• RAG sucks unless you need dynamic in-context chain of thought

  • Forced tool calls via something like Algolia are better 90% of the time.

• Learn your MCPs like you learned your ABCs.

  • Example: if you use LMs for code without basics like Context7 …yeah.

  • Limit ‘em. 10-15 max per context and use RAG to assist with selection.

• Good prompts are as important as they are model-specific.

  • A model-agnostic prompt library will make your life better.

  • Pre-initialized templates are underrated. (ie., you pre-populate the first 1-5 assistant : user message pairs with a focus on what the LM thinks it said)

Effective system prompt example:

ML is a great hammer but not everything is a nail. In general, LMs and RAG are a last resort. They are both slow, expensive, and highly prone to errors. People who are new to ML (aka most of the world) are in a honeymoon phase. Unless you’re an intern, you already know that a system’s observability should be proportional to its autonomy and a system’s reliability should be proportional to its parent ecosystem’s complexity.

In terms of “code paths to cover with unit tests”, ML is a f*cking nightmare.

Conclusion (aka Self-Promotion)

Did you think that today was the day you’d see a post on the internet without self-promotion? Look at the bright side: it’s not “hey bro I’ve got this sick report free(tm) all you need to do is comment on my post or DM me your email” — Sincerely, a “coach” (unemployed) who claims to have scaled startups from 0 to $10M+ / have F500 clients while attempting to sell some “program” for $10-$5000 on LinkedIn.

Unless you want to join our wonderful team over at Sully AI (I’m hiring and build our Platform/AI tools - unlike this 2023 technology, we actually use modern tooling) this should be around 60th-70th percentile in terms of AI tooling you can get for free.

TLDR even if all you do is OpenAI API calls, your users’ TTFT (time to first token) decreases by 395-870+ms, your vendor footprint also becomes smaller, you save money, have better consistency, lower downtime, and your developers love you.

This bridges the gap between companies wanting cross-cloud benefits and those with a Platform team and budget to deploy and maintain K8s clusters in every cloud, with vertically / horizontally scaled Istio sidecars, private SSL certificates, VPC peering, and an Envoy service mesh. Despite its transitional nature, this approach is robust, resilient, maintainable, performant, and scalable. It can and has served 1M+ cross-VPC RPS.

Previous posts: “ML: A Networking Perspective, Pt1/X”, “ML: A Hardware Perspective”.

Benefits (Attempt #2)

1. User Experience

Everything we do is for our users. Seamless UIUX requires sane response times:

• 20-100ms: Feels instant, highly responsive - this is the ideal experience.

• 100-300ms: Noticeable, but still considered near-instant and seamless.

• >300ms: Noticeable delay, mild but acceptable UX degradation.

• >500ms: Perceived lag, sharp decline in UX, and user frustration.

• >1sec: Significant UX degradation and likely user abandonment.

Infra is necessary to build things that are fast. Abstracting that infra is necessary for enabling engineers to build those things fast. On the front-end, that looks like this:

Analysis paralysis caused by free-form text is the silent killer of user experience. A generic, easy-bake recipe for reducing user cognitive load and improving UIUX is:

Superior UIUX for ML products often involves moving away from free-form, text-only interactions as quickly as possible (e.g., Lovelace). This requires a data-driven UI player.

The backend sends a JSON payload representing your users' UI layout (ex: open code canvas, file workspace, query engine, document uploads or downloads, etc.). Like this:

2. Developer Experience

The ideal application developer experience involves at least 70% - ideally over 90% - of what they write being business logic. Less boilerplate equals more customer value.

This series will provide a comprehensive guide for a single engineer to empower your organization with the following advantages in 1-3 days, troubleshooting included:

Front-End Development

• Provide a simple way to use CSR, SSR, SSG, ISR, and SSSR with minimal boilerplate.

• Remove caching, chunking, compression, and asset optimization from app logic.

• Provide a centralized GUI for testing and optimizing front-end performance.

• Remove authorization and authentication from app logic by centralizing it.

• Reliable global <15ms web, mobile, and native front-end performance.

Back-End Development

• Faster TTFT and TPS (tokens per second). OpenAI example:

  • Public API: 600-800ms TTFT / 600ms-1.5sec total latency. Not great UX.

  • Azure API: 200-300ms TTFT / 300-700ms total latency. Much better UX.

  • Azure also offers enterprise security, compliance, and data residency.

  • Services in AWS and GCP will be able to take full advantage of this.

  • Unlike public APIs, API downtime in Azure is virtually unheard of.

• Significantly faster embedding queries.

  • AlloyDB takes 10-20ms to query against millions of vectors.

  • Supabase pgvector takes 30-80ms or more at similar scale.

  • Inter-VPC vs public transit shaves off another 85-200ms+.

  • This makes your RAG queries 95-270ms faster.

  • Your models query their embeddings locally.

• The above takes us from frustrating response times to a great user experience. Every company I've done this for had an immediate increase in user satisfaction.

• Reduce your vendor footprint and allow application developers to call any service or model using unified internal authorization, instead of managing vendor API keys, rotations, etc., on a per-application basis - the definition of insanity.

• Minimize toil by centralizing key management and model / API vendor updates.

• Improve security by never exposing ALBs, NLBs, or services for external access.

  • Explicit and centralized outbound access control and logging.

• Simplify end-user and organization billing and usage tracking by unifying it.

• Centralize vendor failover and adherence to your usage / throttling limits.

There are many other advantages, like enabling access to serverless models on all major clouds (Gemini Flash, for example, has uses in virtually every LLM workflow), but this article's goal is to cover everyday benefits for users and application developers.

A Robust Full-Stack Interface

Enabling your backend and frontend applications to call the same service endpoints will make your full-stack engineers happy and they will say nice things about you.

This also lets you to intercept and centrally authorize any vendor or endpoint. Making a call to Together AI, OpenRouter, Weights & Biases, ScrapingBee, etc.? No problem!

Your service’s SDK called what it thinks is https://scrapingbee/foobar, and your internal DNS routed it 30-100ms+ faster via the proxies and a global backbone while injecting the proper API keys without your back-end service ever knowing about it.

These proxies support DB connections and other cool stuff - they cover HTTP/S/2/3 (QUIC), TCP and UDP, TCP and UDP Streams, GRE, IPIP, IPSec, gRPC and WebSockets.

3. Performance

Fast, predictable infrastructure performance directly improves user satisfaction and reduces infrastructure costs. Here are some real-world benchmarks that show this:

• 800Mbps+ per-node throughput on streams and large RQs.

• 1200-1600 RPS with 3-4ms internal traffic between clouds.

• 350-400 RPS with 13-14ms global front-end performance.

Inter-VPC performance (TLDR, 3-4ms):

This is made possible by physical colocation of the proxy VMs, although it uses public network transit (the internet). I ran these tests for 48 hours and for 11 minutes of that, congestion increased latency to 9-14 ms with zero packet loss. Compared to the VPC peering alternative of $2700-3500 per month, this also gives us higher burst capacity.

Front-end performance (TLDR: 13-14ms):

These were non-cached requests to an NGINX server on the nodes (using more CPU than proxy forwarding), during which none of them exceeded 25% CPU or 630 MB RAM. Speaking of our nodes, this level of throughput costs all of $21 per month:

This also gets us universally accelerated outbound API requests which are normally either unavailable out of the box altogether or cost insane amounts in each cloud.

We will add redundancy and fault tolerance while also making it 60-90% cheaper with auto-healing scaling groups composed of spot instances. This step is also necessary to verify that our design is robust enough for production usage. People talk about resilience and durability, but until you run entirely on spot instances, you have neither.

4. Cost

This setup pays for itself by reducing cloud costs and operational complexity. By the time we’re done, you’ll be running all of it for under $100/mo across all 3 clouds.

Engineering Overhead Savings

• Application development teams move faster because they can self-service infra.

  • No per-service endpoint environment variables and update redeployments.

  • Unified request tracing drastically simplifies debugging of production issues.

  • Unified request logging removes redundant middleware from our services.

• No more per-cloud and per-account SSL certificate rotation (aka Platform KTLO).

  • This also saves us from TLS/SSL termination overhead in all our services.

• Platform is no longer a dependency or blocker and can focus on new features.

Compute Overhead Savings

• Per-account/CDN/ALB AWS Global Accelerators ($18/mo each)

• Per-account/CDN Azure Front Doors ($35/mo standard, $330/mo premium)

  • Cloudflare performance is equivalent to premium Azure Front Doors.

• Per-VPC AWS private DNS ($0.50/mo per zone + $0.40 per 1M requests)

• Per-VPC GCP private DNS ($0.20/mo per zone + $0.40 per 1M queries)

  • GCP NAT Gateway costs (this can get insanely expensive very quickly)

• Per-VPC Azure private DNS ( $0.50/mo per zone + $0.40 per 1M queries)

  • Azure NAT Gateway costs ($32/mo each)

What’s Next

In the next article, Pt3/X, we'll set up a proxy in each cloud that responds to a health-check endpoint and connect it to our public domain via Cloudflared. This entire series will be done in a secure zero trust way that never makes our VMs externally accessible by using neat techniques like UDP hole-punching, reverse tunnels, and relay networks.

After that, the course schedule will be:

• Pt4: Add direct inter-VPC mesh communication to proxies.

• Pt5: Set up automagic DNS routing to proxies in each VPC.

• Pt6: Optimize egress routing through Cloudflare's backbone.

• Pt7: Introduce fault tolerance with spot instance groups.

• Pt8: Expand our mesh a micro-cloud like Lambda Labs.

After that, we’ll do an "ML: A Software Perspective" series, where we write middleware and provide examples using popular frameworks like PyTorch and TensorFlow. We'll also include tools like Autogen, Semantic Kernel, mem0, Composio, LangGraph, AG2, CrewAI, Wetrocloud, Vercel SDK, LlamaIndex, and other easy-bakes for the "AI" crowd.

Let’s make sure the below depicts our competitors’ customers and APIs - not ours.

1. That’s too much. Most companies are LangChain and API bolt-ons.

2. How do we actually set something like this up? Need DevOps help.

This will be a series aimed at addressing both of these things. We’ll set up a global cloud-agnostic backbone with persistent connections between each cloud provider.

This article sets the groundwork for the series with a short action item at the end.

Who This Is For

• Startups working on “AI” - API calls, bolt-ons, frameworks like LangChain.

• Startups working on ML - training, inference, fine-tuning, self-hosted.

• You don’t have a Platform team or their skill ceiling is not the best.

Benefits

• Quickly and safely capitalize on all 3 clouds’ advantages in every cloud:

  • Use GCP’s AlloyDB, Vertex, GAI Studio, Doc AI, BigQuery, etc in AWS/AZU.

  • Use Azure’s OAI, CosmosDB, Dbricks, AML, Cog Serv+Srch, etc in AWS/GCP.

  • Use AWS’s Bedrock, SageMaker, Forecast, Comprehend, etc in GCP/AZU.

• Vastly reduce latency and jitter for most of your external vendor API calls.

• Set up inference endpoints with Lambda Labs’ DGX SuperPods or Scalars.

• Security. Easier IAM/SG/etc management, no more exposing ingress, etc.

• Your future DevOps team will say good things to you AND about you.

• Unified cross-cloud analytics, image optimization, global ingress backbone.

Plus better performance than you’d have with (and save you the monthly charges of) AWS Global Accelerator ($18/mo each), Azure Front Door ($35/mo each), and GCP Premium Tier networking - plus the rather steep bandwidth costs for all of those. A lot more bells and whistles come with this setup but we’ll dig into those in later articles.

Drawbacks

It’s one more place where things can go wrong. This setup is reliable and I’ve used it for many companies, but things CAN go wrong. We will cover DR and redundancy to mitigate that. It’ll cost extra but will still be over 10x cheaper than peering at baseline.

What Are We Doing?

In this series, we’ll implement the diagram below. This involves swapping DNS to Cloudflare and adopting a bit of Pulumi in your infra. I’ve built out MLOps/DevOps infra for dozens of companies, and not a single one has regretted doing either.

I’m not going to bore you with too many of the networking details but here’s the gist:

We’ll go with envoy for management APIs and a few other specific capabilities. Our design goals (ordered): Cost, Security, Maintainability, Performance, and Robustness.

This will be built in a specific way that allows us to easily add Disaster Recovery, Fault Tolerance, Durability, and an Istio control plane for practical management later on.

Alternatives

VPC Peering

The downside to this is cost. Below is one of the most basic peering solutions you can have and the total of the below prices along with Edge Routers ($200-300/mo/cloud) and Monitoring Tools ($100-200/mo) would be at least ~$2,736-3,436 per month.

AWS ↔ GCP

AWS Direct Connect Port: $262/mo 1Gbps, GCP Partner Interconnect: $151/mo 1Gbps, Cross-Connect: $200-300/mo, Subtotal: $613-713/mo

AWS ↔ Azure

AWS Direct Connect Port: $262/mo 1Gbps, Azure ExpressRoute Circuit: $305/mo 1Gbps, Cross-Connect: $200-300/mo , Subtotal: $767-867/mo

GCP ↔ Azure

GCP Partner Interconnect: $151/mo 1Gbps, Azure ExpressRoute Circuit: $305/mo 1Gbps, Cross-Connect: $200-300/mo, Subtotal: $656-756/mo

WireGuard: Advantages

WireGuard is a VPN protocol designed for simplicity and high-performance encrypted networking. Cloudflare Warp (what we’ll be using) is based on the WireGuard but saves us from dealing with things like WG key and configuration management!

• Extreme Performance: Minimal latency and negligible CPU overhead.

• Simplicity and Transparency: Easy-to-understand config, fewer security risks.

• Cost Efficiency: Free and open-source software (FOSS); only server costs.

• Fully Self-Managed: Complete control over security, routing, keys, and privacy.

WireGuard: Disadvantages

• Manual Key & Configuration Management: You have to manually handle cryptographic keys, configurations, updates, deployments, and other stuff.

• Infrastructure Management Required: Self-hosted infra is more complex.

• No Built-in Firewall or Security Policies: it's just a VPN layer.

TailScale: Advantages

Tailscale is built on WireGuard and similar to Cloudflare Warp, but with an automatic mesh that simplifies key distribution and network setup. In our case, TailScale could be 1-3ms faster but it is a direct P2P solution that exposes external IPs, suffers from public network transit jitter, and doesn’t backbone us to external API vendors.

• Zero-Configuration Setup: TailScale is to WireGuard what Istio is to Envoy. It handles WireGuard keys, peer discovery, configuration, and other handy stuff.

• Automatic NAT Traversal and Mesh Routing: Direct peer-to-peer connections even behind NAT are great for warm, low latency, and distributed deployments.

• Strong Security Defaults: Strong auth via Single Sign-On (SSO) integration.

TailScale: Disadvantages

• Dependence on Tailscale Infrastructure: Relies on Tailscale’s control plane.

• Limited Customization: Cannot fully customize encryption, auth, or keys.

• Compliance & Privacy Considerations: Central control plane.

Nebula: Advantages

Nebula is a scalable mesh networking platform developed by Slack. It gives us a secure overlay network and handles dynamic node discovery (kind of like Istio).

• Scalable & Secure by Design: Uses certificate-based authentication (PKI), scales well from 10 to 1000+ nodes. This is extreme overkill in scale and complexity.

• Rich Security Features: Built-in SGs and firewall-like policies give good control.

• Robust NAT Traversal: Traverses firewalls and NAT for hybrid and multi-cloud.

• Open-Source: Highly customizable, transparent, and fully self-hostable.

• Proven in Prod: Heavily battle-tested by Slack in large prod environments.

Nebula: Disadvantages

• Complexity: Initial setup requires PKI certificates and custom configuration files.

• Self-Managed Infra: Similar to WireGuard, we deal with more self-managed infra.

• Operational Overhead of Key Management (PKI): Things can go really wrong.

ZeroTier: Advantages

ZeroTier is software-defined networking (SDN) for cross-cloud virtual Ethernet.

• Simplicity & Rapid Deployment: Very user-friendly with a web GUI.

• Automatic NAT and Firewall Traversal: This saves a lot of headache.

• SDN Capabilities: Routing, bridging, and network segmentation out of the box.

• Hosting Options: ZeroTier’s control plane can be cloud-hosted or self-hosted.

ZeroTier: Disadvantages

• Centralized Control Plane (by Default): they’re not as big as CloudFlare.

• Performance Tradeoffs (Layer-2 Ethernet Emulation): Slightly higher latency due to Layer-2 tunneling overhead versus Layer-3 VPNs (WireGuard/Tailscale).

• Enterprise Scale Costs: can scale quickly and brutally at enterprise levels.

• Complexity at Large Scales: Virtual Ethernet networks are complex in large multi-cloud environments, especially with network loops, multicast, and broadcast.

What Are We Doing In This Post?

In this post we’ll figure out where to put our proxies in each VPC for them to be as physically close to each other as possible. Networking 101 for the Big 3 Clouds:

Region = a geographically wide group of interconnected Availability Zones.
Availability Zone (AZ) = a physical group of data centers in a Region.
Virtual Private Cloud (VPC) = logically isolated network spanning all AZs in a Region.
Subnet = a network segment inside a VPC with its own CIDR IP address range.

The relationship between Subnets and AZs is not the same in AWS, GCP, and Azure.

GCP

A Subnet spans all AZs in a Region by default. Specifying an AZ during creation of a VM guarantees it will be in a specific physical location. Example: us-east4-a, -b, or -c.

Azure

A Subnet also spans all AZs in a Region by default. Specifying an AZ during creation of a VM guarantees it will be in a specific physical location. Example: “eastus” and “1”.

AWS

A Subnet is tied to a specific AZ at creation but the mapping of AZs (“us-east-1a”, “us-east-1b”) to physical locations is randomized per-AWS-account at creation.

In a per-tenant AWS account setup (which you should have, security 101), you need to reverse-map AZs for each account. This is done by spinning up public IP EC2 instances in each account and latency testing between them. Example: if the latency from a VM in “us-east-1a” for AWS Account “Foo” to “us-east-1c” for AWS Account “Bar” is <2ms consistently, you can safely conclude that they are in the same physical location.

Let’s Do The Thing!

We’ll use PBK to find the lowest latency VM combination between the following AZs:

• GCP us-east4-a, us-east4-b, and us-east4-c.

• Azure eastus 1, eastus 2, and eastus 3.

• AWS us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, and us-east-1f

I prepared some code to do this and wrote the guide for it here. In our case, they are:

• AWS us-east-1a

• GCP us-east4-a

• Azure eastus-1

In the next article we’ll set up a proxy in each cloud provider that responds to a simple health check endpoint. We’ll then use WRK and Autocannon to do a few performance tests with and without features like Smart Routing. These will be externally mapped:

• api.[TLD]/api/v5 → Proxy in AWS VPC

• api.[TLD]/api/v6 → Proxy in GCP VPC

• api.[TLD]/api/v7 → Proxy in Azure VPC

VPC-to-VPC communication will be handled later on in a secure and performant way.

Cost Optimization Background

Assumptions

• If you use AMD or TPUs, your technical proficiency exceeds this post.

• Scaling efficiency of ~90% across all generations (close enough).

• You’re going for the cheapest regions, pricing as of 01/28/25.

• You’re not in a specialized HPC environment like UltraCluster.

• Zero consideration for vCPU, bandwidth and SSD allocations.

• …but your pre-processing can keep GPUs fed at near 100%.

• You don’t have the DevOps to leverage spot instances.

• You don’t have the budget for 1-3 year reservations.

• Transformer training when possible, because hype.

• VRAM limits don’t matter as much for training.

NVIDIA A100, 2020

This is when NVIDIA introduced TF32 (Tensor Float). TF32 is supported by TensorFlow and PyTorch, maintains the 8-bit exponent of FP32, but reduces the mantissa to 10 bits. This life hack gives it performance so far beyond FP32 that unless you specifically need the accuracy of FP32 or 64 for advanced quantization techniques (in which case reading this article is a waste of your time), you care mostly about TF32 performance.

Unrelated ML perf: this paper achieves a 95% energy use reduction for multiplication on tensor hardware by almost losslessly approximating it with addition using L-Mul.

TF32 Training Performance with 8xA100 80GB on DGX: ~2,200TF1 (Sparse)

• AWS p4d.24xlarge: 96 vCPU, 1.1TB RAM

  • $32.77/Hr On-Demand ($10.36/Hr Spot)

• GCP a2-ultragpu-8g: 96 vCPU, 1.4TB RAM

  • $40.22/Hr On-Demand

• Azure NDamsrA100v4: 96 vCPU, 1.9TB RAM

  • $32.77/Hr On-Demand ($5.55/Hr Spot)

NVIDIA H100, 2022

TF32 Training Performance with 8x H100 on SXM (DGX): ~7,120TF2

• AWS p5.48xlarge: 192 vCPU, 2TB RAM

  • $98.32/Hr On-Demand ($54.07/Hr Spot)

• GCP a3-highgpu-8g: 208 vCPU, 1.9TB RAM

  • $87.83/Hr On-Demand

• Azure NDisrH100v5: 96 vCPU, 1.9TB RAM

  • $98.32/Hr On-Demand

Practical Cost Optimization

On the Big 3 cloud providers, we pay:

• $14.89/Hr for 1000 TF32 TFLOPS with Ampere

• $12.34/Hr for 1000 TF32 TFLOPS with Hopper

Let’s focus on the lowest micro-cloud cost per 1000 TF32 TFLOPS:

• OVHCloud only offers pre-TensorFloat Voltas, they’re out.

• DigitalOcean is $3.35/Hr for 1000 TF32 TFLOPS …maybe.

  • They’re VERY unclear on whether this is SXM or PCIe.

• Atlantic.Net doesn’t list prices. That about sums that up.

• Lambda Labs are f*cking champs. Clear pricing and platform:

  • $4.69/Hr for 1000 TF32 TFLOPS on SXM 40GB A100s

  • $6.50/Hr for 1000 TF32 TFLOPS on SXM 80GB A100s

  • $3.36/Hr for 1000 TF32 TFLOPS on SXM 80GB H100s

• CoreWeave, not a winner …BUT clear and transparent pricing:

  • $6.91/Hr for 1000 TF32 TFLOPS on SXM 80GB H100s

  • They offer a lot of value-add software (hint hint)

• RunPod’s “community cloud” and no 8x are sus. They’re out.

Lambda Labs vs Big 3 with on-demand 8x H100s SXM? 72.77% Price Reduction.

Now, Vast.ai is a beast of its own. Let’s say this one’s outside of the scope of this post because they deserve their own post and aren’t really a micro-cloud provider. Places like Vast.ai / OpenRouter / etc are excellent for micro-cloud providers monetizing their spare capacity. They’re also a different universe where things like this are real:

…and that’s inside a secure cloud hosted in a trusted datacenter. These are by far the best prices you’ll ever see and they offer value-add software that makes life easy. The whole world isn’t using them because they’re effectively limited to single VM setups.

Technological Barriers to Entry

Ok Yevgen, we’d love to save $7.3K/mo out of our $10K R&D budget, or $22K/mo out of our $30K/mo R&D budget (enough to hire a good MLOps / DevOps engineer with a $210K salary, 1.25x employer overhead accounted for)! How do we get in on this?

Getting a K8s Cluster Up and Running

First off, why? Because that unlocks a large ecosystem of tools: everything you need.

Rejoice! As of October 2024, Lambda Labs has finessed this into a 1-click operation. A massive upgrade from what you have to do on most micro-clouds today, which still resembles Lambda Labs’ messy and time-consuming guide for how to do this in 2022.

Making a Cluster Useful for R&D

Once you have a cluster, a whole world opens up …for you to implement. A brief look:

• SkyPilot is excellent (inference-focused) due to its automatic cost optimization.

• If you want to not hate your life and do things like go directly to distributed training jobs, Kubeflow has an operator for that. It supports PyTorch, XGBoost, Tensorflow, and almost everything else you may want. Kubeflow is my personal favorite, so this may be a biased suggestion since I also know it the best. It has:

  • Easy to use VS Code Server and JupyterLab in-cluster with KF Notebooks!

  • It lets you create E2E ML pipelines natively in Python (researcher-friendly)

  • It has an amazing ez-bake (in beta, I love it) for Hyperparameter Tuning.

  • It has a simple model and artifact registry that works for most use cases.

  • Want to aggregate gradients computed on multiple nodes? You got it.

    • This uses K8s’ MPI operator and works with TensorFlow and PyTorch.

  • Got your shiny new model ready for inference? You can easily do that too.

Making a Cluster Good for R&D

For performant S3-compatible cluster storage, you’ll want MinIO. We love MinIO! After this, you’ll want cluster observability. Here’s a decent diagram I cooked up a while back that is a basic, opinionated, but OK cookie-cutter setup (click for full size)

Now - I wrote that diagram for a different purpose so you’ll want to ignore everything outside of the purple “K8 Cluster” boundary. If this is an R&D cluster skip Envoy, Istio, Kafka, probably ArgoCD too and tweak it to use OpenTelemetry-compatible thingies.

This setup will work for most use-cases because it lets you easily unify observability and operations between AWS/GCP/Azure clusters, or multiple micro-cloud clusters. Having this before something sus happens is good for your health and happiness.

Now that you’ve got basic DevOps observability, you may want MLOps observability. You can always use Comet ML or Weights & Biases like a normal person, or (there are benefits to this) drop MLFlow into your cluster with Bitnami’s ez-bake Helm Chart.

Also, something like Apache Airflow can be a godsend for scheduling / management.

Keep in mind that at this point your tooling will have significant overlap. This will get you everything you need for a complete R&D shop, but it’s most likely unnecessary.

Doing unnecessary things means unnecessary maintenance and overhead. Do be aware of the tech debt you are committing yourself to when considering these.

The Micro-Cloud Dream

You know what myself and a lot of other people who care about GPU pricing would pay a lot of money for that, to the best of my knowledge, does not exist right now?

Imagine this. You’re on Azure, or GCP, or AWS. You have a DevOps or MLOps guy or two, and your R&D budget is $10K-50K/mo - nowhere near enough to justify having a full expert platform team for self-service and optimization. That’s a lot of people.

First off, the micro-cloud contract. Cloud providers always want to sell steady reserved capacity on 9-36 month contracts …but that sucks for most people. What if you could sign on for a set amount of resources but use them when you actually need them?

What if your contract had a fully reserved rate only for the machines you need to be always on and run your K8S control / management planes, and the rest auto-scaled?

What if the onboarding’s timeline, expectations, and engineering requirements were all clearly expressed in the contract? We could just have a few checkboxes, such as:

• We work on [Reinforcement Learning / Computer Vision / Transformers / etc]

• Our Engineers use [VS Code / Jupyter / etc] and [TensorFlow / PyTorch / etc]

• We will need mostly [egress / ingress], [X] Gbps sustained, [Y] Gbps burst.

• Our training data is [X] Gb and we guestimate training [Y] epochs on it.

• We [want / don’t want] them to have awesome, efficient in-cluster IDEs.

• We [want / don’t want] to peer with our main [AWS / GCP / Azure] VPC.

• We [want / don’t want] cluster logs and metrics visible in our main cloud.

• We [want / don’t want] to manage our clusters ourselves. Do it for us!

• Our preferred contract duration is [12/24/36 mo] - include incentives!

• We want Platform support [Nah / As-Needed / Business Hours / 24.7]

Boom! Here are 3-5 options and quotes for what hardware we recommend.

What if micro-cloud providers provided $2-5K worth of professional services free to onboard your R&D team without DevOps/MLOps support and amortized that cost in the contract itself? That’d de-risk potential customers and shift their CAC (Customer Acquisition Cost) from sales to professional services with a notable overall reduction.

You know what almost every R&D shop wants to hear? It’s this:

All done with your checklist? We’re spinning up a cluster for you and you will get SSH keys for non-control plane nodes if you want and VPN credentials to connect to it. We will give your ML engineers docs on how to use IDEs / everything + observability tools.

Whoever gets this right will dominate because the ML Engineers will sell their CTO on it (dope tooling) for you, who in turn won’t be risking his job on this.

“This” being: fill out survey, pick feature and hardware options, sign contract, ENG gets credentials and documentation on how to use their shiny new tools.

R&D orgs are not looking for cheaper servers or GPUs, they’re looking to optimize overall costs - and there is a massive opportunity there for high value add services. You don’t even have to do it yourself. Just find trusted partners who can do it for you, for which you foot the bill (amortize it in your contract to de-risk your customers) and lower the barrier to entry. A $100K+ contract is risky enough. Signing it without a partner heavily invested in making sure it all works out is not an option for most.

Aight, Let’s Wrap it Up

Every startup wants to save costs on GPUs. Less than 1% of them (I can list the ones able to efficiently span micro-clouds on one hand because I know most of them) are capable of it. This is a wheel that has to get re-invented every time a startup wants to save costs on GPUs. A batteries-included end to end setup starting with the cluster, to observability, to docker images, to workload distribution, and IDEs is the R&D dream.

Why leave it to them? Cloud providers have a vested interest. This isn’t even a sales objection, it’s THE sales objection. Please see exhibit below - the ones on top STILL can’t span micro-clouds …and you’re mostly talking to the ones on the bottom.

Meme credit goes to Eduaro Ordax, the man responsible for my 6-pack.

UPDATED 01/13/2025 @ 10:40am PST. Major news outlets & prominent social media accounts are invertedly reporting incorrect information. I’m vouching for the accuracy of the information in this post. It comes directly, first-person from on-site ERTs, CAL FIRE, and FEMA Liaison Notification Officers actively managing the wildfires. We’re doing our best to make this comprehensive, but DO NOT ASSUME THAT IT IS.

Sections: Latest Information || Forecasting || Where To Get Help || How You Can Help

Critical Information

IF YOU ARE IN A LIFE-THREATENING SITUATION, CALL 911.

Stay away from quarantined areas. Pre-70s buildings have combusted asbestos, lead, PCBs, formaldehyde, and other toxins into the air and the ash layering everything. If you are in proximity, get out. N95 masks do not work. Full-face P100/N100/R100 respirators are ineffective against gases and vapors like formaldehyde and mercury.

CITY WATER FROM LADWP IS NOT SAFE in the city of Los Angeles and possibly other areas. Click here for the “Do Not Drink” advisory (was formerly “Boil Water”).

Boiling, freezing, filtering, adding chlorine or other disinfectants, letting water stand, or applying any other form of treatment will NOT make the water safe. Avoid water usage if possible, the city’s systems are under extreme strain to sustain water pressure.

4Ps of Preparation are People, Pets, Photos, and Passport. Have these ready and be prepared to leave immediately if you are within 25 miles of an evacuation order.

If emergency response teams direct you to do something, please comply immediately even if it infringes on your personal rights. Noncompliance causes cascading damage.

You can check an addresses’ status here. This isn’t a substitute for evacuation orders.

Our best community resources at this time are:

• IF YOU NEED HELP: 400+ Locations. (See Where To Get Help for more)

• IF YOU CAN HELP: 40+ Locations. (See How You Can Help for more)

Latest Information

• 01/13 9:45am PST LA County FD Eaton Incident Video Update

• 01/13 9:34am PST Palisades Fire CAL FIRE Update: 14% Contained

• 01/13 8:30am PST LAFD RED FLAG Parking Notice and Map.

• 01/13 8:00am PST LADWP Power and Water Status Update

• 01/13 7:18am PST Eaton Fire CAL FIRE Update: 33% Contained

• 01/13/25 7:00am PST LA County FD Update: Page 1 Link and Page 2 Link

• LA County FD Eaton Fire Update 01/12 7:00pm PST

• LAFD Evacuation Order Downgraded Announcement and Map.

• Eaton Incident Community LAFD Meeting 01/12 @ 04:45pm PST

• LAFD 01/12 ~4:00pm PST RED FLAG Parking Notice and Map.

Celebrating the wins: Kenneth Fire has been contained. Hurst is 89-95% contained.

• Most maps reflect current road closures; check here for upcoming ones.

• PALISADES FIRE status, evacuation warnings and orders.

• EATON FIRE status, evacuation warnings and orders.

• CAL FIRE ArcGIS map of active incidents near LA.

• Genasys Project evacuation zone mapping.

• Metrolink train status service advisories.

• LA Metro evacuation and resource centers near bus and rail systems.

• An AQMD Wildfire Smoke Advisory is in effect for Los Angeles.

• A local health emergency is in effect for Los Angeles County.

Other Credible Sources:

• Palisades Fire LAFD Public Information Officer (Erik Scott)

• Watch Duty Website.

  • iOS Watch Duty App.

  • Android Watch Duty App.

• Up-to-date information on electricity outages.

• Latest information on water system outages.

• Sign up for electricity outage alerts.

• Report an electricity outage.

• X: LA City FD || LA County FD || LADWP

• Complete Palisades Fire Damage Assessments

• Complete Eaton Fire Damage Assessments

Additional Updates:

• A 01/11/24 12:00pm PST Palisades Fire Operations Update.

  • West-bound activity (fire progression) on PCH (coastline) is limited.

  • Favorable winds yesterday evening (01/11) on multiple fronts.

  • Reminder: GROUND ALL CIVILIAN DRONES.

  • Winds are picking up today (01/12).

• LA County FD Eaton Fire Update 01/12 7:00am PST

• LADWP - 01/11 09:00PM Response Update.

• Eaton Fire 7:00pm 01/11 Incident Update

• D29, Malibu Unsafe Water Alert.

• LAFD - Incident 0738 Evacuation Order Map.

• LAFD - Red Flag Parking Restrictions Lifted 01/11.

• LADWP - 01/11 10:00AM Response Update.

• LADWP - 01/11 03:00PM Response Update.

• Pasadena Drinking Water System Impact.

• D21, Kagel Canyon Unsafe Water Alert.

Latest Forecasting

Wind speeds and directionality are the most critical factor. Find them here:

• PALISADES FIRE primary Santa Ana Wind Forecast.

• PALISADES FIRE Malibu-side Wind Forecast.

• EATON FIRE notable Wind Forecast.

Suggested guidelines for consideration:

• Rain and winds of 10-20mph max are ideal.

• Anything dry and over 40mph is not good.

• Anything over 60mph grounds air support.

Where To Get Help

Work In Progress. If you have additional resources: comment, msg on LI, or email me.

Please check active times and availability for all resources in advance.

• Fill this out immediately and/or call the FEMA Helpline at 1-800-621-3362.

• Community disaster assistance resource map with 400+ locations.

• If you need shelter: text SHELTER and your zip code to 43362.

  • AirBnB emergency wildfire intake form.

  • Non-comprehensive list of Evacuation Shelters.

• Find food at hundreds of participating LA Food Bank locations.

• Red Cross Emergency App. 1-800-RED CROSS (800-733-2767)

  • Red Cross Assistance: Finding Friends and Family.

• U-Haul Free Storage for the displaced across 92 centers near LA.

• City of Los Angeles EMD Wildfire Recovery Resources

  • 01/11 9:00am to 5:00pm Family Assistance Center (LA City):

    • 2551 Motor Avenue, Los Angeles, CA 90064 - (310)837-5186

  • 01/11 9:00am to 5:00pm Family Assistance Center (Pasadena):

    • 300 E Green Street, Pasadena, CA 91101 (left of Civic Auditorium)

• Official LA County Palisades Fire Recovery Information.

• LA County Assessor’s Office Disaster Relief for Property Owners.

• Los Angeles Apparel (812 E 59th St.)

Salvation Army Locations (check times, availability, and updates on their page):

• El Camino Real Charter HS, 5440 Valley Circle Blvd, Woodland Hills, CA 91367

• Westwood Recreation Center, 1350 S Sepulveda Blvd, Los Angeles, CA 90025

• Ritchie Vales Recreation Center, 10736 Laurel Canyon Blvd, Pacoima, CA 91331

Salvation Army Press Releases: 01/08 Here. 01/09 Here. 01/10 Here.

Click here for the source spreadsheet of the community disaster assistance map.

How You Can Help

Work In Progress. If you have additional resources: comment, msg on LI, or email me.

A Spreadsheet of Grassroots Help Efforts Where You Can Help!

Civilian volunteers and assistance are currently not accepted due to the inherent dangers associated with the situation. They likely will be in the post-fire efforts.

• LAFD Wildfire Emergency Fund. URGENT HELP IS NEEDED.

• California Community Foundation Direct Donation Page.

• California Community Foundation (Pledge.to) Wildfire Fund.

• World Central Kitchen fund; meals for first responders and evacuees.

• Salvation Army Southern California Wildfire Relief Fund.

• United Way Wildfire Response Donation Page.

• LA Dream Center Donation Page And Locations.

• Official Go Fund Me Wildfire Relief Fund.

• AltaMed Foundation Fire Relief Efforts.

• List of verified Go Fund Mes impacted by the CA Wildfires.

• California Fire Foundation: when the unthinkable happens.

• LARFSC: Text LAFIRE to (844)422-6444 or Donate Online Here.

Other ways you can help! We have not confirmed these.

• Stoney Clover Donation Drive

• Seconds Market LA Donation Program

• Seconds Market LA Fire Donation Match Program

• Riviera Hall Lutheran School Donations

• 818 Night Market

Additional Information

ALL CIVILIAN DRONES MUST BE GROUNDED. Air support is the most critical asset for effective containment, which requires: one plane pass to map the route, a second plane pass to drop fire retardant, and a third plane pass to drop water. Drones force a re-routing of all of these, and drone collisions (which have occurred) ground assets.

The Eaton fire is endangering critical power, water, and telecomm infrastructure. Do not panic if you can’t reach someone. Telecommunications are currently unreliable.

If you see major news outlets and large social media accounts reporting wind speeds or other data incorrectly: comment, msg me on LI, or email me. I will forward it to the correct internal departments. As far as we know there is no official contact at this time.

Misinformation about wind conditions or any of the above can cause loss of life. My company’s systems have all been re-tasked to search for and flag it. Please help us.

Afterword

The current state of things is devastating. Resources have been pulled from as far as Miami, Canada, and Mexico. Please be patient with emergency response teams and our community. We can’t control wind. Everything that can be done, is being done.

Personal note: we lost two homes in the 2018 Woolsey fires and our home in Malibu is currently at risk. We can’t do anything about any of that. Our focus is on what can be done, and giving our devastated communities every last drop of help we possibly can.

Successful founders usually exhibit many traits on the left here:

Personality:

• Takes Extreme Ownership (or Makes Excuses).

• Communicates Proactively (or Reactively).

• Stays Punctual (or Acts Inconsistently).

• Follows Through (or Breaks Trust).

Leadership:

• Commits to SMART Goals (or Avoids Them).

• Perseveres in Vision (or Reacts Myopically).

• Executes Strategically (or Haphazardly).

• Thrives (or Crumbles) Under Pressure.

Startup-specific:

• Focuses on Customer Problems (or Technology).

• Defines Clear Core Competencies (or Sprawls).

• Tests Hypothesis Early (or Iterates Slowly).

• Acts on Data and Feedback (or Wings It).

People hype “Founder Mode” and “The Hustle” and whatnot, which is as cute as it is absolute bullsh*t. Yes, as an individual, nothing beats talent AND hard work (for me, 3 hours daily are spent planning and 12 executing). Alas, very few extraordinary things have been achieved by individuals. Success as a founder boils down to, in order:

• Working on Something You Care Deeply About.

• Working with the Right People.

• Working on the Right Thing, at the Right Time.

• Working Smart.

• Working Hard.

All of those are indispensable. Your success, however, is tied much more to doing the right things at the right time than to how many hours you work or how much you post on LI about “The Struggle (that you signed up for).” Most startups fail because of what they DON’T do: a self-inflicted byproduct of failing to seek advice and feedback.

As a belated Christmas gift to my beloved subscribers, I’ve once again faced my arch-nemesis, Microsoft products, and braved the horrors of Clipchamp to create an apt pictorial representation of how critical working with the right people is, just for you!

(Please don’t sue me, Sam and YC. I love you both equally - the S12 batch was 🔥)