Tags: 0 - Overview 0 - HomeLab 2 - Build Roadmap
AI Inference Infrastructure Roadmap
Objective
The next phase of my homelab is focused on building a deeper understanding of production AI inference infrastructure.
The goal is not simply to deploy LLMs to Kubernetes.
The goal is to understand and measure the systems involved in serving models:
- latency and throughput
- prefill and decode
- KV-cache behavior
- request scheduling
- continuous batching
- accelerator utilization
- quantization
- inference engine architecture
- reliability and SLOs
- GPU scheduling
- autoscaling and capacity planning
- multi-GPU and multi-node inference
The existing Kubernetes homelab provides the foundation.
AWS will be used selectively when experiments require hardware or scale that cannot reasonably be reproduced locally.
Architecture Direction
GitHub
|
GitHub Actions
|
GitOps / Terraform
|
+-------------+-------------+
| |
v v
HOMELAB AWS
Talos Kubernetes EKS Kubernetes
Cilium Karpenter
RTX 4070 Ti Super Ephemeral GPU Nodes
| |
+-------------+-------------+
|
Inference Layer
|
+-----------+-----------+
| | |
v v v
llama.cpp vLLM TensorRT-LLM*
|
Observability
|
Prometheus / DCGM / Grafana
|
Go Benchmarking
|
Load + Experiments
|
Published Engineering
Findings
* TensorRT-LLM introduced when appropriate hardware is available.Phase 1 - Establish the Baseline
Goal
Understand the behavior of the inference system already running in the homelab before introducing additional serving engines.
Current baseline:
- Talos Kubernetes
- Cilium
- ArgoCD / GitOps
- NVIDIA GPU worker
- RTX 4070 Ti Super 16 GB
- llama.cpp
- LiteLLM
- OpenAI-compatible inference API
Observability
Deploy:
- Prometheus
- Grafana
- NVIDIA DCGM Exporter
- inference-server metrics
- gateway/application metrics
Measure:
Accelerator
- GPU utilization
- VRAM utilization
- power consumption
- temperature
- clocks
Inference
- Time to First Token (TTFT)
- Time per Output Token / Inter-Token Latency
- end-to-end latency
- prompt tokens/sec
- generation tokens/sec
- requests/sec
- active requests
- queued requests
- P50 / P95 / P99
- error rate
Exit Criteria
I can trace an inference request from the API to the accelerator and explain what the major metrics are telling me.
Phase 2 - Build inferbench
Goal
Build a Go-based benchmarking and load-generation tool for OpenAI-compatible inference endpoints.
Instead of relying exclusively on existing benchmarking tools, building the client provides an opportunity to learn both Go and inference behavior.
Example:
inferbench \
--endpoint https://inference.example \
--model qwen \
--concurrency 16 \
--requests 500 \
--input-tokens 4096 \
--output-tokens 256Example output:
Requests: 500
Concurrency: 16
TTFT
p50: 240ms
p95: 510ms
p99: 790ms
TPOT
p50: 19ms
p95: 27ms
Throughput: 54 tok/s
Request rate: 2.8 req/s
Errors: 0%Go Engineering
The project should exercise:
- HTTP clients
- streaming responses
- Server-Sent Events
- goroutines
- channels
- synchronization
- context cancellation
- statistics
- structured logging
- CLI design
- unit tests
- integration tests
- JSON/CSV output
Exit Criteria
I can generate repeatable workloads and produce comparable measurements across different inference configurations.
Phase 3 - Characterize Single-GPU Saturation
Goal
Understand where the current accelerator stops scaling efficiently.
Run controlled experiments while changing one variable at a time.
Concurrency
Test approximately:
1
2
4
8
16
32
64Measure:
- TTFT
- TPOT
- total latency
- throughput
- GPU utilization
- VRAM
- queue depth
- P95/P99 latency
Prompt Length
Test:
128
512
2K
4K
8K
16K+Observe:
- prefill time
- memory consumption
- KV-cache pressure
- TTFT
- throughput
Output Length
Compare short and long generations to better understand decode behavior.
Quantization
Where supported, compare different quantization strategies and measure:
- memory consumption
- throughput
- latency
- model-quality tradeoffs
Exit Criteria
I can identify and explain the saturation point of my local inference system.
Specifically:
At what workload does additional concurrency stop producing meaningful throughput improvements and primarily increase latency?
Phase 4 - vLLM
Goal
Move beyond llama.cpp and gain direct experience with an inference engine commonly used for production LLM serving.
vLLM is a required part of this roadmap rather than an optional optimization.
Study:
- PagedAttention
- continuous batching
- KV-cache management
- request scheduling
- prefill
- decode
- chunked prefill
- prefix caching
- quantization support
- speculative decoding
- OpenAI-compatible serving
Run the same inferbench workloads used against llama.cpp.
Compare:
llama.cpp
vs
vLLMInvestigate differences in:
- TTFT
- TPOT
- throughput
- batching behavior
- VRAM consumption
- accelerator utilization
- behavior under concurrency
Deliverable
Publish a measured comparison:
llama.cpp vs vLLM on a 16 GB consumer GPU
The objective is not to declare a universal winner.
The objective is to explain why the systems behave differently.
Exit Criteria
I can explain how vLLM schedules requests and manages memory, and demonstrate those behaviors through measurements.
Phase 5 - Reliability Engineering
Goal
Operate inference as a service rather than a demo.
Establish SLIs around:
- availability
- TTFT
- TPOT
- error rate
- throughput
Establish initial SLOs from measured baseline performance.
Failure Experiments
Test:
- inference pod termination
- gateway termination
- GPU node restart
- model cold start
- VRAM exhaustion
- request overload
- excessive context
- failed readiness
- networking failure
- bad configuration rollout
Measure:
- detection time
- recovery time
- requests lost
- SLO impact
- model loading time
- time until serving readiness
Deliverable
Publish selected incident reports documenting:
failure
|
v
symptoms
|
v
telemetry
|
v
diagnosis
|
v
recovery
|
v
preventive changePhase 6 - AWS GPU Infrastructure
Goal
Extend the inference platform into AWS to learn elastic GPU infrastructure without permanently purchasing additional hardware.
AWS is not a separate homelab project.
It is an extension of the same inference platform.
Infrastructure
Build with Terraform:
- VPC
- EKS
- IAM / IRSA or Pod Identity
- ECR
- S3 model/artifact storage where appropriate
- Karpenter
- GPU NodePools
- NVIDIA device support
- inference workloads
- Prometheus
- DCGM Exporter
Target workflow:
terraform apply
|
v
create GPU capacity
|
v
deploy inference engine
|
v
run experiments
|
v
export results
|
v
terraform destroyGPU infrastructure should remain ephemeral whenever possible.
Experiments
Investigate:
- GPU node provisioning time
- model download/load time
- cold-start behavior
- scaling from zero
- node replacement
- Spot interruption
- scheduling
- GPU utilization
- cost per experiment
- cost per token
- capacity per accelerator
Exit Criteria
I can provision, benchmark, observe, and destroy an accelerator-backed inference environment in AWS reproducibly.
Phase 7 - Multi-GPU Inference
Goal
Learn the problems that do not exist on a single accelerator.
Use short-lived cloud GPU resources where necessary.
Study and experiment with:
- tensor parallelism
- pipeline parallelism
- data parallelism
- model sharding
- NCCL
- GPU topology
- interconnect bandwidth
- distributed scheduling
- distributed failure modes
Questions to answer:
- When does tensor parallelism become necessary?
- What communication overhead does it introduce?
- How does interconnect bandwidth affect inference?
- How does serving change when a model no longer fits on one accelerator?
- What happens when one worker disappears?
- How should replicas be distributed across nodes?
Exit Criteria
I can explain the major architectural and performance differences between single-GPU and distributed inference and have measured at least some of those behaviors directly.
Phase 8 - TensorRT-LLM
Goal
Develop familiarity with NVIDIA’s optimized inference stack after establishing a strong vLLM baseline.
Study:
- TensorRT-LLM architecture
- engine building
- optimized kernels
- FP8
- INT8
- KV-cache optimizations
- inflight batching
- multi-GPU execution
- NVIDIA serving ecosystem
Where suitable hardware is available, benchmark equivalent workloads against the existing inference baseline.
This phase intentionally comes after vLLM.
The objective is to understand the optimization stack rather than merely deploy it.
Phase 9 - Capacity Engineering
Goal
Determine how inference capacity should actually be controlled.
Investigate potential scaling signals:
- GPU utilization
- VRAM
- KV-cache utilization
- queue depth
- active requests
- waiting requests
- tokens waiting
- TTFT
Determine which signals correlate with actual saturation.
Capacity Controller
Extend the planned Go Wake-on-LAN controller into an inference-aware capacity controller.
Potential lifecycle:
request arrives
|
v
no available inference capacity
|
v
capacity controller
|
v
wake/provision accelerator node
|
v
Kubernetes Node Ready
|
v
GPU available
|
v
inference workload scheduled
|
v
model loaded
|
v
readiness succeeds
|
v
traffic admittedMeasure:
Time from zero accelerator capacity to first successful inference request.
Compare this behavior between:
- local GPU capacity
- AWS GPU capacity
Longer-term experiments:
- scale to zero
- queue-based scaling
- inference-aware admission control
- workload-aware scheduling
- capacity forecasting
Phase 10 - Open Source
Open-source work should begin before all previous phases are complete.
Primary projects of interest:
- vLLM
- SGLang
- llama.cpp
- NVIDIA inference tooling
- NVIDIA DCGM Exporter
- Kubernetes GPU ecosystem
Contribution progression:
operate
|
v
measure
|
v
understand
|
v
reproduce problems
|
v
read source
|
v
documentation
|
v
tests
|
v
bug fixes
|
v
larger contributionsThe objective is not to search for arbitrary “good first issues.”
The objective is to become a sufficiently experienced user that useful contributions emerge naturally from operating the software.
Public Engineering Write-Ups
Experiments should produce public artifacts rather than remaining private homelab work.
Potential articles:
- Measuring TTFT and TPOT on my home GPU
- Finding the saturation point of a 4070 Ti Super
- How context length affects KV-cache pressure
- llama.cpp vs vLLM under concurrent inference
- What happens when an inference GPU disappears?
- Scaling an EKS GPU workload from zero
- Measuring GPU cold-start time in AWS
- Tensor parallelism: what changes when one GPU is not enough?
- Building an inference load generator in Go
- Which metric should autoscale an LLM inference service?
Every major experiment should ideally produce:
code
+
benchmark data
+
graphs
+
technical explanation
+
lessons learnedEngineering Questions
Progress is measured by the questions I can answer rather than the number of technologies deployed.
I want to be able to explain:
- What determines Time to First Token?
- Why are prefill and decode different computational workloads?
- How does KV-cache memory scale with context and concurrency?
- How does continuous batching increase accelerator utilization?
- What problem does PagedAttention solve?
- What causes an inference server to saturate?
- Why does P99 latency deteriorate under load?
- Why can an accelerator have low utilization despite queued requests?
- How does chunked prefill affect scheduling?
- When is speculative decoding useful?
- What are the tradeoffs of quantization?
- What signals should drive inference autoscaling?
- How should inference capacity be estimated?
- How should model rollouts interact with SLOs?
- When is tensor parallelism required?
- What does NCCL communication add to distributed inference?
- How does GPU topology affect performance?
- How do vLLM and TensorRT-LLM approach serving differently?
- How should accelerator-backed infrastructure recover from failure?
- How do performance, reliability, and cost interact when operating an inference service?
Deprioritized Work
The following technologies may still be useful but are not currently on the critical path:
- Backstage
- additional generic Kubernetes components
- infrastructure added only for completeness
- extensive Terraform abstraction without a concrete use case
- additional Kubernetes certifications
- generic MLOps tutorial projects
New infrastructure should solve a demonstrated problem.
Working Model
LEARN
|
v
BUILD
|
v
MEASURE
|
v
BREAK
|
v
EXPLAIN
|
v
IMPROVE
|
v
CONTRIBUTE
|
+----------> repeatDeploying a technology is not the outcome.
The outcome is understanding how the system behaves, why it behaves that way, and being able to demonstrate that understanding with measurements.
Immediate Focus
Only the first few phases are active priorities.
The current sequence is:
Observability
|
v
inferbench
|
v
Single-GPU saturation
|
v
vLLM
|
v
Reliability
|
v
AWS GPU infrastructure
|
v
Multi-GPU
|
v
TensorRT-LLM
|
v
Capacity engineering
|
v
Open-source depthEverything after vLLM is direction rather than an immediate checklist.
The immediate objective is simple:
Establish a measured llama.cpp baseline, build repeatable benchmarking tooling, and then use those measurements to understand how and why vLLM behaves differently.