FastAPI Microservices Architecture: A Production Guide

A production FastAPI microservices architecture earns its place for one reason: it is the fastest way to put a typed, async, well-documented HTTP boundary in front of Python code you already need to run, usually machine-learning inference, data work, or glue.

Use it where Python is the right runtime, keep it off your latency-critical hot paths, and run it under a production ASGI server with the worker math done deliberately. Get the async model and the process count right, and a FastAPI service holds its SLO. Get them wrong, and no framework saves you.

Why FastAPI architecture decisions matter

The most common FastAPI failure in production is not a FastAPI failure. It is a deployment that runs one synchronous worker, blocks the event loop on a CPU-bound call, and then someone concludes “Python is slow.”

FastAPI is fast for what it is. The performance you actually get depends almost entirely on whether you respect the async model and size the process pool to the workload. This post is the architecture I run FastAPI with inside a polyglot system, and the specific places it belongs and does not. It is part of the Language choices in polyglot microservices series.

Where FastAPI belongs in a polyglot fleet

The right question is never “FastAPI vs Go.” It is “what is this service’s job, and is Python the right runtime for it?”

FastAPI belongs in front of work that is already Python: model inference, feature engineering, anything leaning on NumPy, Pandas, PyTorch, or the scientific stack. Rewriting that in Go to save a few milliseconds of framework overhead trades a small latency win for a large velocity loss.

It does not belong on a high-throughput, latency-critical edge path. That is Go’s job, for the reasons in Go vs Rust for Microservices: When to Choose Which. A FastAPI service that exists only to forward JSON is a service that should have been written in the language the rest of your edge already uses.

The cleanest pattern is FastAPI as the typed boundary over a Python compute core, called by Go orchestration services over gRPC or HTTP. The Python service does the Python-shaped work and nothing else.

FastAPI vs Go: which runtime for which service?

Use this as a quick triage when a new service shows up. The rule is “match the runtime to the work,” not “use what the team knows.”

Service type	Use	Why
ML inference / model serving	FastAPI	The work is already Python; rewriting loses velocity
Feature engineering, data transforms	FastAPI	NumPy/Pandas ecosystem, fast iteration
High-throughput API edge / gateway	Go	Latency-critical, concurrency-heavy, GC is fine
Control plane / orchestration	Go	Cloud-native ecosystem, fast compile loop
Latency-critical hot path	Rust	Deterministic tail latency (see the hot-path guide)
Glue / JSON forwarding	Go	If it isn’t Python compute, it isn’t a FastAPI job

Should I use async def or def for FastAPI routes?

Use async def only when everything the handler awaits is non-blocking. If the handler calls synchronous libraries, use a plain def so FastAPI runs it in a threadpool. The one combination to avoid is an async def handler that calls a blocking driver, which stalls the event loop.

FastAPI is built on ASGI and Starlette, and its concurrency story is single-threaded cooperative async per worker process. One blocking call freezes that worker for every concurrent request it is handling.

The rule is mechanical. If a path handler is async def, everything it awaits must be non-blocking: an async database driver, an async HTTP client, an async cache client. The moment you call a synchronous library inside an async def, you have stalled the event loop for every other request on that worker.

If you must call blocking code, two correct options exist:

• Define the handler as a plain def. FastAPI then runs it in a threadpool, so it does not block the loop.
• For CPU-bound work, push it to a process pool or a separate worker service. Threads do not buy you parallelism against Python’s Global Interpreter Lock.

# WRONG: blocks the event loop for every concurrent request on this worker
@app.get("/user/{uid}")
async def get_user(uid: int):
    return sync_db.query(uid)        # synchronous call inside async handler

# RIGHT: async driver, never blocks the loop
@app.get("/user/{uid}")
async def get_user(uid: int):
    return await async_db.query(uid)

# ALSO RIGHT: blocking work in a plain def -> FastAPI runs it in a threadpool
@app.get("/report/{rid}")
def build_report(rid: int):
    return sync_db.heavy_query(rid)

How many Uvicorn workers should I run?

Start at roughly one worker per CPU core for I/O-bound services, then tune against real latency. For CPU-bound work the GIL means extra per-worker concurrency does not help, so match workers to cores and move heavy compute elsewhere. Always multiply your worker count by per-worker memory before you size the pod.

A FastAPI deployment runs N worker processes, each with its own event loop. Throughput and resource use both scale with N, and picking N is arithmetic, not a default.

For I/O-bound services, a single worker handles many concurrent requests because it yields the loop on every await. You scale workers to use available cores and to provide failure isolation, not to add concurrency per request.

For CPU-bound work, concurrency per worker is a lie. The GIL serializes Python bytecode, so a worker handling a CPU-bound request blocks. Here you either move the work to a process pool, match workers to cores one-to-one, or move the compute out of Python entirely.

Memory is the constraint people forget. Each worker is a full Python process with its own copy of the loaded model or large in-memory data. Four workers each loading a 2 GB model is 8 GB per pod, not 2 GB.

How do you run FastAPI on Kubernetes?

The deployment shape differs from a Go service in ways that catch teams migrating between them, and most of the differences come from Python’s memory and startup characteristics.

Memory per worker is the sizing constraint. Each Uvicorn worker is a separate process with its own interpreter and its own copy of every imported module. A service importing PyTorch or Pandas can use hundreds of megabytes per worker before serving a single request. Four workers is therefore not “a bit more memory,” it is potentially four times the baseline. Set the pod memory limit against measured per-worker usage times worker count, with headroom — and be aware that an OOM kill during a request looks like a mysterious connection reset to the caller.

Prefer fewer workers per pod and more pods. The instinct from traditional deployments is to run many workers in one large pod. On Kubernetes the opposite is usually better: one or two workers per pod, scaled horizontally. You get finer-grained scheduling, cleaner autoscaling signals, and a smaller blast radius when one pod dies. It also avoids the trap where a single pod’s memory limit is sized for peak across all its workers simultaneously.

Startup is slow and the probes must reflect it. Importing a heavy scientific stack, and loading a model, can take tens of seconds. That is a startup probe with a generous total allowance, not a large initialDelaySeconds, and the readiness probe must not report ready until the model is actually loaded — otherwise Kubernetes routes traffic to a pod that will fail every request until it finishes booting. The distinction and its failure modes are in Readiness Probes That Don’t Lie.

Handle SIGTERM properly. Uvicorn will finish in-flight requests on graceful shutdown, but only if the grace period allows it and the application does not exit early. Combined with the preStop delay that lets endpoint removal propagate, this is what makes rolling deploys invisible to callers rather than producing a burst of errors every release.

How should FastAPI handle background and long-running work?

Python services attract long-running work — a model that takes eight seconds, a report to generate, a batch to process — and FastAPI offers several ways to handle it that differ enormously in durability. Picking the wrong one is a recurring source of silent data loss.

Mechanism	Runs where	Survives restart	Use for
await in the handler	The event loop	N/A — caller waits	Work fast enough to fit the request budget
BackgroundTasks	Same process, after response	No	Short, best-effort, loss-tolerant work
run_in_executor / thread pool	Worker threads	No	Blocking libraries you must call from async code
Process pool	Separate processes	No	CPU-bound work that the GIL would otherwise serialise
External queue (Celery, ARQ, Dramatiq)	Separate workers	Yes	Anything that must actually complete

The dividing line is the “survives restart” column, and it is the only one that matters for correctness. If losing the work would be a bug, it belongs on an external queue. A pod restarts during a deploy — which is routine, not exceptional — and every in-process background task disappears with no error, no retry, and no record that it was pending.

The CPU-bound row deserves emphasis because it is where Python differs most from other runtimes. The GIL means threads do not give you parallelism for CPU work; adding threads to a CPU-bound handler makes it slower, not faster. Genuine parallelism requires separate processes, which is also why worker count rather than thread count is the tuning knob that matters.

For anything longer than a few seconds, the better architecture is usually not “run it in the background” but accept the job and return a handle: the endpoint validates, enqueues, and immediately returns a job ID with a 202, and the client polls or receives a webhook. That keeps request latency bounded, makes the work durable, and makes progress observable — three properties the in-process options cannot offer at any price.

What actually breaks FastAPI services in production?

The failures are consistent across teams, and none of them are FastAPI bugs. They are consequences of async Python that only appear under real concurrency.

A blocking call inside an async def route. This is the dominant production failure. One synchronous database driver, one requests.get, one time.sleep, or one CPU-heavy loop inside an async route blocks the entire event loop — not just that request. Every other request served by that worker stalls behind it. The symptom is baffling: latency spikes across unrelated endpoints, with CPU sitting at 20%. The worker is not busy, it is blocked.

This is why the async def versus def decision matters so much more in FastAPI than the syntax suggests. A plain def route runs in a threadpool and blocking is contained. An async def route that blocks takes the whole loop with it. If you are not certain every call in the path is non-blocking, use def — the framework will do the safe thing for you.

Connection pool sizing that ignores worker count. Pool limits are per process. Four Uvicorn workers with a pool of 20 each is 80 connections to the database, not 20, and databases have connection ceilings that are easy to exhaust. Multiply before you deploy, and remember that a horizontal scale-up multiplies again.

Pydantic validation on large payloads. Validation is the feature and it is not free. Deeply nested models over large request or response bodies consume real CPU, and response serialisation is often more expensive than request parsing because you are validating data you already trust. For hot endpoints returning large collections, consider skipping response-model validation and serialising directly.

Unbounded background tasks. BackgroundTasks runs work after the response is sent, in the same process, with no queue and no limit. It is genuinely useful for short, best-effort work and it is not a job system: the work is lost on restart and nothing bounds concurrency. Anything that must complete belongs on a real queue.

No timeouts on outbound calls. An async client with no timeout holds a slot indefinitely when a dependency hangs. Set them explicitly, derived from the enclosing budget as described in Timeout Budgets Across Service Chains.

How should you structure a FastAPI service?

FastAPI is unopinionated about layout, which is pleasant at 200 lines and a liability at 20,000. A structure that holds up has three properties.

Routes contain no business logic. A route function should validate input, call a service-layer function, and shape the response. When logic lives in routes it cannot be tested without HTTP, cannot be reused by a background worker or CLI, and quietly grows until the route file is the application.

Dependencies are injected, not imported. FastAPI’s Depends is the framework’s best feature and the most underused. Database sessions, the current user, and configuration should arrive as dependencies, which makes them trivially overridable in tests via app.dependency_overrides — no monkeypatching, no import-order games.

Configuration is a typed settings object, loaded once. A Pydantic settings model read from the environment at startup gives you validated config and a single place to see what the service needs. Reading os.environ scattered through modules produces services that start successfully and fail on the first request that touches a missing variable.

A layout that reflects this:

Module	Holds	Must not hold
api/	Routers, request/response schemas	Business logic, direct DB access
services/	Business logic, orchestration	HTTP concepts, framework imports
db/	Models, repositories, session factory	Business rules
core/	Settings, logging, security primitives	Anything domain-specific
deps.py	Shared Depends providers	Logic beyond wiring

The test that tells you the structure holds: can you invoke your core business operation from a script, with no HTTP server running? If not, the logic is entangled with the framework, and every future change — a worker, a scheduled job, a migration to another transport — will be harder than it should be.

The FastAPI production checklist

These settings separate a FastAPI service that holds its SLO from one that pages you.

• ASGI server: Uvicorn workers managed by a process supervisor, or another production ASGI server you have measured a reason to prefer. Never run the development server in production.
• No blocking in async: every await hits an async driver; CPU-bound work goes to a pool or another service.
• Timeouts everywhere: client timeouts on every downstream call, plus a server-side request timeout. A worker waiting forever on a hung downstream takes its whole concurrency slot down with it.
• Health checks that mean something: a readiness probe that fails when the model is not loaded or a critical dependency is down, not one that returns 200 simply because the process started.
• Structured logging and distributed tracing: OpenTelemetry instrumentation so a slow request is debuggable across the boundary, which matters more in a polyglot system.
• Pydantic v2 models at the edge: validate input at the boundary and let the typed model carry through. The validation cost is real but cheap relative to the bugs it stops.

Is FastAPI fast enough for production microservices?

Yes, for the right jobs. When a service is I/O-bound or fronts Python compute, and you run it under a production ASGI server with no blocking in async handlers, FastAPI holds its SLO comfortably. It is the wrong choice for latency-critical, high-throughput edge paths, which belong in Go.

Plan capacity from the bottleneck, and for most FastAPI services the bottleneck is either downstream I/O or model compute, not FastAPI itself. Start by measuring time spent in the service: framework and serialization overhead versus time awaiting downstream calls versus time in actual compute. The split tells you what to scale. If 90% of the time is model inference, more workers will not help a single slow request; a faster model, batching, or a GPU will.

Then size for the tail. An inference service with variable input sizes has a long latency tail, and your p99 is set by your largest realistic input, not your average. Provision and set timeouts against the tail, not the mean.

What I’d do differently

The mistake I see most is using FastAPI as a general-purpose service framework because the team knows Python, then fighting its concurrency model on paths that were never Python-shaped to begin with.

If I were drawing the lines again, I would be stricter: FastAPI only where Python compute is the actual product of the service. Everything else, the routing, the auth edge, the high-throughput glue, goes to Go. The polyglot win comes from each language doing the job it is good at, not from one language doing every job because it is familiar.

The honest tradeoff is velocity against runtime cost. Python ships data and ML work faster than anything else and costs more per request to run. When the work is genuinely Python, that trade is worth it. When it is not, you are paying the runtime cost for none of the velocity benefit. Once a Python service talks to a Go one, the seam itself becomes the risk: see Why Language Boundaries Break Polyglot Microservices.

Sources

• FastAPI documentation, Concurrency and async/await: fastapi.tiangolo.com/async
• Starlette documentation: starlette.io
• Uvicorn documentation, Deployment: uvicorn.org/deployment