gRPC Across Languages: Production Lessons

Running gRPC across languages delivers on its core promise: define a service once in Protobuf, generate clients and servers in every language, and they talk. Where it surprises teams is everything around the call, the load balancing, deadlines, status codes, and schema evolution, because those behaviors are set by each language’s implementation, not by the proto.

The proto gives you a shared vocabulary. It does not give you shared behavior. This post is the set of things you have to configure deliberately to make gRPC dependable across a polyglot fleet, building on the failure modes in Why Language Boundaries Break Polyglot Microservices. It is part of the Language choices in polyglot microservices series.

Why gRPC behaves differently across languages

gRPC’s generated stubs are so good that they hide how much behavior is left to defaults. A Go client, a Java client, and a Python client generated from the same proto will call the service identically and behave differently under load, failure, and idle.

Teams discover this the hard way: a service works perfectly in integration tests, then under real traffic it load-balances to one pod, holds dead connections after a deploy, or burns work on requests that already timed out. None of that is in the proto, so none of it shows up when you only review the schema.

Why does gRPC load-balance poorly on Kubernetes?

Because gRPC multiplexes many requests over one long-lived HTTP/2 connection, and a standard Kubernetes Service balances at the connection level, it pins each client to a single backend pod. You scale to ten replicas and one pod takes nearly all the traffic. Fix it with request-level balancing.

This is the single most common production surprise. Connection-level balancing cannot help, because there is only one connection carrying everything.

How do gRPC deadlines work across services?

You set a deadline on a call, and the remaining budget should propagate to each downstream hop so every service knows how much time is left. A call that has spent 150 ms of a 200 ms budget should pass 50 ms downstream, not a fresh 200. Not every language propagates this identically by default, so verify it under test.

A gRPC call without a deadline can wait forever. In a chain of services, one missing deadline means a slow dependency can pin resources all the way up the call stack. Propagating the remaining budget stops the system from doing work for requests the original caller has already abandoned.

A request that times out at the edge while three downstream services keep working is the canonical wasted-work failure, and it gets worse exactly when you can least afford it, under load.

How should I handle errors in gRPC across languages?

Map your native errors to gRPC status codes with structured error details, and branch on the status code on the client. The status code is the only error signal that means the same thing in every language. Never parse the human-readable status message for control flow.

gRPC defines a fixed set of status codes (OK, INVALID_ARGUMENT, DEADLINE_EXCEEDED, UNAVAILABLE, and so on). Those codes are the one part of your error handling that survives the wire intact. Everything else, your Go error, your Java exception, your Python exception, your Rust Result, is local.

Get the codes right and retries become safe. UNAVAILABLE is generally retryable; INVALID_ARGUMENT never is. If a server returns the wrong code, clients across every language will retry things they shouldn’t or give up on things they should retry. The code is the contract; honor it precisely.

gRPC status code	Retryable?	Typical meaning
UNAVAILABLE	Yes (with backoff)	Transient: server down, connection dropped
DEADLINE_EXCEEDED	Sometimes	Only if the operation is idempotent
RESOURCE_EXHAUSTED	Yes (with backoff)	Rate-limited or quota hit; back off
INVALID_ARGUMENT	No	Bad request; retrying repeats the error
NOT_FOUND	No	The thing isn’t there; a retry won’t change that
ALREADY_EXISTS	No	Duplicate; treat as success or a real conflict
PERMISSION_DENIED	No	Auth problem; fix the caller, don’t retry
INTERNAL	No (usually)	Server bug; retrying rarely helps

How do I evolve a Protobuf schema without breaking clients?

Never reuse or change a field number, never change a field’s type, and reserve the numbers and names of fields you remove. Add new fields with new numbers, and reserve the zero value as an UNKNOWN enum case handled on every client. Follow those rules and old and new clients interoperate safely. For the full ruleset, including the deploy sequencing that keeps a rollout from becoming an outage, see Protobuf Schema Evolution Without Breaking Clients.

Protobuf is designed for backward and forward compatibility, but only if you follow its rules. Adding new fields with new numbers is safe: old clients ignore what they don’t know, and new clients see defaults for what old servers don’t send.

The cross-language wrinkle is enums and unknown values. When a new server sends an enum value an old client has never seen, different languages handle that “unknown” case differently. Reserve the zero value as an explicit UNKNOWN, handle it on every client, and you avoid the silent misroute where one language defaults an unknown enum to the wrong branch.

Lesson: keepalive and connection age need one shared policy

Because gRPC connections are long-lived, how you keep them healthy matters, and the defaults differ by implementation. Keepalive ping intervals, max connection age, and idle timeouts are all configurable, and if each language inherits its own defaults you get intermittent, timing-dependent failures after idle periods or deploys.

Set keepalive and connection-age parameters explicitly, derived from one shared config, identical across every service. A bounded max connection age is also how you get clients to periodically re-resolve and rebalance, which works hand in hand with the load-balancing fix above: connections that live forever never rebalance to pods added after they were established.

Should you retry failed gRPC calls?

Retry only the status codes that are safe to retry, and only with backoff and a budget. UNAVAILABLE and DEADLINE_EXCEEDED on an idempotent operation are reasonable to retry; INVALID_ARGUMENT, NOT_FOUND, and ALREADY_EXISTS never are. Retrying the wrong code turns a small blip into a self-inflicted traffic storm.

The cross-language catch is that retry behavior is configurable and the defaults differ. Some stacks ship a service-config-driven retry policy; others leave it entirely to you. If each language picks its own policy, one client hammers a struggling backend while another gives up immediately on the same failure. Define one retry policy (which codes, how many attempts, what backoff, what overall budget) and apply it identically everywhere.

Two rules keep retries from amplifying an incident. First, use exponential backoff with jitter, so a thousand clients do not retry in lockstep and synchronize into a thundering herd. Second, cap retries with a budget (a ceiling on the fraction of requests that may be retries), so a widespread failure cannot multiply your real traffic by your retry count at the worst possible moment. Retries are a safety mechanism only when they are bounded; unbounded, they are an outage accelerator.

Should you retry gRPC calls, and where?

Retries are the feature most likely to convert a small problem into an outage, and gRPC gives you several places to put them — which means teams frequently end up with retries at more than one layer without realising it.

The multiplication is the danger. A client configured for 3 attempts, calling through a mesh configured for 3 attempts, reaching a service that itself retries its downstream 3 times, produces up to 27 requests from one logical call. Under the exact conditions that trigger retries — a struggling dependency — you have tripled or worse the load on the thing that is already failing. This is a retry storm, and it is self-reinforcing.

Three rules keep it safe:

• Retry at exactly one layer. Pick either the client’s built-in retry policy or the mesh’s, and disable the other. Both is never correct, and “both, but configured carefully” is a claim that survives until someone changes one of them.
• Only retry what is safe and worth retrying. UNAVAILABLE and RESOURCE_EXHAUSTED are usually retryable; INVALID_ARGUMENT and NOT_FOUND never are, since the second attempt will fail identically. DEADLINE_EXCEEDED is the ambiguous one: the work may have completed, so retrying it requires the operation to be idempotent.
• Retries spend the deadline, not a fresh one. A retry that resets the budget defeats the entire timeout-budget model. If the remaining budget cannot accommodate another attempt, do not attempt it.

Add backoff with jitter rather than immediate retries, for the same reason it matters in reconnect storms: synchronised clients retry in synchronised waves and hit the recovering service simultaneously.

The stronger protection above all of this is a circuit breaker. Retries assume the failure is transient; a circuit breaker recognises when it is not and stops sending traffic entirely, letting the dependency recover instead of being held down by the retries of everyone waiting on it. Retries handle the blip. The breaker handles the outage. A system with retries and no breaker has automated exactly the behaviour that turns a degraded dependency into a dead one.

How do you test cross-language gRPC compatibility?

Every problem in this post shares one property: it passes single-language integration tests and fails in production. That is not bad luck — it is what happens when the tests exercise one runtime’s defaults and production exercises four.

The fix is a conformance suite: one set of behavioural tests that every service must pass regardless of language, run in CI. It is deliberately not a test of your business logic. It tests the boundary behaviours the proto does not define.

What it must assert:

• Deadline propagation. Call through a multi-hop chain with a short deadline and assert each hop observed a decreasing remaining budget, and that work stopped when it reached zero.
• Deadline enforcement. Send an already-expired deadline and assert the server refuses immediately rather than doing the work.
• Status code mapping. Force each error class and assert the caller receives the agreed code and structured detail — never a language-native type, never a message string that logic depends on.
• Unknown enum handling. Send an enum value the receiver’s schema version does not know, and assert it lands on the reserved UNKNOWN zero value rather than a meaningful case. This is the silent-misroute bug, and it is only findable with deliberately mismatched schema versions.
• Field presence. Send an absent field, an explicit zero, and a null, and assert all three remain distinguishable end to end.
• Large integers. Round-trip a value above 2^53 and assert exact equality, which catches the JavaScript precision loss before a client does.
• Keepalive and idle behaviour. Idle past the keepalive interval, then send. Assert the call succeeds rather than failing on a half-dead connection.

Two practices make the suite worth its cost. Run it against every language pair you actually deploy, not just against one reference implementation — the failures live in the disagreements between two runtimes, so testing each against a canonical server can miss them. And make passing it the definition of “this language is supported.” A new language entering the fleet is not production-ready until it is green, which converts a vague architectural principle into a merge gate.

There is a second, cheaper test worth adding to CI: schema compatibility checking. Tooling such as buf breaking compares a proto change against the previous version and fails the build on a breaking change — a reused field number, a changed type, a removed field that was not reserved. That catches the entire class of schema-evolution mistakes described above mechanically, before review, and it is a few lines of CI configuration rather than a discipline anyone has to remember.

How do you actually fix gRPC load balancing on Kubernetes?

Diagnosing the problem is the easy half. There are three real fixes, and they trade operational complexity against how much control you keep.

Approach	How it works	Cost
Client-side LB over a headless Service	The client resolves all pod IPs via DNS and balances requests across its own connections	Every client must implement it; DNS re-resolution on scale events needs care
A proxy that speaks HTTP/2 (Envoy, Linkerd, or a gRPC-aware ingress)	The proxy terminates and balances per request	An extra hop and another component to run
Service mesh	Sidecars handle discovery, balancing, retries, and mTLS uniformly	The largest operational surface; justified when you want the other features too

For a polyglot fleet, the comparison tilts differently than it does for a single-language shop. Client-side balancing means implementing and maintaining the same behaviour in every language you run, with the per-runtime differences described above — four implementations that must agree. A proxy or mesh implements it once, outside the application, which is worth real complexity precisely because it removes the cross-language variance rather than multiplying it.

Two details that catch people regardless of approach. MAX_CONNECTION_AGE is the cheapest partial fix available: forcing connections to be recycled periodically makes clients re-resolve and rebalance, which prevents the pathological case where a long-lived connection pins a client to one pod indefinitely. Set it, with jitter, even if you also run a mesh.

And scale-up does not rebalance existing connections. New pods receive traffic only from clients that establish new connections. Without connection-age recycling, you can add ten pods during an incident and watch them stay idle while the original pods stay saturated — the autoscaler reports success and nothing improves.

What changes when you use gRPC streaming?

Almost every cross-language guide covers unary calls and stops. Streaming has its own failure modes, and they are worse because the abstraction differs most between runtimes exactly where it matters.

Flow control is the point, and it is easy to defeat. HTTP/2 gives each stream a flow-control window, so a slow reader naturally exerts backpressure on the writer. That protection only works if your code reads at the pace it can process. A handler that eagerly drains the stream into an in-memory list has converted a flow-controlled stream into an unbounded queue and reintroduced exactly the failure the window existed to prevent — the general pattern in Backpressure Design for Real-Time Systems.

Long-lived streams and deadlines interact awkwardly. A deadline on a stream applies to the whole stream, not to individual messages. For an indefinite stream, that means either no deadline — losing the protection deadlines give you — or a deadline you must handle by reconnecting. Decide deliberately which you want; the default in most stacks is the former, silently.

Error semantics differ from unary calls. A stream can fail after some messages were successfully delivered and processed, so the receiver must be able to resume rather than assume all-or-nothing. In practice that means messages carry sequence numbers or IDs and consumers are idempotent, which is the same discipline described in Idempotency Keys for Distributed Systems.

Load balancing gets worse, not better. A streaming RPC pins to one connection and one backend for its entire life. Long-lived streams therefore produce the most extreme version of the imbalance above, and no amount of connection-age recycling helps a stream that is meant to stay open for hours. If you need both long streams and even distribution, the balancing has to happen when streams are established — which means deliberately capping stream lifetime and reconnecting.

A cross-language gRPC checklist

Before a gRPC service goes to production in a polyglot fleet:

• Load balancing is request-level (client-side LB or an HTTP/2-aware proxy or mesh), not a plain connection-level Service.
• Every call has a deadline, and remaining-budget propagation is verified per language.
• Errors map to gRPC status codes with structured details; no client parses status messages for logic.
• Status codes are correct for retry semantics (UNAVAILABLE retryable, INVALID_ARGUMENT not).
• Schema changes follow Protobuf rules: stable field numbers, reserved on removals, UNKNOWN zero-value enums handled everywhere.
• Keepalive, idle, and max-connection-age come from one shared config, identical across services.
• A cross-language integration test exercises a real client/server pair, not just one language against itself.

What I’d do differently

The lesson that took longest to internalize is that the proto file feels like the whole contract and is only half of it. The schema specifies what you send. Production behavior is decided by how each language’s runtime handles deadlines, balancing, errors, and connections, and those live in config, not in the .proto.

If I were setting up gRPC across languages again, I would establish the behavioral defaults (load balancing, deadline propagation, status-code mapping, keepalive) as a shared, reviewed config before the second language joined the mesh, and I would write a conformance test that every new service must pass. The generated stubs are excellent. They just don’t make these decisions for you, and the defaults are not the decisions you want.

Sources

• gRPC documentation, Load balancing: grpc.io/docs/guides/custom-load-balancing
• gRPC documentation, Deadlines: grpc.io/docs/guides/deadlines
• gRPC documentation, Status codes: grpc.io/docs/guides/status-codes
• Protocol Buffers, Updating a message type: protobuf.dev/programming-guides/proto3/#updating