A cgo call into a C scoring engine takes 50-400 ms; how do you decide whether it stays on the RPC request path?

## Why this is a decision and not a tuning exercise A 50-400 ms C call on the request path is not a slow function. It is a commitment of an operating-system thread per concurrent request, a failure mode that is not contained to one request, and a latency promise you cannot actually keep. Each of those is somebody's budget, which is what makes this an ownership question rather than a code change. ## Capacity: the thread budget is the real number A goroutine inside C holds a thread for the duration and cannot be preempted, so in-flight calls translate almost one-for-one into threads. At 300 ms and a few hundred requests per second you are provisioning hundreds of threads, each with a stack and scheduling cost, and the runtime does not give them back afterwards — the count is a high-water mark for the process's life. So the first decision is a **number you publish**: the maximum number of concurrent calls into the engine, derived from cores, memory, or the engine's own internal parallelism and any licence limit — never from incoming request concurrency, which is not yours to control. Enforce it on entry with a semaphore. The important consequence is that overload becomes queueing in Go instead of thread growth, and queueing is measurable and sheddable: a request whose deadline would expire waiting at the gate never enters C at all. That also changes what you page on. Raw thread count is a lagging, noisy signal; **wait time at the gate** tells you the engine is undersized while there is still time to act. ## Honesty: your deadline is not a deadline Cancellation in Go is cooperative. When the request context expires, the handler can stop waiting, but the C function runs to completion holding its thread. Two things follow. You must not advertise a p99 the engine can violate at will, and you must make sure a client retry does not put a second call in flight while the first is still running — otherwise a latency blip becomes a thread avalanche. If the engine exposes its own timeout or cancel handle, use it and your deadline becomes real; if it does not, that absence is a legitimate reason to push the work off the request path. ## Blast radius: what one bad pointer costs An in-process C dependency shares your address space. A memory fault inside it is not a recoverable panic; the process dies and takes every in-flight request with it. A panic that reaches the boundary from an exported callback does the same. That risk is acceptable for a library your team maintains and fuzzes, and much less acceptable for a vendored numeric engine nobody has touched in three years. The alternative is running the engine as a separate process and talking to it over a socket. You pay serialisation of the feature vector and a copy, and you buy: a blast radius of one worker, the ability to restart it, an operating-system-level memory and CPU limit on it, and a real timeout because you can abandon or kill a process. On a call already costing hundreds of milliseconds, a serialisation cost measured in microseconds is not the deciding factor — the deciding factor is whether you trust the code. ## Off the path entirely Before any of that, ask whether the call must be synchronous. If the score can be precomputed for common inputs, cached with a short lifetime, or computed asynchronously with the response returning a pending state, the cheapest engineering answer is not to make the call per request. If it must be synchronous, batching several requests into one crossing amortises both the boundary cost and the thread. ## What you actually commit to A decision here is worth writing down, because it will be revisited by someone who was not in the room: - the concurrency ceiling and the reasoning behind the number; - the alert on gate wait time, and the thread-count reading that confirms the model; - the latency the service promises, stated as what it can enforce rather than what the engine usually does; - who owns engine upgrades, and that a behaviour change on upgrade is their cost to carry; - the rollback path, which for an in-process engine usually means a flag that routes to the off-path implementation. ## Who overrules you The service owner makes the call; the performance or reliability lead can overrule it, and the strongest grounds for doing so are the ones that are not about this service: a thread budget that threatens co-tenants on the node, an unpatchable dependency, or a stated latency objective the boundary cannot honour. Naming those grounds in advance is what turns a preference into a decision.