skip to content

How would you place Couchbase's data, query, index and search services across nodes for a mixed workload?

level: principalimportance: nice to knowfreq 33%

answer

  1. Nodes run a chosen subset of services
  2. One tier is stateless and scales easily
  3. Another tier's loss removes access paths
  4. Small clusters should keep everything together
  5. Split when one workload starves another

basics

~20 s

Couchbase lets each node run a chosen subset of services, so isolate data, query, index and search onto separate node groups once they contend, and size each group to its own bottleneck: RAM for data, CPU for query, memory plus disk for index.

solid answer

~50 s

Couchbase's Multi-Dimensional Scaling means services are assigned per node rather than every node running everything. The judgment call is when to co-locate and when to split. Small clusters run everything on every node — simplest to operate, and contention is theoretical. You split when a workload starts stealing from another: query CPU starving KV latency, an index build hammering the same nodes that serve reads, full-text search consuming memory the data service needs for its working set. After splitting, each tier scales on a different axis: the Data service on RAM for the working set and on nodes for capacity, the Query service horizontally because it is stateless, the Index service on memory and disk plus index replicas for availability and scan throughput. The cost is more nodes, a minimum viable size per tier, and rebalances that touch more moving parts.

code

sql · 3 lines
sql
CREATE INDEX idx_orders_status
  ON `app`.sales.orders(status)
  WITH {"num_replica": 1};

go deeper

for a junior

Know that a Couchbase node runs only the services assigned to it, and be able to name the main ones: data, query, index and search.

for a middle

Explain what each service does in the path of a query and why they can be placed on different nodes, including that the query tier holds no state of its own.

for a senior

Diagnose from symptoms to topology: which observed contention justifies moving a service off shared nodes, and why index replicas matter for availability rather than only for throughput.

for a principal

Own the whole sizing model — when co-location is still correct, the node-count floor separation imposes, how non-fungible capacity across tiers changes cost, and how divergent growth rates should drive the topology over time.

## What Multi-Dimensional Scaling actually is When a node joins a Couchbase cluster you choose which services it runs: Data (the KV engine), Query, Index, Search, and depending on edition, Analytics, Eventing and Backup. A node runs only the services you assign. That is Multi-Dimensional Scaling — the ability to add capacity along one dimension without paying for the others. The contrast worth naming in an interview is with a design where every node is identical: there, more query capacity means more data nodes and therefore more replication, more rebalance data movement and more memory you did not need. ## When co-location is right Small clusters should co-locate. Three nodes each running Data, Query and Index is operationally simple, gives every service a quorum of hosts, and the contention you are protecting against is hypothetical at that size. Splitting a small cluster into single-node tiers is worse: you lose redundancy per tier and gain nothing. ## The signals that justify splitting Split when you can point at contention: - **KV latency degrading when query load rises.** The Query service is CPU-hungry per statement; on a shared node it competes with the Data service's threads, and KV percentiles are what user-facing paths feel first. - **Index builds or rebuilds disturbing reads.** Building a global secondary index is a sustained memory, CPU and disk burst. On shared nodes it lands on the same hardware serving the working set. - **Search memory pressure.** Full-text indexes are memory-hungry and their footprint is independent of the data service's quota; putting them on their own nodes prevents them eating the cache that determines KV hit rate. - **Different growth rates.** Query volume and data volume rarely grow together. If the ratio has shifted a lot since the cluster was sized, that is a structural argument for separate tiers. ## How each tier scales **Data service.** Sized primarily by memory: the bucket quota must hold the working set (and for the default ejection policy, per-document metadata) if you want cache hits rather than disk reads. Adding data nodes adds both capacity and throughput, but it also triggers rebalance and changes replica placement, so it is the heaviest change to make. **Query service.** Stateless. Every query node can plan and execute any statement, so this tier scales horizontally almost linearly behind a load balancer or via the SDK's own node awareness, and nodes can be added or removed with little ceremony. It is the cheapest tier to grow and the first one to grow when concurrency, not data, is the pressure. **Index service.** Sized by index size (memory and disk) and by scan throughput. Index replicas are the availability and throughput mechanism: creating an index `WITH {"num_replica": 1}` places an additional copy on another index node, so a node loss does not remove the access path and scans can be spread. Without replicas, losing an index node means queries that depended on those indexes start failing for want of an access path — an availability event, not just a slowdown. **Search service.** Its own node group when its memory demand is material, for the reasons above. ## The costs you must own Separation is not free. Each tier needs enough nodes to survive a host failure, so a fully split cluster has a higher floor on node count and therefore on licence and infrastructure spend. Capacity becomes non-fungible: spare RAM on the query tier cannot serve the data tier's working set. Rebalance and upgrade procedures involve more groups and more sequencing. And you now have four sizing models to maintain instead of one. So the principal-level answer is not "always separate". It is: co-locate until measured contention or divergent growth justifies a split, split the noisiest neighbour first (usually Query, then Search), keep every tier at least fault-tolerant, and give the Index service replicas because its failure mode is loss of an access path rather than mere slowness. ## What interviewers are probing They want to see that you reason from measurements to topology rather than reciting a reference architecture; that you know the query tier is stateless and therefore the easy lever; that you understand index replicas as an availability mechanism, not just a performance one; and that you can state the price of separation in nodes and operational complexity rather than presenting it as a free win.

  • Which Couchbase service tier is easiest to scale horizontally, and why?
    The Query service. It is stateless — any query node can plan and execute any statement — so nodes can be added or removed with little coordination and no data movement. That makes it the first lever to pull when concurrency rather than data volume is the pressure, and it is why query load is usually the first workload split off shared nodes.
  • What does WITH {"num_replica": 1} on a Couchbase CREATE INDEX statement give you?
    An additional copy of that index built on another index node. It protects availability — losing one index node no longer removes the access path and does not break the queries that depend on it — and it also lets scan traffic be served by more than one node. It costs the memory, disk and maintenance of a second copy.
  • Why is losing an Index service node more serious than losing a Query service node in Couchbase?
    Query nodes are interchangeable, so the survivors simply absorb the load. Index nodes hold the access paths; if an index has no replica, the statements that depended on it lose their only path and start failing outright rather than merely running slower. That asymmetry is why index replicas are treated as an availability requirement.

saying these in an interview costs you the question

  • Recommends separating every service regardless of cluster size
  • Treats index node loss as a slowdown, not an availability event
  • Thinks adding query nodes increases storage capacity
  • Runs indexes without replicas on a split cluster
  • Presents service separation as free of operational cost

context

resolving links…