An MVP that stopped coping — rebuilt event-driven on Kubernetes

A working MVP began failing as users arrived. Requests queued behind a single slow third-party integration, and the platform could not be scaled by adding servers because the work was happening inside the request. We moved it to an event-driven architecture — orchestrators enqueueing jobs, worker pools consuming them, everything autoscaling on Kubernetes. It now serves more than 10,000 users.

The situation

The product was a working MVP with real customers and growing signups — the good problem. It had been built the way most MVPs are: a service that did the work inline, synchronously, while the user waited. That is the right decision at forty users. It stops being right somewhere before four hundred.

The specific failure was concurrency. Several core operations depended on third-party ecommerce and marketplace APIs, and those APIs are not fast or reliable on anyone's schedule but their own. Every request that touched one of them held a connection open for the duration.

Why adding servers would not have fixed it

The instinct when a system slows under load is to scale it out. That would not have worked here, and it is worth being precise about why.

When slow external work happens inside the request, your throughput is capped by your connection pool multiplied by how fast the slowest third party feels like responding. Doubling the servers doubles the pool and buys a little headroom, but the failure mode is unchanged: one degraded upstream API still backs up requests until the pool is exhausted, and then everything stops — including the parts of the product that had nothing to do with that integration.

What we changed

The core move was to get the slow work out of the request path entirely, and to make each kind of work independently scalable and independently failable.

Orchestrators and queues

Incoming work no longer executes inline. Orchestrator services accept the request, decide what needs to happen, and enqueue jobs onto BullMQ queues backed by Redis. The user gets an immediate response; the work happens behind it.

Separating orchestration from execution matters more than it first appears. The orchestrator holds the business logic — what should happen, in what order, and what to do when a step fails. Workers stay dumb and single-purpose, which is what makes them safe to scale and safe to restart.

Worker pools per workload

Different queues get different worker pools. A slow marketplace sync no longer competes with payment processing, because they are not the same pods and not the same queue. When one upstream degrades, its queue grows and its workers retry — and every other part of the platform carries on unaffected.

That is the real win. Not raw speed, but a blast radius: a bad afternoon at one integration became a delay in one feature instead of an outage.

Retries and idempotency

Moving to queues means accepting that jobs will run more than once — a worker dies mid-task, a pod is rescheduled, an upstream times out after having actually succeeded. Every job was made idempotent so a retry is safe, with backoff on failure so a struggling third party is not hammered into staying down.

Running it on Kubernetes

The architecture only pays off if capacity follows demand, so the whole platform was containerised and deployed to Kubernetes with horizontal pod autoscaling.

• Worker pools scale on queue depth and CPU, so a backlog is absorbed by adding consumers rather than by a human noticing.
• Each service deploys independently, so shipping a fix to one integration does not mean redeploying the platform.
• Rolling deploys with health checks, so releases stop being events that require a quiet moment.
• Pod-level isolation, so a memory leak in one worker cannot take the API down with it.

Integrations

Alongside the re-architecture, the platform's integration surface was built out: payment gateways, and marketplace and ecommerce platform connections including Walmart. Each of these went in as its own service behind its own queue, which is precisely what makes adding the next one cheap — a new integration is a new worker and a new queue, not a change to the core.

Where it landed

The platform now serves more than 10,000 users. The migration was incremental — capability by capability, with the old path in place until the new one was proven — so there was no cutover weekend and no downtime.

The more useful outcome is that the shape of the system now matches the shape of the business. Load grows, pods scale. A third party has an outage, one queue backs up. A new marketplace appears, it is a new worker. None of those are architectural events any more.

What we would tell you to take from this

• Slow third-party work belongs behind a queue, not inside a request. This is the single highest-leverage change for most MVPs under strain.
• Separate queues per workload give you a blast radius: one degraded integration stops being an outage.
• Queues mean retries, and retries mean every job must be idempotent. Design for it up front.
• Autoscaling on queue depth turns a backlog into a cost line rather than an incident.