Backstory first. We run a high-throughput SaaS platform written in Go, and sitting at the heart of it is a messaging service that pushes out time-sensitive outbound messages: OTPs, transactional sends, bulk sendouts, async callback status events. The kind of stuff where if it’s late, someone notices. And not in a good way.

Where We Started

Our pipeline looked like this:

API Request → Main App → Pub/Sub → Subscriber App

It worked. But something kept poking at me.

First, Pub/Sub is an external dependency. Call it lock-in if you want, it’s the same thing: IAM, credentials, an entire fleet of infra we’re suddenly on the hook for. A chunk of our stack rented from someone else’s vendor for no real upside.

Then the dev friction. Wanna test locally? Spin up the emulator first.

And the config. We had something like 13 environment variables floating around just to configure messaging. Thirteen! That’s not a config, that’s an extra job.

Now here’s the honest part. The reason we moved wasn’t performance. It was control. Every complaint above is the same complaint wearing a different hat: the critical path of our product depended on a system we don’t own, can’t debug by writing a query, and can’t run on a laptop without drama. I just wanted that control back.

So we pulled the plug and swapped it for RiverQueue, a Go job queue that’s backed entirely by PostgreSQL. Nothing exotic. It’s a library that runs inside your binary, and your “messages” are just rows sitting in a plain old Postgres table.

What RiverQueue Gives Us Out Of The Box

  • Postgres-backed, so zero new infrastructure to babysit
  • Transactional enqueue, jobs go in atomically with whatever else you’re doing
  • Typed jobs via Go generics (hello, compile-time safety)
  • Per-queue MaxWorkers, so each pipeline gets exactly as many workers as it needs
  • Built-in retry policy via MaxAttempts
  • Native pgx v5 integration, so it slots into our Go code like it was always meant to be there

The New Architecture

API Request → Main App → Postgres → Worker App

Analytics stayed on Kafka, that’s a totally different beast. The messaging path now stays inside our own DB.

We also split things into separate queues by workload, so nothing waits behind something it shouldn’t:

  • Time-critical jobs, the ones with someone waiting in real time
  • Bulk split, the slow process heavy lifting
  • Callback, handling asynchronous callbacks from external services
  • Report / export, generating reports and exporting data
  • Analytics publish, sending analytics events to Kafka

Publishing a Job in Go

Enqueuing a job is boring, but I take that as a compliment:

p.queue.Insert(ctx, dto.TimeCriticalJob{
    LogID:   id,
    Payload: payload,
}, &queue.InsertOpts{
    Queue:       "time_critical_jobs",
    MaxAttempts: 1,
})

A worker is just a method on a struct:

func (w *TimeCriticalWorker) Work(ctx context.Context, job *queue.Job) error {
    return w.Process(ctx, job.Args)
}

And that’s the whole worker. They automatically pick up any available jobs that match their queue and do the work they are assigned to. Just like people queueing up for their turn at a service counter.

The Migration, Step by Step

We didn’t big-bang this. We ran a dual-path for about 2.5 months with a canary percentage per workload. The phases went:

  1. Publish sub only, the new path just watches, nothing acts
  2. Build a worker for each type
  3. Dual-path with a canary %
  4. Ramp slowly to 100%, watching latency and error rates the whole way
  5. Soak for weeks at 100%
  6. Delete all the Pub/Sub code

Why this worked so well: with the default at 0, bugs can’t sneak into the new path; you control it per workload type; zero downtime; rollback is dead simple; and that long soak weeded out the weird stuff a weekend dash never would.

The canary function itself is tiny:

func canary(pct int) bool {
    if pct <= 0 { return false }
    if pct >= 100 { return true }
    return rand.Float64() < float64(pct)/100
}

Where We Landed

  • What used to need two containers now runs in one
  • Messaging env vars: from a dozen-ish down to zero. Yes, zero
  • ~1,000 lines of broker plumbing code deleted, in the bin
  • Local dev is now just docker-compose up, no emulator in sight
  • Retry settings live in code now, a MaxAttempts field right where the job is inserted, instead of in subscription config
  • Debugging is now a SQL query away, nothing quite beats that feeling
  • Each queue scales independently, and it runs anywhere: cloud, bare metal, your laptop

Did It Cost Us Anything In Performance?

Honest answer, no. But we measured anyway, because “we moved for control” sounds nice until it turns out slower. Load test, same workloads on both paths:

  • Insert latency (API → queue): ~5ms with RiverQueue vs teens of ms (mostly two digits) with Pub/Sub
  • Processing throughput: 58–60 rps with RiverQueue vs 57–60 rps with Pub/Sub
  • Pod resources (Kubernetes): cpu: 100m / memory: 200m with RiverQueue vs cpu: 150m / memory: 300m with Pub/Sub

Same throughput, a few milliseconds off the insert path, and it fits in a smaller pod. Nice, but it was never the point. We didn’t move to go faster. We moved so that when something goes wrong, the thing that’s wrong is a table we own, in a database we already run, with a query we can write ourselves.

Wrap Up

We ended up with one less system to own. The broker is gone, the messaging path is a table in our own Postgres, and the ramp was boring on purpose: nudge 10%, watch, ramp, soak, rip it out. Zero downtime, rollback available the whole time, and the only thing left to operate is a database we already run in production.

That’s it for this post. See you on the next one!