A progress bar that just… froze. That’s how this whole thing started.

Another problem we encountered a while back when we were working on our vector migration tool – MING.

Some migrations finish in seconds. Some run for thirty minutes. Either way, the user’s sitting there watching a progress bar, expecting it to actually move in real time.

Sounded simple enough.

The migration itself runs as a background job – the API kicks it off and hands it to a separate worker, then immediately steps out of the way. Which is great for performance, but it creates one annoying problem.

The worker knows exactly how far along the migration is. The browser has no idea.

Apparently that shouldn’t have been a big deal.

Our first attempt was the obvious one – polling. The worker writes progress into the database, and the frontend just asks “are we there yet?” every couple seconds.

It worked. Technically.

But it turned out to be a pain in the back in its own quiet way. Updates felt delayed. Every single browser tab was hammering the database with requests, and most of those requests came back with nothing new. Poll faster and you load the database harder. Poll slower and the progress bar just feels stuck, even when it isn’t.

The information was already there. It just had no real path from the worker to the browser.

So we stopped thinking of it as “how do we ask for updates faster” and started thinking of it as “how do we let the worker just tell the browser directly.”

Turned out we already had the tool sitting right there. Celery – the thing running our background jobs – already uses Redis. So instead of bringing in another piece of infrastructure, we used what was already sitting there. 

We just let the worker publish progress events into a Redis channel, and had the API pick those up and forward them to the browser using Server-Sent Events. One-way, lightweight, no fancy WebSocket setup needed, because the browser only ever needs to listen, never talk back.

But two things almost bit us.

First – Redis Pub/Sub has zero memory. If someone opens the page after the job is already 60% done, they’ll never see those earlier updates, because Redis doesn’t replay anything. So we had to make sure the stream always starts by pulling the current state from the database first, and only then starts listening for new live updates. Current state, then live events. Not one or the other.

Second – if a job finishes before someone even opens the page, no new event is ever going to arrive. And if we’re not careful, that connection just… sits open. Forever. So we had to explicitly check, right after sending the current state, whether the job’s already done. If it is, send the final result and close the door behind you.

Oh, and one more thing that only showed up once we deployed it for real – everything worked fine locally, then broke behind our reverse proxy. Turns out proxies love to buffer responses, which is exactly the opposite of what you want for something that’s supposed to feel instant. We had to explicitly tell it not to buffer, or the user would see updates arrive in weird little bursts instead of smoothly.

Honestly, looking back, this had nothing to do with progress bars specifically.

It was more like – anytime you’re streaming updates to someone who’s just watching, not talking back, three things matter. Use the infrastructure you already have instead of bolting on something new. Remember that live events and current state are two different things, and you usually need both. And never forget that every stream needs a real ending, not just a start.

This is just one of the problems we’ve run into while building MING. I’ll keep sharing the ones that taught us something along the way. Follow along.