Ten databases. Ten completely different ways to say “give me the next thousand.” That was the whole problem.
Our vector database migration tool (MING) moves embeddings across ten different source databases and roughly the same number of targets. Every migration follows the same basic pattern – read a batch, transform it, write it, repeat. Reading sounded like the boring, solved part.
It wasn’t.
First version just loaded everything into memory, transformed it, wrote it out. Worked fine on small test collections. Then we hit production. A single 1536-dimensional vector is roughly 6KB before metadata. A million-vector collection can chew through several gigabytes of memory.
So we had to switch to bounded batches. Fine, simple fix. Except every database wanted “give me the next batch” in a completely different language.
Qdrant hands you a cursor and a next_offset. Chroma, MongoDB, pgvector, SingleStore just use limit and offset – simple, except large offsets get expensive fast. Weaviate doesn’t give you batches at all, just an iterator you accumulate manually. Pinecone makes you list IDs first, then fetch vectors in a second loop. Redis uses SCAN, where the requested batch size is more of a suggestion than a rule, and keys can even show up more than once. LanceDB already thinks in Arrow batches, not rows.
Our first instinct was to build one universal next_batch() function to hide all of that. It didn’t fit. Weaviate isn’t naturally batched. Pinecone’s nested loops don’t collapse into one call. LanceDB shouldn’t be forced to downgrade to Python rows just to match an interface.
So we stopped trying to abstract the read and started abstracting what happens after a batch exists. Every batch – however it got fetched – goes through the same pipeline: filter, transform, write, record IDs, publish metrics.
That one shift meant every batch gave us three really useful things: a memory boundary, a recovery checkpoint if a job dies at 80%, and a natural progress heartbeat.
Ten specialized readers, one shared spine. The differences never really disappear – hiding them just makes them harder to debug later.
More breakdowns like this coming – follow along.
