I believe the most dangerous performance bug is the one that looks fixed.
Our rota planning product at AgileForce started feeling slow. Not one broken endpoint – the whole app. Staff lists, profiles, the earnings dashboard, all just a beat slower than they should be.
Nothing was throwing errors. It just dragged.
The weird part was the code already looked optimised. select_related(’employee’) was sitting right there, exactly where the docs tell you to put it.
Turned out the declaration stopped one hop too early. A shift assignment points to a working-data record, which points to an employee record, which points to a user record – and the name lived on that last one.
So the code fetched employee, felt done, and the serializer quietly walked two more steps to grab the name.
On every row. Every request. The fix was there, it just didn’t reach far enough – and that’s the worst kind of bug, because a line that already has the right function on it is the one line nobody re-reads.
We found more of the same shape elsewhere. The earnings dashboard looped over every staff member and ran two queries each – fifty staff, a hundred queries, before any actual math happened.
We flipped it: fetch the widest date range once for everybody, then sort each row into the right period in plain Python. A hundred queries became two.
Some smaller ones added up too – a list endpoint calling .count() on rows already sitting in memory, a profile endpoint re-fetching a user record the auth layer had already loaded seconds earlier, updates saving entire rows when only one field changed.
But the two biggest wins weren’t in the queries at all. Auth keys were cached in-memory per worker process, wiped on every restart – moving that to Redis gave every worker one shared copy.
And database connections were being opened fresh per request against a managed cloud database over TLS, paying a full handshake cost every single time. Turning on persistent connections removed that entirely.
Real lesson: a fix that’s present but incomplete is more dangerous than one that’s missing – because it stops anyone from ever looking again.
Follow for more of these engineering breakdowns.
