It’s become a bit of a routine over the last two weeks or so. Come in to work in the morning, look at the numbers, find that they have gone off, spend the morning investigating why they’d gone off. Spend the afternoon implementing a patch in response of the findings of the investigation. Leave the system to run for the rest of the day, come in the next morning, check the numbers, find that they have drifted again…
On and on and on, like a never-ending puzzle. Every time, we hose out a category of bugs, only to find another emerge, either from increased clarity, or as a result of the previous fixes.
Everyone was getting frustrated. Not the least, yours truly.
“If you can’t fit the whole picture in your head, something is very wrong”
– Wise words of colleague
It was exactly what I was struggling with. Every morning, I’d come in, look at the numbers going off, and spend a good amount of time just loading up the entire platform in my tiny brain to play out the scenarios, and form a logical explanation for why they were off.
Every patch we added was an added complexity that could very likely come back and bite us further down the track. Some of the actually did.
We needed to take a different approach.
It turned out, we’d made a few design decisions earlier on that didn’t account for one of the “heavier” components coming online more recently. Between the complexities of developing that new component and helping it play well with the rest of the platform, we didn’t have much left behind to even consider revisiting the design of some of the other components.
So we’re going to take some time to reengage the broader design issues and come up with a more permanent solution. Sure it’s weeks of sunk cost down the drain, but it is also potentially months of future debugging that we won’t need to do.
Glad we’ve come this far. No one could have seen this coming, and it’s unlikely that we’d get here by any other way.
So here’s to a new week, a deep overhaul that settles it once and for all, and moving on to productive work.