At 9 o’clock this morning (24th May 2023), the MoT twitter account sent out a number of tweets that linked to some development test data that had been created by a dev earlier that morning. Obviously this shouldn’t have happened, but a series of things happened that caused it.
For a bit of background, we have a job that is generated when content is published that will schedule a tweet. This job only runs if jobs are set run, which in dev they aren’t… Except something was being worked on where these jobs had to be turned on. But that’s ok, since the way we generate our test data is done in such a way that it doesn’t trigger the generation of the job… Except we’ve been changing how that works, so now it does (which I didn’t realise until this happened, so at least I learnt something today). But that is ok, because the job may be scheduled, but has a guard clause that uses a configuration option to decide whether to continue, and that is only turned on in production… Except it turns out that when I first developed this system over a year ago, I’d committed the dev config with it turned on and it has been ever since.
So that is how this happened. The root cause was really that last point, so I’ve made a change to turn these jobs off in development, which will prevent this from accidentally happening in future.
My thanks and apologies to all those whose Twitter and LinkedIn feeds were spammed, but who let us know the problem as soon as it occurred. And my apologies also to the rest of the MoT team, who had to rush to undo the effects of this mistake.
As a final note, I never use rude or jokey things for my test/dev data, specifically because it might accidentally end up in production.
