Six things that broke, and the rules they left behind

The last post said that most of the rules I work by came out of things going wrong. This is where I make good on that. Six things that broke while I was building AlgoForgeX, what each one cost, and the rule it left behind.

None of these were planned for in advance. They were learned, usually the hard way, and then written down so the same thing could not happen twice. I am telling them plainly, without dressing them up. The point is not that the mistakes were clever. Most of them were boring. The point is what they taught.

The afternoon the whole site went blank

One afternoon every page on the site started coming back empty. Not an error page, just blank panels where the content should be.

The cause was one line. A service that gets created once and shared everywhere, a singleton, did a blocking wait on an asynchronous call the first time anything asked for it. On the framework this site runs on, the interface for a given visitor lives on a single thread. Block that thread to wait on async work and it deadlocks. The service happened to get created in the middle of a page render, on that thread, and the whole connection froze. Every panel rendered empty.

The fix was small: read the configuration lazily, the asynchronous way, on first real use, instead of blocking in the constructor. The lasting result was a single hard rule in the conventions file the assistant reads every session, never block on async on any path that can run on one of those interface threads.

The lesson is that the worst outages are boring. One blocking call, in one constructor, took the entire site down.

The job that rewrote history

A background job was supposed to watch for stock splits and flag them for review. The first time it ran on its own, it did not flag them. It applied them, rewriting historical prices across hundreds of instruments.

Price history is the foundation everything else stands on. Charts read it, backtests replay it, risk math is derived from it, portfolio values depend on it, and the AI analysis quotes it. Corrupt the foundation and the damage spreads everywhere at once, quietly, because nothing downstream knows the numbers it was handed are wrong. Cleaning it up meant restoring data.

The rule that came out of it is simple and absolute: detection never changes anything. Anything that could alter production data flags it for a human to look at and approve, and any bulk operation shows a preview of exactly what it would change before it touches a row.

The lesson: never let automation modify production data on its own, least of all the data that everything else trusts.

The emails that just stopped

Password reset emails to outside addresses stopped arriving. There was no error, no bounce, no log line saying anything had gone wrong. From inside the app, every send looked successful.

The email provider at the time was silently dropping outbound mail to external domains under a filtering policy of its own. Because nothing failed in a way the code could see, nothing alerted. It surfaced because a person noticed that resets were not landing, which is the worst way to find out.

The fix was to move all mail, the app's own and the login system's, onto a single different transport. The lesson outlived the fix: the failures that hurt the most are the ones that do not raise an error. "It threw no exception" is not the same as "it worked." Watch what actually happens at the far end, not just whether your own code is happy.

The safety net with a hole in it

AlgoForgeX has an advisor side, where a firm can message its clients. The app leans on a filter at the database level that automatically narrows every query to the right owner, a safety net meant to make leaking someone else's data hard to do by accident.

That net only covers tables marked as belonging to a user. The messaging tables belonged to a firm, not a single user, so the marker did not apply and the filter quietly did nothing for them. A query meant to load one conversation could load every conversation in the database.

It was caught in a security pass at the end of a sprint, before it ever reached a customer. The relief packed into that sentence is the entire reason those passes exist. The fix was an explicit firm check on every one of those queries, plus tests that fail the build if one firm can ever see another firm's data.

The lesson: a safety net has edges, and they are invisible until you find them. The filter is real protection right up until a new idea does not fit the shape it was built to guard, and then it protects nothing, silently. Know exactly where your nets stop.

The fix that was right twice and wrong twenty-eight times

A mobile fix needed the keyboard to open reliably on iOS, which is fussier about it than you would expect. The approach worked. It was tested on the screen that needed it, and it shipped.

Then it got applied to the other two dozen or so places that needed the same behavior, and the build broke for everyone who pulled the branch. The approach relied on an attribute that, on the component library this site uses, collided with one already in use. In the two places it was first tried, there was no collision and it compiled cleanly. Copied to twenty-eight, it did not.

The fix was a different approach that could not collide. The lesson is the one most particular to building with an assistant: a pattern can be correct in the small, confidently explained, and still wrong at scale. The answer is not to trust the tool less in some vague way. It is a habit. Always rebuild after you spread a pattern across the codebase, because two passing examples are not proof of the next twenty-six.

The number that was lying

There was a field called current price, on the object that represents a holding, that did not contain the current price. It held the price of the most recent purchase.

Most of the code that read it had quietly worked around the misnomer by overwriting the value with the latest close before using it. A few places had not. So on several of the more advanced risk and exposure screens, positions were being weighed at what they had cost rather than what they were worth now. A position you had doubled showed half its real weight. No error anywhere, just wrong numbers on exactly the screens where wrong numbers do the most harm.

It was found by the audit that traces every number a user can see back to the data and the code behind it. The fix corrected the handful of screens and, more importantly, renamed the field to say what it actually held, so no one could be misled by it again.

The lesson: a misleading name is a bug sitting in wait inside every file that trusts it. Name things what they are.

What the scars add up to

Every rule in the previous post has a story like one of these underneath it. The conventions file, the habit of flagging instead of applying, the audits that trace numbers and check for leaks, the reflex to rebuild after spreading a change: none of it was drawn up in advance. It accumulated, one incident at a time.

That is the honest shape of building something large and keeping it running. You do not avoid the mistakes. You make sure each one leaves behind something durable, a rule, a test, a renamed field, written down where it will be read again, and then you keep going.

Next, for anyone who wants to see the wiring up close, a technical deep dive or two.