“Business logic” being a term of art which means “your bullshit”, in contrast to “my code,” which is beautiful.

(By the way this was all SO much better than the financial industry job I had before this.)

When we finished, what we had was a Twisted-pure version of the middle layer. Plus a declarative framework which just added lines of code, since declaratively specifying a thousand special cases requires more code than not having the framework at all.

This is a generalizable part of the experience for me—if I could choose a superpower it’d be to appear like Candyman behind any developer saying “it should be easy to make a declarative framework for this.”

The drop-in was also written in callback style with Deferreds, so although logically it was identical to the first version, it was roughly triple the line count and much harder to grok and debug.

If you forgot to return a Deferred, you were shit out of luck since the language obviously couldn’t help you with it.

But eventually after months and months we released this thing on one page, and it burst into flames within milliseconds.

I don’t know what the state of the art is with debugging and profiling nodejs these days. But whatever that story is, I assure you that understanding what a Python reactor loop was doing while it was melting down in 2007 was the bronze age by comparison.

I saved this screenshot of helplessly waving kcachegrind at it and hoping for a miracle

It was at this point that the Twisted consultants were brought back

They said that although Twisted was good at overall throughput, outlying requests could experience severe latency. Which was a problem for the drop-in, because the way the PHP frontend used it was hundreds/thousands of times per web request.

“Yeah sorry folks it’s not good for this.”

So over the course of a few weeks we frantically rewrote the drop-in replacement to use a threadpool instead, exactly like the original heretical one.

Leaving us with literally the same code as the thing it was dropped in to replace, plus a ridiculous declarative framework, plus some tests. It was around this time that everyone got fired (but not me).

One way to look at this is that an entire engineering and ops team lost their jobs because a group of people thought that threads were Wrong.

By the time the drop-in replacement was being systematically eradicated by the drop-in replacement engineering team, this entire saga had been forgotten because it was simply too out there to be believable.

As a younger person I had no power to avert any of this, but I managed to not get fired because through this whole thing I was talking shit about it. Which was not necessarily the lesson I needed to walk away with, but here we are. /end