Monolith Hell + Kubernetes = Monolith Hell

We built a Kubernetes cluster but we still managed to crash our production server. In hindsight, we were living through a Monolithic Hell

Single Point of Failure vs. Monolith Reliability

an ElasticCache (Redis) node was deeply coupled with the monolith Node.js servers. Due to a bad code commit (millisecond vs second), the ElasticCache node ran out of memory soon after deployment. The Kubernetes nodes started to crash one after another. The whole production system was brought down to its knee.

ElasticCache ran out of memory on 1st Mar 2020

Engineering cannot keep up with business growth

more food ordering logic, more POS integrations, more payment processors, all in one Node.js code repo, oh my! Before leaving the startup, I tried really hard to dump my brain into the following diagram, and I felt anything but proud.

Even modularised, things were still bundled in one code repo and deployment.

Afterthought

While many learned Microservcies by reading books, blogs or going to tech talks, I was lucky (or unlucky) enough to have the first-hand experience of Monolithic Hell. I feel a monolithic system is still a viable option for building simple solutions, such as MVP. However, technical leardership needs to be aware of the early signs of Monolithic Hell, and to plan for migrating to Mircorservcies architecture. Conversely, Microservcies comes at its own cost, for a simple CRUD app, it might be overkill.