Hyper-scalars with their inherent nature of distributed networks/systems, have completely changed the outlook of the word “software reliability”. With distributed systems, we can no longer rely just on traditional QA; but start to deal with the chaos that is inherent in communication with multiple systems. Chaos Engineering is the practice of intentionally experimenting on a system, based on our assumptions or understanding, and gaining understanding of the unknown behaviour of our systems, so we can gain confidence in its ability to withstand turbulent conditions in production.Chaos Engineering is not a random discipline, but a highly disciplined, systematic approach to finding vulnerabilities before they turn into 3 AM outages. As our architectures shift from predictable monoliths to micro-services built on hundreds of interdependent components spread over multiple cloud providers and SaaS solutions, the ability to catch failure modes through standard unit or integration testing has hit its limit. To understand this, let's start by understanding the mental model behind it.The Mental Model: Hypothesis Testing at ScaleTo practice Chaos Engineering in the right sense, engineering teams need to fundamentally shift how they think about complex systems. The New mental model goes likes this - Distributed systems are inherently chaotic and that failure isn't just possible, it's inevitable.\ Systems need to embrace failure as a natural occurrence. - Amazon CTO Werner Vogels\Challenging the Fallacies of Distributed ComputingThe reason why we need Chaos Engineering comes down to "Fallacies of Distributed Computing” where a classic set of dangerous assumptions are made, which leads to fragile architectures. When developers assume that the network is reliable, latency is zero, or bandwidth is infinite, they build systems unprepared for production. Chaos experiments are designed to break these assumptions and understand where and when these assumptions break.| Assumptions of Distributed Computing | Production Reality | Chaos Experiment Countermeasure ||----|----|----|| The network is always reliable | Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. | Dropping about 5% of packets or forcing a hard network partition. || Latency is zero | Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. | Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. || Bandwidth is infinite | A sudden, unexpected traffic spike will easily choke our app. | Flooding the network to max out capacity and watching how the system handles the bottleneck. || Topology never changes | Pods spin up, instances die, and IP addresses shift constantly. | Killing random pods, containers, or EC2 instances without any warning. || Transport cost is zero | Moving data across regions actually costs real money and adds noticeable lag. | Cutting off a whole Availability Zone or region to see if fail-overs actually work. |\Moving from Unknowns to Knowns\It helps us to think of our system as a matrix of known-unknowns. Traditional testing covers the "known knowns"—the behaviours we expect and understand. Chaos Engineering targets the "unknown unknowns."There are bizarre, cascading failures and complex component interactions that nobody anticipates until the system falls over in production. By scientifically testing our assumptions, we are dragging these vulnerabilities out of the dark and into the light, building the organisation's infrastructure's immune system.The Scientific Method to ChaosWe can't randomly just unplug/shutdown servers and call it Chaos Engineering. The practice relies on four core principles that ensure experiments yield actual insights without impacting the business.1. Define Your Steady StateFirst, we need to define baseline by answering ‘What does "normal" functioning look like?’.Define steady state business-level metrics that reflect the user experience, instead of focusing on CPU or memory usage.| Steady State Metric Type | Examples | Why do we need this ||----|----|----|| Business Output | Orders processed per second, new signups, or video stream starts. | If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). || User Experience | P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). | We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). || Systemic Boundary | Raw throughput or successful HTTP 200 requests per second. | We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). |Once we've got a solid baseline, it's time to set up a hypothesis that we can actually measure and prove wrong. We can't just assume saying that "the system will handle it”; we need to quantify it.Example Scenario: "If I nuke a container, the load balancer needs to catch it and reroute traffic in under 30 seconds. Meanwhile, our P99 latency shouldn't spike past 250ms, and the error rate needs to stay completely under 0.1%." If the system fails any of those specific checks, our assumption was wrong and we got a bottleneck to fix.2. Simulate Real-World DisruptionsCreate chaos variables which will reflect production reality. Prioritise events based on their likelihood or potential blast radius—think in terms of server crashes, traffic spikes, or malformed API responses.3. Run It in ProductionStaging environments simply don't have the scale, messy data, or complex traffic patterns of the live system. Production is the only place where we can validate the authentic request path.4. Minimize the Blast RadiusWhile running chaos experiments in production is important, protecting the customer is the ultimate priority.Start small, target a single container or a tiny fraction of traffic. Widen the scope only once the system can handle a subset of experiments.| Controlled Chaos Techniques | Implementation Strategy | For what Benefit ||----|----|----|| Targeted Blast Radius | Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. | It stops a controlled burn from turning into a massive forest fire that takes down the whole app. || Strict Time-Boxing | If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. | Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. || Off-Peak Scheduling | Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. | If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. || The "Big Red Button” | Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. | It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. |Chaos Engineering in PracticeWe need to start treating Chaos Engineering as an engineering habit, which can be repeatable across a structured life-cycle.Some Common ScenariosInfrastructure: Killing EC2 instances or maxing out disks to test auto-scaling.Network: Injecting packet loss or DNS failures to validate circuit breakers and retries.Application: Exhausting connection pools or killing processes to ensure graceful degradation.Dependencies: Simulating third-party API timeouts.Automating Chaos EngineeringManual tests don't scale. There is real value when we can automate this by piping these experiments to CI/CD pipelines (like GitHub Actions or Jenkins) so we validate resilience every single time we deploy.| Scheduling Pattern | Frequency | Objective ||----|----|----|| CI/CD Pipeline Gate | Every single deployment. | Make sure the new code doesn't ruin platform resilience. || Scheduled Cadence | Weekly or Monthly. | Catch configuration drift and keep the on-call team on their toes. || Game Days | Quarterly or Ad-hoc. | Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. || Event-Driven Chaos | Real-time | Trigger chaos dynamically based on specific system events or unexpected traffic surges. |\Tooling: AWS FIS vs. LitmusChaosAWS Fault Injection Simulator (FIS)If you’re already paying the AWS tax and locked into their ecosystem, just use the native Fault Injection Simulator (FIS). It’s fully managed, you don't need to install agents, and it hooks right into CloudWatch.| AWS FIS Action Category | Example Actions | Impacted Services ||----|----|----|| Instance Management | stop-instances, terminate-instances | EC2, EKS, ECS || Resource Stress | cpu-stress, memory-stress, io-stress | EC2, Lambda || Connectivity/Network | disrupt-connectivity, inject-api-throttle | VPC, RDS, Kinesis || Managed Failover | failover-db-cluster, zonal-shift | RDS, NLB, ALB || | | |You can use it to kill EC2 instances, starve Lambdas of memory, or sever VPC connections to test your database failovers.Litmus-Chaos (The Cloud-Native)For Kubernetes-heavy shops, Litmus-Chaos is an incredibly powerful open-source (CNCF) alternative. It uses a cloud-native architecture based on Kubernetes Operators and manages experiments via Custom Resource Definitions (CRDs).| Feature Comparison | AWS FIS | Litmus-Chaos ||----|----|----|| Deployment Model | Managed SaaS | Self-managed || Kubernetes Integration | Deep AWS Ecosystem | Native Kubernetes / Cloud-agnostic || Extensibility | API/CLI controlled | CRD-based / Pluggable Probes || Experiment Discovery | FIS Scenario Library | ChaosHub Public Marketplace |\A massive advantage of Litmus is its "Resilience Probes," which constantly verify our steady-state health via HTTP checks or Prometheus queries throughout the experiment life-cycle.Security Chaos Engineering: DevSecOps MaturesThis whole "break it on purpose" mindset is bleeding into cybersecurity, too. Security Chaos Engineering (SCE) is testing defences by simulating common, preventable mistakes.You probably will not get hacked through a nation-state, zero-day exploit. More often, breaches happen because someone leaves an S3 bucket public or misconfigures a production IAM role. SCE validates whether your controls, logging, and alerts actually fire when those things happen. Try quietly disabling a security group rule and see whether your monitoring notices. Or "accidentally" commit fake AWS keys to a GitHub repo and see how quickly your security automation revokes them.Disabling firewall rules to validate logging and alerting.Deploying an overly permissive IAM role to trace lateral movement.Simulating expired SSL certificates.Injecting dummy credentials leaks to trigger automated rotation policies.\The Business Case: ROI of ResilienceBuilding a chaos practice takes time and engineering cycles, so how do we sell it to leadership? Point to the cost of downtime.\Enterprise downtime can easily burn $4,000 to $15,000 per minute. Chaos Engineering attacks that risk by reducing Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR). Teams that actively practice chaos engineering often report a 30% reduction in P1 incidents.| Resilience Metric | Description | Value Proposition ||----|----|----|| MTTD Reduction | Figuring out the system is broken before the roof completely caves in. | Shrinks the window where customers are actually feeling the pain during an outage. || MTTR Improvement | Recovering from a disruption and getting things back online fast. | The shorter the outage, the less money the company bleeds. || Availability Score | Percentage of service uptime (e.g., 99.95%) | Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. || Incident Frequency | How often does the pager go off at 3 AM for a critical, hair-on-fire outage? | Proves the long-term hardening of the system. |For SaaS and e-commerce, availability and uptime are revenue. Framing reliability as a competitive advantage makes the ROI conversation easier.How Pioneers Built ItNetflix: Netflix essentially birthed the movement back in 2010 when they built Chaos Monkey. They literally wrote a script to randomly assassinate their own production servers. It was brutal, but it forced their developers to stop relying on sticky sessions and build truly stateless, resilient apps. A few years later, when an AWS regional outage completely melted down half the internet, Netflix didn't even blink.LinkedIn: LinkedIn approaches it a bit differently with Project Waterbear. They basically treat resilience like an internal product. They use tools to intentionally make the site laggy or break specific features for a microscopic fraction of users. They lean hard on their A/B testing setup to keep the blast radius incredibly tight while they watch what breaks downstream.Amazon: Before the insane traffic crushes of Prime Day or Black Friday, they don't just cross their fingers and hope for the best. They run massive, company-wide "Game Days" where they pretend an entire data centre just got hit by a meteor. It builds organisational muscle memory. When a real Sev-1 fire breaks out, nobody panics because the engineers have already drilled that exact scenario fifty times.Good Observability - A PrerequisiteWe cannot do Chaos Engineering without world-class serviceability. You need deep, world-class observability. If you inject a fault and can't immediately trace exactly how it cascades through your micro-services, you aren't doing science. You’re just a vandal breaking things in production. Lock down your four golden signals—latency, traffic, errors, and saturation. You have to be able to set a baseline, instantly spot the deviation, and mathematically prove the system recovered.The Bottom LineChaos Engineering is not just a testing strategy. It is a shift in how we engineer reliability. In a world of infinite complexity, the only way to build trust in a system is to actively try to break it. By embracing failure, setting strict parameters, and automating experiments, engineering teams can stop putting out fires and start building systems that are far more resilient.\