Red Teaming Meets Chaos Engineering: A Modern Approach to Cyber Resilience

Introduction: What Happens When Security Gets a Surprise?

In today's world, most organizations have policies and processes in place for protecting their systems, monitoring for attacks, and responding to any potential incidents. But what if an attacker is able to exploit an unexpected weakness in the system, perhaps just before a new product release?

Traditional security testing is all about finding vulnerabilities in a system before attackers do. Chaos engineering, on the other hand, is about introducing failures in the system in a controlled way to see how it handles them. When combined, we have an approach to security which could be called security chaos engineering. Security chaos engineering can be thought of as red teaming combined with chaos engineering to inject failures. This allows us to continuously challenge our system’s defenses, which would be done in an adversarial manner. But instead of just looking for potential vulnerabilities, we could focus on understanding how our system’s security controls work when faced with a security failure.

What is Security Chaos Engineering?

Security chaos engineering is the practice of introducing failures (or attack scenarios) into a system in a controlled manner, and evaluating how the system responds. Chaos engineering itself originated in the reliability engineering domain, where it was used to assess the behavior of an application when it encounters failures.

With security chaos engineering, the experiments focus on aspects such as the identity and access controls, monitoring, and network defenses, as well as incident response. For example, one security chaos engineering experiment could be to use a service account in a way it was not intended to be used, as if it had been compromised. Does the system have the appropriate access controls in place to prevent the service account from doing things it shouldn’t be doing? Is the security team alerted if the service account is used in this manner?

In this way, security chaos engineering can be seen as testing the entire process of how a security incident is defended against, rather than just a system’s security defenses.

Red Teaming and Security Chaos Engineering

Red teaming is about simulating an adversary in as realistic a manner as possible to assess the security of a system. This could include simulating an attack using phishing emails, escalating a compromised user’s privileges, or testing network segmentation. Red teaming exercises can provide valuable insights into how a system is secured, as it is essentially viewing security from the perspective of an attacker. The challenge with red teaming is that it is usually not a continuous activity. These are often large, resource-intensive engagements which rely on a team of people with specialized skills.

This means that there is potentially a time gap between red teaming engagements, when the system could be vulnerable. Security chaos engineering helps to mitigate this risk by providing a way to turn some of the attack scenarios used in red teaming exercises into experiments which can be run continuously throughout the software lifecycle.

Why Automated Failure Injection Matters

Automated failure injection is used to intentionally inject a condition into the system to determine the behavior. In the context of security testing, this might be a security service not working, a scenario where credentials are used after their expiry, a scenario with suspicious authentication attempts, a scenario to determine whether access controls are in place even during a failure condition, and more.

The key here is that the conditions should be injected in a controlled manner. This means having a clear understanding of the objective, the scope, the expected outcomes, and the conditions to stop the experiment. If the failure injection can be carried out in an automated and controlled manner, then it is possible to repeat the experiments after changes are made in the infrastructure. This makes it possible to identify regressions earlier and before they make it into production.

How Red Teaming and Chaos Engineering Work Together

Red teaming and chaos engineering are complementary to one another. While red teaming can identify the attack paths for real-world attacks, chaos engineering can be used to test the effectiveness of various defensive measures when a particular failure condition is injected. Combining the two can help to create a feedback loop.

From Attack Scenarios to Repeatable Experiments

Let’s consider an example where a company has a customer-facing portal in the cloud. The red team finds that an attacker can exploit a compromise in a service account to access an internal API that is very permissive. While the red teaming exercise identifies a potential attack path and hence a weakness in the security posture of the company, the organization will be interested in finding out if monitoring and alerting is in place to detect when a service account is misused. It will want to determine if the API is configured to deny unauthorized access. It would also want to know if the incident response team will be able to revoke the account credentials quickly. Security chaos engineering can be used to answer these questions in a controlled manner.

In a security chaos engineering exercise, the typical process usually involves 4 steps: 

1. Identify a particular attack path using techniques like threat modeling, vulnerability assessments, or red teaming exercises. 
2. Once an attack path is identified, the organization needs to define what the expected defense will be for that particular attack path (for example, denying access or triggering a high-priority alert). 
3. After the attach path is known, the organization will inject the condition in a controlled manner in the test environment or the production environment (with appropriate protection). 
4. Finally, the organization will measure the outcome of the experiment and make improvements to fix any issues, and then run the experiment again to ensure that the fix works.

This approach shifts security testing from occasional discovery toward ongoing verification.

Testing More Than the Technology

We should not forget that security is not just about technology. In an incident, one may find weaknesses in technology as well as processes and people. In the example mentioned earlier, an attack may start with stolen credentials and then the attacker may use the permissive API. 

The attack may not be detected because the alerting system is not properly configured. All the different components may appear to be in place, but combining failure injection experiments can help to uncover such dependencies. 

In the above example, one might also simulate a compromise of a user identity, and check if there are proper policies in place to enforce multifactor authentication, restrict access for privileged accounts, and log suspicious events, and whether the correct information is sent to the incident response team. This is a more useful exercise than having a checklist of security products in place.

Practical Security Chaos Engineering Experiments

A practical security chaos engineering experiment is one that focuses on a realistic risk to the business. It does not have to be dramatic, nor should it involve risking critical business operations.

When determining what risks to focus on, it can be a good idea to start with incidents the organization has faced, weaknesses discovered during penetration testing, or scenarios included in threat models.

1. Identity and access control

Identity and access control is often a good starting point for security chaos engineering because it does not require the attacker to gain access to the network in order to be successful. A chaos experiment might introduce an expired credential, a request for access to a resource it does not have, or a service account performing an operation outside of its permissions.

The goal would be to ensure that access controls are designed to fail securely. For example, if an identity provider is unavailable, an application should not grant access automatically just because it is unable to authenticate the request. Metrics in this situation might be unauthorized requests that are not granted, the time to detect the situation, the accuracy of any alerts generated, and the time it takes to revoke the credentials if they were compromised.

2. Gaps in detection and monitoring

Whether it is logs or alerts, endpoint detection or centralized monitoring, security teams often rely on some form of detection to find suspicious behavior. Having detection tools in place does not, however, guarantee that the tools will detect what needs to be detected. The chaos engineering approach to this situation is to inject events that are approved to help test whether or not suspicious events are making it to the detection platform and triggering the appropriate response. This can also help identify issues where the alerts do not have enough context to be useful. For example, the alert might state that something is happening, but not which account or resource is being accessed.

These experiments can help identify areas where telemetry is missing, thresholds for alerts are incorrect, events are not getting processed in a timely manner, and gaps between various security tools.

3. Network segmentation and limiting lateral movement

When an attacker gains access to a network, they typically want to go beyond the single system they initially compromised. Network segmentation, access controls, and other methods help isolate workloads to prevent attackers from moving laterally to other systems. Chaos engineering can be used to test the effectiveness of these controls in a safe manner by using test identities or workloads to attempt to access resources or connections that should be blocked and confirming that appropriate events are generated for these attempts.

4. Security service outages

What happens when a critical security component is not available? An organization could test a loss of a logging collector, a dependency on an identity provider, or a policy enforcement service in a safe environment to confirm that applications continue to work as expected and that events are still being recorded and are recoverable. This type of experiment is particularly useful in a cloud-native environment, where there are many services and dependencies that the application relies on.

Building a Safe and Repeatable Testing Program

Security chaos engineering is a way to improve the security and resilience of your systems, however experiments which are not appropriately designed can potentially cause the very incidents you are trying to prevent. Therefore, it is important to have a security chaos engineering operating model in place.

Start With Hypotheses and Guardrails

Every security chaos experiment should start with a hypothesis. As an example, a security chaos engineering experiment should begin with a hypothesis such as “if a service account attempts to access a resource which it is not authorized to, then the request is denied and an alert is generated within two minutes”. This hypothesis sets out the clear expectations of the experiment.

It is important to define the scope of the system being tested, the maximum level of impact acceptable for that experiment, what monitoring should be in place and what conditions require the experiment to be terminated. 

Special care should be taken when conducting experiments against production systems:

• Experiments should only be conducted in approved windows 
• The appropriate stakeholders and rollback procedures agreed to and in place 
• Appropriate emergency contact details are available in case of unexpected issues

A useful set of safeguards includes:

Scope: Restrict the scope of the experiment, using specific accounts, services and resources where possible.

Safety: Ensure that there are automatic termination conditions in place and that appropriate recovery procedures have been defined.

Operational coordination: Ensure that appropriate stakeholders within the security, infrastructure and incident response teams have been notified.

Evidence collection: Ensure that parameters, timestamps, results and corrective actions are recorded.

This will ensure that experiments can be repeated and do not turn production systems into a playground.

Integrate Experiments Into CI/CD Pipelines

Security chaos experiments can be included as part of an organization's CI/CD pipeline. This allows experiments to be run as part of the software delivery process. This is especially important if there have been changes to an organization's authentication, authorization, network policies or infrastructure configuration. Not all experiments are suitable for every deployment, however lightweight experiments can be run as part of the CI/CD process and more disruptive experiments can be run in isolated environments or as part of a controlled maintenance window.

For instance, a CI/CD pipeline might ensure that a new API is still not accessible by unauthorized users. An experiment may be scheduled to check that a simulated security incident is being received by the incident response platform.

These experiments should produce actionable results. For example, if an experiment reveals a critical authorization regression, it may prevent the deployment from going out. If the risk is not as high, perhaps it creates a remediation task that gets tracked.

By implementing security chaos engineering, organizations begin to move security from being a separate task that is done just prior to release to one that is part of the ongoing responsibility of engineering.

Measuring Cyber Resilience With Meaningful Metrics

Security chaos engineering should have measurable outcomes. The number of security chaos engineering experiments is not an indicator of improved security, nor is the number of events detected an indicator of the quality of the security in the organization.

Rather, organizations should track whether the experiments they perform are validating their security assumptions and whether they are addressing the weaknesses that they identify.

Metrics That Indicate Weaknesses in Defenses

There are a number of metrics that organizations may want to use to track their progress with security chaos engineering:

• Mean time to detect (MTTD). How long it takes to detect suspicious activity.
• Mean time to respond (MTTR). How long it takes to contain or resolve a security event. Exactly what constitutes a "response" should be documented.
• Control effectiveness. The percentage of experiments that exhibit the desired behavior of the control being tested.
• Alert accuracy. Do experiments that simulate security events generate appropriate alerts without excessive false positives?
• Recovery performance. How quickly are critical security services restored?
• Remediation closure. Are security weaknesses being addressed and retested successfully?

These metrics should be tracked over time, so that, for example, if an organization implements a change to improve alerting, it can tell whether that change is having the desired impact and whether the improvement is being maintained over successive deployments.

It is also important to track the impact of a failure on the business. For example, a delay in detecting an event that affects a low-risk internal service is different from a delay in detecting an event that affects a payment platform or a sensitive database.

Common Challenges and How to Avoid Them

Security chaos engineering does pose some challenges in practice, particularly for organizations that have complicated infrastructures and/or lack dedicated resources in security engineering.

It can be tempting to try to automate complex scenarios to save time and effort, but it is better to start with a few well understood risks where the expected outcomes are clearly understood and established. In addition, one must be careful about the operational risks that chaos security testing can pose. For example, disabling a critical authentication system could cause outages or may not allow incident response teams to do their work. So it’s better to start with isolated environments and only once you have established appropriate safeguards and practices, experiment in controlled manner in production as well.

Another important point is not to assume that every failed experiment is the result of a software defect. The root cause could be the way monitoring is configured, or how responsibilities are assigned, or simply a lack of a well defined incident response process. In such situations, collaboration across the security, development, operations and infrastructure teams is required to remediate the issue effectively. Finally, it is important not to measure success by the number of experiments you run. Ten very well chosen experiments that identify real issues and get fixed can be far more beneficial than running hundreds of experiments that end up testing the same thing again and again.

Tools and Technologies For Security Chaos Engineering

Chaos engineering for security does not necessarily require a dedicated tool. Most organizations typically use a combination of existing tools for security testing, infrastructure automation, observability, and incident response. Red teaming and penetration testing tools help you to identify attack paths and test your assumptions. Infrastructure-as-code and configuration management tools are used to create test environments. 

Observability and SIEM (Security Information and Event Management) tools are used for collecting and analyzing metrics, logs, and events. You can also use existing chaos engineering tools to create controlled failures in your infrastructure, but these need to be carefully examined to ensure they meet the specific needs of security chaos engineering. 

Frameworks like the MITRE ATT&CK can also be useful in helping you decide which scenarios to test. This can guide you on the scenarios to test, but it does not take the place of setting appropriate controls and hypotheses.

When selecting security testing tools, it's critical to start with your testing objective. By adding a new platform without a clear understanding of the security problem you are trying to solve, you’re just adding complexity.

The Business Value of Continuous Security Testing

Cyber resilience isn't just about your security department. A successful cyber breach which could have been prevented could affect an organization’s business operations and ability to launch products on time, erode customer confidence, and even have regulatory and contractual consequences.

Security chaos engineering can help you get a clearer picture of how well your security is working before you encounter an incident in the wild. It will also provide evidence that you are continuously testing your security rather than just checking that controls have been implemented.

The business value becomes clear when you tie your chaos experiments to business priorities. For a financial services organization, this might include experiments around identity protection or transaction integrity. For a software organization, this might involve testing the security of your APIs, secrets management, and supply chain. For a healthcare organization, it may be around testing access controls and availability.

Organizations face different risks and their chaos experiments should align with those risks, not a generic chaos testing schedule.

Chaos security testing can also help bridge the gap between technical and business teams by providing evidence that your controls are preventing unauthorized access, alerting you to an incident, and helping with containment. This helps make security investment decisions more practical.

Conclusion: Make Security a Capability That Gets Tested

Red teaming helps you understand how a sophisticated attacker might compromise your organization. Chaos engineering can help you understand how your systems behave when assumptions about the security of your environment fail. Both these practices help build a foundation for continuous security testing and better cyber resilience.

The key is starting with realistic experiments that are narrowly scoped, measurable, and connected to remediation. Rather than starting with a huge, elaborate attack simulation or a major technology investment, you can begin by identifying one of your critical security controls and test its effectiveness.

You can then fix the issues that you find, repeat the experiment and then grow your capabilities over time.

The key takeaway is that security controls should be tested under realistic failure scenarios instead of just trusting that they will work. By making that part of your normal engineering practices, you can find weaknesses before they bite you, improve your response to incidents, and build systems that will cope better with the unexpected.

For any questions about Security Chaos Engineering in Modern Cybersecurity, please contact us at ScreamingBox.

For more info on some of the topics in this blog, check out our podcast on AI, Security & Software Governance