Nowadays, software companies make a lot of effort to ensure the reliability of their systems and services. Carrying out functional and non-functional tests is an integral part of the entire software development process. Resilience testing is a section of the second group which ensures that applications perform well in real-life conditions.
One way of improving the resilience of software is by hosting it on cloud servers. However, it is not enough because the system failures still occur. The conclusion is that the best defense against unexpected failures is to fail often. Such an approach can be adopted by using Chaos Monkey. It follows the Principles of Chaos Engineering.
Why should we use it? Chaos Monkey can help us to verify whether our fallbacks are properly defined, and network latency and service breakdowns do not negatively impact our system. We should run Chaos Monkey in our staging environment and monitor how our system behaves. It would be good to simulate high traffic by load tests. If the metrics confirm that we can control the chaos, then there is no limitation to run Chaos Monkey also in the production environment. Such experiments will make us sure that nothing will surprise us in the future.
In this article we will aim on a dedicated implementation of Chaos Monkey for Spring Boot (CM4SP) applications which is maintained by Codecentric.
Let’s meet Monkeys that make chaos
Chaos Monkey consists of Watchers and Assaults. The graph shows how they are dependent.
Watcheris a Spring component that establishes Aspect for applying additional logic. There is a dedicated watcher created for each of the supported Spring annotations. It can proceed with assaults on all public methods in corresponding classes.
Supported Spring annotations:
Assaultis one of the available attack types conducted by the watcher. The following assaults are possible:
- Latency Assault
- Exception Assault
- AppKiller Assault
- Memory Assault
How does it work?
Chaos Monkey is initialised
by activating the
chaos-monkey application profile. Then Spring Boot loads Chaos Monkey
configuration and creates needed beans. Public classes are scanned to find these that
contain one of the supported annotations. Then watchers can carry out assaults on them.
The set of liable components can be limited by setting configuration property to
explicitly indicate classes we want to attack. There are 4 possible types of attacks. At
the same time, more than one can be active, but some configuration properties are common.
The process of making chaos can be monitored by dedicated metrics.
We can integrate Chaos Monkey into our Spring Boot application in a simple way. No code changes are needed. The only things we should do are:
initialise Chaos Monkey by
activating spring profile
There is only one
requirement. Our project must contain spring-web dependency (
Chaos Monkey is disabled by default, so we can keep calm. No unexpected behaviour will surprise us until we don’t change it in the configuration.
Chaos Monkey gives us the possibility to set the configuration in two different ways:
- statically in
configuration file (
- at runtime through exposed endpoint
Both can be mixed. We can set up all possible properties provided by CM4SB statically in the configuration file. However, it would be a good approach only in small, individual cases. Whereas configuration at runtime has one limitation – we are unable to change the status of active/inactive watchers.
The best approach is to define watchers statically and all other properties later at runtime. Furthermore, Chaos Monkey should not be enabled in the configuration file (just leave it as default – disabled).
The minimal required configuration is to define which Spring components should be attacked by Chaos Monkey. By default, the only service watcher is enabled. If we do not want to have our service components attacked, then we have to explicitly disable it. Setting flags for all other watchers is not required. In the below chunk of configuration, we set them explicitly just to show possible configuration parameters.
There is possibility to
limit Chaos Monkey’s area of destruction. To achieve it we can set
to choose only the services that we want to attack. It is a list of public class and
method names. Suitable watchermust be enabled, otherwise provided
components will not be accessible.
Chaos Monkey offers built-in endpoints exposed via HTTP or JMX that allow us to change the configuration at runtime. We chose HTTP. To enable it, a configuration similar to the below is required:
Exposed endpoints allow to:
get current Chaos Monkey configuration
check status whether Chaos Monkey is enabled or disabled,
enable/disable Chaos Monkey,
get statuses of watchers,
get and modify configuration of assaults
Carry out the assaults
Chaos Monkey for Spring Boot allows us to conduct 4 different types of attacks. They can be grouped by application context or type of activation.
In the first group, we will place Latency Assault and Exception Assault. Both of them depend on HTTP requests. To set the frequency of occurrence we specify the level. For example, level=3 determines that every third application request will be attacked by Chaos Monkey.
To the second group, we will classify Memory Assault and KillApp Assault. They are not dependent on requests. For their activation, you can apply a scheduler.
Applicable cron expression can be set as a value of runtimeAssaultCronExpression configuration parameter.
The latency range is limited
by 2 parameters:
latencyRangeEnd. Their values are validated and can’t
be lower than 1 and higher that in
java.lang.Integer.MAX_VALUE. Time unit is millisecond
Chaos Monkey allows setting custom exception that will be thrown. To apply any exception, its class name with the package has to be passed as type and arguments of its constructor in arguments array.
It attacks the memory of the Java Virtual Machine. For more configuration details take a look at the official documentation.
This one is simple, it just
closes an application according to the scheduler. It is run only when scheduler is defined
– cron expression in
runtimeAssaultCronExpression configuration parameter.
Chaos Monkey for Spring Boot has no built-in generating of reports. It provides metrics that we can use to visualize effects relying on independent tools. We used:
- Micrometerfor exposing metrics in a Prometheus format,
- Prometheusfor gathering metrics,
- Grafanafor visualizing application state during carried out assaults.
To observe the work of Chaos Monkey we generated some traffic. For this purpose, we have used Gatling. It allows us to simulate multiple users that make requests to our application at the same time. Every Gatling test scenario looks as follow:
- enable Chaos Monkey,
- enable and configure assault type,
- make a bunch of requests to the application’s API endpoints,
- disable Chaos Monkey.
In order to expose suitable
metrics , we used
spring-boot-starter-actuator. Versions are
inherited indirectly from Spring Boot Starter Parent.
Prometheus instance gathers metrics from all microservices with integrated Chaos Monkey library. It is used as a data source in Grafana.
A Grafana dashboard id=9845 has been used. Some improvements were made. More readable labels and dedicated panels for Chaos Monkey watchers.
More configuration details and examples can be found over on GitHub.
Chaos Monkey for Spring Boot is a mature tool for conducting resilience tests. The main advantage is the simplicity of applying it in existing systems. It is dedicated to Spring Boot applications. No code modifications are needed. A brief configuration is enough. All other settings can be changed at runtime; it makes it highly flexible.
Chaos Monkey provides 4 different assault types that can be carried out at any time during the runtime. The process of interfering is recorded by using exposed metrics. There is no built-in reporting functionality. However, those metrics can be consumed and visualised in external tools.
Explore more stories like this on our Tech Blog