Archive for security

Infrastructure as Code can Help with Compliance

IaC cycle

This was originally published on Amalgam Insights.


Companies struggle with all types of compliance issues. Failure to comply with government regulations, such as Dodd-Frank, EPA or HIPPA, is a significant business risk for many companies. Internally mandated compliance also represents problems as well. Security and cost control policies are just as vital as other forms of regulation since they protect the company from reputational, financial, the operational risks.

IT helps to manage compliance risks in two ways. First, by deploying systems that detect and assist the company in complying with risk. For example, an analytics system designed to discover Dodd-Frank violations in a bank is a way for IT to help remove some regulatory risk. The second way is to design systems that are compliant with internal and external regulations. That may mean configuring networks such that they address common security holes or data storage so that privacy is maintained. Systems must be configured to meet the needs of regulations and policies.

One of the more important methods of ensuring that systems are compliant is through audits. Different audits are conducted for different purposes, but they operate similarly. The state of a system – business or computer – is evaluated against a series of policies. Policies are statement that describe a required state in a system that ensures compliance. For example, a data privacy policy may require that customer data be only housed on encrypted drives. A security policy may require two-factor authentication for all external logins to a system.

Systems in place can be audited using a variety of tools that match the system state to policies and detect areas out of compliance. The problem with this approach is that it is post-hoc. Finding compliances issues in production is good but finding them before they are in production is better. Pre-production audits often rely on design documentation. This is a limited method because last minute changes or mistakes with configuration can lead to the final state diverging from production.

This is where Infrastructure as Code(IaC) can be helpful. Central to IaC is the idea that a plaintext script – the code part – describes the desired state of the system. The automation server then configures the system to the state indicated in the script. IaC systems may provision and configure both hardware and software components.

IaC presents an advantage when auditing for compliance. The script is the documentation and what is in the script will become the final system state. This makes it easy for compliance and security professionals to understand the system state before it is created. They can then analyze these scripts for potential violations of policy and prevent them from becoming part of the production system. Using IaC allows companies to more easily move from reactive compliance to proactive compliance. Non-compliance can be detected early and corrected before they become the production state. In rapidly changing environments, this is a safer and quicker approach than detecting problems after the fact and correcting them while in production.

Dealing with regulatory, security, and operational compliance of at scale is much becoming more difficult as policies proliferate and systems become more complex. IaC is a tool to deal with systems and their compliance issue in these environments.

Monitoring Containers: Do you know what happening inside your cluster?

container with a spherical object in it

This was originally published on May 18th on the Amalgam Insights. For reasons I can’t fathom, I forgot to push the publish button.


It’s not news that there is a lot of buzz around containers. As companies begin to widely deploy microservices architectures, containers are the obvious choice with which to implement them. As companies deploy container clusters into production, however, an issue has to be dealt with immediately:
container architectures have a lot of moving parts. The whole point of microservices is to break apart monolithic components into smaller services. This means that what was once a big process running on a resource rich server is now multiple processes spread across one or many servers. On top of the architecture change, a container cluster usually encompasses a variety of containers that are not application code. These include security, load balancing, network management, web servers, etc. Entire frameworks, such as NGINX Unit 1.0, may be deployed as infrastructure for the cluster. Services that used to be centralized in a network are now incorporated into the application itself as part of the container network.

Because an “application” is now really a collection of smaller services running in a virtual network, there’s a lot more that can go wrong. The more containers, the more opportunities for misbehaving components. For example:

  • Network issues. No matter how the network is actually implemented, there are opportunities for typical network problems to emerge including deadlocked communication and slow connections. Instead of these being part of monolithic network appliances, they are distributed throughout a number of local container clusters.
  • Apps that are slow and make everything else slower. Poor performance of a critical component in the cluster can drag down overall performance. With microservices, the entire app can be waiting on a service that is not responding quickly.
  • Containers that are dying and respawning. A container can crash which may cause an orchestrator such as Kubernetes to respawn the container. A badly behaving container may do this multiple times.

These are just a few examples of the types of problems that a container cluster can have that negatively affect a production system. None of these are new to applications in general. Applications and service can fail, lock up, or slow down in other architectures. There are just a lot more parts in a container cluster creating more opportunities for problems to occur. In addition, typical application monitoring tools aren’t necessarily designed for container clusters. There are events that traditional application monitoring will miss especially issues with containers and Kubernetes themselves.

To combat these issues, a generation of products and open source projects are emerging that are retrofit or purpose built for container clusters. In come cases, app monitoring has been extended to include containers (New Relic comes to mind). New companies, such as LightStep, have also entered the market for application monitoring but with containers in mind from the onset. Just as exciting are the open source projects that are gaining steam. Prometheus (for application monitoring), OpenTracing (network tracing), and Jaeger (transaction tracing), are some of the open source projects that are help gather data about the functioning of a cluster.

What makes these projects and products interesting is that they place monitoring components in the clusters, close to the applications components, and take advantage of container and Kubernetes APIs. This helps sysops to have a more complete view of all the parts and interactions of the container cluster. Information that is unique to containers and Kubernetes are available alongside traditional application and network monitoring data.

As IT departments start to roll scalable container clusters into production, knowing what is happening within is essential. Thankfully, the ecosystem for monitoring is evolving quickly, driven equally but companies and open source communities.