Where to Begin with Infrastructure Automation

Infrastructure automation relies on technology to automate tasks involved in controlling hardware, provisioning resources, and handling the essential tasks of configuring and deploying servers and environments. Whether your business workloads are running on-premises or are cloud-based, there are always infrastructure components that must be deployed, managed, and cleaned up — consider virtual machines, virtual network switch configuration, network routing settings, and operating system integration.

Infrastructure automation can also be used as a vehicle to (re)deploy development and test scenarios, automate failover and fallback of environments, or quickly provide standby systems whenever needed. The fabric of the workload architecture is based on a collection of scripts, typically managed and deployed using a continuous integration/continuous deployment (CI/CD) pipeline approach. If this is nearly or completely identical to production, deployment can be executed.

The key factor of integrating infrastructure automation into your environment is to improve the weaker aspects of production. In fact, a core principle of infrastructure automation is reducing human workloads by offloading repetitive tasks. This is also an essential component in enabling software development lifecycle (SDL) practices such as CI/CD.

Automating repetitive tasks is neither new nor specific to hybrid or public cloud environments. Task automation has long been used to reduce workload and avoid human errors. Even experienced developers can forget essential steps.

However, humans should still remain 'in-the-loop'. Automation does not adapt well to change, consider unexpected factors, or coordinate with other systems to find a solution. While machines take care of all the predictable, programmable tasks, humans should still come in at crucial points to make decisions that require their insight and expertise.

The growing popularity of orchestrated containers and scalable microservice-based architectures has allowed for the continuous evolution of infrastructure automation. Let’s explore what this looks like in practice.

Steps to infrastructure automation#

Consider the following scenario:

A retail organization is heavily expanding its e-commerce business by deploying scalable, high-availability workloads in a public cloud environment. To enable the platform to run across different cloud vendors, developers rely on containerized microservices architecture. Translated and simplified into a pipeline, the steps might look something like this:

1. Developers build the application source code.
2. Application code is compiled into a containerized artifact.
3. The containerized workload is pushed into a running state, orchestrated by a Kubernetes clustered environment.
4. Once the containerized workload is up and running, it is watched via monitoring and observability.

Steps 1 and 2 would fit in the continuous integration (CI) part of a pipeline. By integrating with source control systems such as Git, source code updates from different developers will be merged into a main branch. It’s this main branch that will be used at the end of a sprint (iteration) and will be pushed into a running state. This repeatable, automated process of developers committing changes is a key characteristic of a CI pipeline component.

The “final state” of the CI pipeline (often called an artifact), which is typically the version at the sprint deadline, will be used as the starting point for deployment, reflecting the CD component in the CI/CD pipeline.

To provide an end-to-end software distribution lifecycle, two distinct steps are required: building a pipeline and integrating infrastructure as code.

Building a pipeline#

Infrastructure automation is just one part of an automated pipeline that enables CI/CD. This means that the actual workload deployment (for example, the microservices containerized object, such as a web application front-end in an e-commerce scenario) should usually be seen as a separate step from deploying the underlying infrastructure.

While not exclusive, CI/CD pipelines and infrastructure automation are techniques best suited for software that is based on containerized or microservices architectures. The main reason for this is that containers and microservices are abstracting the software runtime (“the app”) from the back end (“the underlying infrastructure”).

Organizations hoping to adopt infrastructure automation for their legacy applications should first look into dividing their monolithic applications into microservices. This is especially crucial when considering the second dependency in infrastructure automation — automating the setup and configuration of systems, before an application is even deployed onto the system. That’s where infrastructure as code (IaC) is helpful.

Infrastructure as code#

Infrastructure as code (IaC) is the mechanism by which traditional infrastructure building blocks — including storage, networking, load balancers, and virtual machines — are deployed using a descriptive model known as a deployment template. Similar to what developers use for application source code, system administrators now also rely on source control to define the layout of the underlying infrastructure components.

One key characteristic of infrastructure as code is that the workload infrastructure will be deployed following a set of instructions, configuration parameters, and specifications. The outcome of the deployment will be exactly as the deployment and configuration template prescribed.

Another key characteristic of infrastructure as code is idempotence, which refers to the fact that the end-state of the deployment will always be the same, regardless of the environment in which the resource is being deployed. For instance, if you have an IaC template to deploy a set of infrastructure components to a dev environment, then the exact same setup can be deployed in a production environment.

Just as automation requires tools, infrastructure as code scenarios necessitate the same. By using IaC tools, DevOps engineers can automate the management of critical infrastructure including servers, networks, cloud resources, firewalls, and more.

A major dependency, especially in public cloud environments, is that each vendor is relying on its own proprietary IaC language. For example, Azure is using Azure Resource Manager (ARM) templates while Amazon AWS is using a mechanism called CloudFormation. While the capabilities are approximately the same, the syntax and language are completely different. This works if you are only using a single vendor’s cloud, but many enterprises use a multi-cloud strategy.

That’s where solutions like HashiCorp’s Terraform or Pulumi are advantageous. By providing integrations towards different cloud providers, one can leverage experience and skills from authoring templates for one cloud, then reuse much of the syntax and structure for deploying a similar scenario into a different cloud environment. Nonetheless, you still need to author different deployment templates to target different clouds.

Terraform uses its own proprietary HashiCorp Configuration Language (HCL). HCL is a lot easier from a coding perspective than, for example, JSON, which is used by both Azure ARM and AWS CloudFormation. Terraform also provides a three-step command line-based approach for initializing, validating, and deploying.

Pulumi’s approach is slightly different from Terraform. It leans closer toward familiar development languages such as .NET, Java, or Python. If your teams already have experience with these languages, writing IaC templates with Pulumi is almost like writing code for software.

A final example is Ansible by Red Hat. Ansible gained popularity in the traditional on-premises data center world years ago as a solution for automating and maintaining a server’s configuration state. By regularly updating the configuration of templates (called Ansible Playbooks), DevOps engineers can be sure that all servers running Windows or Linux are continuously up-to-date according to the predefined state.

With the major shift to public cloud providers, as well as the aforementioned shift to containerized microservices architectures such as Docker and Kubernetes, it is convenient that Ansible also integrates with these platforms.

When you choose a third-party IaC framework like the ones above, you gain automation capabilities and standardization. However, it’s a trade-off as you lose access to all the provider’s technological features. For example, when Amazon releases Relational Database Service (RDS) features, it may be many months before those features are available in Terraform. Even if those features would be helpful to your projects or your organization has a need for the latest features, you are stuck with what you already have. 

It is also important to consider who owns the responsibility for writing and maintaining these templates — it could reside with a dedicated operations team or be part of a development team’s standard duties. Since these are functionally new languages to learn, with versions to track, upgrades to make, etc. — even simple implementations may require time and effort from your dev team.

Conclusion#

Infrastructure automation largely relies on a combination of CI/CD pipelines to deploy the software layer and IaC to build out the underlying infrastructure. Several tools are available on the market. The range includes tools for specific cloud platforms and more universally cloud-friendly examples such as Terraform, Pulumi, and Ansible.

Transposit’s IaC solutions enable teams to maintain consistency and accuracy while accelerating development and empowering more employees to participate. Automated workflows and guardrails enable self-service infrastructure so team members get the setup they need within predefined limits, reducing risk. To learn more about how Transposit’s fully-integrated approach to automation can empower your engineering and operations teams, request a demo.