Developing a Simple Python Application and Automating the CI/CD and Containerization Process [Part 2]
Writing Terraform Modules for IaC
In my previous article, I discussed how to set up a simple Python application that can be integrated into the cloud through a comprehensive, automated CI/CD framework. Check it out here for a refresher on those concepts! Now let’s move into the second phase of this effort, which is to set up the cloud infrastructure required for the application we created previously. To do this, we introduce a tool called Terraform, which is the leading Infrastructure as Code (IaC) provider in the industry right now.
So what is Terraform? It’s a tool that allows us to create a template of cloud resources set up as definitions with properties that we can customize. Ideally, this is useful for whenever we want to create new resources, destroy these resources and build them anew, or add more resources to the existing lot, all without having to manually create them on the chosen cloud provider’s portal. Now all this is starting to piece together, but how does Terraform know whether the resources already exist or whether it has to build them from scratch or modify them? To resolve this conundrum, Terraform introduced a concept called a state file, which basically stores all the resources created by it in previous runs into memory. So, in the event that we are adding or destroying resources, or just altering existing definitions, Terraform knows to compare the new changes with the current state and modify the state file accordingly. As we start discussing a more automated cloud/DevOps approach, Terraform becomes exceedingly important as it helps us manage our cloud resources on the fly and integrates seamlessly with most CI/CD platform frameworks and cloud providers.
Alright, now that we’ve gone over an overview of how Terraform works, let’s dive into the cloud resources we’d require for our current example application. As I delineated in my previous article, we have set up a highly scaled down banking system, which will request user input to perform certain operations and then upload the transactions into a sql table. So the first few resources we can already determine as necessary are a SQL server instance and database connection, on which we can create our table to store the uploaded transactions. For this example, I decided to use the Microsoft Azure Cloud Platform as my cloud provider, since it’s the first public cloud platform that I experimented with and learned a lot from when I started working in Cloud & DevOps. So maybe I’m a little biased 😁. But feel free to try this project out with any cloud provider of your choice, since Terraform is cloud agnostic, making it compatible with multiple providers.
Ok, now the first thing we have to do is configure a provider so Terraform knows which cloud platform to connect to and build resources on. Figure 1 shows how to set a provider resource definition in Terraform.
Providers are just plugins to enable interactions with certain APIs. Here, we have added in three separate blocks — one to configure the required providers and the backend, and the other two to tie the providers back to the tenant id that they’re associated with on Azure. A tenant id in Azure is a unique identifier of an organization’s Active Directory instance. It is associated with one or more subscriptions that individuals within the organization may or may not have access to, depending on their RBAC (Role Based Access Control) permissions. Whenever we create a provider in Terraform, we need to refer back to this tenant id so that it will know what organization to create the resources in. In the first block, we see that two providers, one for azurerm (which stands for Azure Resource Manager) and the other for azuread (Azure Active Directory), have been configured with the hashicorp source for each and an updated version. Hashicorp owns Terraform and offers multiple plugins to interact with different APIs owned by the cloud providers. In our case, we will need to interact with the Azure Resource Manager and Azure Active Directory APIs in order to create applications and various other resources on Microsoft Azure.
In addition to this, we can observe a backend configured for the Azure Resource Manager API, referencing the resource group, storage account/container name, as well as key (or file) name. This is none other than a location for the Terraform state file to be uploaded to and read from everytime a new run is invoked. Now the issue is that Terraform will need this path to be set up so that it can upload the state file to it on the first run. So, let’s go ahead and set up the storage account. To do this, we need to head to Azure Portal, login, and click on Storage Accounts under the hamburger icon. We will be redirected to a page like the one shown in Figure 2.
I’ve already created the storage account required for Terraform, but if you’re following along with this tutorial all you have to do is hit the create button and provide information to the UI, such as subscription, resource group, and storage account names, so that it will create the account as shown in Figure 3.
Ensure that public access is enabled to your account so we can easily connect to it for data retrieval. As I mentioned in my previous article, these tutorials are meant to assist developers in a more junior level, so security and role based access control has not been introduced or implemented with grave concern. If you were to implement this within your organization, you would need to include the proper security and access management measures to enforce the principal of least privilege and prevent data breaches.
So coming back to the matter of storage accounts — now we can go ahead and create the resource. Once it has been provisioned, we will click on it and hit the containers tab. Here, I already have my container created, but all you have to do is just click the container option, which will allow you to name your new container and give it public accessibility. Now that we have a path in which the state file will be published, we can proceed to set up the other resource definitions as Terraform scripts.
Figure 4 shows an architecture diagram of the resources we will be provisioning as part of this project.
Earlier, we talked about the importance of a SQL server instance and database for storing transaction data, but here we’re seeing a lot more resources than just those! Let’s look at the bigger picture now that we’ve been presented with it. We’re trying to automate the entire development and devops process, which contains build and release components to it. The build process is triggered through a continuous integration pipeline, that detects any changes made to our code and automatically builds it on our chosen CI/CD platform (in my case this is Azure Devops), converting it into a Docker image that can be pushed to a repository on Azure Container Registry (ACR). This build process will use or modify the resources seen in the above figure and additionally push any artifacts if creates to the release process. The release process is then triggered through a continuous deployment pipeline that starts as soon as the build pipeline has finished execution. This pipeline will connect to the Kubernetes cluster represented in the above diagram, and either run the Docker image it retrieves from ACR (using the cluster role assignment attached to it) as a new deployment, or update an existing deployment. That’s the entire process in a nutshell.
Ok, so now you’ve seen that SQL, AKS, ACR as well as Azure DevOps serve for some importance as shown in Figure 4, but why on earth would we need an Azure Key Vault (AKV) instance for any of this? Well, when we attempt to connect to our database server, we require a username and password to login. While it would be easier to just hardcode these values into our program, that is not a secure and viable option when we’re working with sensitive data like this. So in setting up a key vault instance, we can create secrets on to the vault pertaining to the values we want to encrypt, and those values will be hidden unless we specifically grant access to the vault. Access policies help provide these RBAC permissions and here, I’ve listed three different access policies — one for my python app to directly contact AKV, one for Azure Devops’ in-built service principal (since our code will be run from this platform), and the last one for me as an AAD user (which is useful when I’m testing my code locally).
There are a lot of components to this project, but hopefully that was a helpful overview as to what purpose they each serve and why they’re so crucial in making this process run end to end in an automated manner.
Figures 5- 7 show some Terraform resource definitions that I set up for the components we just talked about.
I haven’t included all the resources we talked about in the above diagrams to save some of your time, but you can always check out my repository (linked in the resources section at the end of this article) to view my code and TF scripts. Now, I’d like to make a note here about something you’ve possibly already observed in these figures. There are a lot of references to data and var. Data refers to data modules in Terraform, which are basically modules that read from existing resources on the cloud. Figure 8 represents a data module set up in order to read information about the resource group that we want our resources to be created in.
Since the resource group has already been set up, we can read information about it into this data module, and pass that information using the module as a reference into the resources to be created.
So what about the var keyword? Var is an abbreviation for variables, which are just containers to hold data values that could possibly keep changing. In our case, we have quite a few variables, mostly to hold the values of our Azure Organization entities, as well as some other sensitive data. Figures 9 and 10 display some variable definitions and where their values are stored.
The variable values shown in Figure 10 are stored in a file called a tfvars file. This file extension is known to Terraform as a variables file, so it will use it and supplement the values into the resource definitions when a plan or apply is invoked. There is a special extra extension that must be added in this case called an auto extension, since these scripts will be triggered on a CI/CD platform and not locally. This extension is required since Azure DevOps (or your chosen CI/CD platform), does not know to look for the variable values locally within the file system, as it doesn’t connect the tfvars file with the resource definitions by default. Adding the auto extension to tfvars (like fileName.auto.tfvars) allows it to understand that it should automatically use this file to set up any environment variables necessary for the Terraform resources to be created.
That should pretty much sum up most of the doubts that could arise from the Terraform scripts I included in Figures 5–7. But if you navigated over to my repository and looked at the actual TF scripts, you probably noticed one other thing that I haven’t talked about yet — a reference to two service principals, one for the python application and one for Azure DevOps. We’ll talk about the latter in our next article, where the focus will be on CI/CD, but let’s take a look at the former. What exactly is a service principal in the first place and why would we need it for our application to connect to resources on the cloud? A service principal is an entity registered on the cloud that is provided with certain roles and permissions to access applications. Users define these roles according to what permissions they want to grant their applications, and the service principal acts on behalf of the application to utilize these permissions, since the app can’t connect to the cloud by itself. In this case, I needed the application to access AKV in order to read the database password secret; so I provided secret reading permissions to the service principal that I created.
Creating a service principal on Azure involves setting up an AAD app registration and creating a client secret. This secret will be used in the actual code to access Azure resources through authentication to AAD. For more information as well as a guided tutorial on how to set up a service principal, check out the resources section at the end of this article!
And with that, the IaC portion of this project comes to a wrap! All we have to do is run terraform init, so Terraform can initialize in the directory hosting the tfscripts, and then run terraform plan & apply. Terraform plan displays a list of the resources to be created as a plan, and terraform apply executes the plan to create the resources on the chosen cloud provider. We can do all of this locally and manually, but when automation is our end goal, we will have to integrate this with a CI/CD platform so that the init, plan, and apply processes are automatically triggered through a build pipeline. Sounds interesting? You’ll have to wait until my next article to learn more about this! See y’all there 😉
Find my code here 👉 https://abujji.visualstudio.com/banking-app-python/_git/terraform_scripts
Resources:
- https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
- https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal
- https://stackoverflow.com/ (If you’re in Tech, then you know how big of a lifesaver this forum is 😂)
- Can’t forget, my Brain 😎
