How to Automate Azure Storage Using Pulumi ESC & Automation API

This is a submission for the Pulumi Deploy and Document Challenge: Shhh, It's a Secret!

What I Built

I developed a Python-based automation tool that streamlines the provisioning of Azure Storage Accounts using Pulumi ESC (Environment, Secrets, and Config) and the Pulumi Automation API. This tool enables infrastructure as code (IaC) by dynamically retrieving environment-specific configurations, securely handling authentication credentials, and deploying cloud resources in a fully automated manner.

The implementation leverages Pulumi ESC SDK to fetch environment variables, secrets, and configuration details, ensuring a secure and scalable infrastructure deployment process. Additionally, the Pulumi Automation API is used to programmatically create and manage stacks, enabling seamless provisioning, updating, and destruction of cloud resources.

Key features of the tool include:

Environment-aware provisioning: Automatically fetches required configurations and credentials.
Secure authentication: Uses OIDC-based authentication to interact with Azure.
Infrastructure as Code (IaC): Automates the creation of Azure Resource Groups and Storage Accounts.
Automated stack management: Supports creation, update, and teardown of resources with minimal manual intervention.

Live Demo Link

https://github.com/Ajanhari/azure-pulumi-esc/blob/main/README.md#Demo

Project Repo

https://github.com/Ajanhari/azure-pulumi-esc

My Journey

Create a new pulumi program:

'ajan@LAPTOP-I71Q9TDP:~/pulumi-esc-demo$' pulumi new python
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press .
Press ^C at any time to quit.

Project name (pulumi-esc-demo):
Project description (A minimal Python Pulumi program): Automate Azure Storage Creation Using Pulumi ESC & Automation API
Created project 'pulumi-esc-demo'

Please enter your desired stack name.
To create a stack in an organization, use the format / (e.g. `acmecorp/dev`).
Stack name (dev): DemoESCOrg/prod
Created stack 'prod'

The toolchain to use for installing dependencies and running the program pip
Installing dependencies...

Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (22.0.2)
Collecting pip
  Using cached pip-25.0.1-py3-none-any.whl (1.8 MB)
  ...
  ...
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (59.6.0)
Installing collected packages: six, semver, pyyaml, protobuf, grpcio, dill, debugpy, pulumi
Successfully installed debugpy-1.8.13 dill-0.3.9 grpcio-1.66.2 protobuf-4.25.6 pulumi-3.159.0 pyyaml-6.0.2 semver-3.0.4 six-1.17.0
Finished installing dependencies
Finished installing dependencies

Your new project is ready to go!

To perform an initial deployment, run `pulumi up`

'ajan@LAPTOP-I71Q9TDP:~/pulumi-esc-demo$' ls
Pulumi.yaml  __main__.py  requirements.txt  venv

We now have a pulumi project YAML configuration file and the beginnings of a small python program all set up.

According to the pulumi documentation, we need to add a pulumi cloud access token

export PULUMI_ACCESS_TOKEN="your pulumi cloud org token"

Setup Azure Service Principal and Configure Federated Credentials for pulumi ESC

This guide will walk you through creating a Service Principal in Azure with Federated Credentials using the Azure Portal.

Steps to Create a Service Principal with Federated Credentials

1. Create an App Registration

• • In the Azure portal, navigate to Azure Active Directory.
  • Select App registrations and then click New registration.
  • Provide a name for your application (e.g., pulumi-azure-esc-auth).
  • In the Supported account types section, select Accounts in this organizational directory only.
  • Click Register.

After the Microsoft Entra application has been created, take note of the following details:

• Subscription ID
• Application (client) ID
• Directory (tenant) ID

These values will be necessary when enabling OIDC for your service.

2. Configure Federated Credentials

Once your application is registered, navigate to the Certificates & secrets pane in the left navigation menu.

• Select the Federated credentials tab.
• Click on Add credential to start the Add a credential wizard.
• In the wizard, select Other Issuer as the Federated credential scenario.
• Fill in the remaining form fields as follows:
  • Issuer: https://api.pulumi.com/oidc
  • Subject Identifier: pulumi:environments:org:DemoESCOrg:env:DevDemo/prod
  • Name: An arbitrary name for the credential, e.g., "pulumi-oidc-credentials".
  • Audience:
  • For Pulumi Deployments, this is only the name of your Pulumi organization.
  • For ESC (Enterprise Service Connection), this is the name of your Pulumi organization prefixed with azure: (e.g., azure:DemoESCOrg).

3. Assign Roles

• Go to the resource group/subscription you want the service principal to access.
• Select Access control (IAM).
• Click on Add role assignment, choose the appropriate role (e.g., Contributor), and select your newly created application.
• Click Save.

Once these steps are completed, your Service Principal with Federated Credentials will be set up and ready to use.

Creating Pulumi ESC environment:

ESC environments can be established using two methods: through the ESC CLI or via Pulumi Cloud. This guide specifically outlines the process for creating an environment using the ESC CLI.

Before proceeding, please ensure that the ESC CLI is installed on your system.

• Create a New Organization (Optional): If desired, you can create a new organization to help manage your projects and environments more effectively. This step is optional and can be skipped if you prefer to use your existing organization.

• Initialize a New Environment: Use the following command to create a new environment
  

  esc login
  esc env init DemoESCOrg/DevDemo/prod
  

  • Pulumi uses OIDC (OpenID Connect) to authenticate with Azure via Microsoft Entra Workload Identity Federation. This requires configuring ESC environment to pass the correct credentials.
  • Run the following command to modify the environment configuration:

    esc env edit DemoESCOrg/DevDemo/prod
  

  • Replace Default Content with OIDC Configuration and add the following environment variables:

    values:
      azure:
        login:
          fn::open::azure-login:
            clientId: 
            tenantId: 
            subscriptionId: /subscriptions/
            oidc: true
      environmentVariables:
        ARM_USE_OIDC: 'true'
        ARM_CLIENT_ID: ${azure.login.clientId}
        ARM_TENANT_ID: ${azure.login.tenantId}
        ARM_OIDC_TOKEN: ${azure.login.oidc.token}
        ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
  

  • Replace the placeholder 'your-client-id', 'your-tenant-id' and 'your-subscription-id' with actual value based your service principle and azure subscription.
  • To confirm that the environment variables were set correctly, run below and output verify values are set properly:

  esc env get DemoESCOrg/DevDemo/prod
  

Retrieve Secrets from Azure Key Vault using Pulumi ESC

• The azure-secrets provider enables you to dynamically import Secrets and Configuration from Azure Key Vault into your Environment. The provider will return a map of names to Secrets.

  • Run the following command to modify/update the environment configuration:

    esc env edit DemoESCOrg/DevDemo/prod
  

  • Replace Default Content with OIDC Configuration and add the following environment variables:

    values:
      azure:
        login:
          fn::open::azure-login:
              ...
              ...
        secrets:
          fn::open::azure-secrets:
            login: ${azure.login}
            vault: example-vault-name # your keyvault name
            get:
              api-key:
                name: api-key # Replace with actual secret name from keyvault
        app-secret:
          name: app-secret
      environmentVariables:
          ...
          ...
          ...
  

  • Replace the placeholder 'your-client-id', 'your-tenant-id' and 'your-subscription-id' with actual value based your service principle and azure subscription.

  • To confirm that the environment variables were set correctly, run below and output verify values are set properly:
    

     esc env get DemoESCOrg/DevDemo/prod OR
     esc env open DemoESCOrg/DevDemo/prod
    

  • You should see output similar to the following and the value for "api-key" is fetched from azure key vault:
    

      {
      "azure": {
        "login": {
          "clientId": "aaaa....",
          "oidc": {
            "token": "ey...."
                  },
          "subscriptionId": "000....",
          "tenantId": "8888...."
                },
        "secrets": {
          "api-key": "my-api-key-value",
                 }
            }
      }
  

Using Pulumi ESC

Using Pulumi ESC, achieved below goals

• Establish Secrets and Configurations: Utilize Pulumi ESC environments within Pulumi Cloud to create and manage secrets and configurations effectively.
• Securely Retrieve Secrets: Fetch secrets from Azure Key Vault and securely store them in the Pulumi Cloud environment for streamlined access and management.
• Integrate and Provision Resources: Developed a program that leverages Pulumi ESC configurations to provision an Azure Resource Group and a Storage Account.

Challenges:

According to this document (https://www.pulumi.com/docs/pulumi-cloud/access-management/oidc/provider/azure/#configure-oidc-in-the-pulumi-console), enabling Azure integration automatically stores the fetched credentials as environment variables. However, since I am unable to save the configuration, the credentials are not stored automatically.

As a workaround, I retrieved the values using the ESC SDK in a Python program and set the required values through code. This enables authentication to Azure using the OIDC token.

Note: Pulumi has very decent documentation, which helps me a lot to configure the project setup.

Conclusion.

This script seamlessly integrates Pulumi with ESC to retrieve environment values and authenticate with Azure using OIDC. It automates the creation and management of Azure resources, ensuring a streamlined and secure deployment process. Additionally, it allows you to easily update or destroy the stack as needed, providing flexibility and control over your infrastructure.

If you have any queries, let me know in the comments section. Happy to assist you.