1. Answers
  2. Setting up AWS Batch batch computing workloads with Pulumi

How Do I Set Up AWS Batch Batch Computing Workloads With Pulumi?

Introduction

AWS Batch is a service that enables developers to run batch computing workloads on the AWS Cloud. By using Pulumi, an infrastructure as code tool, you can define and manage your AWS Batch resources programmatically. This guide will walk you through the process of setting up AWS Batch workloads using Pulumi, detailing each step from defining a compute environment to creating job definitions.

Step-by-Step Setup Process

1. Define IAM Roles

First, create the necessary IAM roles for AWS Batch. These roles will allow AWS Batch to interact with other AWS services securely.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create IAM role for AWS Batch
const batchServiceRole = new aws.iam.Role("batch_service_role", {
    name: "batch_service_role",
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Effect: "Allow",
            Principal: {
                Service: "batch.amazonaws.com",
            },
            Action: "sts:AssumeRole",
        }],
    }),
});
// Attach the necessary permissions for the batch service role
const batchServicePolicy = new aws.iam.RolePolicyAttachment("batch_service_policy", {
    role: batchServiceRole.name,
    policyArn: "arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole",
});
// Create an instance role for AWS Batch to use
const batchInstanceRole = new aws.iam.Role("batch_instance_role", {
    name: "batch_instance_role",
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Effect: "Allow",
            Principal: {
                Service: "ec2.amazonaws.com",
            },
            Action: "sts:AssumeRole",
        }],
    }),
});
// Attach the necessary permissions for the batch instance role
const batchInstancePolicy = new aws.iam.RolePolicyAttachment("batch_instance_policy", {
    role: batchInstanceRole.name,
    policyArn: "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
});

2. Create a Compute Environment

The compute environment specifies the computing resources that will be available for your batch jobs.

// Create Compute Environment for AWS Batch
const myComputeEnv = new aws.batch.ComputeEnvironment("my_compute_env", {
    computeEnvironmentName: "my_compute_environment",
    serviceRole: batchServiceRole.arn,
    type: "MANAGED",
    computeResources: {
        type: "EC2",
        minVcpus: 0,
        maxVcpus: 16,
        desiredVcpus: 8,
        instanceRole: batchInstanceRole.arn,
        instanceTypes: ["m4.large"],
        subnets: ["subnet-0123456789abcdef0"],
        securityGroupIds: ["sg-0123456789abcdef0"],
    },
});

3. Set Up a Job Queue

The job queue manages the scheduling and execution order of your batch jobs.

// Create a Job Queue
const myJobQueue = new aws.batch.JobQueue("my_job_queue", {
    name: "my_job_queue",
    state: "ENABLED",
    priority: 1,
    computeEnvironments: [myComputeEnv.arn],
});

4. Define a Job

Finally, define the job that specifies the container image, vCPUs, memory, and the command to be executed.

// Create a Job Definition
const myJobDef = new aws.batch.JobDefinition("my_job_def", {
    name: "my_job_definition",
    type: "container",
    containerProperties: pulumi.jsonStringify({
        image: "busybox",
        vcpus: 1,
        memory: 256,
        command: [
            "sleep",
            "10",
        ],
        jobRoleArn: batchInstanceRole.arn,
    }),
});
export const batchComputeEnvironmentArn = myComputeEnv.arn;
export const batchJobQueueArn = myJobQueue.arn;
export const batchJobDefinitionArn = myJobDef.arn;

Key Points

  • IAM Roles: Essential for AWS Batch to operate securely.
  • Compute Environment: Defines the resources for batch processing.
  • Job Queue: Manages job execution order and priorities.
  • Job Definition: Specifies how each job should be executed.

Conclusion

By following these steps, you have successfully set up an AWS Batch environment using Pulumi. This setup allows you to efficiently manage and run batch computing workloads on AWS, leveraging the scalability and flexibility of the cloud. The ARNs for the compute environment, job queue, and job definition are exported for easy reference and integration into your workflow.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up