Search-Enabled AI Applications using AWS OpenSearch Domain

Question

Pulumi · Accepted Answer

To create a search-enabled AI application using AWS OpenSearch (formerly known as Elasticsearch), you will need to set up an OpenSearch Domain, which will serve as the foundation for indexing and searching your application data. Pulumi allows you to define this infrastructure as code, making it easy to deploy and manage.

Here is a step-by-step guide along with a Pulumi Python program that demonstrates how to create an OpenSearch Domain on AWS.

### Step 1: Create an OpenSearch Domain
We will start by creating an AWS OpenSearch Domain with encryption at rest, VPC options, and node-to-node encryption for security. The domain will require an access policy to define permissions, such as who can access the domain.

### Step 2: Configure Access Policies
Access policies are essential for securing your OpenSearch Domain. We will use an AWS Identity and Access Management (IAM) policy to control access to the domain. This policy can be as open or restrictive as needed; for this example, we will give open access to the domain from any AWS resource (note that this is not recommended for production environments).

### Step 3: Export the Domain Endpoint
After creating the domain, we'll want to export the endpoint URL, which applications will use to connect to the OpenSearch service.

### The Pulumi Program
Below is the Python program that sets up an AWS OpenSearch Domain with the steps mentioned above. Make sure you have your AWS credentials configured correctly before running the program. You can either set them up via the AWS CLI or the Pulumi configuration.

```python
import pulumi
import pulumi_aws as aws

# Create an AWS OpenSearch Domain with encryption, VPC options, and node-to-node encryption enabled.
opensearch_domain = aws.opensearch.Domain("ai-search-domain",
    engine_version="OpenSearch_1.0", # Specify the OpenSearch version to use
    cluster_config=aws.opensearch.DomainClusterConfigArgs(
        instance_type="t2.small.search", # Choose the instance type for data nodes in the cluster
        instance_count=1, # Number of instances. More instances increase reliability.
    ),
    ebs_options=aws.opensearch.DomainEbsOptionsArgs(
        ebs_enabled=True,
        volume_size=10, # Size in GiB. Adjust according to your needs.
    ),
    encrypt_at_rest=aws.opensearch.DomainEncryptAtRestArgs(
        enabled=True # Enable encryption at rest.
    ),
    node_to_node_encryption=aws.opensearch.DomainNodeToNodeEncryptionArgs(
        enabled=True # Enable node-to-node encryption.
    ),
    vpc_options=aws.opensearch.DomainVpcOptionsArgs(
        # Specify the VPC, subnet, and security group for the OpenSearch domain.
        # These must be set if you want to place your domain within a VPC.
        subnet_ids=["subnet-xxxxxxxx"], # Replace with your actual subnet ID
        security_group_ids=["sg-xxxxxxxx"], # Replace with your actual security group ID
    ),
    # Open access policy (not recommended for production).
    # You should restrict it based on your requirements.
    access_policies=pulumi.Output.all(opensearch_domain.arn).apply(lambda arn: f"""
        {{
          "Version": "2012-10-17",
          "Statement": [
            {{
              "Effect": "Allow",
              "Principal": "*",
              "Action": "es:*",
              "Resource": "{arn}"
            }}
          ]
        }}
    """),
)

# Export the endpoint of the OpenSearch Domain.
pulumi.export('domain_endpoint', opensearch_domain.endpoint)
```

In this program, we start by importing `pulumi` and `pulumi_aws` modules. We then create an instance of `aws.opensearch.Domain`, which represents an AWS OpenSearch Domain. The `engine_version` parameter specifies the OpenSearch version we want to create. The `cluster_config` options such as `instance_type` and `instance_count` allow you to specify the computing resources for your domain.

The `ebs_options` parameter enables Elastic Block Store (EBS) for your domain and sets the volume size. `encrypt_at_rest` and `node_to_node_encryption` ensure that data is encrypted while stored and as it travels between nodes.

We configure the domain to be inside a VPC for added security and network isolation by setting `vpc_options`. The `subnet_ids` and `security_group_ids` are utilized here to place the domain within your existing VPC.

The access policy controls who has access to the OpenSearch Domain. Here, we've provided an open access policy—which is *not* recommended for production—for instructional purposes.

Finally, we use `pulumi.export` to make the domain endpoint available outside of Pulumi, so you can use it in your application or other parts of your infrastructure.

Remember to replace the placeholder `subnet-xxxxxxxx` and `sg-xxxxxxxx` with your actual subnet and security group IDs. You'll also want to fine-tune the access policy based on your actual requirements to enforce proper security measures.