Exoscale Object Storage for Dataset Versioning

Question

Pulumi · Accepted Answer

In order to set up an object storage solution on Exoscale with dataset versioning, we will make use of the Exoscale Object Storage service. Exoscale isn't listed as having a unique object storage resource in the given Pulumi Registry results, however, it does provide an S3-compatible API that can be used with Pulumi's AWS provider by configuring it to point to Exoscale's S3-compatible endpoints.

Exoscale Object Storage, which is S3-compatible, allows you to store and manage your datasets with versioning. Versioning in an S3-compatible object storage works by keeping multiple variants of an object in the same bucket. You can use this to recover objects from accidental deletions or overwrites.

Here's how you would use Pulumi to create a bucket with versioning enabled on Exoscale:

### Pulumi Python Program for Exoscale Object Storage with Versioning

```python
import pulumi
import pulumi_aws as aws  # Import Pulumi's AWS SDK which we'll configure for Exoscale

# Exoscale provides an S3-compatible API endpoint which can be targeted by configuring the AWS provider accordingly.
exoscale_endpoint = "https://sos-{zone}.exo.io"  # Replace `{zone}` with your Exoscale zone, e.g., 'ch-dk-2'.

# Configure the AWS provider to point to Exoscale.
# The Exoscale secret key and access key should already be configured in the environment or the Pulumi config.
aws_provider = aws.Provider("exoscale-s3",
    access_key=pulumi.Config("aws").require("accessKey"),
    secret_key=pulumi.Config("aws").require("secretKey"),
    skip_credentials_validation=True,  # Since we're not using real AWS, we skip credentials validation.
    skip_metadata_api_check=True,      # We won't use the AWS metadata API.
    s3_force_path_style=True,          # Use path-style S3 addressing.
    endpoint=exoscale_endpoint)

# Create a new S3 bucket with versioning enabled on Exoscale.
bucket = aws.s3.Bucket("my-dataset-bucket",
    versioning=aws.s3.BucketVersioningArgs(
        enabled=True
    ),
    opts=pulumi.ResourceOptions(provider=aws_provider))

# Output the bucket name and endpoint.
pulumi.export("bucket_name", bucket.id)
pulumi.export("bucket_endpoint", exoscale_endpoint)
```

In this program, we are doing the following:
- Importing the required modules.
- Configuring the AWS (S3) provider with Exoscale's S3-compatible API endpoint. Here, you should replace `{zone}` with the Exoscale zone you are using.
- Creating a new S3 bucket with versioning enabled. Our `BucketVersioningArgs` with `enabled=True` is what turns on versioning for the bucket.
- Exporting the bucket name and endpoint as stack outputs so they can be easily retrieved.

Remember to replace the placeholder `{zone}` with your actual Exoscale zone. Also, ensure that your AWS credentials (`accessKey` and `secretKey`) are correctly configured in your environment or Pulumi configuration.

After running this Pulumi program, you will have a bucket on Exoscale where you can upload your datasets, and the versioning will keep a history of every change made to each object in the bucket, which can be vital for dataset management and recovery.