How Do I Collect Application Logs in Kubernetes With Elastic APM Agents?

In this guide, we will set up a Kubernetes cluster to collect application logs using Elastic APM agents. Elastic APM agents will be deployed as a DaemonSet to ensure that they run on every node in the cluster. We will also create a namespace and necessary Kubernetes resources for the Elastic APM setup.

Key Points

Create a Kubernetes Namespace for Elastic APM.
Deploy Elastic APM agents as a DaemonSet.
Configure the necessary resources for Elastic APM to collect logs.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a Kubernetes Namespace for Elastic APM
const apmNamespace = new k8s.core.v1.Namespace("apm-namespace", {
    metadata: {
        name: "elastic-apm",
    },
});

// Define the Elastic APM DaemonSet
const apmDaemonSet = new k8s.apps.v1.DaemonSet("apm-daemonset", {
    metadata: {
        namespace: apmNamespace.metadata.name,
        name: "elastic-apm-agent",
    },
    spec: {
        selector: {
            matchLabels: {
                app: "elastic-apm-agent",
            },
        },
        template: {
            metadata: {
                labels: {
                    app: "elastic-apm-agent",
                },
            },
            spec: {
                containers: [{
                    name: "elastic-apm-agent",
                    image: "docker.elastic.co/apm/apm-server:7.10.0",
                    ports: [{
                        containerPort: 8200,
                        name: "apm-server",
                    }],
                    env: [
                        {
                            name: "ELASTIC_APM_SERVER_URL",
                            value: "http://apm-server.elastic-apm.svc:8200",
                        },
                        {
                            name: "ELASTIC_APM_SECRET_TOKEN",
                            value: "your-secret-token",
                        },
                    ],
                }],
            },
        },
    },
});

// Create a Service for Elastic APM Server
const apmService = new k8s.core.v1.Service("apm-service", {
    metadata: {
        namespace: apmNamespace.metadata.name,
        name: "apm-server",
    },
    spec: {
        selector: {
            app: "elastic-apm-agent",
        },
        ports: [{
            protocol: "TCP",
            port: 8200,
            targetPort: "apm-server",
        }],
    },
});

// Export the URLs of the Elastic APM Server
export const apmServerUrl = apmService.status.loadBalancer.ingress[0].hostname;

Summary

In this guide, we created a Kubernetes namespace for Elastic APM, deployed Elastic APM agents as a DaemonSet, and configured a service to expose the APM server. This setup ensures that Elastic APM agents run on every node in the cluster, collecting application logs and sending them to the Elastic APM server for analysis.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.