OpenTelemetry Exporters

When you first turn on OpenTelemetry logging or metrics in CPK with no additional configuration, CPK will automatically configure default exporters for the logging and metrics pipelines. Read on to learn about the default exporters for logging and metrics, as well as how to configure exporters of your choosing.

Default Logging Exporter

Without explicit exporter configuration in place, the logs that are collected are sent to the Debug Exporter, which outputs the logs to the console. Since the collector is running in a sidecar container in a Kubernetes Pod, that console output is added to the container logs which you can retrieve with the kubectl logs command. If you were running a logging-enabled PostgresCluster named hippo in the postgres-operator namespace and wanted to see your postgres, patroni, and pgbackrest logs from the primary instance pod, the commands to retrieve those logs would look like this:

PG_CLUSTER_PRIMARY_POD=$(kubectl get pod -n postgres-operator -o name -l postgres-operator.crunchydata.com/cluster=hippo,postgres-operator.crunchydata.com/role=master)
kubectl -n postgres-operator logs "${PG_CLUSTER_PRIMARY_POD}" -c collector
Copy

However, this output is not the easiest to read and is not well organized or easily filtered or searched. You will therefore almost certainly want to export your logs to a dedicated backend or service of some kind where you can more easily search and read through your logs. Details on configuring exporters can be found in the Configuring OpenTelemetry Exporters section below.

Default Metrics Exporter

Without explicit exporter configuration in place, the metrics that are collected are sent to a Prometheus Exporter that is preconfigured to work seamlessly with our Monitoring stack. However, some users will want to use their own monitoring tools to visualize and analyze their metrics or might want to further transform the metrics data in some way.

Starting in CPK v5.8.3, you can now add exporters to your metrics pipelines to export the data as you set fit. Details on configuring exporters can be found in the Configuring OpenTelemetry Exporters section below.

Configuring OpenTelemetry Exporters

The OpenTelemetry Collector that we use has a plethora of exporters built into it that should satisfy most needs.

To use an exporter, you define it in the instrumentation.config.exporters section. Fields in this section should follow the type[/name] "component identifier" format where the type is the exporter you want to use and name is the optional name you want to give this configuration. The optional name allows you to define multiple configurations of the same exporter type. For example, you could have two configurations of the otlp exporter where one is called otlp/logging and the second is called otlp/metrics. The value for each field is the configuration for that exporter. For example:

spec:
  instrumentation:
    config:
      exporters:
        otlp/logging:
          endpoint: a-standalone-collector:4317
          tls:
            insecure: true
        otlp/metrics:
          endpoint: another-collector:4317
Copy

The configuration you define will differ depending on the exporter you are using. Please follow the documentation for your chosen exporter to determine what configuration to provide. OpenTelemetry keeps a list of exporters that are specific to the "contrib" collector, along with their documentation. There are also exporters available in the "contrib" collector that we use that have their documentation in the base collector repo.

Once the exporter is configured, you lastly need to tell the collector to use the exporter in the logging and/or metrics pipeline by adding the name of the exporter to the instrumentation.logs.exporters and/or instrumentation.metrics.exporters array, respectively. For example:

spec:
  instrumentation:
    config:
      exporters:
        otlp/logging:
          endpoint: a-standalone-collector:4317
          tls:
            insecure: true
        otlp/metrics:
          endpoint: another-collector:4317
    logs:
      exporters: ['otlp/logging']
    metrics:
      exporters: ['otlp/metrics']
Copy

Note: Exporters can only be added to metrics pipelines in CPK v5.8.3 and higher.

You will find more examples of exporter configurations for commonly used logging and metrics backends in the Example Exporter Configurations section below.

Files

Some exporters might require that configuration be provided via files, such as separate config files, certificates, etc. This can be done via the instrumentation.config.files section, which allows you to project files that are in Kubernetes Secrets, ConfigMaps, etc., into the volume that is mounted into the collector container. For example, creating a Secret with the following command:

kubectl -n postgres-operator create secret tls some-otel-exporter-certs --cert=server.crt --key=server.key
Copy

And then adding the following to your instrumentation spec:

spec:
  instrumentation:
    config:
      files:
        - secret:
            name: some-otel-exporter-certs
Copy

Will result in the server.crt and server.key files being mounted in the /etc/otel-collector directory of the collector container.

Environment Variables

Some exporters might also utilize environment variables for their configuration. You can add environment variables to the collector container by defining them in the instrumentation.config.environmentVariables section. The value for any given variable can either be defined in plaintext in the manifest, or can be defined by referencing a ConfigMap or Secret. You can see the different value definition methods in the examples shown below.

One of the primary uses that we foresee for adding environment variables to the collector container is to give exporters the requisite information to authenticate to the backend services that receive the OTel data. For example, we might be using the googlecloud and awss3 exporters to send our logging data to Google's monitoring service for analysis and to an AWS S3 bucket for safe keeping. Both of these exporters can use environment variables to retrieve the credentials and connection information needed to authenticate to their respective cloud services.

spec:
  instrumentation:
    config:
      environmentVariables:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /etc/otel-collector/credentials/my-gcs-creds.json
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            configMapKeyRef:
              name: aws-credentials-configmap
              key: access-key
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials-secret
              key: secret-key
      files:
        - secret:
            name: google-credentials-secret
            items:
              - key: my-gcs-creds.json
                path: credentials/my-gcs-creds.json
Copy

As you can see, in this example the AWS specific environment variables are referencing values from a ConfigMap and a Secret, whereas the Google specific environment variable is referencing a credentials file that was mounted into the container using the instrumentation.config.files section described earlier in our documentation.

Example Exporter Configurations

This section provides example configurations for a variety of different OpenTelemetry-compatible logging and metrics services and backends.

Google Cloud

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: otel-hippo
  namespace: postgres-operator
spec:
  instrumentation:
    config:
      detectors:
        - name: gcp
      exporters:
        # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter#configuration-reference
        googlecloud: 
          log:
            default_log_name: "collector-exported-log"
            resource_filters:
              - prefix: "k8s"
              - prefix: "db"
    logs:
      exporters: ['googlecloud']
Copy

AWS Cloudwatch

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: otel-hippo
  namespace: postgres-operator
spec:
  instrumentation:
    config:
      environmentVariables:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            configMapKeyRef:
              name: aws-credentials-configmap
              key: access-key
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials-secret
              key: secret-key
      exporters:
        # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awscloudwatchlogsexporter
        awscloudwatchlogs:
          log_group_name: "otel-hippo-logs"
          log_stream_name: "otel-hippo-logging-stream"
          region: "us-east-1"
          endpoint: "https://logs.us-east-1.api.aws"
    logs:
      exporters: ['awscloudwatchlogs']
Copy

Azure Monitor

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: otel-hippo
  namespace: postgres-operator
spec:
  instrumentation:
    config:
      environmentVariables:
        - name: APPLICATIONINSIGHTS_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: azure-appinsights-secret
              key: connection-string
      exporters:
        # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuremonitorexporter
        azuremonitor: {}
    logs:
      exporters: ['azuremonitor']
    metrics:
      exporters: ['azuremonitor']
Copy

OTLP

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: otel-hippo
  namespace: postgres-operator
spec:
  instrumentation:
    config:
      exporters:
        # https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter#getting-started
        otlp: # for exporting to another collector
          endpoint: "otel-collector:4317"
          tls:
            insecure: true
    logs:
      exporters: ['otlp']
Copy