Distributed tracing

Distributed tracing is an Evolving feature in PingIDM. It is subject to change without notice, even in a minor or maintenance release.

Distributed tracing helps you monitor and observe system requests as they flow through IDM. A request is tracked and analyzed with a unique identifier (ID) used to troubleshoot requests that show errors or performance issues. This unique ID remains with a transaction as it interacts with microservices, containers, and infrastructure.

When a user interacts with the Ping Identity Platform, the request can travel through multiple services before it completes. Distributed tracing lets you monitor the request flow through the Ping Identity Platform.

Tracing benefits

Distributed tracing makes it easier to:

Provide a single view of a request’s journey
Locate bottlenecks and errors
Identify slow services
Optimize application performance and reduce debugging time
Improve the end-user experience

IDM and OpenTelemetry

IDM supports the OpenTelemetry framework (OTEL) for collecting distributed tracing data.

OpenTelemetry handles:

Generation
Collection
Management
Export of telemetry

OpenTelemetry does not handle telemetry storage and visualization. You can use a trace collector, such as Jaeger, to collect and visualize trace data.

Request types supported in IDM

IDM supports distributed tracing for the following request types:

Incoming HTTP requests
Outgoing HTTP requests to PingAM (Ping Identity Platform deployments only)
Outgoing LDAP requests

These requests are searchable and identifiable for the following LDAP operations:
- ADD
- MODIFY
- SEARCH
- DELETE
- BIND
Outgoing scripting HTTP requests

Outgoing scripting HTTP requests must use the openidm.action function to make an external call using IDM’s external REST service.

Understand a trace object

A trace represents the path of a request through an application. A trace is made up of one or more spans.

Each span includes the following elements:

traceId: Representing the trace that the span is a part of
spanId: A unique ID for that span
parentSpanId: The ID of the originating request

The root span indicates the start and end of an entire operation. The parentSpanId of the root span is null because the root span isn’t part of an existing trace.

Subsequent spans in the trace have their own unique spanId. The traceId is the same as that of the root span. The parentId matches the spanId of the root span.

Learn more in Traces in the OpenTelemetry documentation.

Enable distributed tracing

Distributed tracing is disabled by default.

To enable distributed tracing:

In the /path/to/openidm directory, create a /trace directory, for example, where you can place the tracing configuration file.

The configuration file is not required to be in the /conf directory. You can place this file in any location readable by IDM.
In the /trace directory, for example, create an OTEL configuration JSON file with the following information and set "enabled": true:
```
{
  "tracing": {
    "enabled": true,
    "exporter": {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4318/v1/traces"
      }
    }
  }
}
```
You can find information on additional configuration properties in Configure distributed tracing.

If the content of the configuration file is invalid JSON, distributed tracing remains disabled, even if you set "enabled": true.
IDM uses the environment variable OPENIDM_TRACING_CONFIG_PATH. Set this environment variable to point to the configuration file in /path/to/openidm or to any location readable by IDM, for example:
```
OPENIDM_TRACING_CONFIG_PATH=/path/to/openidm/tracing/tracing.json
```
After you create the OTEL configuration file in the directory that you choose and point the environment variable to the OTEL configuration file, start IDM.

Starting IDM launches a service that reads the OTEL configuration file and monitors it for changes to perform runtime updates to the distributed tracing service.

Disable distributed tracing

To disable distributed tracing:

In the OTEL configuration file, set "enabled": false.

After you’ve changed the configuration file, the service that detects configuration changes relaunches the distributed tracing service in a disabled state.

Configure distributed tracing

The Ping Identity Platform supports a common set of configuration properties for OpenTelemetry support.

The stability of this configuration interface is classified as Evolving.
Any changes to the configuration require a server restart.

To change the default OpenTelemetry configuration, add the configuration properties to your configuration file, for example:

{
  "tracing": {
    "enabled": true,
    "resourceAttributes": {
      "service.instance.id": "idm-server-1"
    },
    "exporter": {
      "config": {
        "headers": {
          "X-CUSTOM-HEADER": "custom-value"
        }
      }
    },
    "spanLimits": {
      "maxNumberOfAttributesPerEvent": 128
    }
  }
}

Distributed tracing configuration properties

enabled: boolean, optional

Set to true to enable OpenTelemetry tracing.

Default: false

resourceAttributes: object, optional

A map of additional resource attributes for processing traces. Find more information in the OpenTelemetry documentation on Semantic Attributes with SDK-provided Default Value.

For example, if there are multiple Ping Identity Platform instances in a deployment, you could set the "service.instance.id" resource attribute differently for each one to distinguish between them:

{
  "resourceAttributes": {
    "service.instance.id": "idm-server-1"
  }
}

exporter: object, optional

Configuration for the exporter, which pushes traces to the OpenTelemetry service:

type: string, optional

Set to otlp for OpenTelemetry Protocol (OTLP) support. This is currently the only supported protocol.

Default: otlp

config: object, optional

Endpoint and timeout configuration:

compressionMethod: enumeration, optional

Method used to compress trace data; either gzip or none.

Default: gzip
connectionTimeout: duration, optional

Time out a connection to the endpoint after this duration.

Default: 10 seconds.
endpoint: string, optional

The endpoint to publish traces to.

For HTTPS, IDM trusts the default JVM CAs. To override this, set the -Djavax.net.ssl.trustStore and associated JVM settings when starting IDM. Learn more about the optional settings in the Java Secure Socket Extension (JSSE) Reference Guide.

IDM doesn’t support TLS configuration for the tracing endpoint at this time.

Default: http://localhost:4318/v1/traces
headers: object, optional

Map of additional headers to include in the export span request.

The following example sets the authorization header, Authorization: Bearer ${bearer.token}:
```
"headers": { "Authorization": "Bearer ${bearer.token}" }
```
retries: object, optional

Defines a retry policy for the export span requests.

Default: Enabled
- backoffMultiplier: number, optional Multiplier for the backoff wait time before retries.
  
  Default: 1.5
- enabled: boolean, optional
  
  Retry failed requests.
  
  Default: true
- initialBackoff: duration, optional
  
  How long to wait before the first retry.
  
  Default: 1 second
- maxAttempts: number, optional
  
  Maximum number of retries.
  
  Default: 5
- maxBackoff: duration, optional
  
  Maximum wait time between retries.
  
  Default: 5 seconds
timeout: duration, optional

Time out a request to publish data to the endpoint after this duration.

Default: 10 seconds.

batch: object, optional

Enable and configure batch processing for trace data.

compressionMethod: enumeration, optional

Method used to compress trace data; either gzip or none.

Default: gzip
enabled: boolean, optional

Leave batch processing enabled in deployment.

Default: true
exporterTimeout: duration, optional

Time out a data exporter after this duration.

Default: 30 seconds
exportUnsampledSpans: boolean, optional

Whether to report on unsampled spans.

Default: false
maxExportBatchSize: number, optional

Maximum number of spans in a batch.

Default: 512
maxQueueSize: number, optional

Maximum number of spans to queue before dropping them.

Default: 2048
scheduleDelay: duration, optional

Maximum interval between sending batches of trace data.

Default: 50 seconds

sampler: object, optional

Configuration for sampling spans.

ratio: number, optional

For ratio-based types, a percentage of spans to process.

Default: 50 (percent)

type: string, optional

The sampler strategy to use is one of the following:

alwaysOn: Send every span for processing.
alwaysOff: Never send any span for processing.
traceIdRatio: Sample the specified ratio of spans deterministically based on the trace IDs of the spans.
parentBasedAlwaysOn: Always send the span for processing if the parent span was sampled. (Default)
parentBasedAlwaysOff: Never send the span for processing if the parent span was sampled.
parentBasedTraceIdRatio: Send the specified ratio of spans for processing if the parent span was sampled.

spanLimits: object, optional

Configuration for limits enforced when recording spans.

maxNumberOfAttributes: number, optional: The maximum number of attributes per span.

Default: 128
maxNumberOfAttributesPerEvent: number, optional: The maximum number of metadata items (attributes) attached to a span per event. An event is an annotation to span at a particular, meaningful point in time during the span’s duration.

Default: 128
maxNumberOfAttributesPerLink: number, optional: The maximum number of attributes per link.

Default: 128
maxNumberOfEvents: number, optional: The maximum number of events per span.

Default: 128
maxNumberOfLinks: number, optional: The maximum number of links per span. Links associate the current span with one or more other spans.

Default: 128

Visualize traces with a trace collector

Trace collectors work alongside IDM, allowing the service to offload data quickly. A collector manages retries, batching, encryption, and data filtering.

You can use a trace collector to collect trace data from the OpenTelemetry Collector and visualize that data.

Example visualization with a trace collector

This example assumes a local IDM deployment.

Start the trace collector.

Send a request against IDM:

curl --location 'http://localhost:8080/openidm/schema/managed/user' \
--header 'x-openidm-username: openidm-admin' \
--header 'x-openidm-password: openidm-admin'

Query the trace collector’s localhost URL for the request:

 curl --location 'http://localhost:portnumber/api/traces?service=idm'

You should receive a query response similar to the following:

{
  "data": [
    {
      "traceID": "09cb4130f4c8803011b3996f8bda6b8c",
      "spans": [
        {
          "traceID": "09cb4130f4c8803011b3996f8bda6b8c",
          "spanID": "e49cd1d70d65502d",
          "operationName": "GET /openidm",
          "references": [],
          "startTime": 1741892475358333,
          "duration": 165774,
          "tags": [
            {
              "key": "forgerock.transaction_id",
              "type": "string",
              "value": "c246e9ea-e596-4a99-a0eb-5072806cefe3-890"
            },
            {
              "key": "http.request.method",
              "type": "string",
              "value": "GET"
            },
            {
              "key": "network.protocol.name",
              "type": "string",
              "value": "http"
            },
            {
              "key": "otel.scope.name",
              "type": "string",
              "value": "idm"
            },
            {
              "key": "span.kind",
              "type": "string",
              "value": "server"
            },
            {
              "key": "url.full",
              "type": "string",
              "value": "http://localhost:8080/openidm/schema/managed/user"
            },
            {
              "key": "url.path",
              "type": "string",
              "value": "/openidm/schema/managed/user"
            }
          ],
          "logs": [],
          "processID": "p1",
          "warnings": null
        }
      ],
      ...
    }
  ]
}