IMPORTANT: Developer documentation for the current main branch. This content is unreleased and may change before the next Polaris release. For stable user documentation, see the latest release docs.

Telemetry

Metrics🔗

Metrics are published using Micrometer; they are available from Polaris’s management interface (port 8182 by default) under the path /q/metrics. For example, if the server is running on localhost, the metrics can be accessed via http://localhost:8182/q/metrics.

Metrics can be scraped by Prometheus or any compatible metrics scraping server. See: Prometheus for more information.

Additional tags can be added to the metrics by setting the polaris.metrics.tags.* property. Each tag is a key-value pair, where the key is the tag name and the value is the tag value. For example, to add a tag environment=prod to all metrics, set polaris.metrics.tags.environment=prod. Many tags can be added, such as below:

polaris.metrics.tags.service=polaris
polaris.metrics.tags.environment=prod
polaris.metrics.tags.region=us-west-2

Note that by default Polaris adds one tag: application=Polaris. You can override this tag by setting the polaris.metrics.tags.application=<new-value> property.

Realm ID Tag🔗

Polaris can add the realm ID as a tag to all API and HTTP request metrics. This is disabled by default to prevent high cardinality issues, but can be enabled by setting the following properties:

polaris.metrics.realm-id-tag.enable-in-api-metrics=true
polaris.metrics.realm-id-tag.enable-in-http-metrics=true

You should be particularly careful when enabling the realm ID tag in HTTP request metrics, as these metrics typically have a much higher cardinality than API request metrics.

In order to prevent the number of tags from growing indefinitely and causing performance issues or crashing the server, the number of unique realm IDs in HTTP request metrics is limited to 100 by default. If the number of unique realm IDs exceeds this value, a warning will be logged and no more HTTP request metrics will be recorded. This threshold can be changed by setting the polaris.metrics.realm-id-tag.http-metrics-max-cardinality property.

HTTP Request Histogram Buckets🔗

By default, the HTTP server request duration timer exports only the count, sum, and max series. These support average and worst-case latency but cannot be used to compute aggregable percentiles (e.g. p95, p99) across instances.

To publish histogram buckets for HTTP server request duration, configure the Service Level Objective (SLO) boundaries to publish:

polaris.metrics.http-server-requests.histogram-slos=10ms,50ms,100ms,1s,5s

These buckets can be used by Prometheus histogram_quantile to estimate p95/p99; accuracy depends on the configured bucket boundaries.

NoSQL Distributed Cache Invalidation🔗

The NoSQL distributed cache invalidation sender and receiver publish the following Micrometer meters when a MeterRegistry is available. Names below share the prefix polaris.cache.invalidation.; Prometheus converts dots to underscores and adds the usual counter, summary, and duration suffixes.

Meter suffix	Type	Labels	Meaning
`entries`	Counter	`stage=enqueued,dispatched`, `type=obj,ref`	Entries accepted into the sender queue, or entries whose local eviction handler returned successfully on the receiver.
`skipped`	Counter	`reason=no_discovery,no_token,no_peers`	Entries skipped before any peer delivery attempt. When neither discovery nor a token is configured, `no_discovery` takes precedence.
`batch.size`	Distribution summary	None	Number of entries in a submitted batch, recorded once before fan-out, including batches with no peers.
`queue.depth`	Gauge	None	Entries waiting for submission; excludes outstanding HTTP requests.
`send.inflight`	Gauge	None	Outstanding peer HTTP requests, including waiting for the response body, until completion, failure, or timeout.
`send.duration`	Timer	`outcome=success,timeout,transport_error,http_rejected`	Duration and count of delivery attempts, once per destination peer. Only HTTP 200 and 204 count as success.
`receive.duration`	Timer	`outcome=processed,self,invalid_token,invalid_content_type,invalid_payload,handler_error`	Duration and count of requests handled by the receiver, including parsing and local dispatch.
`discovery.duration`	Timer	`outcome=success,failure`	Duration and count of peer discovery attempts, including initial discovery.
`discovery.peers`	Gauge	None	Usable peers from the last successful discovery after local-address filtering. Failures retain the previous value.
`discovery.last_success.timestamp`	Gauge	None	Unix timestamp in seconds of the last successful discovery, or zero before any success.

These meters use fixed label values and do not include peer addresses, sender instance IDs, realms, object IDs, or reference names. Use scrape target labels to locate an affected instance and logs to identify individual failing peers. Histogram buckets and precomputed percentiles are disabled by default. Timers provide event counts, total durations, and maximum durations without separate request counters.

Monitor queue depth together with in-flight requests and send outcomes: the sender drains batches without waiting for HTTP completion, so an empty queue does not establish delivery health. Zero peers can be normal in a single-instance deployment. The last-success timestamp helps distinguish an empty discovery result from a stale retained peer list.

Sender success describes HTTP delivery, while receiver outcomes describe processing. In particular, malformed JSON currently produces an empty batch and an HTTP 204 response; the receiver records invalid_payload rather than processed. Unhandled exceptions during request processing, including handler and response-writing failures, record handler_error and continue to propagate as before. Successfully dispatched entries preceding a handler failure remain counted. A dispatched entry does not establish that the entry was present in the cache, and these metrics do not prove cluster-wide cache freshness. Failed asynchronous sends are not retried by this mechanism.

Traces🔗

Traces are published using OpenTelemetry.

By default OpenTelemetry is disabled in Polaris, because there is no reasonable default for the collector endpoint for all cases.

To enable OpenTelemetry and publish traces for Polaris set quarkus.otel.sdk.disabled=false and configure a valid collector endpoint URL with http:// or https:// as the server property quarkus.otel.exporter.otlp.traces.endpoint.

If these properties are not set, the server will not publish traces.

The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317), e.g. “http://otlp-collector:4317”.

By default, Polaris adds a few attributes to the OpenTelemetry Resource to identify the server, and notably:

service.name: set to Apache Polaris Server;
service.version: set to the Polaris version.

You can override the default resource attributes or add additional ones by setting the quarkus.otel.resource.attributes property.

This property expects a comma-separated list of key-value pairs, where the key is the attribute name and the value is the attribute value. For example, to change the service name to Polaris and add an attribute deployment.environment=dev, set the following property:

quarkus.otel.resource.attributes=service.name=Polaris,deployment.environment=dev

The alternative syntax below can also be used:

quarkus.otel.resource.attributes[0]=service.name=Polaris
quarkus.otel.resource.attributes[1]=deployment.environment=dev

Finally, two additional span attributes are added to all request parent spans:

polaris.request.id: The unique identifier of the request, if set by the caller through the X-Request-ID header.
polaris.realm: The unique identifier of the realm. Always set (unless the request failed because of a realm resolution error).

Troubleshooting Traces🔗

If the server is unable to publish traces, check first for a log warning message like the following:

SEVERE [io.ope.exp.int.grp.OkHttpGrpcExporter] (OkHttp http://localhost:4317/...) Failed to export spans.
The request could not be executed. Full error message: Failed to connect to localhost/0:0:0:0:0:0:0:1:4317

This means that the server is unable to connect to the collector. Check that the collector is running and that the URL is correct.

Logging🔗

Polaris relies on Quarkus for logging.

By default, logs are written to the console and to a file located in the ./logs directory. The log file is rotated daily and compressed. The maximum size of the log file is 10MB, and the maximum number of backup files is 14.

JSON logging can be enabled by setting the quarkus.log.console.json.enabled and quarkus.log.file.json.enabled properties to true. By default, JSON logging is disabled.

The log level can be set for the entire application or for specific packages. The default log level is INFO. To set the log level for the entire application, use the quarkus.log.level property.

To set the log level for a specific package, use the quarkus.log.category."package-name".level, where package-name is the name of the package. For example, the package io.smallrye.config has a useful logger to help debugging configuration issues; but it needs to be set to the DEBUG level. This can be done by setting the following property:

quarkus.log.category."io.smallrye.config".level=DEBUG

The log message format for both console and file output is highly configurable. The default format is:

%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n

Refer to the Logging format guide for more information on placeholders and how to customize the log message format.

MDC Logging🔗

Polaris uses Mapped Diagnostic Context (MDC) to enrich log messages with additional context. The following MDC keys are available:

requestId: The unique identifier of the request, if set by the caller through the X-Request-ID header.
realmId: The unique identifier of the realm. Always set.
traceId: The unique identifier of the trace. Present if tracing is enabled and the message is originating from a traced context.
parentId: The unique identifier of the parent span. Present if tracing is enabled and the message is originating from a traced context.
spanId: The unique identifier of the span. Present if tracing is enabled and the message is originating from a traced context.
sampled: Whether the trace has been sampled. Present if tracing is enabled and the message is originating from a traced context.

Other MDC keys can be added by setting the polaris.log.mdc.* property. Each property is a key-value pair, where the key is the MDC key name and the value is the MDC key value. For example, to add the MDC keys environment=prod and region=us-west-2 to all log messages, set the following properties:

polaris.log.mdc.environment=prod
polaris.log.mdc.region=us-west-2

MDC context is propagated across threads, including in TaskExecutor threads.

Links🔗

Visit Using Polaris with telemetry tools to see sample Polaris config with Prometheus and Jaeger.