* Use String.replace() instead of replaceAll() for non-regexp replacements
When arguments do not make use of regexp features replace() is a more efficient option, especially the char-variant.
Telemetry / APM settings are renamed from "tracing.apm.{name}" to "telemetry.tracing.{name}" for tracing related settings. General APM settings are renamed to "telemetry.{name}". The old legacy settings are kept for now and applied as fallback.
This implements metrics for the threadpools.
The aim is to emit metrics for the various threadpools, the metric callback should be created when the threadpool is created, and removed before the threadpool is shutdown.
The PR also includes a test for the new metrics, and some additions to the metrics test plugin.
Finally the metric name check has been modified to allow some of the non compliant threadpools (too long, includes - )
Co-authored-by: Przemyslaw Gomulka <przemyslaw.gomulka@elastic.co>
This implements metrics for the threadpools.
The aim is to emit metrics for the various threadpools, the metric callback should be created when the threadpool is created, and removed before the threadpool is shutdown.
The PR also includes a test for the new metrics, and some additions to the metrics test plugin.
Finally the metric name check has been modified to allow some of the non compliant threadpools (too long, includes - )
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Przemyslaw Gomulka <przemyslaw.gomulka@elastic.co>
Prevent invalid and miss-configuration of APM agent using an explicit allow-list of setting keys.
Additionally, configuration defaults of APMAgentSettings are consolidated in APMJvmOptions to keep defaults in a single location.
(ES-6916)
This commit adds a minimum metric name validation which checks:
metric name starts with es. prefix
metric name is using . as a separator of elements
metric name is using characters from a white list
validate min number of elements = 3 elements ( prefix, group and the suffix name)
validate max number of elements and max characters per element
validate the suffix element in a metric name to be from the enumerated allow list
It also modifies existing metric names to adhere to those rules
`Tracer.startTrace(ThreadContext threadContext, Traceable traceable, String name, Map<String, Object> attributes)` takes in a `ThreadContext` which creates a dependency on `server`. This change adds a new interface, `ThreadContext` with those methods required from `ThreadContext` and uses that as the parameter to `startTrace`.
The methods in the `TraceContext` interface are
```
<T> T getTransient(String key);
void putTransient(String key, Object value);
String getHeader(String key);
void putHeader(String key, String value);
```
which are needed for getting the parent context, the remote headers, the x-opaque-id and setting the remote headers for a trace.
This is an ugly but functional way to remove a dependency on server to be able to move telemetry in to a library for static metric registration.
`Tracer.startTrace` has overloaded methods for accepting `RestRequest` and `Task.` In order to implement static registration, we want to move APM into a library. Accepting these two types as parameters creates a dependency on server that needs to be removed for telemetry into lib.
This change removes the overloaded method in favor of the `Traceable` interface allows `Tracer` to get the `SpanId`.
The `ThreadContext` argument also creates a dependency on server, but that is more involved and will be addressed in another change.
This commit adds a log message at debug level when a metric is registered. This allows to start elasticsearch with a level changed for the apm package and get a list of metric names.
It should be useful when trying to came up with a new meteric name the command to run
./gradlew run -Dtests.es.logger.org.elasticsearch.telemetry.apm=debug
Avoid null pointer error when an observation is null [0].
* To get the tests working, replace the this-escape buildInstrument with a Builder.
* Removes locking around close
{Double,Long}AsyncCounters were not in the registrar list so they were not included when switching providers.
Other changes:
* Moved DoubleAsyncCounter's register and get to match the order of the rest of APMMeterRegistry.
* Fixed RecordingOtelMeter's Histogram and DoubleUpDownCounter
* Fixed String format in RecordingMeter's asserts.
apm-data module should be more explicit that it is for the apm server usage. It is confusing when starting up ES to see APM is disabled log line. Especially since we also have :modules:apm which is meant to be for sending apm metrics and traces
This commit rephrases the log messages and rename APMPlugin class to mention APM Server.
asynchronous counters (available in otel) are instruments which allow to update the counter's value with the callback style api. The callback has to report always the latest value. Upon restart the value might go back to 0, but in the backend apm server will know that since the value is always increasing.
The test was checking for the wrong meter instance, it should be checking
for the updated meter, `noopMeter`.
Removed the assertBusy(() -> assertThat...) which was hiding the problem.
The goal of the test is that we serialize access to registration and the
provider. A nice way to do this is wait for the contending threads to
join (with the added benefit of avoiding thread leaks). After the threads
join, we check that we have the expected state.
Fixes: #101725
Open Telemetry supports instrument names lengths 63 characters or
less for versions before 1.30.
Reject those names early to avoid runtime failures that hit the log.
Refs: #101679
Replace synchronous gauges with gauges that accept an observer which is called once per reporting interval.
Calling code may register one observer at a time. If the observer is null, there is no observation in that reporting interval.
Adds an implementation of Otel's Meter that records all instrument calls made through the open telemetry interface.
This allows the registry to avoid mocking out otel in testing.
Updates the GaugeAdapterTests to use the recording meter.
This splits out the registry and the service, which makes testing easier and removes much of the delegation from the old `APMMeter` to `Instruments` (now renamed `APMMeterRegistry`).
APMMeterService takes care of the lifecycle and APMMeterRegistry holds the instruments.
OTEL gauges should follow the callback model otherwise they will not be sent by
apm java agent. (or use BatchCallback)
This commit changes the gagues creation model to return Observable*Gauge
and uses AtomicLong/Double to store current value which will be polled when
metrics are exported (and callback is called)
`SpanId` is used when explicitly closing the trace in `executeQueryPhase` to avoid double closing the associated task.
`doPrivileged` avoids hitting `java.lang.UnsupportedOperationException: Cannot define class using reflection: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")` when classes are sometimes injected while switching spans.
Removed `default Releasable withScope(Task task)` from the Tracer API because it automatically created a span id and, in one of the three uses, that SpanId was necessary to close the span.
Fixes: #100072
The latest version contains a fix to allow sending metrics to APM server. also adds a apm agent jvm options
"enable_experimental_instrumentations", "true"
which is required to enable the otel-metrics-instrumentation.
relates https://github.com/elastic/elasticsearch/pull/99832
Adds Metering instrument interfaces and adapter implementations for opentelemetry instrument types:
* Gauge - a single number that can go up or down
* Histogram - bucketed samples
* Counter - monotonically increasing summed value
* UpDownCounter - summed value that may decrease
Supports both Long* and Double* versions of the instruments.
Instruments can be registered and retrieved by name through APMMeter which is available via the APMTelemetryProvider.
The metering provider starts as the open telemetry noop provider.
`telemetry.metrics.enabled` turns on metering.
in order to avoid adding yet anther parameter to createComponents
a Tracer interface is replaced with TelemetryProvider.
this allows to get both Tracer and Metric (in the future) interfaces
with the support of metrics the TracerPlugin name is no longer adequate. Renaming this to TelemetryPlugin.
Also introducing TelemetryProvider interface. While it is only used in Node.java at the moment to fetch Tracer instance, it is intended to be used in Plugin::createComponents (to be done in separate commit due to
the broad scope of this method)
This will allow for plugins to get access to both Tracer and Metric interfaces
without the need to add yet another argument to createComponents
Also adding internal subpackage in module/apm so that it is more obvious
which packages are not exported
