Observability

Quartz publishes traces and metrics through System.Diagnostics: an ActivitySource and a Meter, both named Quartz. No Quartz package is needed; install only the collector. Subscribe with the public constants:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(QuartzInstrumentation.ActivitySourceName)
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddMeter(QuartzInstrumentation.MeterName)
        .AddOtlpExporter());

QuartzInstrumentation is in the Quartz.Diagnostics namespace. Both constants are "Quartz", so an existing AddSource("Quartz") keeps working.

Traces

The thirty-four store operations are those that change something or hand work to the scheduler:

AcquireNextTriggers, TriggersFired, TriggeredJobComplete, ReleaseAcquiredTrigger, ScheduleJob, ScheduleJobs, AddJob, AddTrigger, StoreTrigger, AddCalendar, DeleteJob, DeleteJobs, DeleteTrigger, DeleteTriggers, DeleteCalendar, ReplaceTrigger, UpdateTriggerDetails, PauseTrigger, PauseTriggers, PauseTriggerGroups, PauseJob, PauseJobs, PauseJobGroups, ResumeTrigger, ResumeTriggers, ResumeTriggerGroups, ResumeJob, ResumeJobs, ResumeJobGroups, PauseAll, ResumeAll, ResetTriggerFromErrorState, ResetTriggersFromErrorState, Clear.

• Pausing by key and pausing by group matcher are separate spans. The first moves the given triggers; the second records that a group is paused and also catches triggers added to it later.
• Reads (GetJob, Exists, the Query* members) are not spans, so a dashboard listing triggers does not produce a span per page.

How the spans are shaped into traces

• A firing is its own trace. Quartz.Job.Execute and Quartz.Job.Veto are always trace roots and never take the ambient Activity as parent. What the job traces (an HttpClient call, an EF Core query, your own spans) are children of the firing.
• A store span belongs to the caller. scheduler.ScheduleJob(…) inside an HTTP request puts Quartz.JobStore.ScheduleJob in that request's trace.
• The scheduler loop's store calls are roots. AcquireNextTriggers, TriggersFired and TriggeredJobComplete belong to no request, and the loop runs for the life of the process.

Span names are constants on Quartz.Diagnostics.OperationName. Attributes are namespaced quartz.* and are constants on Quartz.Diagnostics.ActivityTags:

quartz.fire.instance.id identifies one firing; it is the id IScheduler.InterruptFireInstance takes. quartz.job.result is the value the duration histogram and the execution history record; see Reading the numbers.

Failed spans

A span ends in Error only when something threw:

quartz.job.result is failed for both kinds of failed run.

Exceptions as span events

OpenTelemetry deprecated span events, exception events included, in favour of exceptions as logs. Quartz records them by default. To stop:

services.AddQuartz(q => q.ConfigureScheduler(options => options.RecordExceptionSpanEvents = false));

• The span keeps its status, description and error.type either way.
• The option wins over the variable. The variable is read when the scheduler is built; its value is case-insensitive. The variable is OpenTelemetry's semantic-convention opt-in.
• A job's exception is logged through ILogger while its span is current, so an OpenTelemetry log exporter ties it to the trace. A store operation's exception goes to its caller.

Linking a firing to what scheduled it

When the scheduling call runs inside an Activity, the scheduler stores its W3C trace context on the trigger, under the reserved keys SchedulerConstants.TraceParent and SchedulerConstants.TraceState. The firing's Quartz.Job.Execute span, and a Quartz.Job.Veto span, carry an ActivityLink back to it.

• The firing may run much later and on another node. It is linked, not parented, so no trace stays open across the wait. This is OpenTelemetry's shape for an asynchronous producer and consumer.
• Nothing needs configuring. An HTTP API request gets it automatically, because the endpoint runs inside ASP.NET Core's server span.
• Cost: two string entries on each trigger's data map, visible in MergedJobDataMap, the dashboard and GET /triggers.

Turn it off with:

q.ConfigureScheduler(options => options.PropagateTraceContext = false);

Metrics

Eleven instruments, all on the Quartz meter. Every measurement carries quartz.scheduler.name and quartz.scheduler.id: the name identifies the scheduler, the id its node in a cluster.

Each instrument name is a const string on QuartzInstrumentation.Instruments; for example QuartzInstrumentation.Instruments.JobExecutionDuration is quartz.job.execution.duration. Tests hold the constants equal to what the meter emits and snapshot the name, kind, unit and description of every instrument.

¹ Only when the trigger names an execution group. A trigger in no group has no such attribute (not an empty one), so the two are separate series. ² Only when the operation failed. The value is the exception type's fully-qualified name. ³ Always. succeeded, failed, cancelled or skipped: the run's JobRunResult, lower-cased, as the execution history records it. Constant: ActivityTags.JobResult. ⁴ On the ADO.NET store, from 4.6: TRIGGER_ACCESS or none, the lock the round ran under. The in-memory store has no such attribute, so its series keep their shape. ⁵ Only on a round that took the lock: why. The values are the table in Lock-free firing rounds. Constant: ActivityTags.JobStoreLockReason.

• quartz.trigger.retry counts retries scheduled, not attempts configured: an unused policy adds nothing.
• quartz.trigger.retries_exhausted counts once per occurrence that gave up, never per attempt. It has the same attributes as quartz.trigger.retry, so the two divide.
• quartz.cluster.checkin.duration records each attempt: a retried check-in is two measurements.
• quartz.jobstore.lock.wait.duration does not record a re-entrant acquisition, which waited for nothing.
• quartz.jobstore.operation is one of the thirty-four Quartz.JobStore.* names above, so one string finds a slow operation in traces and metrics. The histogram count is the number of each operation; the error.type-tagged part is the failures.
• quartz.jobstore.lock is the LOCK_NAME value: TRIGGER_ACCESS (taken by scheduling operations) or STATE_ACCESS (taken by cluster check-in). On the two acquisition instruments it is also none, for a firing round that took no lock.

The lock-wait histogram is the one instrument that reports during a stall. A lock statement blocked behind another session returns and throws nothing, so no operation or failure is recorded. The matching warning is event 3716 in Log Events; the case is A Lock Held by a Connection That Is Gone.

The two cluster instruments and the lock-wait histogram come from the ADO.NET store, the only clustered one. The other eight work with any store.

Reading the numbers

• There is no execution counter and no error counter. Executions are the count of quartz.job.execution.duration; group it by quartz.job.result for each result.

• error.type names what a run threw, so it is on thrown failures only:

  quartz.job.result	error.type
  succeeded, skipped, cancelled	absent
  failed, the job threw	the exception type
  failed, from JobRunReport.Failed	absent

• error.type is the OpenTelemetry convention. Neither it nor quartz.job.result is on quartz.job.execution.active: an up-down counter's increment and decrement need identical attributes, and the result is not known at start.

The meter is created from the container's IMeterFactory when there is one. AddMetrics(), and so every generic-host application, registers it. This keeps measurements of two schedulers, or two hosts in one test process, apart.

OpenTelemetry.Instrumentation.Quartz

OpenTelemetry.Instrumentation.Quartz is the OpenTelemetry community's instrumentation library, written for 3.x.

In the OpenTelemetry SDK a "legacy source" is an activity with no ActivitySource. 4.x activities have one, and 4.x writes nothing to a DiagnosticListener, so neither half of the subscription matches.

Replace it with the two calls at the top of this page, AddSource(QuartzInstrumentation.ActivitySourceName) and AddMeter(QuartzInstrumentation.MeterName), and remove the package:

- builder.Services.AddOpenTelemetry()
-     .WithTracing(tracing => tracing.AddQuartzInstrumentation());
+ builder.Services.AddOpenTelemetry()
+     .WithTracing(tracing => tracing.AddSource(QuartzInstrumentation.ActivitySourceName))
+     .WithMetrics(metrics => metrics.AddMeter(QuartzInstrumentation.MeterName));

- <PackageReference Include="OpenTelemetry.Instrumentation.Quartz" Version="1.*" />

• Lost: the package's QuartzInstrumentationOptions.TracedOperations filter. Subscribing directly records both Quartz.Job.Execute and Quartz.Job.Veto; drop one with an OpenTelemetry processor or sampler.
• Gained: everything 4.0 added, including the store spans and all eleven instruments.

Older packages

Quartz.OpenTelemetry.Instrumentation is obsolete and not part of 4.x. Subscribe to the activity source directly, as at the top of this page.

Coming from Quartz.OpenTracing

Quartz.OpenTracing is not part of 4.x and has no 4.x release; the OpenTracing project is archived. It used the DiagnosticSource events that 4.x replaced with System.Diagnostics.Activity. Replace an AddQuartzOpenTracing call with the OpenTelemetry setup at the top of this page.

Logging

Quartz logs through Microsoft.Extensions.Logging, using the application's configuration; nothing to wire up. Types no container builds (a listener or trigger you constructed, the static helpers, the jobs in Quartz.Jobs) use the factory set with Quartz.Diagnostics.LogProvider.SetLogProvider(loggerFactory).

The names are the span attributes above. Set IncludeScopes = true on the OpenTelemetry logger to export them.

For job and trigger history as log entries rather than traces, use the history plugins.