Observability

Quartz publishes traces and metrics through System.Diagnostics — an ActivitySource and a Meter, both named Quartz — so nothing has to be installed to make a scheduler observable. What is installed is whatever collects them.

The two names are public constants, so they can be subscribed to without typing a string twice:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(QuartzInstrumentation.ActivitySourceName)
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddMeter(QuartzInstrumentation.MeterName)
        .AddOtlpExporter());

QuartzInstrumentation is in the Quartz.Diagnostics namespace, and both constants are "Quartz", so an existing AddSource("Quartz") keeps working.

Traces

Three kinds of span: the execution, the veto, and one per job store round trip.

The thirty-three store operations are the ones that change something or hand work to the scheduler:

AcquireNextTriggers, TriggersFired, TriggeredJobComplete, ReleaseAcquiredTrigger, ScheduleJob, ScheduleJobs, AddJob, AddTrigger, AddCalendar, DeleteJob, DeleteJobs, DeleteTrigger, DeleteTriggers, DeleteCalendar, ReplaceTrigger, UpdateTriggerDetails, PauseTrigger, PauseTriggers, PauseTriggerGroups, PauseJob, PauseJobs, PauseJobGroups, ResumeTrigger, ResumeTriggers, ResumeTriggerGroups, ResumeJob, ResumeJobs, ResumeJobGroups, PauseAll, ResumeAll, ResetTriggerFromErrorState, ResetTriggersFromErrorState, Clear.

Pausing by key and pausing by group matcher are separate spans, because they are separate operations: one moves the triggers it was given, the other records that a group is paused and catches what is added to it afterwards.

Reads — GetJob, Exists, the Query* members — are deliberately not spans. A dashboard listing triggers would otherwise produce a span per page.

How the spans are shaped into traces

A firing is a trace of its own. Quartz.Job.Execute and Quartz.Job.Veto are always trace roots: they never take the ambient Activity as a parent, whatever was current on the worker that ran them. Anything the job traces — an HttpClient call, an EF Core query, a span of your own — is a child of the firing, and that is the whole of the trace.

A store span belongs to whoever made the call. scheduler.ScheduleJob(…) inside an HTTP request puts Quartz.JobStore.ScheduleJob in that request's trace, which is what you want: it is a round trip the request paid for. The store calls the scheduler's own loop makes — AcquireNextTriggers, TriggersFired, TriggeredJobComplete — are roots, because the loop belongs to no request. It runs for the life of the process, and a trace has to end.

The span names are constants, on Quartz.Diagnostics.OperationName. Attributes are namespaced quartz.*, and are constants on Quartz.Diagnostics.ActivityTags:

Linking a firing to what scheduled it

A job runs minutes, hours or days after the call that scheduled it, quite possibly on another node. When that call was made inside an Activity, the scheduler records its W3C trace context on the trigger — under the reserved keys SchedulerConstants.TraceParent and SchedulerConstants.TraceState — and the firing's Quartz.Job.Execute span carries an ActivityLink back to it. A Quartz.Job.Veto span carries the same link, because a fire a listener refused is worth walking back from too. Nothing needs configuring, and an HTTP API request gets it without asking, because the endpoint runs inside ASP.NET Core's server span.

It is a link rather than a parent on purpose: the firing is its own trace root, so a trace never has to stay open across the wait. That is the shape OpenTelemetry gives an asynchronous producer and the consumer that eventually picks the work up, and every backend that renders links will walk from the firing back to the request that asked for it.

The cost is two string entries on each trigger's data map, visible wherever trigger data is — MergedJobDataMap, the dashboard, GET /triggers. Turn it off with:

q.ConfigureScheduler(options => options.PropagateTraceContext = false);

Metrics

Eleven instruments, all on the Quartz meter. Every measurement carries quartz.scheduler.name and quartz.scheduler.id — the name says which scheduler, the id says which node of it, and a cluster is several nodes sharing one name.

Each name is a const string on QuartzInstrumentation.Instruments, so a view, an alert rule or a dashboard built in code can name one without spelling it: QuartzInstrumentation.Instruments.JobExecutionDuration is quartz.job.execution.duration. The meter builds its instruments from those constants and a test holds the two sets equal in both directions, so the table below cannot drift from what is emitted. A second test snapshots the whole catalogue — name, kind, unit and description — so an instrument that arrives, leaves, changes kind or changes unit is a reviewed diff rather than a chart that quietly stops reading anything.

¹ Only when the trigger names an execution group. A trigger in no group carries no such attribute rather than an empty one, so the two are not folded into one series. ² Only when the operation failed. The value is the fully-qualified name of the exception type.

quartz.jobstore.operation's value is one of the thirty-three Quartz.JobStore.* names above, so the same string finds a slow operation in a trace and in a metric. Its histogram's count is how many of each operation there were, and the error.type-tagged part of that count is how many failed.

quartz.jobstore.lock's value is the LOCK_NAME column's — TRIGGER_ACCESS, which every scheduling operation takes, or STATE_ACCESS, which the cluster check-in takes. This is the one instrument that says something while a scheduler is stalled rather than after: a lock statement blocked behind another session returns nothing and throws nothing, so no operation is recorded and no failure is counted. The warning that goes with it is event 3716 — Log Events — and the case behind both is A Lock Held by a Connection That Is Gone.

The two cluster instruments and the lock-wait histogram come from the ADO.NET store, which is the only clustered one. The other eight are store-agnostic.

Reading the numbers

A histogram carries its own count, which is why there is no execution counter and no error counter: the number of executions is quartz.job.execution.duration's count, and the number of failures is the part of that count tagged with error.type — which also says which exception, something a plain error counter never could.

error.type is the OpenTelemetry convention rather than a Quartz name. It is deliberately not on quartz.job.execution.active: an up-down counter's increment and decrement must carry identical attributes, and whether a job will fail is not known when it starts.

The meter is created from the container's IMeterFactory when there is one — which AddMetrics(), and therefore every application built on the generic host, registers. That is what lets two schedulers, or two hosts in one test process, keep their measurements apart.

OpenTelemetry.Instrumentation.Quartz

OpenTelemetry.Instrumentation.Quartz is the OpenTelemetry community's Quartz instrumentation library, and it was written for 3.x.

The reason is that the two versions publish through different System.Diagnostics mechanisms. 3.x wrote to a DiagnosticListener named Quartz, creating each Activity with new Activity(...) and no ActivitySource behind it. The package subscribes to exactly that: a DiagnosticSourceSubscriber filtered to the listener named Quartz, plus AddLegacySource("Quartz.Job.Execute") and AddLegacySource("Quartz.Job.Veto") — and "legacy source" in the OpenTelemetry SDK means precisely an activity that has no ActivitySource.

4.x emits from an ActivitySource named Quartz. Its activities are therefore not legacy activities, and nothing writes to a DiagnosticListener at all, so both halves of the subscription match nothing.

The 4.0 way is the two lines at the top of this page — AddSource(QuartzInstrumentation.ActivitySourceName) and AddMeter(QuartzInstrumentation.MeterName). There is no package to install:

- builder.Services.AddOpenTelemetry()
-     .WithTracing(tracing => tracing.AddQuartzInstrumentation());
+ builder.Services.AddOpenTelemetry()
+     .WithTracing(tracing => tracing.AddSource(QuartzInstrumentation.ActivitySourceName))
+     .WithMetrics(metrics => metrics.AddMeter(QuartzInstrumentation.MeterName));

- <PackageReference Include="OpenTelemetry.Instrumentation.Quartz" Version="1.*" />

What is lost with the package is its QuartzInstrumentationOptions.TracedOperations filter. Subscribing directly records both Quartz.Job.Execute and Quartz.Job.Veto; drop one with an OpenTelemetry processor or a sampler if a vetoed fire is not worth a span to you. What is gained is everything 4.0 added — the store spans and all eleven instruments — none of which the package knows about.

Older packages

Quartz.OpenTelemetry.Instrumentation is obsolete and is not part of 4.x. Subscribe to the activity source directly, as at the top of this page.

Coming from Quartz.OpenTracing

Quartz.OpenTracing is not part of 4.x either. It was built on the DiagnosticSource events that 4.x replaced with System.Diagnostics.Activity, and there is no 4.x release of it — the OpenTracing project itself is archived. Replace an AddQuartzOpenTracing call with the OpenTelemetry setup at the top of this page.

Logging

Quartz logs through Microsoft.Extensions.Logging and uses whatever the application has configured; there is nothing to wire up. The types no container builds — a listener or trigger you constructed, the static helpers, the jobs in Quartz.Jobs — are pointed at a logger factory with Quartz.Diagnostics.LogProvider.SetLogProvider(loggerFactory).

For a history of every job and trigger event as log entries, rather than as traces, the history plugins write one.