Clustering

A cluster is several scheduler instances sharing one database.

• Load balancing: whichever node acquires a trigger runs it.
• Failover: when a node dies, the others recover the jobs it was running, if the job asked for it with RequestRecovery().

Clustering needs a persistent store; RAMJobStore has nothing to share. Use LocalTransactionJobStore, which UsePersistentStore registers.

Enabling it

builder.Services.AddQuartz(q =>
{
    q.ConfigureScheduler(options =>
    {
        // every node in the cluster shares this name: it is what makes them one cluster
        options.InstanceName = "orders";

        // ...and each needs its own id. Generating one is the easy way to be sure.
        options.GenerateInstanceId = true;
    });

    q.UsePersistentStore(store =>
    {
        store.UseSqlServer(connectionString);
        store.UseClustering();
    });
});

• InstanceName must be the same on every node. It is the SCHED_NAME column of every row. Nodes with different names sharing a database are two schedulers that cannot see each other's work, not a cluster.
• InstanceId must be different on every node. A node recognises its own check-in row and fired triggers by it. GenerateInstanceId = true derives one at startup from the registered IInstanceIdGenerator (by default host name and timestamp); the flat value AUTO used to mean this. Set an explicit id when something outside Quartz names the node, such as node affinity, which pins a trigger to an id and needs one that survives a restart.
• The store configuration must match. Nodes may differ in thread pool size and other process-local settings, but must use the same tables, table prefix and serializer.

UseClustering() also turns on database locking; clustering does not work without it.

Tuning the check-in

store.UseClustering(cluster =>
{
    cluster.CheckinInterval = TimeSpan.FromSeconds(10);
    cluster.CheckinMisfireThreshold = TimeSpan.FromSeconds(20);
});

• A shorter interval notices a dead node sooner, at the cost of more database traffic from every node.
• Raise the threshold if a node is declared dead while merely busy or on a slow database. A false positive recovers its running jobs, so they run twice.

Batching trigger acquisition

A scheduler acquires the triggers it is about to fire in batches. MaxBatchSize defaults to 0, automatic:

• A persistent store batches. A round there is round trips and a commit, so one round for every trigger already due is cheaper per firing.
• On a cluster a batch takes the lock, and still pays. Above 1, every acquisition cycle takes the TRIGGER_ACCESS row lock, including cycles that acquire nothing. The measured gain is in Sizing a cluster.
• MaxBatchSize = 1 keeps a cluster's acquisition off the lock. With AcquireTriggersWithinLock off (the default), a round of one takes no cluster-wide lock.
• In memory a batch is not a clear win. It measured faster for repeating triggers and slower for a burst of one-offs, so the default stays at one.
• An explicit value wins. MaxBatchSize = 1 is 4.2's behaviour on every store.

Two settings decide a batch's size:

• MaxBatchSize is the upper bound on how many triggers one acquisition takes.
• The window decides how many it actually takes: a batch ends at the first trigger's fire time, or now if that is later, plus the window. At zero, the default, a batch takes the triggers already due, or those due at the same instant as the first. Raising MaxBatchSize alone batches a backlog, not spread-out fire times.

For fire times that are close but not equal, change both:

q.ConfigureScheduler(options =>
{
    options.MaxBatchSize = 10;
    options.BatchTriggerAcquisitionFireAheadTimeWindow = TimeSpan.FromSeconds(1);
});

• Batch when many triggers fire at once (a few hundred at the top of the hour): one acquisition and one TRIGGERS_FIRED round trip replace one of each per trigger. Do not batch triggers a minute apart.
• Triggers fire early, by up to the window. A one-second window on a schedule with second-level precision changes behaviour.
• MaxBatchSize may not exceed the thread pool's MaxConcurrency; startup rejects it. Triggers acquired beyond the available threads are held by this node, unfireable by any other, until the pool drains.

A serial job across nodes

A [DisallowConcurrentExecution] job runs on one node at a time, and its other triggers wait. The run's end wakes only the node it ran on, which fires the ones it may. On a cluster, a node with nothing to do looks again sooner when a run elsewhere holds back a trigger of its own:

• While the store still reports the trigger held, the wait stops growing at 5 seconds, or IdleWaitTime if that is shorter. A held trigger fires within about 5 seconds of the run ending, and a run that lasts hours costs the node a look every 5 seconds.
• Once nothing is held, the wait doubles on to IdleWaitTime and stays there.
• When a pinned trigger's serial job has changed hands since the last look, the wait starts again at 100 ms, once it has grown to 400 ms.
• Off a cluster nothing changes: a run's end wakes its own node. Through a subclassed driver delegate a held pinned trigger is found after IdleWaitTime; see What the delegate cannot reach.

Before 4.4 a node waited the whole IdleWaitTime in each case, and a pinned trigger fired that late every time (#3988).

Seeing the cluster

Check-ins land in QRTZ_SCHEDULER_STATE. IScheduler.QueryClusterNodes() reads them without SQL:

List<ClusterNode> nodes = await scheduler.QueryClusterNodes();

foreach (ClusterNode node in nodes)
{
    string marker = node.IsCurrentNode ? " (this node)" : "";
    Console.WriteLine($"{node.InstanceId}{marker}: {node.State}, last check-in {node.LastCheckInUtc:u}");
}

// The verdicts come from the same predicate the failover sweep applies, so a node reported
// Failed is one whose in-flight work the cluster is about to take over.
List<ClusterNode> failed = nodes.FindAll(node => node.State == ClusterNodeState.Failed);

Each ClusterNode carries InstanceId, LastCheckInUtc, the CheckInInterval that node was configured with, IsCurrentNode, and a State:

• The answering node is listed first, then the rest by instance id. The answering node is always listed, even before its first check-in.
• The state comes from the same predicate the failover sweep uses, so the listing and the recovery cannot disagree. A failed node is listed for a while, then disappears.
• The states are what this node believes, by its own clock; another reason the clocks must agree.
• A scheduler that is not clustered returns itself, Alive, with both times null, so a caller need not check whether clustering is on.

To see what each node is doing, join the listing to QueryFireInstances on SchedulerInstanceId:

List<ClusterNode> nodes = await scheduler.QueryClusterNodes();
PagedResult<FireInstance> firings = await scheduler.QueryFireInstances(new FireInstanceQuery
{
    // both states: what a node is holding is as interesting as what it is running, and a
    // reservation left behind by a dead node is what recovery is about to clear
    State = null
});

foreach (ClusterNode node in nodes)
{
    int running = firings.Items.Count(firing =>
        firing.SchedulerInstanceId == node.InstanceId && firing.State == FireInstanceState.Executing);

    Console.WriteLine($"{node.InstanceId} ({node.State}) is running {running} job(s)");
}

The same listing is behind GET /schedulers/{name}/nodes in the HTTP API and the Cluster page of the dashboard.

When a node leaves

A node that is shut down, rather than killed, hands back what it can.

It stops firing before it stops running jobs. The scheduling loop is halted and waited for first:

• every trigger it had reserved but not fired is released to WAITING for another node's next pass;
• a firing it had already committed (fired-trigger row written, trigger moved to its next instant) is dispatched, not dropped.

This holds whether or not the shutdown waits for jobs. Before 4.1 an unwaited shutdown could close the thread pool under its own loop and lose that occurrence, recoverable only through RequestRecovery.

It completes the firings it can.

A completion issued after the store closes is refused: the firing stays EXECUTING and, for a [DisallowConcurrentExecution] job, its trigger stays BLOCKED. A job still working when the window closes is abandoned and leaves what a crashed node leaves. A peer resolves it once the check-in lapses, after CheckinInterval + CheckinMisfireThreshold plus a grace period for an execution that may still be alive.

It does not delete its check-in row. The QRTZ_SCHEDULER_STATE row keeps its last timestamp until a peer declares the node failed and recovers it. So a node that leaves is declared failed on the same schedule as one that crashed. Wait for jobs, and allow the window above, so the node leaves nothing to recover.

Asking for recovery

Failover recovers a dead node's executions, only for jobs that call RequestRecovery():

q.AddJob<ChargeInvoicesJob>(j => j
    .WithIdentity("charge-invoices")
    .RequestRecovery());

• It is off by default and does nothing without a persistent store.
• When a clustered scheduler decides a peer stopped checking in, it examines every fired-trigger row the peer left. If the job requested recovery, the row becomes a new trigger in the RECOVERING_JOBS group with the original trigger's job data. Otherwise the row is deleted and that occurrence is lost.
• Read it with IJobDetail.RequestsRecovery. A recovery firing sets IJobExecutionContext.Recovering.
• A firing a graceful shutdown cancels is not recovered unless the store says so; see A job a shutdown stops.

Which jobs should ask, what recovery re-runs and what it does not, and how a recovered firing identifies itself: What RequestsRecovery re-runs, and when, which also covers making a job safe to re-run. Recovery is for an interrupted execution. A job that threw completed its firing; to re-run it, use a retry policy on its trigger.

Related:

• Operating a Cluster: rolling upgrades, instance ids in containers, sizing, backup
• Running under an External Leader Election: when the election already exists
• Quartz.Examples.Worker: a worker service with a persistent store, configured as this lesson describes