Retrying Failed Jobs

A retry policy says how many times, and how far apart, the scheduler re-fires a trigger when its job fails. With a policy set, a job that throws is retried and one that succeeds is not. The policy is the trigger's own, its job type's or the scheduler's default: see Declare it on the job, or set a default.

Give the trigger a policy

There are exactly three kinds: fixed, exponential and explicit. A fixed policy waits the same time between retries:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // Three retries, five minutes apart, after a failure.
        .WithRetryPolicy(RetryPolicy.Fixed(3, TimeSpan.FromMinutes(5))));
});

An exponential policy backs off, optionally up to a ceiling:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // 30s, 1m, 2m, 4m, 8m — but never longer than ten minutes.
        .WithRetryPolicy(RetryPolicy.Exponential(
            maxAttempts: 5,
            initialDelay: TimeSpan.FromSeconds(30),
            factor: 2,
            maxDelay: TimeSpan.FromMinutes(10))));
});

An exponential policy can be jittered, so triggers that failed together do not return together. Each wait is multiplied by a value drawn uniformly from [1 - jitter, 1 + jitter], still bounded by the ceiling:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // The same backoff as above, spread by a fifth either way: the first retry lands
        // between 24 and 36 seconds after the failure, the second between 48 and 72, and so
        // on. A hundred triggers that failed on the same outage come back at a hundred
        // different instants instead of all at once.
        .WithRetryPolicy(RetryPolicy.Exponential(
            maxAttempts: 5,
            initialDelay: TimeSpan.FromSeconds(30),
            factor: 2,
            maxDelay: TimeSpan.FromMinutes(10),
            jitter: 0.2)));
});

A jitter of 0 (the default) equals the four-argument overload, down to the bytes in the RETRY_POLICY column.

An explicit policy lists the waits; the list's length is the number of attempts, and its last entry repeats:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // Try again quickly twice, then give the upstream system an hour.
        .WithRetryPolicy(RetryPolicy.Explicit(
            TimeSpan.FromSeconds(10),
            TimeSpan.FromMinutes(1),
            TimeSpan.FromHours(1))));
});

MaxAttempts counts retries after the first failure: Fixed(3, …) runs a job that keeps failing four times.

Declare it on the job, or set a default

The first policy found applies:

RetryPolicy.None found at any level means no retry.

Declare the policy on the job when whether it is safe to retry is a property of its code:

// Five retries, 30s, 1m, 2m, 4m and 8m apart, but never more than ten minutes, for
// every trigger of this job that names no policy of its own.
[RetryPolicy(5, "00:00:30", 2, MaxDelay = "00:10:00")]
public sealed class FeedImportJob : IJob
{
    private readonly IImportService importer;

    public FeedImportJob(IImportService importer)
    {
        this.importer = importer;
    }

    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default)
    {
        await importer.Run(cancellationToken);
    }
}

Durations are invariant TimeSpan strings. An attribute whose arguments are not a policy fails AddJob and ScheduleJob with a SchedulerException naming the job type.

Set a default for everything else:

builder.Services.AddQuartz(q =>
{
    // Any trigger whose own policy and job type name none: three retries, one minute apart.
    q.UseDefaultRetryPolicy(RetryPolicy.Fixed(3, TimeSpan.FromMinutes(1)));
});

Opt a job type or a trigger out of what it would inherit:

// Charging a card twice is worse than not charging it: never retried, whatever the
// scheduler's default says.
[RetryPolicy(0)]
public sealed class ChargeCardJob : IJob
{
    public ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default) => default;
}

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // Never retried, whatever ImportJob declares or the scheduler defaults to.
        .WithRetryPolicy(RetryPolicy.None));
});

The policy is looked up when the job fails. Nothing is written to the trigger, so the job type's policy and the default cover triggers stored before them, and the RETRY_POLICY column stays the trigger's own. IJobExecutionContext.RetryPolicy is the policy that applies to a firing.

Every node decides for itself. A node reads the attribute from its own build and the default from its own configuration, so deploy the same on every node. A node older than 4.4 ignores both.

Manual runs and recovery inherit too. TriggerJob, the dashboard's Run now and Run again, and the trigger that recovers a job a dead node was running all fire on a trigger with no policy of its own, so the job type's policy or the default retries them. [RetryPolicy(0)] on the job stops that.

Mark a job that must not run twice with [RetryPolicy(0)]. It lives in code, so no node can drop it. A trigger's RetryPolicy.None is a stored value, which a 4.3 node empties when it fires the trigger. SendMailJob and NativeJob carry it.

What counts as a failure

A job fails when Execute throws anything; there is nothing to implement or annotate. Not failures:

• A JobExecutionException asking for RefireImmediately, UnscheduleFiringTrigger or UnscheduleAllTriggers: the job's own decision wins over the policy.
• A cancellation on the scheduler's own token (shutdown, interrupt). For a node vanishing mid-execution, use RequestsRecovery; for a shutdown, see A job a shutdown stops.
• Anything, when no policy applies (the default).

A job can catch a failure that retrying cannot fix and return, saving its attempts:

public sealed class SelectiveImportJob : IJob
{
    private readonly IImportService importer;
    private readonly ILogger<SelectiveImportJob> logger;

    public SelectiveImportJob(IImportService importer, ILogger<SelectiveImportJob> logger)
    {
        this.importer = importer;
        this.logger = logger;
    }

    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await importer.Run(cancellationToken);
        }
        catch (TransientImportException)
        {
            // Let it out. Throwing is what asks for a retry, so the trigger's policy takes over.
            throw;
        }
        catch (InvalidOperationException e)
        {
            // A failure no amount of retrying can fix - bad input, not a flaky dependency. Report
            // it and return: the occurrence is over, and the trigger goes back to its ordinary
            // schedule instead of spending its attempts on a certainty.
            logger.LogError(e, "Import cannot succeed for this occurrence and will not be retried");
        }
    }
}

A job a shutdown stops

A shutdown's cancellation is not a failure, so no policy retries it. From 4.4 a persistent store can hand the firing back for recovery instead: the job runs again on a peer's next acquisition, or on this node when it starts. Off by default.

builder.Services.AddQuartz(q =>
{
    // The shutdown asks running jobs to stop, and waits for them.
    q.ConfigureScheduler(options => options.ShutdownJobInterruption = ShutdownJobInterruption.WhenWaitingForJobs);

    q.UsePersistentStore(store =>
    {
        store.UsePostgres(connectionString);
        store.ConfigureStore(options => options.RecoverFiringsCancelledByShutdown = true);
    });

    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly").RequestRecovery());
});

builder.Services.AddQuartzHostedService(options => options.WaitForJobsToComplete = true);

What a hand-back does:

• Stores a recovery trigger in RECOVERING_JOBS, in the completion's transaction, as cluster recovery does after a crash. The job sees Recovering and RecoveringTriggerKey.
• Counts no retry: RetryAttempt does not move, and the policy is not consulted.
• Moves what awaits the trigger onto the recovery trigger, so the replay's outcome settles it: an OnSuccess continuation runs once the replay succeeds, an OnFailure one once it fails with no retry left. A spent one-shot trigger is deleted as usual.
• Does not write the job's data, even with [PersistJobDataAfterExecution]: the replay starts from the data the cancelled run started with.
• Releases a [DisallowConcurrentExecution] job's other triggers, and a BufferOne or CancelPrevious trigger's hold, as any completion does.
• Records a Cancelled history row with the summary Handed back for recovery: the scheduler shut down while it ran. Log event 3054 names the recovery trigger.

The replay is an ordinary recovery trigger, as after a crash. It has no policy of its own, nor the original trigger's, so a failed replay is retried by the job type's [RetryPolicy] or the scheduler's default, from attempt 0; see Declare it on the job, or set a default. Handed back again, it keeps the markers of the original firing.

A completion that arrives after the store has closed is refused, as before. Its fired-trigger row stays, and recovery replays the job when a peer, or this node at its next start, recovers it. A 4.3 node ignores the setting, and fires a recovery trigger a 4.4 node stored.

What the job sees

IJobExecutionContext.RetryAttempt is 0 on a regular fire and n on the n-th retry:

public sealed class RetryAwareImportJob : IJob
{
    private readonly IImportService importer;
    private readonly ILogger<RetryAwareImportJob> logger;

    public RetryAwareImportJob(IImportService importer, ILogger<RetryAwareImportJob> logger)
    {
        this.importer = importer;
        this.logger = logger;
    }

    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default)
    {
        if (context.RetryAttempt > 0)
        {
            logger.LogWarning(
                "Import retry {Attempt} for the occurrence scheduled at {Scheduled}",
                context.RetryAttempt,
                context.ScheduledFireTimeUtc);
        }

        // Throwing anything is what asks for a retry. There is nothing to opt into.
        await importer.Run(cancellationToken);
    }
}

ScheduledFireTimeUtc is the same on every retry: the occurrence the schedule called for, not when the retry ran.

When the policy gives up

1. The execution context. IJobExecutionContext.Outcome is how the firing ended; IJobExecutionContext.RetryScheduled is whether another attempt follows. Both are set before completion notifications, so JobWasExecuted and TriggerComplete can read them:

/// <summary>
/// A job listener that tells an attempt from a verdict, which before 4.2 only a trigger listener
/// could do — and only by comparing an instruction against <c>RetryTrigger</c>.
/// </summary>
public sealed class OutcomeReadingListener : IJobListener
{
    private readonly ILogger<OutcomeReadingListener> logger;

    public OutcomeReadingListener(ILogger<OutcomeReadingListener> logger)
    {
        this.logger = logger;
    }

    public ValueTask JobWasExecuted(
        IJobExecutionContext context,
        JobExecutionException? jobException,
        CancellationToken cancellationToken = default)
    {
        if (context.Outcome == ExecutionOutcome.Failed && !context.RetryScheduled)
        {
            logger.LogError("{Job} failed for the last time", context.JobDetail.Key);
        }

        return default;
    }
}

A job that threw is ExecutionOutcome.Failed whether or not it will be retried; RetryScheduled says whether the occurrence is finished.

2. ITriggerListener.TriggerRetriesExhausted, raised once per occurrence, between JobWasExecuted and TriggerComplete, when a policy applies, the job threw, and no attempt follows because:

• the policy's attempts are spent;
• a retry was declined for lack of room: it would land at or within a second of the next occurrence, after the trigger's end time, or past the end of representable time (the rules), so attempts are left. Only under the trigger's own policy: see the warning below;
• the job's JobExecutionException asked for UnscheduleFiringTrigger or UnscheduleAllTriggers, so no retry was attempted.

Tell them apart with context.RetryAttempt against context.RetryPolicy.MaxAttempts, and TriggerComplete's instruction.

Only a trigger whose completion TriggerBase.ExecutionComplete decides applies an inherited policy. A trigger of your own that does not derive from TriggerBase, or overrides ExecutionComplete without calling the base, is reported by its own policy alone.

/// <summary>
/// Raises an alert when an occurrence has run out of retries, and says nothing while it is still
/// trying.
/// </summary>
public sealed class GaveUpListener : ITriggerListener
{
    private readonly ILogger<GaveUpListener> logger;

    public GaveUpListener(ILogger<GaveUpListener> logger)
    {
        this.logger = logger;
    }

    public ValueTask TriggerRetriesExhausted(
        ITrigger trigger,
        IJobExecutionContext context,
        JobExecutionException exception,
        CancellationToken cancellationToken = default)
    {
        // context.RetryAttempt is how many retries this occurrence spent before giving up, and
        // context.RetryScheduled is false: there is no further attempt coming.
        logger.LogError(
            exception,
            "{Job} gave up after {Attempts} retries; the occurrence scheduled for {Scheduled} never succeeded",
            context.JobDetail.Key,
            context.RetryAttempt,
            context.ScheduledFireTimeUtc);

        return default;
    }
}

builder.Services.AddQuartz(q =>
{
    q.AddTriggerListener<GaveUpListener>(Matchers.AllTriggers());
});

Never raised when no policy applies, nor for a failure being retried, a cancelled, vetoed or successful firing, or a RefireImmediately run other than the one that ends the firing. It is a default interface member, so listeners written for 4.0 or 4.1 are unchanged.

3. The execution history. Each row carries RetryAttempt and RetryScheduled. A failed row with RetryScheduled false is a final failure, selected by ExecutionHistoryQuery.FailedFinally (HTTP API: failedFinally on GET …/history/executions). Succeeded alone cannot do this: a policy of three writes four failed rows for one bad night.

The dashboard's History page shows Failed (retrying) or Failed, has a Failed after retries option in its Outcome filter, and gives a final failure a Run again button (recorded in the action log; absent when read-only).

4. Pause the trigger instead. From 4.3, q.PauseTriggerWhenRetriesExhausted() pauses a trigger that gives up, with the exception's message as the reason, until someone resumes it. See Pausing when retries run out.

The rules worth knowing

A retry never displaces the trigger's next occurrence. One that would land at, or within a second of, the next fire time is dropped: an hourly trigger with a 90-minute wait is never retried. Nor is a retry scheduled past EndTimeUtc or the end of the calendar (an exponential wait can outgrow DateTimeOffset). The occurrence ends and the schedule continues.

A retry burns nothing: no SimpleTrigger repeat count, recurrence COUNT slot or TimesTriggered. The schedule afterwards is the one there would have been without the failure.

Running out of attempts is not an error. The trigger returns to its schedule with the attempt reset, not to TriggerState.Error, unless PauseTriggerWhenRetriesExhausted() pauses it.

A missed retry is an ordinary misfire. The trigger's misfire instruction decides, and the attempt is cleared with its occurrence. There is no separate retry-misfire policy.

Changing a policy on a stored trigger

UpdateTriggerDetails changes the policy without rescheduling:

await scheduler.UpdateTriggerDetails(
    new TriggerKey("nightly", "imports"),
    new TriggerDetailsUpdate().WithRetryPolicy(RetryPolicy.Fixed(5, TimeSpan.FromMinutes(2))),
    cancellationToken);

RetryPolicy.None stops retries:

await scheduler.UpdateTriggerDetails(
    new TriggerKey("nightly", "imports"),
    new TriggerDetailsUpdate().WithRetryPolicy(RetryPolicy.None),
    cancellationToken);

null removes the trigger's own policy, so it inherits its job type's or the scheduler's default. With neither, that also stops retries:

await scheduler.UpdateTriggerDetails(
    new TriggerKey("nightly", "imports"),
    new TriggerDetailsUpdate().WithRetryPolicy(null),
    cancellationToken);

The change applies from the next failure; a pending retry keeps its time. The attempt cannot be set: it belongs to the occurrence in flight.

Watching it happen

• Meter quartz.trigger.retry: each scheduled retry, tagged with scheduler, trigger group and execution group (the same tags as quartz.trigger.misfire).
• Meter quartz.trigger.retries_exhausted: each occurrence that gave up, same tags. Compare the two to see where a policy gains nothing.
• Log event 1056 (Information): trigger, attempt and retry instant. 1057: the occurrence that gave up, attempts spent, and the last exception.
• ITriggerListener.TriggerComplete with SchedulerInstruction.RetryTrigger, and ITriggerListener.TriggerRetriesExhausted once per occurrence that gives up.
• QRTZ_TRIGGERS columns: RETRY_POLICY (the trigger's own policy in its stored string form, none, or empty when it inherits) and RETRY_ATTEMPT (progress of the current occurrence, whichever policy applies). Both queryable; shown on the dashboard's trigger page.
• QRTZ_EXECUTION_HISTORY, where history is in the database: RETRY_ATTEMPT and RETRY_SCHEDULED per row.

See also

• More About Triggers — misfire instructions, priorities and calendars
• Rescheduling Jobs — changing a live schedule, and recovering a trigger in error
• Pausing with a Reason — pausing a trigger whose retries ran out