Retrying Failed Jobs

A trigger can carry a retry policy: how many times, and how far apart, the scheduler re-fires it when its job fails. Give a trigger one and there is nothing else to do — a job that throws is retried, and a job that succeeds is not.

Give the trigger a policy

Three shapes, and the only three ways to make one:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // Three retries, five minutes apart, after a failure.
        .WithRetryPolicy(RetryPolicy.Fixed(3, TimeSpan.FromMinutes(5))));
});

An exponential policy backs off, optionally up to a ceiling:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // 30s, 1m, 2m, 4m, 8m — but never longer than ten minutes.
        .WithRetryPolicy(RetryPolicy.Exponential(
            maxAttempts: 5,
            initialDelay: TimeSpan.FromSeconds(30),
            factor: 2,
            maxDelay: TimeSpan.FromMinutes(10))));
});

An exponential policy can also be jittered, so that triggers which failed together do not come back together. Each wait is multiplied by a value drawn uniformly from [1 - jitter, 1 + jitter], and the ceiling still bounds the result:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // The same backoff as above, spread by a fifth either way: the first retry lands
        // between 24 and 36 seconds after the failure, the second between 48 and 72, and so
        // on. A hundred triggers that failed on the same outage come back at a hundred
        // different instants instead of all at once.
        .WithRetryPolicy(RetryPolicy.Exponential(
            maxAttempts: 5,
            initialDelay: TimeSpan.FromSeconds(30),
            factor: 2,
            maxDelay: TimeSpan.FromMinutes(10),
            jitter: 0.2)));
});

A jitter of 0 — the default — is the four-argument overload: the same waits, and the same bytes in the RETRY_POLICY column.

Or spell the waits out. The table's length is the number of attempts, and its last entry repeats:

builder.Services.AddQuartz(q =>
{
    q.AddJob<ImportJob>(j => j.WithIdentity("import", "nightly"));
    q.AddTrigger<ImportJob>(t => t
        .ForJob("import", "nightly")
        .WithCronSchedule("0 0 2 * * ?")
        // Try again quickly twice, then give the upstream system an hour.
        .WithRetryPolicy(RetryPolicy.Explicit(
            TimeSpan.FromSeconds(10),
            TimeSpan.FromMinutes(1),
            TimeSpan.FromHours(1))));
});

MaxAttempts counts retries after the first failure, not fires: Fixed(3, …) means a persistently failing job runs four times in all.

What counts as a failure

A job fails, for retry purposes, when Execute throws — anything at all. There is no interface to implement and no attribute to add.

Three things are deliberately not failures:

• A JobExecutionException that asks for something itself. RefireImmediately, UnscheduleFiringTrigger and UnscheduleAllTriggers are decisions the job made, and they win over the trigger's policy.
• A cancellation on the scheduler's own token. Shutdown and interrupt are operator decisions; a node that vanishes mid-execution is what RequestsRecovery is for.
• Anything at all, on a trigger with no policy. That is the default, and it behaves exactly as it always did.

A job that knows a failure is not worth retrying can say so, and keep its attempts for the failures that are:

public sealed class SelectiveImportJob : IJob
{
    private readonly IImportService importer;
    private readonly ILogger<SelectiveImportJob> logger;

    public SelectiveImportJob(IImportService importer, ILogger<SelectiveImportJob> logger)
    {
        this.importer = importer;
        this.logger = logger;
    }

    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await importer.Run(cancellationToken);
        }
        catch (TransientImportException)
        {
            // Let it out. Throwing is what asks for a retry, so the trigger's policy takes over.
            throw;
        }
        catch (InvalidOperationException e)
        {
            // A failure no amount of retrying can fix - bad input, not a flaky dependency. Report
            // it and return: the occurrence is over, and the trigger goes back to its ordinary
            // schedule instead of spending its attempts on a certainty.
            logger.LogError(e, "Import cannot succeed for this occurrence and will not be retried");
        }
    }
}

What the job sees

IJobExecutionContext.RetryAttempt is 0 on a regular fire and n on the n-th retry:

public sealed class RetryAwareImportJob : IJob
{
    private readonly IImportService importer;
    private readonly ILogger<RetryAwareImportJob> logger;

    public RetryAwareImportJob(IImportService importer, ILogger<RetryAwareImportJob> logger)
    {
        this.importer = importer;
        this.logger = logger;
    }

    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken = default)
    {
        if (context.RetryAttempt > 0)
        {
            logger.LogWarning(
                "Import retry {Attempt} for the occurrence scheduled at {Scheduled}",
                context.RetryAttempt,
                context.ScheduledFireTimeUtc);
        }

        // Throwing anything is what asks for a retry. There is nothing to opt into.
        await importer.Run(cancellationToken);
    }
}

ScheduledFireTimeUtc is the same on every one of those firings — a retry is another attempt at one occurrence, so it reports the occurrence the schedule called for rather than the instant it actually ran.

When the policy gives up

Running out of attempts used to be silent. A listener could work it out, but only by comparing TriggerComplete's instruction against SchedulerInstruction.RetryTrigger and knowing what the trigger's policy said. Three things now say it outright.

The execution context says how the firing ended. IJobExecutionContext.Outcome is what the scheduler classified the firing as, and IJobExecutionContext.RetryScheduled is whether the trigger answered it with another attempt. Both are set before the completion notifications go out, so JobWasExecuted and TriggerComplete read them:

/// <summary>
/// A job listener that tells an attempt from a verdict, which before 4.2 only a trigger listener
/// could do — and only by comparing an instruction against <c>RetryTrigger</c>.
/// </summary>
public sealed class OutcomeReadingListener : IJobListener
{
    private readonly ILogger<OutcomeReadingListener> logger;

    public OutcomeReadingListener(ILogger<OutcomeReadingListener> logger)
    {
        this.logger = logger;
    }

    public ValueTask JobWasExecuted(
        IJobExecutionContext context,
        JobExecutionException? jobException,
        CancellationToken cancellationToken = default)
    {
        if (context.Outcome == ExecutionOutcome.Failed && !context.RetryScheduled)
        {
            logger.LogError("{Job} failed for the last time", context.JobDetail.Key);
        }

        return default;
    }
}

A job that ran and threw is ExecutionOutcome.Failed whether or not it is going to be retried — the outcome says what the firing did. RetryScheduled is what says whether the occurrence is finished.

A trigger listener is told once per occurrence that gave up.ITriggerListener.TriggerRetriesExhausted is raised between JobWasExecuted and TriggerComplete, and only when the trigger has a policy, the job failed, and there is no further attempt coming:

/// <summary>
/// Raises an alert when an occurrence has run out of retries, and says nothing while it is still
/// trying.
/// </summary>
public sealed class GaveUpListener : ITriggerListener
{
    private readonly ILogger<GaveUpListener> logger;

    public GaveUpListener(ILogger<GaveUpListener> logger)
    {
        this.logger = logger;
    }

    public ValueTask TriggerRetriesExhausted(
        ITrigger trigger,
        IJobExecutionContext context,
        JobExecutionException exception,
        CancellationToken cancellationToken = default)
    {
        // context.RetryAttempt is how many retries this occurrence spent before giving up, and
        // context.RetryScheduled is false: there is no further attempt coming.
        logger.LogError(
            exception,
            "{Job} gave up after {Attempts} retries; the occurrence scheduled for {Scheduled} never succeeded",
            context.JobDetail.Key,
            context.RetryAttempt,
            context.ScheduledFireTimeUtc);

        return default;
    }
}

builder.Services.AddQuartz(q =>
{
    q.AddTriggerListener<GaveUpListener>(Matchers.AllTriggers());
});

A failure on a trigger with no policy never raises it: nothing gave up, because nothing was going to try again. Neither does a failure that is being retried. Like every other member of ITriggerListener it is a default interface member, so a listener written against 4.0 or 4.1 compiles and runs unchanged.

The history says which row was the last word. Every execution the history records carries RetryAttempt — which attempt at the occurrence it was — and RetryScheduled. A row that did not succeed and has RetryScheduled false is a final failure, and ExecutionHistoryQuery.FailedFinally selects exactly those. It is a question Succeeded alone cannot ask: a job under a policy of three writes four failed rows for one bad night, and a page filtered on failure shows the same occurrence four times over. The HTTP API takes it as failedFinally on GET …/history/executions.

The dashboard's History page is built on all three: a row says Failed (retrying) or Failed, the Outcome filter offers Failed after retries, and a final failure carries a Run again button that fires the job by hand — recorded in the action log like every other mutation, and absent when the dashboard is read-only.

The rules worth knowing

A retry never displaces the trigger's next scheduled occurrence. If the retry would land at, or within a second of, the next fire time, it is dropped and the occurrence wins. So a policy whose waits are longer than the gap between occurrences quietly does nothing — an hourly trigger with a 90-minute retry wait is never retried, because the next hour comes first. A retry is not scheduled past the trigger's EndTimeUtc either, nor past the end of the calendar: an exponential policy's waits grow until one of them is longer than the room left in a DateTimeOffset, and a retry with nowhere to land is declined for the same reason as one that would land too late. In every case the occurrence settles and the trigger keeps its ordinary schedule.

A retry burns nothing. It does not consume a SimpleTrigger repeat count, a recurrence rule's COUNT slot, or a TimesTriggered. The schedule after a retry is exactly the schedule there would have been if nothing had failed.

Running out of attempts is not an error. The trigger goes back to its ordinary schedule with the attempt reset — it is not moved to TriggerState.Error, because one bad hour must not kill a cron trigger.

A missed retry is an ordinary misfire. If the scheduler never got to the retry, the trigger's own misfire instruction decides what happens, and the attempt is cleared: the occurrence it belonged to is gone. There is no separate retry-misfire policy.

Changing a policy on a stored trigger

UpdateTriggerDetails changes the policy without rescheduling the trigger:

await scheduler.UpdateTriggerDetails(
    new TriggerKey("nightly", "imports"),
    new TriggerDetailsUpdate().WithRetryPolicy(RetryPolicy.Fixed(5, TimeSpan.FromMinutes(2))),
    cancellationToken);

Passing null stops it retrying:

await scheduler.UpdateTriggerDetails(
    new TriggerKey("nightly", "imports"),
    new TriggerDetailsUpdate().WithRetryPolicy(null),
    cancellationToken);

The new policy applies from the next failure. An occurrence already waiting on a retry keeps the schedule it was given, and there is deliberately no way to set the attempt through an update: it belongs to the occurrence in flight, and setting it would either grant a running job extra attempts or take away ones it has already spent.

Watching it happen

• Meter quartz.trigger.retry counts each retry the scheduler schedules, tagged with the scheduler, the trigger group and the execution group — the same tags quartz.trigger.misfire carries.
• Meter quartz.trigger.retries_exhausted counts each occurrence that gave up, under the same tags. The two divide: a group whose retries are nearly all exhausted is a group the policy is buying nothing for.
• Log event 1056 reports the trigger, the attempt and the retry instant at Information, and 1057 reports the occurrence that gave up, the attempts it spent and what the last one threw.
• ITriggerListener.TriggerComplete is called with SchedulerInstruction.RetryTrigger, and ITriggerListener.TriggerRetriesExhausted once when the attempts run out.
• On a persistent store the two columns are on QRTZ_TRIGGERS: RETRY_POLICY holds the policy's stored string form and RETRY_ATTEMPT how far through it the current occurrence is. Both are queryable, and the dashboard's trigger page shows them.
• Where the history is kept in the database, QRTZ_EXECUTION_HISTORY carries RETRY_ATTEMPT and RETRY_SCHEDULED per row.

See also

• More About Triggers — misfire instructions, priorities and calendars
• Rescheduling Jobs — changing a live schedule, and recovering a trigger in error