Scheduling, Delayed Jobs, and Time-Based Work

By the end of this lesson, you will be able to...

• Trace a future obligation from its creation through a durable scheduler, queue handoff, worker, and completion record.

• Distinguish a one-off delayed job from a recurring schedule rule.

• Choose a lateness, missed-run, and overlap policy from the meaning of a time-based task.

Idea in one sentence: A scheduler keeps durable promises about future work; when a promise becomes due, it creates work for the pool rather than trying to execute everything itself.

Core Insight

On Tuesday at 10:00, a learner registers for a live class that starts on Wednesday at 10:00. The platform promises to send a reminder 24 hours before the class.

At the same time, the platform has a different promise: every night at 02:00 it prepares an instructor report. The reminder is created once from a class event. The report is created repeatedly from an ongoing calendar rule.

Both tasks involve time. They are not the same mechanism.

class registration -> one reminder due at Wednesday 10:00
nightly rule       -> one report run for each calendar slot

The worker pool from lesson 005 can execute a reminder or report once it exists in the ready queue. It cannot, by itself, remember a promise for tomorrow after a process restart. That is the scheduler's job.

time rule or event -> durable future intent -> due run -> queue -> worker pool

The hard part is not writing a cron expression. It is deciding what the system owes when the ideal clock and real life disagree: a scheduler crashes, a deployment crosses the due moment, a run is late, or a previous run is still active.

The Situation: Two Promises With Different Origins

Keep these two cases separate.

Plain meaning:

A delayed job is one future task created from one event. A recurring job is a standing rule that keeps creating future runs.

In this scenario:

The class produces one reminder record with a run_at timestamp. The report scheduler stores a rule and calculates the next 02:00 run after each slot.

Technical name:

These are delayed scheduling and recurring scheduling. Both need durable timing state, but their ownership and recovery semantics differ.

This distinction prevents a common mistake: modeling every time-based task as a timer hidden inside one application process. A process timer may work in development. In production it forgets when its process restarts, cannot explain missed work, and makes two replicas liable to fire the same rule.

The Naive Model: Cron Is the Whole Design

The first design writes:

0 2 * * * generate_instructor_report

and adds an in-memory timer for the class reminder.

It works while one host stays healthy and every run finishes before the next one. Then a deployment happens at 01:59, the process restarts, and the report does not run. Or the report takes 90 minutes, so the next day's slot arrives while the old report is still writing data. Or a reminder is held only in memory and disappears overnight.

The failure is not that cron syntax is bad. The failure is confusing an ideal firing time with the whole product contract.

For each schedule, the team must state:

• who owns the durable schedule or one-off promise;
• what counts as the unique run identity;
• whether a late run should execute, be skipped, or be merged;
• whether two runs may overlap;
• what happens after a scheduler or worker failure;
• which signals reveal late, missing, duplicate, or stuck work.

The schedule string is input to that policy. It is not the policy itself.

The Moving Parts

The scheduler is not a fast worker. Its main job is to make time-based intent durable, then turn due intent into a normal queue job with an identity that can be tracked.

A Worked Trace: One Reminder, One Missed Report

The table below makes the intermediate states visible.

The report's late behavior was not discovered at 02:06. The policy already said that a delayed report is still useful and should run once. A different task might make another choice:

• a cleanup sweep can coalesce several missed runs into one;
• a business-close report may need every missed slot replayed;
• a reminder after its class starts may be skipped because it no longer helps;
• a billing task may require a stronger, separately designed execution and idempotency contract.

The naive failure contrast is:

naive claim: “run daily at 02:00” defines success
better claim: success includes the intended slot, lateness policy, overlap rule, handoff, and recorded outcome

So far, the scheduler has not guaranteed exactly-once email delivery. It has preserved and dispatched one identifiable obligation. The worker still needs a safe side-effect and completion boundary.

Claiming Due Work Safely

Several scheduler instances may run for availability. If all scan for “due now” and publish freely, the report can be enqueued twice. The scheduler therefore needs a claim transition:

pending due run -> claimed for dispatch -> enqueued -> completed or recoverable

A simplified shape is:

def dispatch_due_runs(now):
    for run in scheduler.claim_due(before=now, limit=100):
        queue.publish(
            name=run.job_name,
            payload=run.payload,
            dedupe_key=run.fire_id,
        )
        scheduler.mark_enqueued(run.fire_id)

The important transition is claim_due, not the function name. It gives one scheduler temporary ownership of a due run. If that process dies after the claim, recovery must eventually make the run visible again or record that it was already handed off. If it dies after publishing but before recording enqueued, the system may try again; the run identity helps the queue consumer detect or safely tolerate the duplicate.

This is ordinary distributed uncertainty. Durable scheduling reduces forgotten promises. It does not remove the need for idempotent side effects, acknowledgement, and observability in the worker path.

Misses, Lateness, and Overlap Are Product Policies

Choose the policy by asking what the task means after time has passed.

The central trade-off is strict time semantics versus coordination and recovery cost. “Replay every missed run” preserves more history but can create a burst after downtime. “Skip if late” is simpler but can violate a user promise. “Never overlap” needs locks, leases, or run-state rules and may let backlog grow.

Do not apply one answer to every schedule. State the useful delay, acceptable duplicate risk, and desired behavior under a slow previous run.

Cost, Limits, and Signals

Durable intent costs state and operation

The scheduler needs a store, a claiming rule, and monitoring. That is more work than a local timer. It buys recoverable, inspectable promises when processes restart or replicas change.

The scheduler is not the throughput control

Dispatching 10,000 overdue jobs immediately can overwhelm the worker pool and its dependencies. The scheduler should hand off work with bounded claims and the queue/pool controls still apply. The next lesson addresses rate limits and backpressure for that pressure.

Clock time has ambiguity

Time zones, daylight-saving changes, and clock skew can change what “02:00” means. A product-facing schedule must name its time zone and its policy for repeated or missing local times. This lesson does not require a calendar-library implementation, but it does require that the ambiguity be owned rather than hidden.

A run record is not proof of the external side effect

enqueued means the scheduler handed off work. completed should mean the worker recorded enough evidence for the product. Email delivery, payment, and external APIs may need their own confirmation and retry rules.

This helps when future work matters after restarts and outages. It costs durable state, coordination, and recovery logic. It does not guarantee on-time execution under every failure or exactly-once effects. You see the boundary when due-run lag rises, missed runs accumulate, duplicate fire ids appear, or overlapping work consumes the same resource.

Common Confusions

Confusion: A delayed job and a recurring job are the same kind of record

Why it is tempting:

Both eventually put a job into a queue.

Better model:

A delayed job is normally one obligation derived from one event. A recurring schedule is a rule that continually produces new obligations. They need different edit, recovery, and missed-run reasoning.

Confusion: A cron expression specifies correct behavior

Why it is tempting:

The expression visibly names the desired time.

Better model:

It names an ideal slot. Correctness also needs a time zone, late policy, overlap policy, durable ownership, and a way to inspect the actual run.

Confusion: Durable scheduling gives exactly-once execution

Why it is tempting:

The schedule record survives restart and has one fire id.

Better model:

The handoff can fail or be repeated around crashes. Use a run identity and make the worker's side effect safe to retry; durable intent prevents forgotten work, not all duplicate work.

Check Your Understanding

Check: A scheduler was down from 02:00 to 02:20. A cleanup job is configured to run hourly. What question must be answered before deciding whether to enqueue twenty missed cleanup jobs?

Think first, then reveal.

Answer: Ask whether each missed slot has independent value. Cleanup often needs one eventual pass, so the missed work can be coalesced. A task whose output represents every hour may instead need replay or an explicit gap report.

Check: Two scheduler replicas both see the same reminder due at 10:00. What mechanism prevents them from freely publishing two identical jobs?

Think first, then reveal.

Answer: A durable claim or lease transition gives one replica ownership of that fire id before dispatch. Recovery still needs to handle a crash during handoff, so the downstream job remains identifiable and safe to retry.

Check: A daily report starts at 02:00 and is still running at 03:00 when the next rule would create another report. What is missing if the team says only “it runs daily”?

Think first, then reveal.

Answer: An overlap policy. The team must decide whether concurrent runs are safe, whether the second waits, or whether it is skipped or merged. The answer comes from output ownership and product meaning.

Practice

Design the timing policy for a platform that has: a reminder 24 hours before a class, a nightly report, and a temporary-upload cleanup pass.

For each, state:

• whether it is delayed or recurring;
• the durable record and run identity;
• the late or missed-run policy;
• whether overlap is allowed;
• the queue handoff and one signal that detects failure.

A strong answer treats the reminder as a one-off promise with a deadline, treats the report as a recurring rule with a visible late policy, and allows cleanup to coalesce when exact slots do not matter. It separates scheduler dispatch from worker completion and names a safe response to duplicate handoff.

Resources

• [DOC] BullMQ: Delayed Jobs — Focus: inspect how one-off future work is stored before becoming ready work.
• [DOC] Celery: Periodic Tasks — Focus: compare recurring rules, dispatch, and overlap cautions.
• [DOC] Quartz Scheduler Documentation — Focus: study durable jobs, triggers, and missed-fire behavior after the core model is clear.

Key Takeaways

1. Time-based work starts as durable intent. A scheduler remembers a future obligation; a worker pool executes it only after it is due.
2. Delayed and recurring work differ. One is an event-derived promise; the other is a rule that creates many runs.
3. A slot is not a complete policy. Lateness, missed runs, overlap, time zone, and handoff failure determine real behavior.
4. Run identities make uncertainty visible. Claims and fire ids support recovery, but workers still need safe repeatable side effects.