Scheduling, Delayed Jobs, and Time-Based Work

Day 074: Scheduling, Delayed Jobs, and Time-Based Work

The moment time becomes the trigger, the system stops just processing jobs and starts keeping promises about the future.

Today's "Aha!" Moment

A queue usually begins with reactive work: a user uploads a video, so the system creates a transcode job now. Time-based work is different. The job may need to exist tomorrow at 09:00, every night at 02:00, or exactly 24 hours after a trial starts. The difficult question is no longer only "How do workers execute it?" but also "Who remembers that this must happen later?"

Keep one example in view. The learning platform sends reminder emails 24 hours before a live class, generates an instructor report every night, and deletes temporary uploads after they expire. All three are background jobs, but they are not triggered the same way. The reminder is a one-off promise tied to one class. The report is a recurring promise tied to the calendar. The cleanup pass is maintenance work that may tolerate being late, but must not silently stop forever.

That is the aha. Scheduling is a layer that turns time rules into durable intent. It answers questions a plain queue does not answer by itself: what if the process restarts before the job is due, what if the scheduler is down at the scheduled moment, what if the previous run is still active, and what if a late execution is still useful? A cron expression describes the ideal clock. The real architecture is the policy for when reality diverges from that clock.

Once you see scheduling that way, the system boundary becomes much clearer. The scheduler remembers promises about future work. The queue and worker pool execute work that is due now. Mixing those responsibilities carelessly is how teams end up with duplicate reminders, overlapping reports, or important delayed jobs that vanish after a deploy.

Why This Matters

The problem: Many teams treat scheduled work like a small configuration detail, then discover too late that "run every night" is actually a correctness problem with edge cases, policy decisions, and failure modes.

Before:

After:

Real-world impact: Fewer missed reminders, fewer duplicate reports, safer cleanup jobs, and a scheduling system operators can actually reason about during incidents and deploys.

Learning Objectives

By the end of this session, you will be able to:

  1. Distinguish recurring and delayed jobs - Explain why calendar-driven work and event-driven future work are not the same trigger model.
  2. Reason about scheduling correctness - Identify overlap, catch-up, skip, and lateness decisions before they become production bugs.
  3. Explain why scheduling needs durable state - Connect due-time tracking and scheduler recovery to reliable execution.

Core Concepts Explained

Concept 1: Delayed Jobs and Recurring Jobs Are Different Kinds of Promises

The reminder email for a live class and the nightly instructor report are both "scheduled," but they come from different sources of truth.

The reminder starts with an event: a class exists, so the system creates one delayed job to fire at class_start - 24h. The nightly report starts with a rule: every day at 02:00, produce a report for instructors. One follows a timestamp derived from an event. The other follows a continuing calendar policy.

class created at T0 ---------> reminder due at T0 + delta
every day at 02:00 ----------> report run for that day

That distinction matters because the lifecycle is different. A delayed job is often created once and then waits. A recurring job is often materialized repeatedly from a schedule definition. If you blur the two, you make it harder to answer simple but important questions: where did this run come from, can it be edited, and what counts as "missed"?

The trade-off is simplicity versus clarity. One unified scheduling subsystem is useful operationally, but only if it keeps the semantics explicit. Treating everything as "just cron" or "just a delayed message" makes the system harder to reason about as soon as product rules get more specific.

Concept 2: Scheduling Correctness Is Mostly About What Happens When the Ideal Clock Fails

Suppose the nightly report is due at 02:00, but the scheduler is down until 06:00. Or the previous run is still executing when the next slot arrives. Those are not strange edge cases. They are the normal moments where scheduling policy becomes visible.

At that point the real questions are:

02:00 due ---- scheduler unavailable ---- 06:00 recovery
                    |-> skip missed run
                    |-> run once late
                    |-> replay every missed slot

These choices are not purely infrastructural. They depend on product semantics. A reminder sent two minutes late may still be useful. A duplicate billing run may be unacceptable. A cleanup sweep may be fine to coalesce into one later run. Scheduling correctness is therefore a policy problem before it is a library problem.

The trade-off is stronger guarantees versus more coordination and more state. Once you require "never overlap," "replay every missed window," or "fire exactly one reminder per class," the scheduler needs explicit locking, bookkeeping, and recovery logic. That complexity is often worth paying because vague time semantics become vague user-facing behavior.

Concept 3: A Production Scheduler Needs Durable Intent and a Clean Handoff to Workers

Important scheduled work cannot live only inside a process timer. If the process restarts before the reminder is due, the promise disappears. A production scheduler therefore needs durable records for schedule definitions, due times, and execution attempts, plus a reliable way to hand due work to the normal execution path.

The architecture usually looks like this:

event or schedule rule
        |
        v
durable scheduler store
  - run_at / next_run_at
  - payload
  - execution state
        |
        v
claim due runs -> enqueue job -> worker pool executes

The scheduler is not the worker. Its job is to notice that work is due, claim it safely, and hand it off. The worker still needs the same discipline as any other async consumer: retries, idempotency, and observability.

def dispatch_due_runs(now):
    due_runs = scheduler.claim_due(before=now, limit=100)
    for run in due_runs:
        queue.publish(
            job_name=run.job_name,
            payload=run.payload,
            dedupe_key=run.fire_id,
        )
        scheduler.mark_enqueued(run.fire_id)

The point of the code is not the API shape. The point is the handoff boundary: due work is claimed from durable state, published into the execution system, and tracked so operators can tell whether it is pending, late, enqueued, or completed.

The trade-off is operational cost versus reliability. Durable schedulers require storage, monitoring, and explicit ownership, but they let the system survive restarts, recover after downtime, and explain what happened to time-based work instead of guessing.

Troubleshooting

Issue: Treating cron syntax as if it fully defines job correctness.

Why it happens / is confusing: The clock expression is visible, so teams mistake it for the whole design.

Clarification / Fix: Define overlap, missed-run, and lateness policy separately. The schedule string describes intent, not the full behavior under failure.

Issue: Assuming recurring work can always overlap safely.

Why it happens / is confusing: The job may seem independent until a slow run collides with the next one in production.

Clarification / Fix: Decide per job whether overlap is safe, must be serialized, or should cause the next run to be skipped or merged.

Issue: Keeping important delayed jobs only in memory or in opaque host-level cron.

Why it happens / is confusing: Local development makes process timers and manual cron entries feel good enough.

Clarification / Fix: If the job matters to the product or to operations, its schedule should be durable and visible inside the application's operational model.

Advanced Connections

Connection 1: Scheduling ↔ Queue Reliability

The parallel: Once due work is handed to a queue, all the usual distributed uncertainty returns: retries, duplicates, and consumer crashes.

Real-world case: A reminder job can be scheduled exactly once and still execute more than once unless the consumer path remains idempotent.

Connection 2: Scheduling ↔ Product Semantics

The parallel: Time-based work encodes promises made to users or operators, so scheduling policy should reflect what "late," "skipped," or "duplicated" means for the product.

Real-world case: A late report may be acceptable, but a duplicate billing run or a missed token rotation may not be.

Resources

Optional Deepening Resources

Key Insights

  1. Recurring and delayed jobs are different trigger models - One comes from a continuing rule, the other from a one-off future promise tied to an event.
  2. The real design is in the failure policy around time - Overlap, lateness, and missed runs matter more than the schedule string itself.
  3. A scheduler needs durable intent and a clear handoff boundary - It should remember promises about future work and then feed due work into the normal execution path safely.

Knowledge Check (Test Questions)

  1. What best distinguishes a delayed job from a recurring job?

    • A) A delayed job is created for one future timestamp tied to an event, while a recurring job is generated repeatedly from an ongoing schedule rule.
    • B) A delayed job always uses fewer workers.
    • C) A recurring job never needs persistence.

  2. What is the main architectural question when a scheduled run is missed because the scheduler was down?

    • A) Whether the system should skip, run late, or replay missed executions based on product semantics.
    • B) Which cron syntax extension the team prefers.
    • C) Whether the worker code should be rewritten in a faster language.

  3. Why should important scheduled work be stored durably before it is due?

    • A) Because the system may restart or recover late, and the promise about future execution still needs to survive.
    • B) Because durable storage guarantees perfect exactly-once execution.
    • C) Because delayed jobs never use queues or workers.

Answers

1. A: A delayed job is usually created once from an event and waits for its due time, while a recurring job is emitted repeatedly from a standing schedule definition.

2. A: Missed-run behavior is a policy decision about correctness. The key question is whether late execution, replay, or skipping matches the product's meaning of success.

3. A: Durable state preserves the system's memory of future obligations across restarts and outages. It improves reliability, but it does not magically remove the need for safe handoff and idempotent execution.

← Back to Learning