Background tasks and scheduled jobs with Celery

Some work is too slow to do inside a web request. Generating a PDF report, sending appointment-reminder SMS to a hundred patients, processing an uploaded scan — if the view does these, the user stares at a spinner for ten seconds, or the request times out. The answer is to move slow work out of the request into a background task: the view hands off the work and returns immediately, and a separate worker does it. In the Django world the standard tool is Celery. This lesson is why, when, and how — at the level you need to use it well.

The problem: slow work in the request cycle

A request should be fast — ideally tens of milliseconds. When a view does something slow, three things go wrong: the user waits (bad experience), a web worker is tied up for the whole duration (fewer requests served), and long operations risk hitting the server's request timeout and failing outright. Sending reminder messages to every patient with an appointment tomorrow might take thirty seconds — doing that in the request handler is simply the wrong place for it. The work needs to happen, but not while the user waits.

The solution: hand off to a task queue

A task queue splits the work in two: the web process enqueues a task (fast — just drops a message on a queue) and returns to the user immediately; a separate worker process picks the task off the queue and runs it. The pieces:

Celery — the task queue library: you write tasks as Python functions and call them asynchronously.
A broker (usually Redis or RabbitMQ) — the queue itself, where enqueued tasks wait for a worker.
Worker processes — separate processes (celery worker) that consume tasks and execute them, entirely outside the web server.

So a slow operation becomes: the view enqueues, returns instantly; the worker does the slow part in the background. The user is not kept waiting, and the web workers stay free to serve requests.

Writing and calling a task

A Celery task is an ordinary function with a decorator:

# patients/tasks.py
from celery import shared_task
from django.core.mail import send_mail

@shared_task
def send_appointment_reminder(appointment_id):
    appt = Appointment.objects.select_related("patient").get(id=appointment_id)
    send_mail(
        subject="Appointment reminder",
        message=f"Reminder: your appointment is on {appt.scheduled_for:%d %b}.",
        from_email="clinic@nidaan.example.com",
        recipient_list=[appt.patient.email],
    )

The view enqueues it with .delay() instead of calling it directly:

def book_appointment(request):
    # ... create the appointment ...
    send_appointment_reminder.delay(appointment.id)   # enqueue — returns immediately
    return redirect("appointment_list")               # user is not kept waiting

.delay(appointment.id) puts the task on the queue and returns at once; a worker runs send_appointment_reminder moments later. Pass IDs, not objects — a task receives simple, serialisable arguments (an integer id), and re-fetches the object inside the task, because the object may have changed by the time the worker runs and complex objects do not serialise cleanly onto a queue. This is the single most common Celery mistake to avoid.

Scheduled jobs: Celery Beat

The other kind of background work is scheduled — "every morning at 7, send reminders for today's appointments" — with no request triggering it at all. Celery Beat is the scheduler: it enqueues tasks on a timetable.

# celery config
from celery.schedules import crontab

app.conf.beat_schedule = {
    "daily-reminders": {
        "task": "patients.tasks.send_todays_reminders",
        "schedule": crontab(hour=7, minute=0),      # every day at 07:00
    },
}

Beat runs as its own process and, at the scheduled time, drops the task on the queue for a worker to run. So Celery covers both shapes of background work: on-demand (enqueued by a view with .delay()) and scheduled (enqueued by Beat on a timetable). Appointment reminders, nightly report generation, cleaning up expired records — all are scheduled tasks.

The operational reality — and the lighter alternatives

Be honest about the cost: Celery adds moving parts to your deployment. You now run web processes and worker processes and a broker (Redis), and must monitor all three — a worker that dies silently means tasks pile up undone. That is real operational weight, justified when you genuinely have slow or scheduled work, and overkill when you do not. Two honest caveats:

Do not reach for Celery for trivial deferral. If you occasionally need to run something slightly later, simpler options exist — a management command run by cron, small packages like django-q or huey, or Django's own background-tasks framework being introduced in recent releases. Celery is the heavyweight, full-featured choice; match the tool to the need.
Tasks must be idempotent and handle failure. A worker can crash mid-task, or a task can be retried — so a task should be safe to run more than once (sending a reminder twice is bad; charging a card twice is a disaster). Design tasks to be retry-safe, and configure retries for transient failures.

The rule: move slow and scheduled work out of the request into a task queue — but treat the queue as real infrastructure with real failure modes, and pick the lightest tool that meets the need. For a clinic sending reminders and generating nightly reports, Celery with Redis is the standard, sound choice; for a single occasional deferred email, it is more than you need.

Check your work

Why move work out of the request. Slow work in a view makes the user wait, ties up a web worker, and risks a request timeout — the work should happen, but not while the user waits.

The pieces. Celery (task library), a broker (Redis/RabbitMQ — the queue), and separate worker processes that run tasks outside the web server.

Writing and calling a task. @shared_task on a function; enqueue with .delay(...) which returns immediately. Pass IDs, not objects, and re-fetch inside the task.

Scheduled jobs. Celery Beat enqueues tasks on a timetable (crontab(...)) with no request — for daily reminders, nightly reports, cleanup.

The operational cost. Celery adds processes (workers, broker) to run and monitor; use it for genuine slow/scheduled work, and lighter tools (cron, django-q/huey, or Django's newer built-in tasks framework) for trivial deferral.

Task safety. Tasks must be idempotent (safe to run more than once) and handle/retry failures, because workers can crash or retry.

Practice

Write a @shared_task that sends a reminder given an appointment_id; call it with .delay(id) from a view and confirm the view returns immediately.
Deliberately pass the whole appointment object to .delay() instead of its id and reason about why passing the id and re-fetching is safer.
Sketch a Celery Beat schedule that runs a "send today's reminders" task every morning; explain what runs it (Beat) and what executes it (a worker).
List the processes a Celery deployment adds beyond the web server, and what monitoring each needs.
For three slow operations, decide Celery versus a lighter option (cron/management command/Django tasks) and justify each.
Take a task that sends a message and make it idempotent, so a retry does not send twice.

Official documentation

Celery — First steps with Django — Wiring Celery into a Django project.
Celery — Periodic tasks (Beat) — Scheduled jobs.
huey / django-q — Lighter task queues for simpler needs than Celery.

Next: caching — the cache framework and where it actually helps.