Django Tasks Integration

Warning

This integration is experimental. Its behaviour, events, metadata and signals may change in future releases.

We are referring to Django’s django.tasks and not django-tasks backport.

This integration requires Django 6.0+ and a task backend compatible with Django’s task framework.

Getting Started with Django Tasks

In order to be able to support django.tasks you need to configure both your webapp and your workers.

How it works

django.tasks does not provide a way to attach custom metadata to a task. django-structlog stores the logging context in a dedicated Django cache instead:

  1. When a task is enqueued (task_enqueued signal), the current structlog context is saved in the django_structlog cache using the task’s id as key.

  2. When the task starts (task_started signal), the context is cleared, task_id is bound and the saved context is retrieved from the cache and bound again.

  3. When the task finishes (task_finished signal), task_succeeded or task_failed is logged.

Warning

The context is stored in the django_structlog cache. It should only contain data that can be serialized by your cache backend (pickle for Django’s built-in backends). You can use modify_context_before_task_publish to make sure this is the case.

Replace your requirements

First of all, make sure your django-structlog installation knows you use django.tasks in order to validate compatibility with your installed version. See Installing “Extras” for more information.

Replace django-structlog with django-structlog[django-tasks] in your requirements.txt.

django-structlog[django-tasks]==X.Y.Z

Enable Django Tasks integration in your web app

In your settings.py

MIDDLEWARE = [
    # ...
    'django_structlog.middlewares.RequestMiddleware',
]

DJANGO_STRUCTLOG_DJANGO_TASKS_ENABLED = True

Configure the django_structlog cache

The integration requires a cache named django_structlog. Django will fail to start if it is missing while the integration is enabled.

CACHES = {
    "default": {
        # ...
    },
    "django_structlog": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379",
        # Must be longer than the delay between a task's enqueue and its start
        "TIMEOUT": 60 * 60 * 24,
    },
}

Warning

The cache must be shared between the processes enqueueing the tasks (ex: your web app) and the workers running them (ex: Redis, Memcached or database caching).

LocMemCache is only suitable for tests or when the task runs in the same process (ex: django.tasks.backends.immediate.ImmediateBackend).

Note

Context entries are stored without an explicit timeout, therefore the cache’s TIMEOUT applies (Django’s default is 300 seconds). If a task starts after its entry expired, the task’s logs will not contain the caller’s context. Set a TIMEOUT greater than the longest delay you expect between a task being enqueued and being started (including tasks scheduled with run_after).

Entries are not deleted once the task has started, they are evicted when they expire.

Configure your workers

Your workers must use the same settings as your web app: django_structlog in INSTALLED_APPS, DJANGO_STRUCTLOG_DJANGO_TASKS_ENABLED = True and the same django_structlog cache. This way the receivers are connected when the worker initializes Django.

Make sure your worker also configures logging and structlog (see Getting Started), otherwise the task events will not be rendered as expected.

Signals

modify_context_before_task_publish

You can connect to modify_context_before_task_publish signal in order to modify the metadata before it is stored in the cache.

By example you can strip down the context to keep only some of the keys:

from django.dispatch import receiver
from django_structlog.tasks import signals

@receiver(signals.modify_context_before_task_publish)
def receiver_modify_context_before_task_publish(sender, signal, context, **kwargs):
    keys_to_keep = {"request_id", "parent_task_id"}
    new_dict = {key_to_keep: context[key_to_keep] for key_to_keep in keys_to_keep if key_to_keep in context}
    context.clear()
    context.update(new_dict)

See django_structlog.tasks.signals.modify_context_before_task_publish.

bind_extra_task_metadata

You can optionally connect to bind_extra_task_metadata signal in order to bind more metadata to the logger or override existing bound metadata. This is called when the task starts, after the context was restored and before task_started is logged.

from django.dispatch import receiver
from django_structlog.tasks import signals
import structlog

@receiver(signals.bind_extra_task_metadata)
def receiver_bind_extra_task_metadata(sender, signal, task_result=None, logger=None, **kwargs):
    structlog.contextvars.bind_contextvars(task_backend=task_result.backend)

See django_structlog.tasks.signals.bind_extra_task_metadata.

pre_task_succeeded

You can optionally connect to pre_task_succeeded signal in order to bind more metadata before task_succeeded is logged.

from django.dispatch import receiver
from django_structlog.tasks import signals
import structlog

@receiver(signals.pre_task_succeeded)
def receiver_pre_task_succeeded(sender, signal, logger=None, task_result=None, **kwargs):
    structlog.contextvars.bind_contextvars(result=str(task_result.return_value))

See django_structlog.tasks.signals.pre_task_succeeded.

Events and Metadata

See Django Tasks.