Celery task schedule (Celery, Django and RabbitMQ)

Question

I want to have a task that will execute every 5 minutes, but it will wait for last execution to finish and then start to count this 5 minutes. (This way I can also be sure that there is only one task running) The easiest way I found is to run django application manage.py shell and run this:

while True:
    result = task.delay()
    result.wait()
    sleep(5)

but for each task that I want to execute this way I have to run it's own shell, is there an easy way to do it? May be some king custom ot django celery scheduler?

Answer 1

All you need is specify in celery conf witch task you want to run periodically and with which interval.

Example: Run the tasks.add task every 30 seconds

from datetime import timedelta

CELERYBEAT_SCHEDULE = {
    "runs-every-30-seconds": {
        "task": "tasks.add",
        "schedule": timedelta(seconds=30),
        "args": (16, 16)
     },
}

Remember that you have to run celery in beat mode with the -B option

manage celeryd -B

You can also use the crontab style instead of time interval, checkout this:

http://ask.github.com/celery/userguide/periodic-tasks.html

If you are using django-celery remember that you can also use tha django db as scheduler for periodic tasks, in this way you can easily add trough the django-celery admin panel new periodic tasks. For do that you need to set the celerybeat scheduler in settings.py in this way

CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"

Answer 2

Wow it's amazing how no one understands this person's question. They are asking not about running tasks periodically, but how to ensure that Celery does not run two instances of the same task simultaneously. I don't think there's a way to do this with Celery directly, but what you can do is have one of the tasks acquire a lock right when it begins, and if it fails, to try again in a few seconds (using retry). The task would release the lock right before it returns; you can make the lock auto-expire after a few minutes if it ever crashes or times out.

For the lock you can probably just use your database or something like Redis.

Answer 3

You may be interested in this simpler method that requires no changes to a celery conf.

@celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=5))
def my_task():
    # Insert fun-stuff here

Answer 4

To expand on @MauroRocco's post, from http://docs.celeryproject.org/en/v2.2.4/userguide/periodic-tasks.html

Using a timedelta for the schedule means the task will be executed 30 seconds after celerybeat starts, and then every 30 seconds after the last run. A crontab like schedule also exists, see the section on Crontab schedules.

So this will indeed achieve the goal you want.

Answer 5

Because of celery.decorators deprecated, you can use periodic_task decorator like that:

from celery.task.base import periodic_task
from django.utils.timezone import timedelta

@periodic_task(run_every=timedelta(seconds=5))
def my_background_process():
    # insert code

Answer 6

Add that task to a separate queue, and then use a separate worker for that queue with the concurrency option set to 1.