Celery UnicodeDecodeError with django

Question

I have some issues with celery==3.0.23 on Django==1.5,

Let me put you all in context, i'm using Celery with RabbitMQ as message broker to maintain a queue of asynchronous tasks in my django project because they involve too much interaction with remote services to do it online.

Here's my django-celery configuration in my settings.py, I also have celery and djcelery in my INSTALLED_APPS tuple

import djcelery

BROKER_URL = 'amqp://guest:guest@localhost:5672'
CELERY_ROUTES = {
    'users.tasks.sync_messages': {'queue': 'cron'},
    'users.tasks.update_balance': {'queue': 'cron'},
    'profiles.tasks.*': {'queue': 'job'}
}

CELERYBEAT_SCHEDULE = {
    'sync-messages': {
        'task': 'users.tasks.sync_messages',
        'schedule': crontab(minute='*/4'),
    },
    'update-balance': {
        'task': 'users.tasks.update_balance',
        'schedule': crontab(minute='*/5'),
    },
}

(I have some scheduled jobs too)

I defined some methods in profiles.helpers and I call this methods as explained in the celery's docs:

method.delay(arg1, arg2...)

and it rejects the job due to a UnicodeDecodeError

I ran celeryd with -l DEBUG option and this is the output when I try to enqueue the job:

[2013-09-13 17:19:56,108: DEBUG/MainProcess] consumer: Ready to accept tasks!
[2013-09-13 17:21:13,970: WARNING/MainProcess] Traceback (most recent call last):
[2013-09-13 17:21:13,970: WARNING/MainProcess] File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
[2013-09-13 17:21:14,020: WARNING/MainProcess] msg = self.format(record)
[2013-09-13 17:21:14,021: WARNING/MainProcess] File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
[2013-09-13 17:21:14,021: WARNING/MainProcess] return fmt.format(record)
[2013-09-13 17:21:14,021: WARNING/MainProcess] File "/home/israelord/.virtualenvs/ringtu/local/lib/python2.7/site-packages/celery/utils/log.py", line 110, in format
[2013-09-13 17:21:14,041: WARNING/MainProcess] return safe_str(logging.Formatter.format(self, record))
[2013-09-13 17:21:14,041: WARNING/MainProcess] File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
[2013-09-13 17:21:14,041: WARNING/MainProcess] record.message = record.getMessage()
[2013-09-13 17:21:14,041: WARNING/MainProcess] File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
[2013-09-13 17:21:14,042: WARNING/MainProcess] msg = msg % self.args
[2013-09-13 17:21:14,042: WARNING/MainProcess] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 58: ordinal not in range(128)
[2013-09-13 17:21:14,042: WARNING/MainProcess] Logged from file consumer.py, line 589

I know there is not much information here but I have no idea on how to debug this

I think it has to do with how Celery serializes the tasks, but I tried several serializers and got the same result, I was reading some blog posts and found that this was a very common issue on Celery < 3.0 versions and that it was solved on Celery 3.0

Can anybody give me a light on this?

Thank you very much taking the time to read and help.

Answer 1

Generally it's not a good idea to pass Django queryset objects or lazily evaluated objects like User into Celery tasks as they don't serialize well. I would suggest simplifying what you're sending by only passing the username string and then inside the celery task query the DB for that user again.

Answer 2

You can try to add following code at the beginning of the py files where you emit the log.

import sys
reload(sys)
sys.setdefaultencoding('utf8')