Skip to content
This repository has been archived by the owner on Feb 4, 2022. It is now read-only.

♻️ Migrate from RQ to Celery for async jobs #766

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

jozsefsallai
Copy link
Member

@jozsefsallai jozsefsallai commented Aug 17, 2021

TODO

  • refactor tasks to run through celery instead of rq
  • make unit tests be aware of the change*
  • make celery beat schedules configurable
  • reconfigure docker/supervisord for celery
  • use rabbitmq as a message broker instead of redis
  • reconfigure docker for rabbitmq
  • orchestrate containers using k8s for better scalability
  • port health check tools (rq_helpers) to celery (maybe we dont have to)

*) There are a few unit tests regarding media uploads in posts that will also check if the post is in "pending" state; since the @celery_use_eager decorator will make it so jobs are run as if they were regular synchronous function calls, the API endpoint will not send back a response until the post media is processed, so the post is never in "pending" state. For now, I've uncommented the lines that test for the pending state, but we will need to employ a strategy to monitor the relevant job and block the unit test until it finishes, instead of blocking the execution of the API endpoint.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant