Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workers can stream results to Consumer Group subscribers for results from a Queue #16

Open
codecakes opened this issue May 11, 2023 · 0 comments

Comments

@codecakes
Copy link
Contributor

codecakes commented May 11, 2023

Many requesting clients should be able to access results from the same queue. This is not possible using the redis-queue structure.

The nature of a queue is that each item can only be dequeued (or popped) once. Once an item is dequeued by a client, it is removed from the queue and no longer available for other clients to dequeue.

This proposal should allow multiple clients to subscribe to a consumer group just like using redis streams yet all of them can fetch the result from the enqueued job.

As a worker client, Given #15,
When a job for a queue is alloted for which that worker is assigned,
The worker should store the job_id|worker_id|timestamp into redis,
So that it can:

  1. process that job AND
  2. synchronously fetch results from redis AND
  3. stream it to a redis stream key which is a consumer group.
@codecakes codecakes changed the title Dynamic Consumer Group subscribers to Queue Workers can stream results to Consumer Group subscribers for results from a Queue May 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant