Dead letter handling

[ewhauser]

This might be a lack of comprehension from the docs, but I'm trying to understand what the pattern for dead letter handling is in river. My understanding is that if a job is an error state that it is eventually cancelled after exhausting retries. Once a job is cancelled it will eventually be cleaned from the database. What's the preferred pattern for moving the jobs to the side on error and then explicitly queuing again?

[bgentry]

Hi @ewhauser! After a job has run out of attempts, it is moved to the discarded state. This means no further attempts will be performed on that job. It will also be pruned from the jobs table after the DiscardedJobRetentionPeriod has elapsed.

You have the opportunity to take an action when the job errors for a final time using an error handler, or you can write your own logic to periodically look for discarded jobs and then alert you or whatever you’d like to do in response to those.

Let me know if you have any ideas on how this could be made clearer in the docs! One thing I know we’re missing is higher level stuff like just describing all the different job states, and the general flow that jobs go through during their lifetime.

[ewhauser]

Yes, this does make sense. Can the error handler participate in the transaction? i.e. if you insert the JobRow into another table could you transactional commit that with the status change for the job to be discarded?

I do think dead letter queue functionality is pretty standard functionality for queue systems. Would there be openness for feature request for this?

[bgentry]

No, the transaction is not available to the error handler and it wouldn’t make sense to have it push back against the client’s ability to mark the job as discarded so it can move on.

It’s hard to know exactly what to recommend without hearing more about how you plan to use this, but you can achieve whatever transactional behavior you want on the job rows after they are marked as discarded. For example you could insert additional cleanup jobs at the same time as deleting the discarded job rows yourself (instead of waiting for the job cleaner to delete them). Imagine a unique periodic job that runs every X minutes and looks for jobs that were discarded within the past X minutes, performing whatever dead letter logic you want on them at that time.

[ewhauser]

We have a queue that is updating an external system via API (at least once behavior is fine as it idempotent). Sometimes, the external system can have a configuration issue that will cause the update to fail. However, even if that is the case, we need an engineer to investigate the issue as opposed having the job being dropped. Our internal SLA on resolving these errors is quite lengthy; since they rare we only need to validate them prior to running invoicing for the next month.

The suggestion to use a periodic job to move the rows to a dead letter queue makes sense. Thank you.