Sudden error: "No staging files found

[jannikbend]

Describe the bug
After running rudderstack for a while a “no staging files found” message gets logged with error log level pretty frequently. See below:

“{"level":"WARN","ts":"2024-08-07T05:26:49.148Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/upload.go:600","msg":"upload error","uploadJobID":30514,"namespace":"<…>","sourceID":"<…>","sourceType":"Node","destinationID":"CLICKHOUSE_DESTINATION","destinationType":"CLICKHOUSE","workspaceID":"<…>","useRudderStorage":false,"uploadStatus":"internal_processing_failed","error":"no staging files found","priority":100,"retried":false,"attempt":0,"loadFileType":"csv","errorMapping":"uncategorised","destinationCredsValid":null}”
{"level":"ERROR","ts":"2024-08-07T06:32:13.889Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/router.go:274","msg":"[WH] Failed in handle Upload jobs for worker: %!w(*errors.errorString=&{s:no staging files found})"}

The error is often combined with the following:

“{"level":"WARN","ts":"2024-08-07T05:26:49.148Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/upload.go:505","msg":"error setting upload status","uploadJobID":30514,"namespace”:”<…>”,”sourceID":"<…>","sourceType":"Node","destinationID":"CLICKHOUSE_DESTINATION","destinationType":"CLICKHOUSE","workspaceID":"<…>","useRudderStorage":false,"error":"upload not found"}”,

however that message is logged as warning and not as an error.

The Job IDs mentioned in the log message (30514) does indeed not exist in the Postgres database, but we do also observe no missing or failed events in any case.

Steps to reproduce the bug
Unclear: On a freshly installed rudderstack instance this does not occur.
The error suddenly occurs after running low impact load tests for a few hours.

Expected behavior
No errors should occur

Screenshots
None

Any additional context
Rudder-Server: 1.29.0
Transformer: 1.49.0
Related Slack Post: https://rudderstack.slack.com/archives/C01E4PLB135/p1723477378948789

[contributor-support]

Thanks for opening this issue! We'll get back to you shortly. If it is a bug, please make sure to add steps to reproduce the issue.

[gitcommitshow]

@jannikbend Ideally it should not happen if it is a valid upload job. Although there is one scenario where we merge old waiting job with a new one and delete the old upload job. If for some reason because of race condition the old upload job got picked up to process it will not see any staging files because all of them will be marked for the new one.