Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize parallel listing strategy #625

Closed
alexvanin opened this issue Jul 26, 2022 · 2 comments · Fixed by #655
Closed

Optimize parallel listing strategy #625

alexvanin opened this issue Jul 26, 2022 · 2 comments · Fixed by #655
Assignees
Labels
enhancement Improving existing functionality
Milestone

Comments

@alexvanin
Copy link
Contributor

Current PR just reduce likelihood of error.

Agree. Let's think about it in separate issue. I thought about combining expectations and dynamic feedback to nodesGenerator()

  1. User expects 5 items so nodesGenerator() generate exactly 5 node versions (I will call it jobs) into the channel
  2. In no request's failing, then it is optimal solution
  3. If some requests fail we should update strategy by giving feedback to nodesGenerator():
    3.1) Re-request the same object in the same go-routine without feedback
    3.2) Produce feedback so nodesGenerator() will produce one more job to do.

This solution might slower than current implementation in case of network fails. However it should be robust and consistent when network is healthy. I think it is more important.

Originally posted by @alexvanin in #622 (comment)

@KirillovDenis
Copy link
Contributor

KirillovDenis commented Aug 4, 2022

Let's check if tree-service can provide wanted metadata. Maybe we can get rid of head requests at all.

@KirillovDenis
Copy link
Contributor

Let's check if tree-service can provide wanted metadata

PR #598 add size and etags in NodeVersion but this is still not enough to get rid of head operation during listing. We need:

  • LastModified time
  • Owner info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improving existing functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants