Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Writable Warm] Composite Directory implementation and integrating it with FileCache #12782

Merged
merged 22 commits into from
Jun 20, 2024
Merged

[Writable Warm] Composite Directory implementation and integrating it with FileCache #12782

merged 22 commits into from
Jun 20, 2024

Commits on Jun 20, 2024

  1. Composite Directory POC 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    3c6ad60 View commit details
    Browse the repository at this point in the history

  2. Refactor TransferManager interface to RemoteStoreFileTrackerAdapter 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    30ddaea View commit details
    Browse the repository at this point in the history

  3. Implement block level fetch for Composite Directory 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    a7361d2 View commit details
    Browse the repository at this point in the history

  4. Removed CACHE state from FileTracker 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    54beed2 View commit details
    Browse the repository at this point in the history

  5. Fixes after latest pull 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    87772f0 View commit details
    Browse the repository at this point in the history

  6. Add new setting for warm, remove store type setting, FileTracker and … 

    …RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    8fb76d1 View commit details
    Browse the repository at this point in the history

  7. Modify TransferManager - replace BlobContainer with Functional Interf… 

    …ace to fetch an InputStream instead

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    932ad92 View commit details
    Browse the repository at this point in the history

  8. Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockComposi… 

    …teIndexInput

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    3e74318 View commit details
    Browse the repository at this point in the history

  9. Modify constructors to avoid breaking public api contract and code re… 

    …view fixes

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    dca1a07 View commit details
    Browse the repository at this point in the history

  10. Add experimental annotations for newly created classes and review com… 

    …ment fixes

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    723aba0 View commit details
    Browse the repository at this point in the history

  11. Use ref count as a temporary measure to prevent file from eviction un… 

    …til uploaded to Remote

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    959f90c View commit details
    Browse the repository at this point in the history

  12. Remove method level locks 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    2632738 View commit details
    Browse the repository at this point in the history

  13. Handle tmp file deletion 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    ba34798 View commit details
    Browse the repository at this point in the history

  14. Nit fixes 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    85003ee View commit details
    Browse the repository at this point in the history

  15. Handle delete and close in Composite Directory, log current state of … 

    …FileCache and correct it's clear method and modify unit and integration tests as per review comments

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    de1895e View commit details
    Browse the repository at this point in the history

  16. Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_IN… 

    …DEX_SETTING

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    f9d880b View commit details
    Browse the repository at this point in the history

  17. Add tests for FileCachedIndexInput and review comment fixes 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    e1b18ad View commit details
    Browse the repository at this point in the history

  18. Add additional IT for feature flag disabled 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    5e008db View commit details
    Browse the repository at this point in the history

  19. Move setting for Partial Locality type behind Feature Flag, fix bug f… 

    …or ref count via cloneMap in FullFileCachedIndexInput and other review fixes

Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    c1e6b18 View commit details
    Browse the repository at this point in the history

  20. Minor test and nit fixes 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    74f9c29 View commit details
    Browse the repository at this point in the history

  21. Add javadocs for FullFileCachedIndexInput 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    7fce3d3 View commit details
    Browse the repository at this point in the history

  22. Minor precommit fixes 

    Signed-off-by: Shreyansh Ray <[email protected]>
    @rayshrey
    rayshrey committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    901849b View commit details
    Browse the repository at this point in the history