Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add workflow support #23

Merged
merged 34 commits into from
Nov 10, 2023
Merged

Add workflow support #23

merged 34 commits into from
Nov 10, 2023

Commits on Nov 9, 2023

  1. Initial structure for out of the application workflow orchestration

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    4ce7a63 View commit details
    Browse the repository at this point in the history

  2. Add Python Applications of the AMS-Orchestrator 

    * Add wrapper to start flux with proxy
* Add infrastructure of connecting to and receiving from RMQ
* Add initial daemon implementation
    @milroy @koparasy
    milroy authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    f75dc88 View commit details
    Browse the repository at this point in the history

  3. Restructure some of the orchestrator work

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b2afa65 View commit details
    Browse the repository at this point in the history

  4. Add Python AMS package structure and setup scripts

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    e8799fe View commit details
    Browse the repository at this point in the history

  5. Fixed several problems in AMS packages and added a AMSBroker to send … 

    …messages to a RabbitMQ queue

Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b37c397 View commit details
    Browse the repository at this point in the history

  6. Added initial scripts to launch AMS workflow (#4) 

    * Added initial script to bootstrap Flux on CORAL/IBM machine
* Added support Slurm based system, tested with flux-core 0.49 on Ruby/Lassen
* Added script to launch AMS miniapp with Flux
* Reverted script to support older version of flux, Lassen bootstrap only works wityh Flux<= 0.45 (tested with 0.45)
* Added scripts to add secrets on OC
* Added new scripts to launch the entire AMS workflow
* Upgrade all scripts, they are now fully functional (main script communicates with AMS daemon via RMQ)

---------

Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    3e38de9 View commit details
    Browse the repository at this point in the history

  7. Minor fix, removed absolute path 

    Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    8c8596d View commit details
    Browse the repository at this point in the history

  8. Added database backend for CSV 

    Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    e259d1f View commit details
    Browse the repository at this point in the history

  9. This commit addresses multiple problems in the broker part of AMS (#9) 

    * This commit addresses multiple problems in the broker part of AMS
- we are not sending input/output as encoded string anymore, we send binary blobs
- base64 has been removed
- a bug has been fixed with (very) old libevent version (<= 2.0.21-stable)
- offloading inputs/outputs to the thread managing RMQ is now much faster

* Moved to ResourceManager, created AMSMessage structures, moved to smart pointers.

* Complete re-design of the RabbitMQ backend

* Removed EventBuffer, removed pthread and signals. Big cleanup of the code.

* Added documentation and new AMSMsgHeader class + moved from memcpy to ResourceManager::copy
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    9ff3f3d View commit details
    Browse the repository at this point in the history

  10. Format code through black

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    053a12a View commit details
    Browse the repository at this point in the history

  11. Add tests for store, database

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    dd8a0cc View commit details
    Browse the repository at this point in the history

  12. Flake/Black/isort

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    3461b28 View commit details
    Browse the repository at this point in the history

  13. Adding pyproject.toml

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    72fb4cc View commit details
    Browse the repository at this point in the history

  14. Comments for store

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    8150a14 View commit details
    Browse the repository at this point in the history

  15. Add comments

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    5c1212e View commit details
    Browse the repository at this point in the history

  16. Renamed file and fixed imports

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    84ae52d View commit details
    Browse the repository at this point in the history

  17. Add comments on Stager

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    659fa3e View commit details
    Browse the repository at this point in the history

  18. Fix Staging mechanism

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    3d88e83 View commit details
    Browse the repository at this point in the history

  19. Add a random prune action

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    1f82753 View commit details
    Browse the repository at this point in the history

  20. Example conifuration

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    07a3c06 View commit details
    Browse the repository at this point in the history

  21. Action abstraction class

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c4dceeb View commit details
    Browse the repository at this point in the history

  22. Working version of data-sub-selection example and train script. Provi… 

    …des 'views' for kosh data
    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    2e7068e View commit details
    Browse the repository at this point in the history

  23. Provides CLI interface to the ams store. Adds end to end example scr… 

    …ipt to run AMS in sequential steps.
    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    9edf947 View commit details
    Browse the repository at this point in the history

  24. Added initial support for RMQ staging mechanism 

    Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    cd4ecde View commit details
    Browse the repository at this point in the history

  25. Add example file

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    fa59358 View commit details
    Browse the repository at this point in the history

  26. Fix compilation issues introduced by rebase

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    60e1564 View commit details
    Browse the repository at this point in the history

  27. Fixed some bugs introduced by previous rebase 

    Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    e2d28f2 View commit details
    Browse the repository at this point in the history

  28. Fixed resourcemanager bug 'makeAllocator Allocator with name HOST is … 

    …already registered.'

Signed-off-by: Loic Pottier <[email protected]>
    @lpottier @koparasy
    lpottier authored and koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    5e6a083 View commit details
    Browse the repository at this point in the history

  29. Revert "Fixed resourcemanager bug 'makeAllocator Allocator with name … 

    …HOST is already registered.'"

This reverts commit 50d1588.
    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    df9a7aa View commit details
    Browse the repository at this point in the history

  30. Fix test compilation issues

    @koparasy
    koparasy committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    a520727 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Fix python tests

    @koparasy
    koparasy committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d74e584 View commit details
    Browse the repository at this point in the history

  2. Fix end to end example

    @koparasy
    koparasy committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    b8445ab View commit details
    Browse the repository at this point in the history

  3. Make cmake aware of ctests

    @koparasy
    koparasy committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    baeb41e View commit details
    Browse the repository at this point in the history

  4. Make container aware of python code

    @koparasy
    koparasy committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    8aacc35 View commit details
    Browse the repository at this point in the history