Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add multiprocessing #92

Open
wants to merge 99 commits into
base: developer
Choose a base branch
from
Open

add multiprocessing #92

wants to merge 99 commits into from

Commits on Apr 18, 2024

  1. Add files for multiprocessing

    qgh533 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    0e2242e View commit details
    Browse the repository at this point in the history

  2. Update identify_associations_multiprocess.py

    @LFT18
    LFT18 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    2add9b9 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Clean multiprocessing script

    @LFT18
    LFT18 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    f645ea4 View commit details
    Browse the repository at this point in the history

  2. Update __main__.py multiprocessing

    @LFT18
    LFT18 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    f471704 View commit details
    Browse the repository at this point in the history

  3. Update schema.py multiprocessing

    @LFT18
    LFT18 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    85c28e5 View commit details
    Browse the repository at this point in the history

  4. Update __init__.py multiprocessing

    @LFT18
    LFT18 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    6330e92 View commit details
    Browse the repository at this point in the history

  5. Update preprocessing.py

    @LFT18
    LFT18 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    bbe1b4e View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. 🔥 clean-up duplicated src/move files (pkg was in main folder)

    Henry Webel committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    820c554 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. ✨ add identify_associations_multiprocess to src/move/tasks 

    - use pip install to make it available
-minor adaptions to original scritp w.r.t. to imputs
    Henry Webel committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    ce9a9dc View commit details
    Browse the repository at this point in the history

  2. 🐛 make mutliprocessing not stale: assign # of threads for each process 

    ref: https://pytorch.org/docs/stable/notes/multiprocessing.html#avoid-cpu-oversubscription
    Henry Webel committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    5327223 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #1 from enryH/main 

    Fix Multiprocessing
    @LFT18
    LFT18 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    eaa858a View commit details
    Browse the repository at this point in the history

  4. Updated identify_associations_multiprocess.py

    qgh533 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    5ab5e59 View commit details
    Browse the repository at this point in the history

  5. Update config files for small tries

    qgh533 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    33f565a View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Multiprocessing for analyze_latent

    qgh533 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    63f128b View commit details
    Browse the repository at this point in the history

  2. Analyze latent multiprocessing

    qgh533 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    ca389d2 View commit details
    Browse the repository at this point in the history

  3. Analyze latent multiprocessing

    qgh533 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e08a94b View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Fix bayes_k calculation

    qgh533 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    e94ef90 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Fix analyze_latent_multiprocessing

    qgh533 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    f4f0aa3 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. Update and new functions

    qgh533 committed May 11, 2024
    Configuration menu
    Copy the full SHA
    6a0b665 View commit details
    Browse the repository at this point in the history

  2. Delete files and fix multiloop

    qgh533 committed May 11, 2024
    Configuration menu
    Copy the full SHA
    a5310a6 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Clean identify_association_multiprocess.py

    qgh533 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    e67bb75 View commit details
    Browse the repository at this point in the history

  2. Clean analyze_latent multiprocessing.py

    qgh533 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    86bfed5 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Update perturbations.py

    @LFT18
    LFT18 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    f9d4961 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Update perturbations.py

    @LFT18
    LFT18 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    c2c49e8 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Delete src/move/tasks/analyze_latent_efficient.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    0a4bcae View commit details
    Browse the repository at this point in the history

  2. Delete src/move/tasks/analyze_latent_multiprocessing.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ce20dac View commit details
    Browse the repository at this point in the history

  3. Delete src/move/tasks/identify_associations_multiprocess_loop.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    4a72842 View commit details
    Browse the repository at this point in the history

  4. Delete src/move/tasks/identify_associations_multiprocess_may.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f537d21 View commit details
    Browse the repository at this point in the history

  5. Delete src/move/tasks/identify_associations_selected.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    568aaa8 View commit details
    Browse the repository at this point in the history

  6. Delete src/move/tasks/analyze_latent_original.py

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ede3707 View commit details
    Browse the repository at this point in the history

  7. Remove multiprocess_loop

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    5df8a01 View commit details
    Browse the repository at this point in the history

  8. Remove multiprocess_loop

    @LFT18
    LFT18 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    52c37fd View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. 🎨 format with black

    Henry committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    2e29f23 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into LFT18-main

    Henry committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    13678cb View commit details
    Browse the repository at this point in the history

  3. Merge branch 'developer' into LFT18-main

    Henry committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    f92a862 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. 🎨 add trigger of actions from PR

    Henry committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    b7824f7 View commit details
    Browse the repository at this point in the history

  2. 🎨 format with black

    Henry committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    e5253a2 View commit details
    Browse the repository at this point in the history

  3. 🔥 remove duplicated code and intermediate scripts

    Henry committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    d4118a3 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Merge branch 'developer' into LFT18-main

    Henry committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    3e19e24 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. 🐛 fix f-string formatting errors 

    - multiline { not allowed
    Henry committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    dbc0238 View commit details
    Browse the repository at this point in the history

  2. 🐛 remove unused imports

    Henry committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    80352e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. ⏪ add configuration files back in from developer branch

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    488b4a4 View commit details
    Browse the repository at this point in the history

  2. 🎨 isort imports

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    9b3a27e View commit details
    Browse the repository at this point in the history

  3. 🚧 see if this advances CI to the next step

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    05d4c34 View commit details
    Browse the repository at this point in the history

  4. 🔥 remove intermediate files of development 

    - saved in separate branch for archiving: https://github.com/LFT18/multiprocessing_bayes_MOVE/tree/archiv_intermediate_exp
    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    ebb72ad View commit details
    Browse the repository at this point in the history

  5. 🚧 multiprocess only defined for bayes factors

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    efbfd5c View commit details
    Browse the repository at this point in the history

  6. 🐛 remove non-existing, intermediate tasks (used for developing), no m… 

    …ultiprocessing for t-tests

- document that t-test is not available for multiprocessing -> key is still needed in Schema
- do not set key in tutorial config as it it not implemented for t-test
    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    fbbeb19 View commit details
    Browse the repository at this point in the history

  7. 🐛 also deactivate mutliprocessing for KS as it's not implemented

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    cb3ad30 View commit details
    Browse the repository at this point in the history

  8. 🎨 fix flake8-bugbear issues except missing multiprocessing of t-test … 

    …and KS approach
    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8c61d35 View commit details
    Browse the repository at this point in the history

  9. 🐛 format and fix import

    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    5aa03ff View commit details
    Browse the repository at this point in the history

  10. 🐛 use perturb_continuous_data_extended from perturbations 

    (as defined on developer branch)
- remove debug log statements here
- the version in "identify_associations.py" had a hardcorded number of parameters (5): num_features = 5
    Henry committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8b06298 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. 🔥 comments and old configurations; format

    Henry committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    6cfd1f8 View commit details
    Browse the repository at this point in the history

  2. 🔥 remove duplicated functionality 

    - use only one version in extra module
    Henry committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    8277891 View commit details
    Browse the repository at this point in the history

  3. ✨ integrate multiprocessing into analyze_latent.py 

    - add test with multiprocessing activated
- unify modules into one (with the option for both approaches)
    Henry committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    44802eb View commit details
    Browse the repository at this point in the history

  4. ✨ merge multiprocessing bayes factors into identify_associations 

    - add possibility to multiprocess to itdentify associations
    Henry committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    f64d779 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. 🔥 remove old schema entries, increase run time

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    6a17110 View commit details
    Browse the repository at this point in the history

  2. :zip: do no save intermediate files for single-process bayes_approach

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    b8b4769 View commit details
    Browse the repository at this point in the history

  3. 🔥 remove comments

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    2927d3f View commit details
    Browse the repository at this point in the history

  4. 🔥 remove unused code

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    202eb74 View commit details
    Browse the repository at this point in the history

  5. 🎨 reorder functions

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    d6bc896 View commit details
    Browse the repository at this point in the history

  6. 🚧 move bayes_parallel to own module 

    - start rewriting _bayes_approach to contain more well defined steps
  (define separate functions for each step)
- check that the results will match roughly
    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    b99ce97 View commit details
    Browse the repository at this point in the history

  7. 🚧 unify interface

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    171c915 View commit details
    Browse the repository at this point in the history

  8. 🔥 remove not-used code

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    95c9f40 View commit details
    Browse the repository at this point in the history

  9. 🎨 start separating recurrent code into fcts

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    e556fae View commit details
    Browse the repository at this point in the history

  10. 🎨 adapt to look more similar to single-core bayes factor fct

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    1a2774d View commit details
    Browse the repository at this point in the history

  11. ✨ add back masking of self-perturbed feat.

    Henry committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    680c164 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. 🎨 initiailize logger at the top of the module

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    8f43c90 View commit details
    Browse the repository at this point in the history

  2. ✨ pass feature_mask to bayes_parallell 

    - pass feature mask
    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    6178c8f View commit details
    Browse the repository at this point in the history

  3. 🐛 add condition for masking 

    - finish aligning variable names (bayes_k)
    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    60ed227 View commit details
    Browse the repository at this point in the history

  4. 🎨 align masking strategies

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    f5da671 View commit details
    Browse the repository at this point in the history

  5. 🐛 fix cont perturbation 

    - before more than one feat was changed,
and probably not the desired one.
- out of index slice  in numpy returns just an empty array!

see: https://stackoverflow.com/questions/54613753/why-does-python-allow-out-of-range-slice-indexes-for-sequences
    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    7eae82d View commit details
    Browse the repository at this point in the history

  6. 🐛 remove redefintion of nan_mask

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    50a623e View commit details
    Browse the repository at this point in the history

  7. 🎨 only define logger once in module

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2df057b View commit details
    Browse the repository at this point in the history

  8. 🎨 align single process bayes and multiprocess bayes fct 

    - allows easier comparison of functionality
    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2e87e12 View commit details
    Browse the repository at this point in the history

  9. 🎨 just document in code that this cannot happen 

    - 🐛 also add missing imports
    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    aa2e5d9 View commit details
    Browse the repository at this point in the history

  10. ⚡ improve CI speed, reduce stability (-> one refit only)

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4041fcb View commit details
    Browse the repository at this point in the history

  11. 🐛 use default no. of epochs + t-test needs 4 refits

    Henry committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2d004e0 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. ⚡ do not run t-test check (for now) 

    - see if results betw. single and multirun match.
    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8d65528 View commit details
    Browse the repository at this point in the history

  2. ⚡ bump up bayes factor training

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1dd6788 View commit details
    Browse the repository at this point in the history

  3. 🎨 train both refits with 100 epochs

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    dc9020e View commit details
    Browse the repository at this point in the history

  4. ✨ add log2 option 

    - decide wheather to log2 transform
- default: do not in order to allow negative features which are then standard normalized
    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    9cd2a7b View commit details
    Browse the repository at this point in the history

  5. 🎨 document some more

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    a4911d7 View commit details
    Browse the repository at this point in the history

  6. ⚡ test multiprocess on continuous tutorial

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    e0421bd View commit details
    Browse the repository at this point in the history

  7. 🐛 remove non-exisitng key

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    c70d328 View commit details
    Browse the repository at this point in the history

  8. ✨ build dataloader fct 

    - use everywhere.
    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1c72316 View commit details
    Browse the repository at this point in the history

  9. 🐛 fix minor bug (wrongly assigned feat)

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    58f08e4 View commit details
    Browse the repository at this point in the history

  10. ⚡ move masking code into main fct of module

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    f895237 View commit details
    Browse the repository at this point in the history

  11. 🎨 move feat_mask creation out

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    5eb7954 View commit details
    Browse the repository at this point in the history

  12. 🚑 temp. fix of CI 

    - need to switch dataloader constructor for type of pertubation (cat or cont)
    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8c4e53b View commit details
    Browse the repository at this point in the history

  13. ⚡ do not build dataloaders for multiprocessing

    Henry committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    49a93d0 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. 🚧 test t-test again, re-run pert. w/o model training

    Henry committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    709c674 View commit details
    Browse the repository at this point in the history

  2. ✨ add categorical pert. to multiprocessing

    Henry committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    6e65cc6 View commit details
    Browse the repository at this point in the history

  3. 🔥 remove unused code

    Henry committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    4efbdd9 View commit details
    Browse the repository at this point in the history

  4. 🎨 remove unused argument

    Henry committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    dab767a View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. ⏪ checkout developer version

    Henry committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    980bbce View commit details
    Browse the repository at this point in the history

  2. 🎨 move shared key to base class

    Henry committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    c26b2dd View commit details
    Browse the repository at this point in the history

  3. 🔥 remove comments and code duplications

    Henry committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    05c1735 View commit details
    Browse the repository at this point in the history

  4. 🎨 update type hints, remove unused import

    Henry committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    c5002cd View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Merge branch 'developer' into main

    @enryH
    enryH committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    fe8c48b View commit details
    Browse the repository at this point in the history