Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Echometrics improvements early 2023 #22

Closed
wants to merge 37 commits into from

Commits on Mar 20, 2023

  1. Echometrics improvements early 2023

     * Rename pseudogram to echogram
     * Rename ecometrics to echometrics
     * Ensure all tests pass before proceeding with additional updates
     * convertDBD.sh
       * Better match conditions for input files (rt[tbd] or delayed mode[EBD]) to decoder script
       * Remove use of slocum dbd2asc decoder in favor of dbdreader (next)
       * Move echogram files separately from dba_extension files since they normally do not exist
     * Update templates to track updated names
    jr3cermak committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    0c1e179 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Configuration menu
    Copy the full SHA
    cf49b84 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2023

  1. Part 2 of Echometrics improvements early 2023

        Documentation
    
         * Update README.pdf for echotools
    
        General Notes
    
         * Rename driver script
         * Remove unneeded scripts
         * Most of the updates are within echotools processing
         * Rename as we go: pseudogram => echogram
         * Rename as we go: echosounder => echogram
         * Hopefully not breaking any APIs in the process
         * Set title to {segment} for echograms
         * Note deprecated arguments
         * Remove debug print statement
         * Allow user supplied arguments to override deployment.conf
           arguments
         * Print help menu if a data type is not provided
         * Convert limits to python lists if provided
         * Use segment for filenames if "default" is part of the
           filename
         * Default plot type is "binned"
         * Fix up "time" axis for echogram plots
         * Allow deployment.conf to update arguments not provided on
           the command line
    
       Specific
    
         * tests: TestEchoMetricsTwo: echometrics2; test_slocum.py
           * Fix date in deployment.json for echometrics2
           * Replace with modern example
           * Includes test of teledyne module and creation
             of numpy, pandas and xarray data frames from
             echogram to provide an example of how it can
             start to be leveraged from within the GUTILS
             framework.
    
         * gutils: slocum/__init__.py
           * move setting of enable_image to a better spot
           * Update calls to convertDBD.sh to match API changes
           * Update calls to decodePseudogram.py => processEchograms.py
    
         * gutils: tests/resources/slocum/*
           * Eco => Echo
           * Update extra_kwargs to match API changes/updates
           * Rename as we go: pseudogram => echogram
           * Rename as we go: echosounder => echogram
           * Add more tests to Echodroid/Echometrics tests to
             attempt to capture backend failures
           * Remove some some unneeded files
    
         * echotools: teledyne.py
           * specify matplotlib backend to potentially make
             faster; defaults tend to trigger X11 layer
           * rename [pP]seudo => [eE]cho
           * Check arguments for None
           * default plot type => pcolormesh => binned
           * set plot facecolors to lightgrey
           * recalculate range bin size if defaults are changed
           * fix an inventory filtering bug
           * add a function to create groups of DBD files based on
             a raw DBD file listing
           * a deployment directory is not required if not writing
             netcdf files
           * a template file is required when writing netcdf files
           * add a function that overrides key arguments if a
             deployment file is read.  Mainly applies to echograms.
           * allow plot types to be specified in a comma delimited
             list or with "all"
           * rearrange plotting code so all three types can be
             produced
           * ensure the color bar is consistent between the three
             plotting techniques
           * Rename as we go: pseudogram => echogram
           * Rename as we go: echosounder => echogram
    
         * echotools: processEchograms.py
           * Rename as we go: pseudogram => echogram
           * Rename as we go: echosounder => echogram
           * Always look for a deployment configuration file
           * Update arguments based on deployment configuration file
    jr3cermak committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    7effd9f View commit details
    Browse the repository at this point in the history
  2. Fixes for flake8 tests

    jr3cermak committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    a66e956 View commit details
    Browse the repository at this point in the history
  3. Pass 2 for flake8

    jr3cermak committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    573ad45 View commit details
    Browse the repository at this point in the history
  4. Pass 3 for flake8

    jr3cermak committed Mar 25, 2023
    Configuration menu
    Copy the full SHA
    90d9b3e View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Verified "egram" mode support for gliders

     * slocum/__init__.py
       * Set default plot type to 'binned'
       * Emit a log message for debugging
     * convertDBD.sh
       * Remove other debugging materials
     * echotools/teledyne.py
       * Leave default cmap selection as ascii and convert to
         actual colormap when requested
       * remove debugging material
       * if the date/time ticks are days that line up
         with 00:00:00, omit the time portion of the string
       * More migration of Pesudo => Echo
       * Bug/Enhancement: echogram is now properly time
         adjusted for "combo" and "egram" glider modes
    jr3cermak committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    54eef98 View commit details
    Browse the repository at this point in the history
  2. Initial documentation update

     * Ability to create timeseries views of echogram profiles
    jr3cermak committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    0273431 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2023

  1. Product update

     * Add new default product plot type "profile"
     * Update examples to use default product type
    jr3cermak committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    677f1ea View commit details
    Browse the repository at this point in the history
  2. flake8 fix

    jr3cermak committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    ccfdd67 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Quick fix ups and dependency dumps for echogram merge (#4)

    * Reduce testing resource size and allow testing in Docker image
    * Update dependencies and Docker image version
    * Pseudogram -> Echogram
    * Whoops, the binary utilities still require focal
    * dbdreader to the rescue... eventually!
    * Run the watch tests in Docker CI testing
    kwilcox committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    b0f918c View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. First attempt at the parquet update

     * gutils/__init__.py
       * set logging level
     * gutils/filters.py
       * pass extra_kwargs as optional argument to reader_class
       * extra_kwargs now part of the class, remove from extras()
     * gutils/nc.py
       * Do not emit debug log message for orphans when the set is empty
       * Do not cross fill variables (is this needed for the DAC?)
     * gutils/slocum/__init__.py
       * Add datetime, warnings, dbdreader and pyarrow.parquet to needed modules
       * Reduce logging output for dbdreader and matplotlib
       * Add optional kwargs to SlocumReader()
       * Add flag to enabling parquet processing
       * Set processing mode based on ascii_file
       * Use internal kwargs to class
       * Let routine fail gracefully if parquet processing is enabled but
         unable to find appropriate modules
       * Fix: echometrics data was not being propogated to _extra_*.nc du
         to restrictive time filter.
       * Be sure to drop echogram_sv from main profile data
       * All echo data now stored in _extras.
       * Add block_read() to perform same task as dbd2asc
       * Add read_header() to fetch certain metadata from dbd
       * Adapt convert() routine to work as ascii or parquet
       * Updates to ascii processing for echograms
     * gutils/slocum/bin/convertDbds.sh
       * Add two arguments for echogram processing via ascii method
       * Add some proper defaults
     * Update gutils/slocum/echotools/teledyne.py large amount of
       upstream fixes and improvements.
     * pytest
       * New default: echograms produce two image files for each profile
       * echometrics5 and echometrics7 use ascii method
       * echometrics6 and echometrics8 use parquet method
    jr3cermak committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    e3d3a8f View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2023

  1. flake8 fixes

     * gutils/slocum/echotools/teledyne.py now flake8 compliant
     * gutils/nc.py
       * move imports to top
       * add debug flag to get at bugs at deep levels
     * gutils/tests/test_slocum.py
       * update answers with bug fixes and updates
     * setup.cfg
       * remove exception for echotools
    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    e771458 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5f813fd View commit details
    Browse the repository at this point in the history
  3. Requirements.txt and mamba

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    3e0ea0c View commit details
    Browse the repository at this point in the history
  4. requirements

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    3159b77 View commit details
    Browse the repository at this point in the history
  5. test2

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    725495b View commit details
    Browse the repository at this point in the history
  6. try setup.py

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    05fa9cb View commit details
    Browse the repository at this point in the history
  7. try3

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    bd3737c View commit details
    Browse the repository at this point in the history
  8. try4

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    8ca83cb View commit details
    Browse the repository at this point in the history
  9. try5

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    26fd439 View commit details
    Browse the repository at this point in the history
  10. docker try #2

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    821dd6d View commit details
    Browse the repository at this point in the history
  11. docker try#3

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    0feb736 View commit details
    Browse the repository at this point in the history
  12. docker try #4

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    5c9158a View commit details
    Browse the repository at this point in the history
  13. docker 3.8 try #1

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    41a760c View commit details
    Browse the repository at this point in the history
  14. docker 3.8 try #1b

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    a816e38 View commit details
    Browse the repository at this point in the history
  15. docker 3.8 try #2

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    c9f2a6b View commit details
    Browse the repository at this point in the history
  16. docker try #1

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    a217e19 View commit details
    Browse the repository at this point in the history
  17. docker try #2

    jr3cermak committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    2465e7e View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2023

  1. docker try #3

    jr3cermak committed Jun 25, 2023
    Configuration menu
    Copy the full SHA
    86e194f View commit details
    Browse the repository at this point in the history
  2. docker try #4

    jr3cermak committed Jun 25, 2023
    Configuration menu
    Copy the full SHA
    2db1a29 View commit details
    Browse the repository at this point in the history
  3. docker try #5

    jr3cermak committed Jun 25, 2023
    Configuration menu
    Copy the full SHA
    ee741ee View commit details
    Browse the repository at this point in the history
  4. docker try #6

    jr3cermak committed Jun 25, 2023
    Configuration menu
    Copy the full SHA
    dca53e2 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. echotools/GUTILS sync

     * __init__.py
       * SciPy 1.13.0 warnings
     * nc.py
       * Skip writing extras file for echograms that are short
     * slocum/__init__.py
       * enable deeper debugging
     * slocum/bin/convertDbds.sh
       * enforce grep to use ascii on binary files
     * slocum/echotools/teledyne.py
       * allow PYTHONWARNINGS to override coded warnings
       * utilize dbdreader skip_inital_line feature
       * numpy datetime conversion warning
       * flake8 file matching update
       * ensure file descriptors are released
       * merge duplicate time coordinate data from DBD
       * flake8 updates
    jr3cermak committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    cdd8f41 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Finalize major updates and begin workflow tests

     * improve handling when using the parquet format
     * use glider terms to refer to data file sources (glider -> flight)
     * use python classes instead of [] and ()
     * set output directory appropriately when argument is not set
     * add metadata to data dict for tracking of created files
     * add env var GUTILS_PYTEST_NOTEARDOWN to tests to defer removal of ascii and netcdf directories during testing (debugging)
     * fix checks in a few tests with recent improvements to echotools processing
    jr3cermak committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    5a3e795 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    29beb4a View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2023

  1. Bug fixes to latest provided PR

     - slocum/__init__.py
       - Final alignment of data types: rt, rtd, delayed
     - bin/convertDbds.sh
       - platform compatability fix for unit tests (see below)
     - tests/test_slocum.py tests/test_watch.py
       - do additional clean up of cache files after test completes
    
    Unit tests failed on different platforms due to either magic
    or operational diffences of file.  Testing for data or ASCII
    allows all the unit test to successfully pass.
    
    -    ftype=$(file $dbdSource | grep data);
    +    ftype=$(file $dbdSource | grep 'data\|ASCII');
    jr3cermak committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    7c96143 View commit details
    Browse the repository at this point in the history