-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Echometrics improvements early 2023 #22
Commits on Mar 20, 2023
-
Echometrics improvements early 2023
* Rename pseudogram to echogram * Rename ecometrics to echometrics * Ensure all tests pass before proceeding with additional updates * convertDBD.sh * Better match conditions for input files (rt[tbd] or delayed mode[EBD]) to decoder script * Remove use of slocum dbd2asc decoder in favor of dbdreader (next) * Move echogram files separately from dba_extension files since they normally do not exist * Update templates to track updated names
Configuration menu - View commit details
-
Copy full SHA for 0c1e179 - Browse repository at this point
Copy the full SHA 0c1e179View commit details
Commits on Mar 21, 2023
-
Configuration menu - View commit details
-
Copy full SHA for cf49b84 - Browse repository at this point
Copy the full SHA cf49b84View commit details
Commits on Mar 25, 2023
-
Part 2 of Echometrics improvements early 2023
Documentation * Update README.pdf for echotools General Notes * Rename driver script * Remove unneeded scripts * Most of the updates are within echotools processing * Rename as we go: pseudogram => echogram * Rename as we go: echosounder => echogram * Hopefully not breaking any APIs in the process * Set title to {segment} for echograms * Note deprecated arguments * Remove debug print statement * Allow user supplied arguments to override deployment.conf arguments * Print help menu if a data type is not provided * Convert limits to python lists if provided * Use segment for filenames if "default" is part of the filename * Default plot type is "binned" * Fix up "time" axis for echogram plots * Allow deployment.conf to update arguments not provided on the command line Specific * tests: TestEchoMetricsTwo: echometrics2; test_slocum.py * Fix date in deployment.json for echometrics2 * Replace with modern example * Includes test of teledyne module and creation of numpy, pandas and xarray data frames from echogram to provide an example of how it can start to be leveraged from within the GUTILS framework. * gutils: slocum/__init__.py * move setting of enable_image to a better spot * Update calls to convertDBD.sh to match API changes * Update calls to decodePseudogram.py => processEchograms.py * gutils: tests/resources/slocum/* * Eco => Echo * Update extra_kwargs to match API changes/updates * Rename as we go: pseudogram => echogram * Rename as we go: echosounder => echogram * Add more tests to Echodroid/Echometrics tests to attempt to capture backend failures * Remove some some unneeded files * echotools: teledyne.py * specify matplotlib backend to potentially make faster; defaults tend to trigger X11 layer * rename [pP]seudo => [eE]cho * Check arguments for None * default plot type => pcolormesh => binned * set plot facecolors to lightgrey * recalculate range bin size if defaults are changed * fix an inventory filtering bug * add a function to create groups of DBD files based on a raw DBD file listing * a deployment directory is not required if not writing netcdf files * a template file is required when writing netcdf files * add a function that overrides key arguments if a deployment file is read. Mainly applies to echograms. * allow plot types to be specified in a comma delimited list or with "all" * rearrange plotting code so all three types can be produced * ensure the color bar is consistent between the three plotting techniques * Rename as we go: pseudogram => echogram * Rename as we go: echosounder => echogram * echotools: processEchograms.py * Rename as we go: pseudogram => echogram * Rename as we go: echosounder => echogram * Always look for a deployment configuration file * Update arguments based on deployment configuration file
Configuration menu - View commit details
-
Copy full SHA for 7effd9f - Browse repository at this point
Copy the full SHA 7effd9fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a66e956 - Browse repository at this point
Copy the full SHA a66e956View commit details -
Configuration menu - View commit details
-
Copy full SHA for 573ad45 - Browse repository at this point
Copy the full SHA 573ad45View commit details -
Configuration menu - View commit details
-
Copy full SHA for 90d9b3e - Browse repository at this point
Copy the full SHA 90d9b3eView commit details
Commits on Mar 31, 2023
-
Verified "egram" mode support for gliders
* slocum/__init__.py * Set default plot type to 'binned' * Emit a log message for debugging * convertDBD.sh * Remove other debugging materials * echotools/teledyne.py * Leave default cmap selection as ascii and convert to actual colormap when requested * remove debugging material * if the date/time ticks are days that line up with 00:00:00, omit the time portion of the string * More migration of Pesudo => Echo * Bug/Enhancement: echogram is now properly time adjusted for "combo" and "egram" glider modes
Configuration menu - View commit details
-
Copy full SHA for 54eef98 - Browse repository at this point
Copy the full SHA 54eef98View commit details -
* Ability to create timeseries views of echogram profiles
Configuration menu - View commit details
-
Copy full SHA for 0273431 - Browse repository at this point
Copy the full SHA 0273431View commit details
Commits on Apr 4, 2023
-
* Add new default product plot type "profile" * Update examples to use default product type
Configuration menu - View commit details
-
Copy full SHA for 677f1ea - Browse repository at this point
Copy the full SHA 677f1eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for ccfdd67 - Browse repository at this point
Copy the full SHA ccfdd67View commit details
Commits on Jun 16, 2023
-
Quick fix ups and dependency dumps for echogram merge (#4)
* Reduce testing resource size and allow testing in Docker image * Update dependencies and Docker image version * Pseudogram -> Echogram * Whoops, the binary utilities still require focal * dbdreader to the rescue... eventually! * Run the watch tests in Docker CI testing
Configuration menu - View commit details
-
Copy full SHA for b0f918c - Browse repository at this point
Copy the full SHA b0f918cView commit details
Commits on Jun 22, 2023
-
First attempt at the parquet update
* gutils/__init__.py * set logging level * gutils/filters.py * pass extra_kwargs as optional argument to reader_class * extra_kwargs now part of the class, remove from extras() * gutils/nc.py * Do not emit debug log message for orphans when the set is empty * Do not cross fill variables (is this needed for the DAC?) * gutils/slocum/__init__.py * Add datetime, warnings, dbdreader and pyarrow.parquet to needed modules * Reduce logging output for dbdreader and matplotlib * Add optional kwargs to SlocumReader() * Add flag to enabling parquet processing * Set processing mode based on ascii_file * Use internal kwargs to class * Let routine fail gracefully if parquet processing is enabled but unable to find appropriate modules * Fix: echometrics data was not being propogated to _extra_*.nc du to restrictive time filter. * Be sure to drop echogram_sv from main profile data * All echo data now stored in _extras. * Add block_read() to perform same task as dbd2asc * Add read_header() to fetch certain metadata from dbd * Adapt convert() routine to work as ascii or parquet * Updates to ascii processing for echograms * gutils/slocum/bin/convertDbds.sh * Add two arguments for echogram processing via ascii method * Add some proper defaults * Update gutils/slocum/echotools/teledyne.py large amount of upstream fixes and improvements. * pytest * New default: echograms produce two image files for each profile * echometrics5 and echometrics7 use ascii method * echometrics6 and echometrics8 use parquet method
Configuration menu - View commit details
-
Copy full SHA for e3d3a8f - Browse repository at this point
Copy the full SHA e3d3a8fView commit details
Commits on Jun 24, 2023
-
* gutils/slocum/echotools/teledyne.py now flake8 compliant * gutils/nc.py * move imports to top * add debug flag to get at bugs at deep levels * gutils/tests/test_slocum.py * update answers with bug fixes and updates * setup.cfg * remove exception for echotools
Configuration menu - View commit details
-
Copy full SHA for e771458 - Browse repository at this point
Copy the full SHA e771458View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5f813fd - Browse repository at this point
Copy the full SHA 5f813fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e0ea0c - Browse repository at this point
Copy the full SHA 3e0ea0cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3159b77 - Browse repository at this point
Copy the full SHA 3159b77View commit details -
Configuration menu - View commit details
-
Copy full SHA for 725495b - Browse repository at this point
Copy the full SHA 725495bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 05fa9cb - Browse repository at this point
Copy the full SHA 05fa9cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for bd3737c - Browse repository at this point
Copy the full SHA bd3737cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8ca83cb - Browse repository at this point
Copy the full SHA 8ca83cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 26fd439 - Browse repository at this point
Copy the full SHA 26fd439View commit details -
Configuration menu - View commit details
-
Copy full SHA for 821dd6d - Browse repository at this point
Copy the full SHA 821dd6dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0feb736 - Browse repository at this point
Copy the full SHA 0feb736View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c9158a - Browse repository at this point
Copy the full SHA 5c9158aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 41a760c - Browse repository at this point
Copy the full SHA 41a760cView commit details -
Configuration menu - View commit details
-
Copy full SHA for a816e38 - Browse repository at this point
Copy the full SHA a816e38View commit details -
Configuration menu - View commit details
-
Copy full SHA for c9f2a6b - Browse repository at this point
Copy the full SHA c9f2a6bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a217e19 - Browse repository at this point
Copy the full SHA a217e19View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2465e7e - Browse repository at this point
Copy the full SHA 2465e7eView commit details
Commits on Jun 25, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 86e194f - Browse repository at this point
Copy the full SHA 86e194fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2db1a29 - Browse repository at this point
Copy the full SHA 2db1a29View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee741ee - Browse repository at this point
Copy the full SHA ee741eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for dca53e2 - Browse repository at this point
Copy the full SHA dca53e2View commit details
Commits on Jul 10, 2023
-
* __init__.py * SciPy 1.13.0 warnings * nc.py * Skip writing extras file for echograms that are short * slocum/__init__.py * enable deeper debugging * slocum/bin/convertDbds.sh * enforce grep to use ascii on binary files * slocum/echotools/teledyne.py * allow PYTHONWARNINGS to override coded warnings * utilize dbdreader skip_inital_line feature * numpy datetime conversion warning * flake8 file matching update * ensure file descriptors are released * merge duplicate time coordinate data from DBD * flake8 updates
Configuration menu - View commit details
-
Copy full SHA for cdd8f41 - Browse repository at this point
Copy the full SHA cdd8f41View commit details
Commits on Aug 28, 2023
-
Finalize major updates and begin workflow tests
* improve handling when using the parquet format * use glider terms to refer to data file sources (glider -> flight) * use python classes instead of [] and () * set output directory appropriately when argument is not set * add metadata to data dict for tracking of created files * add env var GUTILS_PYTEST_NOTEARDOWN to tests to defer removal of ascii and netcdf directories during testing (debugging) * fix checks in a few tests with recent improvements to echotools processing
Configuration menu - View commit details
-
Copy full SHA for 5a3e795 - Browse repository at this point
Copy the full SHA 5a3e795View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29beb4a - Browse repository at this point
Copy the full SHA 29beb4aView commit details
Commits on Oct 15, 2023
-
Bug fixes to latest provided PR
- slocum/__init__.py - Final alignment of data types: rt, rtd, delayed - bin/convertDbds.sh - platform compatability fix for unit tests (see below) - tests/test_slocum.py tests/test_watch.py - do additional clean up of cache files after test completes Unit tests failed on different platforms due to either magic or operational diffences of file. Testing for data or ASCII allows all the unit test to successfully pass. - ftype=$(file $dbdSource | grep data); + ftype=$(file $dbdSource | grep 'data\|ASCII');
Configuration menu - View commit details
-
Copy full SHA for 7c96143 - Browse repository at this point
Copy the full SHA 7c96143View commit details