Bnb/dh refactor #220

…handlers to factory. Tests passing

…han manual dict creation

…erent being used anymore after changing spatial agg to use mean over overlapping gids. Dont need exo_resolution input resolution input anymore either.

…y construction

…tf-8 headers

…ng for dc handlers. old doc string clean up and some simplifying combinations in batch qeue.

…metry

… kwargs from bias and qa

…handler.

…_Dataset`

…pdates to dc exo training tests

…tory meta class. moved bias correct from ForwardPass, done for each chunk, to ForwardPassStrategy init.

…xoData.split` method.

…lternative

…dler which invoked log interp. Thus the log interp methods from era_downloader can be removed and LogLinInterpolator can also be removed.

…e dask vs numpy arrays

…old Feature class since we now have `parse_feature` method.

…aHandler, following common signature

… returns type specific handler based on file type

…e why the hanging tests.

…hunk` instance method

… input type check in fwp and exo handler name arg in exo data handler.

… vars in pytest_config fixture. used these vars to clean up tests. lin_bc added to bias cli test.

…added pytest.warns() catches for some intentional checks.

….dumps. fwp with bc in cli test. updated nc output.

…ed to string with ellipsis as a datetimeindex for the command string when time index is long.

…e too long to pass through cli and are non serializable.

…rg - can be defined right before writing.

… files

feat: default fwp_chunk_shape for None input fix: moved get chunk to after check for exisitng output file.

…ines when running on a head_node

fix: use low res to compute means / stds if features are lr only. fix: ensure correct feature order and feature divisions for dual samplers.

test: added full domain loading for rasterized h5 file

…om legacy np.random

… micro opts in loaders

…fixed cacher dims parsing

…downloader and type conversion to np.float32.

…s collection and batch handler init

…ys, which slows down the fwp.

…mputing param arrays passed to QDM() bc rex asserts that these are ndarrays and doesnt allow for da.core.Array types.

…et checks in h5 loading

…ata volume prior to node kick off.

…re then reshaped into n_obs. This is more efficient than building batches from indendent samples. This adds batch_size are an argument to sampler objects. This also allows us to remove lots of the previous batch building methods, like those using tf.data.Dataset

…ate collect test code to conftest

… level rename in cdsapi

…all available from dict and add if needed. cache loading for data handlers.

Use `rex Regridder` port

… test with cache data loading. DualSamplerCC test fix.

…ode.

@ppinchuk

-T_Dataset type removed. -Exo datahandler stuff moved to data_handlers/exo -typeagnostic rasterizer, loader, datahandler favored as base objects intead of type specific ones. -more explicit with args / kwargs in composite objects. -signature tests added for composite objects: data handler / batch handlers and https://github.com/NREL/gaps used to correctly resolve composite signatures and composite doc strings. thanks @ppinchuk! - docs branch built and checked. -cleaned up some sphinx syntax. -added some examples and explanation to Sup3rDatasets

vindex method for fancy indexing added signature check in batch factory and gaps derived docs / sigs in derived classes.

…c to Sup3rMeta meta class which all containers inherit.

…arison in stats test

…oved dim ordering logic in Sup3rX initialization. Only enforce dim order when returned from a numpy style indexing request

…mbers. Prep for replacing model step iteration with coarsening of highest res for each step. graceful batch handler thread exit.

…and then use xr.Dataset().coarsen for each model step. moved that simplified code from exo dir contents back into exo.py

…d check. fixed sphinx doc render - needed to define __name__ appropriately.

… addressed other pr comments.

…iner` data setter. sphinx_book_theme.

…map_blocks so this could be done with efficient mem use.

…e it was default without netcdf4 installed.

…h default engine + format

… can also be used to replace process pools just with scheduler='processes'

…emporal chunks for collection.

…n bump

…e cache if using hr topo in a temporal model.

…features and hr_exo_features

…e. much faster for large forward passes.

…ng now. added s_enhancements and t_enhancements to abstract models and now use this to determine exo shapes. removed temporal pad from SolarMultiStepGan, since this can be determined from the t_enhance arg and low res shape

…5 output_handlers

…anch is merged

…r modular calls

…y from parameters not file

…hould be a bool

…M to prevent large delta values

… issues with small window size so increased default to 120 (seasonal correction)

…tor formatting

… round data to zero anymore

…er data

… pass with correct output shape

…reduced mem use since number of spatial chunks tends to be much lower than number of time chunks.

… for now.

…ata time index can be shifted to start at the beginning of the day instead of at noon. GCM data frequently stamps daily data at noon instead of the beginning of the day. This caused an issue with the solar module thinking that given gan data had 48 time steps, since the time index had two unique day values, even though there were only 24 time steps from noon to noon on each day.

Gb/presrat updates 2

…ut files, with irradiance suffixes

… config file.

…ed to start at the zeroth hour, if it has not been shifted already.

…parate script if this is an issue.

…t features. now just repeat over time dimension in the forward pass padding method. this is much more performant for time independent features. this doesnt solve slow down issues with time dependent exo features like sza though.

…ndent features

…pendent

Commits on Aug 19, 2024

test fix

bnb32 committed Aug 19, 2024

Configuration menu

View commit details

Copy full SHA for a00fdaf

Browse repository at this point

Copy the full SHA

a00fdaf View commit details

Browse the repository at this point in the history

Bnb/dh refactor #220

Are you sure you want to change the base?

Bnb/dh refactor #220

Commits on Jul 23, 2024

Commits on Jul 24, 2024

Commits on Jul 25, 2024

Commits on Jul 27, 2024

Commits on Jul 28, 2024

Commits on Jul 29, 2024

Commits on Jul 30, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 3, 2024

Commits on Aug 4, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 11, 2024

Commits on Aug 12, 2024

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Commits on Aug 18, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 22, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 30, 2024

Commits on Aug 31, 2024

Commits on Sep 4, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 15, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 20, 2024

Commits on Sep 21, 2024