v0.11.0

Changelog:

• Breaking: Replace apache-arrow with @uwdata/flechette to access Apache Arrow IPC data with better performance and smaller footprint. While the API for Flechette tables mirrors many aspects of the Arrow JS reference implementation, it is not a perfect one-to-one replacement. It is possible that some Mosaic users may need to adjust query result processing in response.
• Breaking: Redesign the data cube indexer to write index tables into a named schema (default 'mosaic'). This allows data cube index tables to be managed in a largely isolated environment and to be reused across users and sessions. Temporary index tables are no longer supported, as temporary tables can not be created within a named schema.
• Breaking: Remove temp option for data cube index tables.
• Breaking: Remove data cube indexer enabled method, instead use get/set properties.
• Add data cube indexer schema get/set properties.
• Add data cube indexer dropIndexTables() method. This method issues a query to remove the entire data cube index table schema. It should be called if base tables are updated, causing index tables to become stale and inaccurate.
• Add Selection include to define a Selection that includes the clauses from one or more upstream selections. This addition allows selections to be composed to build up larger clause sets from sub-selections.
• Add client initialize method. This method supports dynamic client re-initialization, issuing new field info and client query requests.
• Fix throttle utility to be resilient to callback rejections. (thanks @andyrooger!)
• Fix canceled queries by rejecting them, rather than leaving them pending. (thanks @andyrooger!)
• Fix index creation queries to no longer be canceled when clearing the data cube indexes. These indexes may already be used in other queued requests. (thanks @andyrooger!)
• Fix Plot class to re-initialize mark clients upon dynamic field updates.
• Fix data cube indexer to generate correct averages in variance and covariance calculations. (#487)
• Fix broken usage of json_format and add loadJSON test. (thanks @geoffreyd!)
• Fix management of cancelled queries, remove race condition. (thanks @andyrooger!)
• Fix query consolidation to flag aggregate queries with no groupby. (#479, thanks @andyrooger!)
• Fix variable access typo in DuckDBStatement within node.js DuckDB lib.
• Fix time interval year span calculation. (#484)
• Fix nearest interactor with param-compatible clause for single-field selections. (#502)
• Fix nearest interactor for consistent param and selection updates. (#508)
• Fix Coordinator queryRequest to always return a Promise. (#507)
• Update Jupyter widget, remove temp_indexes property and add dataCubeSchema property.
• Update types, method signatures in Coordinator and DataCubeIndexer.
• Update test runner from mocha to vitest. (thanks @domoritz!)
• Update Coordinator API documentation.
• Update athletes example to use Selection include. We now use an upstream selection to limit text search options to the current sport and sex.

v0.10.0

Changelog:

• Breaking: Update bin transform declarative spec to support top-level option properties, remove options object as second argument.
• Breaking: Remove coordinator configure method. It was confusing and involved interactions among settings.
• Breaking: Refactor clause generators for better clarity (clauseInterval instead of just interval).
• Add default client set construction to selection clause helpers.
• Add date interval support to bin transform. A new interval option specifies a desired date/time unit (hour, day, month, etc) or forces standard numerical binning (number). If not specified, a reasonable date/time interval and step size will be determined automatically for temporal data. (#427)
• Add dateBin SQL helper method.
• Add empty flag to selections. If true, selections with no clauses will not select any values. If false (the default), selections with no clauses select all data values.
• Add as selection option to Table input, for clauses corresponding to selected table row values.
• Add rLabel plot attribute.
• Add support for plot aspectRatio property when auto-calculating heights at the Mosaic level.
• Add groupby and flexible column reference support to hexbin mark queries.
• Add more flexible groupby handling in data cube index column analysis.
• Add pending() method to AsyncDispatch to track param/selection event processing.
• Fix plot legend to check for plot label attributes.
• Fix table offset reset upon external updates. (#363)
• Fix empty data handling in regression mark.
• Fix query error handling in coordinator and clients.
• Update data cube indexer to operate over selection-client pairs directly. Use a single indexer per coordinator. Remove coordinator-internal FilterGroup class.
• Update toDataColumns utility, move from mosaic-plot to mosaic-core. This method maps Apache Arrow data to native JS values.
• Update internal query result promise to a proper class.
• Update test cases for client and selection lifecycle methods. (h/t @matys1)
• Update python libs to enable uv via installer key. (thanks @peter-gy)
• Update dependencies, use DuckDB 1.0.0.

v0.9.0

Changelog:

Features:

• Breaking: Rename selection clause schema property to meta.
• Breaking: Change nearest interactor options to support multidimensional point selection and separate pointer targets from selected data channels.
• Add nearest interactor for selecting nearest 2D point.
• Add errorbarX and errorbarY marks for parametric confidence intervals over unaggregated data. The existing ruleX/Y marks should be used for pre-computed interval boundaries or custom aggregate expressions.
• Add mark-level select option for internal filtering of mark data, for example first or last points in a series or the current nearest mark to the pointer. Mark-level select simplifies creating various hover effects and annotations, though may query data in an unoptimized fashion, and does not interoperate with params or selections.
• Add covariance aggregate method.
• Add data cube support in cases where only some filtered clients are indexable.
• Add stddev, variance, covariance, and regression aggregates to data cube indexer.
• Add stddev, variance, covariance transforms in declarative specs.
• Add binning method hints for data cube indexer.
• Add selection clause generation methods (point, interval, etc).
• Add selection clause typings.
• Add Selection valueFor method to look up source-specific values.
• Add selection clause methods to inputs and interactors.
• Add value label to slider input.
• Add interval selection support to slider input.
• Add field option to inputs (menu, slider, search) to control the column referenced in selection clause predicates. (#406)
• Add toggleZ interactor directive to select line or area series using the z channel.
• Add QueryManager logQueries method to toggle logging of queries submitted to connector.
• Add FilterGroup index method to toggle data cube indexing.
• Add SQLExpression export to mosaic-sql package.
• Add Query.select column override support. Newer as values overwrite prior ones, and a null expression value removes any prior statements.

Examples:

• Add linear-regression-10m example with dynamic regression calculation over 10m points.
• Add athlete-height example with error bars and slider-based selection.
• Add line-multi-series example with interactive hover highlights and annotations.

Developer Support:

• Add options for logging and optimization to the dev (npm run dev) example runner.
• Add JSON schema publishing (e.g., to https://uwdata.github.io/mosaic/schema/latest.json).
• Add project vscode settings for YAML schemas (requires the YAML extension)

Bug Fixes, etc.:

• Fix bin transform to support extents for rule marks.
• Fix data type lookup in plot renderer.
• Fix SQL generation escaping of strings with single quotes.
• Fix Menu value initialization and updates. (#401)
• Fix legend interactor peers and selection value extraction.
• Fix interval 1D brush extent sort order.
• Fix raster mark color domain sorting.
• Fix data lookup from SVG in toggle and highlight interactors to support lines/areas.
• Update dependencies, documentation, and types.

v0.8.0

Changelog:

• Add use of columnar data throughout vgplot internals.
• Add interactive brushing for continuous color and opacity legends.
• Add meta-key input support to toggle interactor.
• Add symbol scale directives to vgplot.
• Add legends test example spec.
• Add preamble option to astToESM() to insert custom code.
• Add data cube index support for ARG_MIN, ARG_MAX aggregates.
• Add data cube index support for PRODUCT, BIT_AND, BIT_OR, BIT_XOR, BOOL_AND, BOOL_OR aggregates.
• Add xLabelArrow, yLabelArrow plot attributes.
• Add (x|y|color|opacity|r|length)Percent plot attributes.
• Remove invalid fxNice and fyNice plot attributes.
• Fix legend sizing based on the presence of a label (legend title).
• Fix regression mark tip option to pass to error band only, for a single informative tooltip.
• Fix plot directive to call update, which is needed for plots without marks.
• Fix raster mark transient scale domain updates to get appropriate domain definitions.
• Fix contour mark fill/stroke/opacity encoding channel generation for density thresholds.
• Fix hspace/vspace codegen to properly stringify values.
• Fix namedPlots.clear() to also clear queued requests.
• Fix proper context use in AST node instantiate() calls.
• Fix JSON DuckDB type handling for field info.
• Refactor handleParam mark utility for better clarity and type-friendliness.
• Update dev example runner.
• Update some jsdoc typings throughout Mosaic packages.
• Add and publish TypeScript types for declarative JSON specs.
• Add and publish JSON schema for declarative JSON specs.
• Add test TypeScript files for all specs, run tsc on them as part of @uwdata/mosaic-spec tests.
• Add JSON schema validation to spec tests.
• Update dependencies, including latest version of TypeScript.

v0.7.1

Changelog

• Fix astToDOM() to respect pre-defined Params/Selections passed via the params options.
• Add tests for arrow value conversions.
• Bump dependencies.

v0.7.0

Changelog:

• Add Query.describe(query) to generate DESCRIBE ... queries for metadata.
• Add support for arbitrary SQL expressions (not just column names) as field info metadata. Passing direct SQL expressions to vgplot marks such as {x: vg.sql`-ra`} should now work!
• Add support to (lossily) convert Apache Arrow DECIMAL values to JavaScript Number values.
• Update query consolidator to handle DESCRIBE queries, which require filtering rather than projection.
• Remove table catalog from Coordinator, rely on cache and consolidation instead.
• Refactor fieldInfo metadata management within vgplot Mark implementations.
• Refactor Apache Arrow utilities to be exports of @uwdata/mosaic-core.
• Update dependencies.

As a result of the above, the catalog option of Coordinator.clear is no longer supported. Clients can still pass the option, but it will have no effect. Instead, clearing the cache will now also drop any cached table metadata.

v0.6.1

Changelog:

• Add raster mark interpolation options nearest, barycentric, and random-walk.
• Add initial plot auto-height support, fixes case of raster dimensions without a specific plot height.
• Add aeromagnetic survey example to test raster interpolation and smoothing.
• Update example preparation script, consolidate processing for both tests and docs.
• Fix error reduction in initialization of Deriche convolution.
• Bump dependencies.

v0.6.0

This release adds support for non-linear scales across binning methods, both in gridded marks like raster, and for the bin transform. It also improves marks that use gridded data, such as density, raster, and the new heatmap convenience mark.

Breaking Changes:

This release renames some of the options for density/contour/raster marks and adjusts their defaults. This simplifies some naming and usage, and better aligns with the underlying Observable Plot library.

• binType is now interpolate. The "normal" option value is now "none".
• binWidth is now pixelSize.
• binsX is now width.
• binsY is now height.

Changelog:

• Breaking: Rename options for density/contour/raster marks, change defaults.
• Add support for non-linear scales (log, symlog, sqrt) to pixel-level binning methods.
• Add support for non-linear scales (log, symlog, sqrt) to the bin transform.
• Add SQL scale transform utilities to the mosaic-sql package.
• Add richer option support to raster mark, including gridded ordinal data alongside numeric data.
• Add heatmap mark, a raster mark with convenient defaults for density estimation.
• Add additional plot attributes for scale transforms.
• Add more SQL transforms to the declarative specs.
• Update mark implementations to use improved binning.
• Update density1d example spec to use a log-scaled domain.
• Update examples, including new Observable latency example.
• Update documentation.
• Update dependencies, use DuckDB v0.10.0.
• Fix range transformation bug in DataCubeIndexer.
• Fix filter range bug in DensityLineMark.
• Fix M4 optimization bug in ConnectedMark (line and area).

v0.5.0

This release provides a number of major updates, including:

• Spatial data support via the DuckDB spatial extension
• A redesigned specification parsing and code generation package
• API contexts for vgplot: use multiple coordinators or named plot sets simultaneously
• A new Python-based local server supporting the latest DuckDB version
• Watch out for the multiple breaking changes below

Changelog:

Update declarative spec parser and code generator

• Breaking: Remove spec parser and code generator from vgplot. The parseSpec method is no longer exported by vgplot. Instead, use the new methods from the @uwdata/mosaic-spec package.
• Breaking: Remove support for client-side data loading. Previously this was used to load GeoJSON and TopoJSON data. The DuckDB spatial extension is now used to load and manage such data directly in the database.
• Breaking: The expr construct has been removed from declarative specs, instead use sql. This change better aligns the specs with the underlying vgplot API (where the sql tagged template is used).
• Add spec package with updated spec parser and AST framework.
• Add extension loading to spec processing, using the config.extensions spec property.

Spatial data support

• Add support for the DuckDB spatial extension.
• Add loadSpatial helper and spatial methods (e.g., centroidX, centroidY) to mosaic-sql.
• Remove client-side GeoJSON/TopoJSON loading, route through DuckDB instead.
• Add mapping and spatial analysis examples to tests and docs.

Python-based DuckDB server

• Add duckdb-server package for a Python-based HTTP/WebSocket server. This server supercedes the earlier Node-based server in the duckdb package, due to persistent bugs arising in the Node client and Arrow extension.

Refactor packages

• Add top-level specs folder for shared access to example specs.
• Add plot package for plot, mark, and interactor component classes.
• Re-focus vgplot package solely on providing an exported API for Mosaic applications.

Other changes

• Modify the wasmConnector method to be synchronous. DuckDB initialization is now perform lazily.
• Add options to the wasmConnector to use an existing DuckDB instance.
• Add vgplot API contexts (createAPIContext()) to permit multiple coordinators or named plot sets on the same page.
• Add sql loadExtension() utility, which generates commands to load external DuckDB extensions.
• Add array support to Coordinator.exec(). An array of queries can now be provided, issued to DuckDB as a single multi-statement query.
• Fix vgplot margin() attribute directive.
• Fix line/area marks to not apply M4 optimization to unoriented marks.
• Fix menu input to support to non-truthy values such as 0 and null. (thanks @FrTennis!)
• Breaking: Replace selections with params in Jupyter widget, include all params, and index params by name.

v0.4.0

Changelog:

• Breaking: Use mark-level orderby (not order) to convey query orderby criteria. Avoids conflict with Plot's order option for implicit stack transforms.
• Breaking: Rename channel window transform keys to orderby and partitionby (not order and partition) to more closely match SQL and improve consistency with Mosaic's SQL helpers and mark-level orderby criteria.
• Update highlight interactor to automatically add mark orderby when needed to ensure consistent data order. (#248)
• Bump dev dependencies.