matcher: new database schema

[hdonnay]

This is a new database schema for the matcher subsystem, focused on:

1. enabling queries across ecosystem silos
2. reducing write traffic during an update
3. trusting identifiers in the data instead of using client-side hashing schemes
4. bulk load via COPY

Explicit non-goals:

1. maintaining backwards compatibility
2. JOIN avoidance

TODO for this PR:

• exercise GC down to attr and the "name" tables
• benchmark a likely query, perhaps add a relevant VIEW

[codecov]

Codecov Report

Attention: Patch coverage is 60.00000% with 12 lines in your changes missing coverage. Please review.

Please upload report for BASE (dev/matcher-v2@20f7be7). Learn more about missing BASE report.

Files	Patch %	Lines
datastore/postgres/migrations/migrations.go	60.00%	6 Missing and 6 partials ⚠️

Additional details and impacted files

@@                Coverage Diff                @@
##             dev/matcher-v2    #1340   +/-   ##
=================================================
  Coverage                  ?   56.25%           
=================================================
  Files                     ?      267           
  Lines                     ?    16876           
  Branches                  ?        0           
=================================================
  Hits                      ?     9493           
  Misses                    ?     6418           
  Partials                  ?      965           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hdonnay]

I cooked up a monster "show me everything" view that took ~38ms to return the first row.

CREATE TYPE display_reference AS (
	namespace TEXT,
	name TEXT,
	uri TEXT[]
);

CREATE TYPE display_package AS (
	name TEXT,
	kind PackageKind,
	arch Architecture[],
	vulnerable_range VersionMultiRange,
	version_upstream TEXT[],
	version_kind TEXT,
	purl TEXT,
	cpe TEXT
);

CREATE TYPE display_attr AS (
	mediatype TEXT,
	data JSONB
);

CREATE OR REPLACE VIEW latest_advisory_full AS
SELECT
	latest.id,
	latest.added,
	latest.generation,
	latest.updater,
	latest.name,
	meta.issued,
	meta.summary,
	meta.description,
	meta.uri,
	meta.severity,
	meta.normalized_severity,
	ref.refs,
	pkg.pkgs,
	attr.attrs
FROM
	matcher_v2.latest_advisory AS latest
	JOIN matcher_v2.advisory_meta AS meta ON meta.advisory = latest.id
	JOIN (
		SELECT
			ar.advisory,
			array_agg((
			r.namespace,
			r.name,
			r.uri)::matcher_v2.display_reference) AS refs
		FROM
			matcher_v2.advisory_reference AS ar
			JOIN matcher_v2.reference AS r ON r.id = ar.reference
		GROUP BY
			ar.advisory
	) AS ref ON ref.advisory = latest.id
	JOIN (
		SELECT array_agg((n.name, p.kind, p.arch, p.vulnerable_range, p.version_upstream, p.version_kind, p.purl, p.cpe)::matcher_v2.display_package) AS pkgs, p.advisory
		FROM matcher_v2.package AS p
		JOIN matcher_v2.package_name AS n ON n.id = p.name
		GROUP BY p.advisory
	) AS pkg ON pkg.advisory = latest.id
	JOIN (
		SELECT aa.advisory,
			array_agg((mt.mediatype, a.data)::matcher_v2.display_attr) AS attrs
		FROM matcher_v2.advisory_attr AS aa
		JOIN matcher_v2.attr AS a ON a.id = aa.attr
		JOIN matcher_v2.mediatype AS mt ON a.mediatype = mt.id
		GROUP BY aa.advisory
	) AS attr on attr.advisory = latest.id
;

I don't think this is a typical query, as the normalized parts (package, attr, ref) won't all be joined to the package at the same time, and the meta table definitely shouldn't be.

[hdonnay]

Decided to try merging this stuff into a feature branch as things are going.

[crozzy]

Just a couple of comments while trying to stand up the schema that I noticed (i.e. not a full review)

[crozzy]

I don't see a table to connect attrs with references

[hdonnay]

Right, attr and reference are associated with an advisory.

[crozzy]

Are we missing a VersionMultiRange definition here? Can't see to be able to run the migration without an error

[hdonnay]

No, it's created automatically with a RANGE type: https://www.postgresql.org/docs/16/sql-createtype.html#SQL-CREATETYPE-RANGE