Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data migrations with Julia functions on attributes #823

Merged
merged 20 commits into from
Sep 6, 2023
Merged

Data migrations with Julia functions on attributes #823

merged 20 commits into from
Sep 6, 2023

Conversation

KevinDCarlson
Copy link
Contributor

This PR upgrades the @migration macros with the ability to plug Julia code into appropriate locations. For instance:

  • If a is an attribute of the target schema, one can write a=> x->body, where body may depend on x and all the names in the source schema. This allows the produced migration to mutate the attributes in a more-or-less arbitrary way. The generality of this mutation is illustrated in the tests by TuringMachine.
  • Similarly, if h is a hom in the target schema, we can write h's image in terms of an anonymous function whose value depends on all the names in the source schema, but one whose values will be combinatorial data rather than attribute values.
  • More generally, the above hold when a hom or an attribute is sent to a multi-line diagram hom, for each line of that hom independently.
  • If an object of the target schema is sent to a conjunctive diagram, this diagram may include morphisms over expressions of the same form as in the previous bullet, allowing the migration to effectively filter the migrated acset via arbitrary code applied to its attributes.
  • When an attribute a of the target schema has as domain an object sent to a conjunctive diagram containing an object y, we may write a=> (b(y)|>(x->body)) to disambiguate the input.
  • The single-line -> expressions can be replaced with blocks whose last line is such an expression.

The behavior is basically that the macro builds a DataMigration superficially just as it used to, but where some values of the hom_map of the underlying functor may be AttrExpr{:nothing}, and those and perhaps other homs have their values modified upon actually migrated using the Julia functions stored in the migration's params.

The most immediate remaining concern is that the code relies on changes in lots of places, at all three of the key steps of the migration macros (parsing into values from DiagrammaticPrograms.AST, then into actual Diagrams and DiagramHoms, then doing the actual migration), while also allowing for several different syntaxes, and so is not so clean.

The biggest large-scale concern is that we can't yet use multi-input Julia functions, as for instance you'd want to take a product of weighted graphs with the weights of a pair of edges given by the sum of the pair's member's weights. This is an actual semantic step beyond the model we've used so far, as the intermediate migration of something like weight => (x,y) -> weight(x)+weight(y) would need to send Weight to Tuple{Weight,Weight} or to @cases W:Weight; U:Weight, and this splitting would need to be collapsed upon the actual migration. None of this shifting of attribute types happens with examples we can do up till now. The Tuple solution seems easiest, but it does lead to the question of what happens if you have two different combinatorial types using the same attribute type a different number of times each. So I don't think there's an obvious choice of direction here.

@KevinDCarlson KevinDCarlson marked this pull request as draft July 3, 2023 23:09
@KevinDCarlson KevinDCarlson marked this pull request as ready for review July 4, 2023 00:38
@KevinDCarlson KevinDCarlson marked this pull request as draft July 4, 2023 00:38
@epatters epatters marked this pull request as ready for review July 14, 2023 22:55
@epatters epatters changed the title Migrations with code Data migrations with Julia functions on attributes Jul 14, 2023
Copy link
Member

@epatters epatters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kevin, this will be an awesome feature. First round of comments below.

test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinSets.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinSets.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/theories/Category.jl Outdated Show resolved Hide resolved
src/theories/Category.jl Outdated Show resolved Hide resolved
src/gats/SyntaxSystems.jl Outdated Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
Copy link
Member

@epatters epatters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kevin! Here's a batch of small questions/comments. I'll pick up with more reviewing later but I wanted to get some stuff down now.

src/categorical_algebra/DataMigrations.jl Show resolved Hide resolved
src/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
src/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
src/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
src/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
@KevinDCarlson KevinDCarlson self-assigned this Aug 24, 2023
src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
src/categorical_algebra/Diagrams.jl Show resolved Hide resolved
src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Show resolved Hide resolved
Copy link
Member

@epatters epatters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, with this round of comments I am through the code itself! Now I just need to go through the tests.

src/categorical_algebra/Diagrams.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/categorical_algebra/FinCats.jl Outdated Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Show resolved Hide resolved
src/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
src/theories/Category.jl Show resolved Hide resolved
Copy link
Member

@epatters epatters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final round of comments on the tests.

test/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
test/categorical_algebra/DataMigrations.jl Outdated Show resolved Hide resolved
test/categorical_algebra/DataMigrations.jl Show resolved Hide resolved
test/categorical_algebra/DataMigrations.jl Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
test/programs/DiagrammaticPrograms.jl Outdated Show resolved Hide resolved
test/runtests.jl Outdated Show resolved Hide resolved
@KevinDCarlson
Copy link
Contributor Author

Ready for re-review (!!!!!)

I should note that there aren't any tests for aggregation and such unintended applications yet.

@epatters epatters merged commit d308a61 into main Sep 6, 2023
11 of 13 checks passed
@epatters epatters deleted the attrMigr branch September 6, 2023 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants