Suite of integration tests for Minion

[gskorokhod]

We need a large suite of problems to solve with minion and compare our solutions to conjure (see #247).

Having a large test suite would probably flag up some issues (unsupported Essence features, unsupported Minion constraints, issues with the rewriting engine, issues in minion_rs) and hopefully provide more direction for further work

[ChrisJefferson]

This is probably a long term project, but I strongly believe fuzzing is the only way to get top quality coverage. Once we had a program which could generate valid minion inputs, and tested with those, we found dozens of bugs in minion I'd never have found otherwise.

[ozgurakgun]

I agree that fuzzing will be good and I agree that we are not there yet :)

We probably want a native parser first, before we get to fuzzing.

[niklasdewally]

I agree that fuzzing will be good and I agree that we are not there yet :)

We probably want a native parser first, before we get to fuzzing.

We can fuzz into rust types fairly easily using the abitrary crate (ie from bytes to structs without a parser)

[niklasdewally]

See #231 - could do something similar for the conjure oxide ast?

[niklasdewally]

My plan for minion_rs was to use the standard integration tests for "valid models" and "right solutions" and then test safety and ffi and the c++ by fuzzing over the minion_rs ast.

If we fuzz the bindings well, we may not need to fuzz in conjure-oxide itself for Minion stuff, as minion_rs would be more or less safe

[gskorokhod]

I agree that fuzzing will be good and I agree that we are not there yet :)

We probably want a native parser first, before we get to fuzzing.

I think long term it would be useful to have both a Rust native parser for Essence and pretty printing of our AST back to essence. Then it would probably be relatively easy to procedurally generate some valid models and pretty print them to an Essence file that we can feed to old conjure to compare results?

[gskorokhod]

This is probably a long term project, but I strongly believe fuzzing is the only way to get top quality coverage. Once we had a program which could generate valid minion inputs, and tested with those, we found dozens of bugs in minion I'd never have found otherwise.

We should definitely work on fuzzing, but maybe it should be a separate issue?

I was thinking we could start with some existing Essence files just to quickly find some things that aren't working. (At this stage there will probably be obvious bugs in the rules, or some constraints just not being supported etc)

And later, when we have a working base, we can try to test more exhaustively with fuzzing