Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make booster default in Foundry proofs #2044

Closed
wants to merge 6 commits into from
Closed

Conversation

palinatolmach
Copy link
Contributor

Partially addresses #1958.

This PR is aimed to improve the default observable performance of Kontrol by using the Booster backend in foundry-prove and generating an LLVM library during foundry-kompile by default.

@palinatolmach palinatolmach marked this pull request as ready for review August 25, 2023 09:39
Comment on lines 455 to 467
foundry_kompile.add_argument(
'--with-llvm-library',
dest='llvm_library',
default=True,
action='store_true',
help='Make kompile generate a dynamic llvm library.',
)
foundry_kompile.add_argument(
'--no-llvm-library',
dest='llvm_library',
action='store_false',
help='Do not generate a dynamic llvm library during kompile.',
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're already actually building with the llvm library every time, so this isn't needed. Also, this is made more explicit here: #2034

So we can just remove this option altogether (and the conflict_handler option above).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Removed in 18f4e77.

Comment on lines 502 to 534
default=False,
default=True,
action='store_true',
help="Use the booster RPC server instead of kore-rpc. Requires calling foundry-kompile with the '--with-llvm-library' flag",
help='Use the booster RPC server instead of kore-rpc.',
)
foundry_prove_args.add_argument(
'--no-booster',
dest='use_booster',
action='store_false',
help='Do not use the booster RPC server instead of kore-rpc.',
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should name this option --no-use-booster, not --no-booster. The convention is to prefix the exact same argument name with no-.

Copy link
Contributor Author

@palinatolmach palinatolmach Aug 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to --no-use-booster in 45680be; added this flag to the test-prove-foundry CI job, as suggested below.

@ehildenb
Copy link
Member

We should also adjust here:

run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make test-foundry-prove PYTEST_ARGS=-vv'
, to add the option --no-use-booster, so that we're still testing non-booster on CI.

@ehildenb
Copy link
Member

This PR shuold probably wait for #2034 to merge, which simplifies the booster building process a bit and build sit by default every time we build Foundry proofs anyway.

@@ -198,7 +198,7 @@ jobs:
- name: 'Build Foundry'
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make build-foundry -j2'
- name: 'Foundry Prove'
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make test-foundry-prove PYTEST_ARGS=-vv'
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make test-foundry-prove PYTEST_ARGS=-vv --no-use-booster'
Copy link
Contributor

@iFrostizz iFrostizz Aug 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This --no-use-booster flag logic should be added somewhere, you could either:

  • rename all uses of use_booster in pytest tests and invert their logic
  • handle the logic in the Makefile by writing --use-booster if the flag NO_USE_BOOSTER is false
    As you want to make the booster the default, 1. might be a better choice and a less confusing one

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, what about writing

diff --git a/kevm-pyk/src/tests/conftest.py b/kevm-pyk/src/tests/conftest.py
index 7a2b381d3..58cd4dcd6 100644
--- a/kevm-pyk/src/tests/conftest.py
+++ b/kevm-pyk/src/tests/conftest.py
@@ -30,4 +30,4 @@ def update_expected_output(request: FixtureRequest) -> bool:

 @pytest.fixture(scope='session')
 def use_booster(request: FixtureRequest) -> bool:
-    return request.config.getoption('--use-booster')
+    return not request.config.getoption('--no-use-booster')

Copy link
Contributor Author

@palinatolmach palinatolmach Sep 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @iFrostizz for the suggestion, looks very good!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iFrostizz @ehildenb I realized that the --use-booster option is now also used in the CI Haskell proof tests (test-prove-haskell-booster), where the booster is not used by default. Changing the interface for use_booster in pytest tests will affect Haskell proofs tests too, i.e., we'll have to add --no-use-booster to no-booster test-prove-haskell in this line.

As far as I understand, the CLI's (--no)-use-booster behavior doesn't impact the usage of booster in tests, as the latter relies on the pytest --use-booster argument defined in conftest.py:

def pytest_addoption(parser: Parser) -> None:
...    
    parser.addoption(
        '--use-booster',
        action='store_true',
        default=False,
        help='Use the kore-rpc-booster binary instead of kore-rpc',
    )

@pytest.fixture
def update_expected_output(request: FixtureRequest) -> bool:
    return request.config.getoption('--update-expected-output')

It's further supported by CI job timings: the "legacy" Foundry proofs job still takes more than an hour to complete, while the booster job took less than 20 minutes.

I agree that it'd be better to unify the behavior between CLI and pytest arguments, but, considering different default booster behavior between kevm prove and kevm foundry-prove (and, therefore, test-prove-haskell and test-prove-foundry), I'm thinking about leaving the CI commands as they are. @iFrostizz @ehildenb could you please let me know what you think? Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it's okay to leave the CI commands as they are @palinatolmach , because they are not user facing, and because we may drop non booster CI jobs in the future. Also, the --use-booster option is more straightforward, having --no-use-booster features a double negation unless you edit further the tests, it looks like unnecessary effort!

@palinatolmach palinatolmach changed the title Make booster, with-llvm-library default in Foundry proofs Make booster default in Foundry proofs Sep 4, 2023
@palinatolmach palinatolmach force-pushed the default-booster branch 2 times, most recently from 9805031 to aa03b3b Compare September 4, 2023 09:49
@palinatolmach
Copy link
Contributor Author

Decided not to merge this PR and enable booster by default until its behavior is stabilized. Moving to Blocked.

@yale-vinson
Copy link

@palinatolmach What is this blocked by?

@palinatolmach
Copy link
Contributor Author

@palinatolmach What is this blocked by?

The instability of the booster backend — technically, we can merge it and there's no specific issue that should be fixed first, but it just doesn't make sense to do it now as the booster is quite problematic now.

@palinatolmach
Copy link
Contributor Author

Closing as we decided not to use booster by default until the booster stabilizes. Will open a similar PR against Kontrol repository when that happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants