-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make booster default in Foundry proofs #2044
Conversation
kevm-pyk/src/kontrol/__main__.py
Outdated
foundry_kompile.add_argument( | ||
'--with-llvm-library', | ||
dest='llvm_library', | ||
default=True, | ||
action='store_true', | ||
help='Make kompile generate a dynamic llvm library.', | ||
) | ||
foundry_kompile.add_argument( | ||
'--no-llvm-library', | ||
dest='llvm_library', | ||
action='store_false', | ||
help='Do not generate a dynamic llvm library during kompile.', | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're already actually building with the llvm library every time, so this isn't needed. Also, this is made more explicit here: #2034
So we can just remove this option altogether (and the conflict_handler
option above).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Removed in 18f4e77.
kevm-pyk/src/kontrol/__main__.py
Outdated
default=False, | ||
default=True, | ||
action='store_true', | ||
help="Use the booster RPC server instead of kore-rpc. Requires calling foundry-kompile with the '--with-llvm-library' flag", | ||
help='Use the booster RPC server instead of kore-rpc.', | ||
) | ||
foundry_prove_args.add_argument( | ||
'--no-booster', | ||
dest='use_booster', | ||
action='store_false', | ||
help='Do not use the booster RPC server instead of kore-rpc.', | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should name this option --no-use-booster
, not --no-booster
. The convention is to prefix the exact same argument name with no-
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to --no-use-booster
in 45680be; added this flag to the test-prove-foundry
CI job, as suggested below.
We should also adjust here: evm-semantics/.github/workflows/test-pr.yml Line 172 in ba06470
--no-use-booster , so that we're still testing non-booster on CI.
|
This PR shuold probably wait for #2034 to merge, which simplifies the booster building process a bit and build sit by default every time we build Foundry proofs anyway. |
.github/workflows/test-pr.yml
Outdated
@@ -198,7 +198,7 @@ jobs: | |||
- name: 'Build Foundry' | |||
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make build-foundry -j2' | |||
- name: 'Foundry Prove' | |||
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make test-foundry-prove PYTEST_ARGS=-vv' | |||
run: docker exec -u github-user kevm-ci-foundry-${GITHUB_SHA} /bin/bash -c 'make test-foundry-prove PYTEST_ARGS=-vv --no-use-booster' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This --no-use-booster
flag logic should be added somewhere, you could either:
- rename all uses of
use_booster
in pytest tests and invert their logic - handle the logic in the
Makefile
by writing--use-booster
if the flagNO_USE_BOOSTER
is false
As you want to make the booster the default, 1. might be a better choice and a less confusing one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, what about writing
diff --git a/kevm-pyk/src/tests/conftest.py b/kevm-pyk/src/tests/conftest.py
index 7a2b381d3..58cd4dcd6 100644
--- a/kevm-pyk/src/tests/conftest.py
+++ b/kevm-pyk/src/tests/conftest.py
@@ -30,4 +30,4 @@ def update_expected_output(request: FixtureRequest) -> bool:
@pytest.fixture(scope='session')
def use_booster(request: FixtureRequest) -> bool:
- return request.config.getoption('--use-booster')
+ return not request.config.getoption('--no-use-booster')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @iFrostizz for the suggestion, looks very good!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@iFrostizz @ehildenb I realized that the --use-booster
option is now also used in the CI Haskell proof tests (test-prove-haskell-booster
), where the booster is not used by default. Changing the interface for use_booster
in pytest
tests will affect Haskell proofs tests too, i.e., we'll have to add --no-use-booster
to no-booster test-prove-haskell
in this line.
As far as I understand, the CLI's (--no)-use-booster
behavior doesn't impact the usage of booster in tests, as the latter relies on the pytest --use-booster
argument defined in conftest.py:
def pytest_addoption(parser: Parser) -> None:
...
parser.addoption(
'--use-booster',
action='store_true',
default=False,
help='Use the kore-rpc-booster binary instead of kore-rpc',
)
@pytest.fixture
def update_expected_output(request: FixtureRequest) -> bool:
return request.config.getoption('--update-expected-output')
It's further supported by CI job timings: the "legacy" Foundry proofs job still takes more than an hour to complete, while the booster job took less than 20 minutes.
I agree that it'd be better to unify the behavior between CLI and pytest arguments, but, considering different default booster behavior between kevm prove
and kevm foundry-prove
(and, therefore, test-prove-haskell
and test-prove-foundry
), I'm thinking about leaving the CI commands as they are. @iFrostizz @ehildenb could you please let me know what you think? Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that it's okay to leave the CI commands as they are @palinatolmach , because they are not user facing, and because we may drop non booster CI jobs in the future. Also, the --use-booster
option is more straightforward, having --no-use-booster
features a double negation unless you edit further the tests, it looks like unnecessary effort!
c5a7557
to
45680be
Compare
with-llvm-library
default in Foundry proofs9805031
to
aa03b3b
Compare
Decided not to merge this PR and enable booster by default until its behavior is stabilized. Moving to Blocked. |
@palinatolmach What is this blocked by? |
The instability of the booster backend — technically, we can merge it and there's no specific issue that should be fixed first, but it just doesn't make sense to do it now as the booster is quite problematic now. |
Closing as we decided not to use booster by default until the booster stabilizes. Will open a similar PR against Kontrol repository when that happens. |
Partially addresses #1958.
This PR is aimed to improve the default observable performance of Kontrol by using the Booster backend in
foundry-prove
and generating an LLVM library duringfoundry-kompile
by default.