-
-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
model contract enforce is not done at compile time #496
Comments
@faizhasan did you check if this is related to dbt-core and not to dbt-athena? in my understanding this is the expected behvior. Maybe @Jrmyy can also help out here. |
Unsure what you mean by testing with dbt-core, do you mean with a different adapter like dbt-spark? In general a Compilation error is thrown before runtime execution, not after. At the moment the behavior of contract enforcement in dbt-athena is destructive - the model table gets removed if contract does not match. The dbt documentation refers to contract enforcement in 2 places both clearly saying this should happen before model is executed. reference/resource-configs/contract
|
The destructive behaviour is indeed far from ideal. Could you tell me in version of dbt-core and dbt-athena you saw this behavior? |
Our implementation is slightly different from dbt-core because we are limited by what can be done in Trino. This means that indeed, in the case the materialization is table and
We can try to make the contract check before dropping the table. This is a good catch, feel free to submit a PR if you feel like it, otherwise we will try to tackle it as soon as we can |
@Jrmyy i missed that contract check was inside the cta, we should definitely try to move it out and put in the materialization |
This could be tricky because we want to do it also for all tables we create along the way (to stay consistent with dbt-core implementation), which means temp tables, ha tables, etc |
Shall we recommend users to use iceberg or ha=true or incremental model for contracts enforcement? @faizhasan could you try with ha=True for your use cases and see if you get a less destructive behaviour? |
We can but the long term solution should be to have a non-destructive behavior for contract enforcement or not, independent from the high availability feature. |
When trying to enforce contracts on a model schema, the
dbt run
command is supposed to throw a Compilation Error before dbt has materialised the table when the contract does not match - instead the run continues to happen and a Compilation error is thrown after the run is executed.If the table already exists, it gets dropped, making the situation worse for downstream systems.
tested with dbt-athena 1.5.2 and 1.6.4 versions.
test_contract.sql
schema.yml
Result:
Expected:
contract is checked at Compile time before dbt run runs the materialisation on the table.
The text was updated successfully, but these errors were encountered: