Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify base and view #2

Closed
wants to merge 61 commits into from
Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
1b6a5df
Unify base and view
agnessnowplow Sep 12, 2023
b0a7fdb
Postgres changes
agnessnowplow Sep 12, 2023
95e6d1e
Remove limit pv to session
agnessnowplow Sep 12, 2023
65e7f7c
Remove enable_load_tstamp var
agnessnowplow Sep 12, 2023
d316ff3
Unify unified events this run
agnessnowplow Sep 12, 2023
4607119
Make yauaa coalesce null
agnessnowplow Sep 12, 2023
5fb587a
Rename web page context field macro
agnessnowplow Sep 12, 2023
83ac335
Rename page_views cluster by fields
agnessnowplow Sep 12, 2023
4149042
Add redshift to default views this run
agnessnowplow Sep 12, 2023
e4a4576
Remove weird renaming from redshift this run
agnessnowplow Sep 13, 2023
0a104f4
Rename web in macro
agnessnowplow Sep 13, 2023
f9d66c4
Rename pageviewstitching
agnessnowplow Sep 13, 2023
932ea30
Use set block for engaged time calc
agnessnowplow Sep 13, 2023
dddf9c5
Rename cluster by part 2
agnessnowplow Sep 13, 2023
98d2365
Unify base events this run
agnessnowplow Sep 13, 2023
cd73c29
Unify views this run
agnessnowplow Sep 13, 2023
f78ce57
Change device category logic
agnessnowplow Sep 13, 2023
e688846
Rename name to schema
agnessnowplow Sep 13, 2023
3d62453
Update macros/unify_fields_query.sql
agnessnowplow Sep 13, 2023
c83aa9c
Update macros/unify_fields_query.sql
agnessnowplow Sep 13, 2023
da452c3
Change device cat logic 2
agnessnowplow Sep 13, 2023
4b7a2aa
Comma beautification
agnessnowplow Sep 13, 2023
ec47def
Remove custom example
agnessnowplow Sep 13, 2023
cf58e70
Fix databricks commas
agnessnowplow Sep 13, 2023
472cfd5
Change os_type priority for mobile
agnessnowplow Sep 13, 2023
3eeb4ee
Move device fields more to the front
agnessnowplow Sep 13, 2023
9bbf2d6
Make engaged time calc understandable hopefully
agnessnowplow Sep 13, 2023
6bae568
Update macros/config_check.sql
agnessnowplow Sep 13, 2023
7c894d3
Update macros/config_check.sql
agnessnowplow Sep 13, 2023
71386af
Update macros/config_check.sql
agnessnowplow Sep 13, 2023
a662ae9
Update macros/identifiers.sql
agnessnowplow Sep 13, 2023
cd65a09
Fix indentation
emielver Sep 13, 2023
5487499
FIx indentation
emielver Sep 13, 2023
f4ad3e5
More identation stuff
emielver Sep 13, 2023
02bd8cf
More indentation
emielver Sep 13, 2023
1a6d99a
Indentation
emielver Sep 13, 2023
83270a5
Update macros/field_extractions/get_app_context_fields.sql
agnessnowplow Sep 13, 2023
483788a
Update macros/field_extractions/get_iab_context_fields.sql
agnessnowplow Sep 13, 2023
4bad3b5
Update macros/field_extractions/get_deep_link_context_fields.sql
agnessnowplow Sep 13, 2023
b1aca43
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
d5e7349
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
e2456e0
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
e93d809
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
5ca4262
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
962910a
Update macros/field_extractions/get_app_error_context_fields.sql
agnessnowplow Sep 13, 2023
0e61ed6
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
eed9a5a
Update macros/field_extractions/get_browser_context_fields.sql
agnessnowplow Sep 13, 2023
0238a59
remove comma
agnessnowplow Sep 13, 2023
22409d6
Fix more indentation stuff
emielver Sep 13, 2023
0e25783
Swap sessionindex around
agnessnowplow Sep 13, 2023
aaead5a
Unify domain userid under device identifier
agnessnowplow Sep 14, 2023
6c6b2f8
Rename cte to views
agnessnowplow Sep 14, 2023
3c8f071
Rename session_index and add missed device fields
agnessnowplow Sep 14, 2023
ccb028f
Bug fixes
agnessnowplow Sep 14, 2023
c5a4ace
Point utils to release branch
agnessnowplow Sep 14, 2023
9c39a70
Simplify views for readability
agnessnowplow Sep 14, 2023
94102e0
Enable multi version support for bq contexts
agnessnowplow Sep 14, 2023
c2de2be
Bq fixes
agnessnowplow Sep 14, 2023
d3ad7d0
Change nulls strings to max macro due to redshift
agnessnowplow Sep 14, 2023
0ce05ae
Add config to unified this run
agnessnowplow Sep 14, 2023
ac931b3
Change dummy to model_tstamp
agnessnowplow Sep 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @agnessnowplow
68 changes: 68 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
name: Bug report
about: Report a bug or an issue you've found with this package
title: ''
labels: type:bug
assignees: ''

---

## Describe the bug
<!---
A clear and concise description of what the bug is.
--->

## Steps to reproduce
<!---
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.
--->

## Expected results
<!---
A clear and concise description of what you expected to happen.
--->

## Actual results
<!---
A clear and concise description of what actually happens.
--->

## Screenshots and log output
<!---
If applicable, add screenshots or log output to help explain your problem.
--->

## System information
**The contents of your `packages.yml` file:**
```yml
# contents goes here
```

**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] databricks
- [ ] other (specify: ____________)

**The output of `dbt --version`:**
```
<output goes here>
```

**The operating system you're using:**


**The output of `python --version`:**


## Additional context
<!---
Add any other context about the problem here. For example, if you think you know which line of code is causing the issue.
--->

## Are you interested in contributing towards the fix?
<!---
Let us know if you want to contribute towards the fix, and whether you would need a hand getting started
--->
25 changes: 25 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
name: Feature request
about: Suggest an idea for this package
title: ''
labels: type:enhancement
assignees: ''

---

## Is your feature request related to a problem? Please describe.
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

## Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

## Describe alternatives you've considered
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## Additional context
<!-- Add any other context or screenshots about the feature request here. -->

## Are you interested in contributing towards this feature?
<!---
Let us know if you want to contribute towards the feature, and whether you would need a hand getting started
--->
16 changes: 16 additions & 0 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: Question
about: A more general question about the package. Consider using discourse if more applicable.
title: ''
labels: type:question
assignees: ''

---

<!-- Please consider if your question would be better placed on the Snowplow Discourse (https://discourse.snowplow.io/) for more open conversations -->

### Question
<!-- What is your question -->

### Additional context
<!-- Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here. -->
81 changes: 81 additions & 0 deletions .github/labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
- name: "category:breaking_change"
color: '1e1037'
description: "A breaking change will be introduced if this issue is completed."
- name: "category:documentation"
color: '3849b8'
description: "About the documentation."
- name: "cla:no"
color: '313131'
description: "[Auto generated] Snowplow Contributor License Agreement has not been signed."
- name: "cla:yes"
color: 'c1c1c1'
description: "[Auto generated] Snowplow Contributor License Agreement has been signed."
- name: "good first issue"
color: '69f6ff'
description: "Good issue for a first time contributor."
- name: "priority:high"
color: 'f1ff1d'
description: "To fix as soon as possible."
- name: "priority:low"
color: 'ffd0d4'
description: "Not on the roadmap."
- name: "priority:medium"
color: 'ff6372'
description: "On the roadmap."
- name: "status:blocked"
color: '042e31'
description: "Maintainers can't work on this right now."
- name: "status:completed"
color: '0ea417'
description: "Completed - but might not be released yet."
- name: "status:duplicate"
color: 'b6e1e3'
description: "Duplicates another issue."
- name: "status:has_pr"
color: '0ea462'
description: "A PR exists for this issue."
- name: "status:in_progress"
color: '0e9ba4'
description: "Maintainers are working on this."
- name: "status:needs_triage"
color: 'b6e3cf'
description: "Needs maintainer triage."
- name: "status:wont_fix"
color: '04311d'
description: "Can't or won't be fixed."
- name: "status:info_needed"
color: '56B9BF'
description: "More information is needed from filer for issue to be actionable"
- name: "status:do_not_merge"
color: 'D93F0B'
description: "Flag to denote a Issue or PR which should not yet be merged (usually pending a release)"
- name: "status:help_wanted"
color: '69f6ff'
description: "Help is wanted to help solve this issue."
- name: "type:admin"
color: 'fdffd2'
description: 'About code or repo management.'
- name: "type:bug"
color: '4c381f'
description: 'Bugs or weaknesses. The issue has to contain steps to reproduce.'
- name: "type:enhancement"
color: 'ffbd69'
description: 'New features or improvements to existing features.'
- name: "type:question"
color: 'ffebd2'
description: "Questions to the maintainers."
- name: "category:models"
color: '6738b8'
description: "Related to the models in the package."
- name: "category:macros"
color: 'f1ff1d'
description: "Related to the macros in the package."
- name: "category:tests"
color: 'a738b8'
description: "Related to the tests in the package."
- name: "category:other"
color: '6372ff'
description: "Related to other parts of the package or repo itself."
#- name: "category:4"
# color: '9e62dd'
# description: "Desc"
22 changes: 22 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!--
If this is your first time contributing you will be asked to sign the Individual Contributor License Agreement.
If you would prefer to read this in advance of submitting your PR you can find it here https://docs.google.com/forms/d/e/1FAIpQLSd89YTDQ1XpTZbj3LpOkquV_h1Y8k9ay3iFbJsZsJrz18I23Q/viewform
-->

## Description & motivation
<!--
Describe your changes, and why you're making them.
-->

## Checklist
- [ ] I have verified that these changes work locally
- [ ] I have updated the README.md (if applicable)
- [ ] I have added tests & descriptions to my models (and macros if applicable)
- [ ] I have raised a [documentation](https://github.com/snowplow/documentation) PR if applicable (Link here if required)
- [ ] Is your change a breaking change?

<!--
## Release Only Checklist
- [ ] I have updated the version number in all relevant places
- [ ] I have updated the CHANGELOG.md
-->
26 changes: 26 additions & 0 deletions .github/workflows/no-response.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: No Response

# Both `issue_comment` and `scheduled` event types are required for this Action
# to work properly.
on:
issue_comment:
types: [created]
schedule:
# Schedule for 08:00 every day
- cron: '0 8 * * *'

jobs:
noResponse:
runs-on: ubuntu-latest
steps:
- uses: lee-dohm/[email protected]
with:
token: ${{ github.token }}
daysUntilClose: 14
responseRequiredLabel: "status:info_needed"
closeComment: >
This issue has been automatically closed because there has been no response
to our request for more information from the original author within 14 days. With only what
is currently in the issue, we don't have enough information
to take action. Please comment with this information if you have it to reopen the issue so
that we can investigate further.
138 changes: 138 additions & 0 deletions .github/workflows/pr_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
name: pr_tests

on:
pull_request:

concurrency: dbt_integration_tests

env:
# Set profiles.yml directory
DBT_PROFILES_DIR: ./ci

# Redshift Connection
REDSHIFT_TEST_HOST: ${{ secrets.REDSHIFT_TEST_HOST }}
REDSHIFT_TEST_USER: ${{ secrets.REDSHIFT_TEST_USER }}
REDSHIFT_TEST_PASS: ${{ secrets.REDSHIFT_TEST_PASS }}
REDSHIFT_TEST_DBNAME: ${{ secrets.REDSHIFT_TEST_DBNAME }}
REDSHIFT_TEST_PORT: ${{ secrets.REDSHIFT_TEST_PORT }}

# BigQuery Connection
BIGQUERY_TEST_DATABASE: ${{ secrets.BIGQUERY_TEST_DATABASE }}
BIGQUERY_LOCATION: ${{ secrets.BIGQUERY_LOCATION }}
BIGQUERY_SERVICE_TYPE: ${{ secrets.BIGQUERY_SERVICE_TYPE }}
BIGQUERY_SERVICE_PROJECT_ID: ${{ secrets.BIGQUERY_SERVICE_PROJECT_ID }}
BIGQUERY_SERVICE_PRIVATE_KEY_ID: ${{ secrets.BIGQUERY_SERVICE_PRIVATE_KEY_ID }}
BIGQUERY_SERVICE_PRIVATE_KEY: ${{ secrets.BIGQUERY_SERVICE_PRIVATE_KEY }}
BIGQUERY_SERVICE_CLIENT_EMAIL: ${{ secrets.BIGQUERY_SERVICE_CLIENT_EMAIL }}
BIGQUERY_SERVICE_CLIENT_ID: ${{ secrets.BIGQUERY_SERVICE_CLIENT_ID }}
BIGQUERY_SERVICE_AUTH_URI: ${{ secrets.BIGQUERY_SERVICE_AUTH_URI }}
BIGQUERY_SERVICE_TOKEN_URI: ${{ secrets.BIGQUERY_SERVICE_TOKEN_URI }}
BIGQUERY_SERVICE_AUTH_PROVIDER_X509_CERT_URL: ${{ secrets.BIGQUERY_SERVICE_AUTH_PROVIDER_X509_CERT_URL }}
BIGQUERY_SERVICE_CLIENT_X509_CERT_URL: ${{ secrets.BIGQUERY_SERVICE_CLIENT_X509_CERT_URL }}

# Snowflake Connection
SNOWFLAKE_TEST_ACCOUNT: ${{ secrets.SNOWFLAKE_TEST_ACCOUNT }}
SNOWFLAKE_TEST_USER: ${{ secrets.SNOWFLAKE_TEST_USER }}
SNOWFLAKE_TEST_PASSWORD: ${{ secrets.SNOWFLAKE_TEST_PASSWORD }}
SNOWFLAKE_TEST_ROLE: ${{ secrets.SNOWFLAKE_TEST_ROLE }}
SNOWFLAKE_TEST_DATABASE: ${{ secrets.SNOWFLAKE_TEST_DATABASE }}
SNOWFLAKE_TEST_WAREHOUSE: ${{ secrets.SNOWFLAKE_TEST_WAREHOUSE }}

# Postgres Connection
POSTGRES_TEST_HOST: ${{ secrets.POSTGRES_TEST_HOST }}
POSTGRES_TEST_USER: ${{ secrets.POSTGRES_TEST_USER }}
POSTGRES_TEST_PASS: ${{ secrets.POSTGRES_TEST_PASS }}
POSTGRES_TEST_PORT: ${{ secrets.POSTGRES_TEST_PORT }}
POSTGRES_TEST_DBNAME: ${{ secrets.POSTGRES_TEST_DBNAME }}

# Databricks Connection
DATABRICKS_TEST_HOST: ${{ secrets.DATABRICKS_TEST_HOST }}
DATABRICKS_TEST_HTTP_PATH: ${{ secrets.DATABRICKS_TEST_HTTP_PATH }}
DATABRICKS_TEST_TOKEN: ${{ secrets.DATABRICKS_TEST_TOKEN }}
DATABRICKS_TEST_ENDPOINT: ${{ secrets.DATABRICKS_TEST_ENDPOINT }}

jobs:
pr_tests:
name: pr_tests
runs-on: ubuntu-latest
defaults:
run:
# Run tests from integration_tests sub dir
working-directory: ./integration_tests
strategy:
matrix:
dbt_version: ["1.*"]
warehouse: ["postgres", "bigquery", "snowflake", "databricks"] # TODO: Add RS self-hosted runner

services:
postgres:
image: postgres:latest
env:
POSTGRES_DB: ${{ secrets.POSTGRES_TEST_DBNAME }}
POSTGRES_USER: ${{ secrets.POSTGRES_TEST_USER }}
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_TEST_PASS }}
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
# Maps tcp port 5432 on service container to the host
- 5432:5432

steps:
- name: Check out
uses: actions/checkout@v3

# Remove '*' and replace '.' with '_' in DBT_VERSION & set as SCHEMA_SUFFIX.
# SCHEMA_SUFFIX allows us to run multiple versions of dbt in parallel without overwriting the output tables
- name: Set SCHEMA_SUFFIX env
run: echo "SCHEMA_SUFFIX=$(echo ${DBT_VERSION%.*} | tr . _)" >> $GITHUB_ENV
env:
DBT_VERSION: ${{ matrix.dbt_version }}

- name: Set DEFAULT_TARGET env
run: |
echo "DEFAULT_TARGET=${{ matrix.warehouse }}" >> $GITHUB_ENV

- name: Python setup
uses: actions/setup-python@v4
with:
python-version: "3.8.x"

- name: Pip cache
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ matrix.dbt_version }}-${{ matrix.warehouse }}
restore-keys: |
${{ runner.os }}-pip-${{ matrix.dbt_version }}-${{ matrix.warehouse }}

# Install latest patch version. Upgrade if cache contains old patch version.
- name: Install dependencies
run: |
pip install --upgrade pip wheel setuptools
pip install -Iv dbt-${{ matrix.warehouse }}==${{ matrix.dbt_version }} --upgrade
dbt deps
if: ${{matrix.warehouse != 'spark'}}


- name: Install spark dependencies
run: |
pip install --upgrade pip wheel setuptools
pip install -Iv "dbt-${{ matrix.warehouse }}[ODBC]"==${{ matrix.dbt_version }} --upgrade
dbt deps
if: ${{matrix.warehouse == 'spark'}}

- name: "Pre-test: Drop ci schemas"
run: |
dbt run-operation post_ci_cleanup --target ${{ matrix.warehouse }}

- name: Run tests
run: ./.scripts/integration_test.sh -d ${{ matrix.warehouse }}

# post_ci_cleanup sits in utils package
- name: "Post-test: Drop ci schemas"
run: |
dbt run-operation post_ci_cleanup --target ${{ matrix.warehouse }}
Loading