Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sending Raw Data to Datalake #157

Merged
merged 42 commits into from
Jul 26, 2024
Merged

Sending Raw Data to Datalake #157

merged 42 commits into from
Jul 26, 2024

Commits on Jul 22, 2024

  1. chore: Including basedosdados and upload_to_datalake function

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    6c638d0 View commit details
    Browse the repository at this point in the history

  2. feat: sending data to datalake

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    3baf859 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into send_data_to_datalake

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    b21abf7 View commit details
    Browse the repository at this point in the history

  4. fix: parquet wrong casting to timestamp; implement new unnesters

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    887c758 View commit details
    Browse the repository at this point in the history

  5. fix(lint): unused imports

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    0adc858 View commit details
    Browse the repository at this point in the history

  6. chore: Add option to upload data to datalake in create_raw_data function

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d1b1b40 View commit details
    Browse the repository at this point in the history

  7. fix: tests don't send to datalake

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    24b0740 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #153 from prefeitura-rio/send_data_to_datalake 

    Sending RAW data to datalake
    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    3b56a76 View commit details
    Browse the repository at this point in the history

  9. fix: prepare gcp credentials

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    c224639 View commit details
    Browse the repository at this point in the history

  10. Merge pull request #154 from prefeitura-rio/send_data_to_datalake 

    fix: prepare gcp credentials
    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a280e1c View commit details
    Browse the repository at this point in the history

  11. fix: config of basedosdados

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    0ee63e3 View commit details
    Browse the repository at this point in the history

  12. Merge pull request #155 from prefeitura-rio/send_data_to_datalake 

    Fixing config of basedosdados
    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    427a355 View commit details
    Browse the repository at this point in the history

  13. fix: Copy payloads before iterating in unnester functions

    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    7b2bbd7 View commit details
    Browse the repository at this point in the history

  14. Merge pull request #156 from prefeitura-rio/send_data_to_datalake 

    Copy payloads before iterating in unnester functions
    @TanookiVerde
    TanookiVerde committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    2df25a3 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. feat: rename class + upload folder not files

    @TanookiVerde
    TanookiVerde committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    5b9da0c View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    336489c View commit details
    Browse the repository at this point in the history

  3. feat: remove unused import in datalake.py

    @TanookiVerde
    TanookiVerde committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    fd5f41e View commit details
    Browse the repository at this point in the history

  4. Merge pull request #158 from prefeitura-rio/feat/improved-datalake-up… 

    …load

feat: rename class + upload folder not files
    @TanookiVerde
    TanookiVerde committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    b7b493c View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Refactor Datalake Sending logic

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    7cf6b35 View commit details
    Browse the repository at this point in the history

  2. feat: implementing encounter formatters

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    68b60a2 View commit details
    Browse the repository at this point in the history

  3. chore: automatically reformat file

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    7e254e9 View commit details
    Browse the repository at this point in the history

  4. chore: quick docs in datalake files

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    310e678 View commit details
    Browse the repository at this point in the history

  5. feat: rollback of upload_to_datalake option

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    bb59558 View commit details
    Browse the repository at this point in the history

  6. feat: Update cod_ciap2 field to be optional in VitacareCondicao model

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    69c4929 View commit details
    Browse the repository at this point in the history

  7. feat: Refactor Datalake Sending logic and improve error handling

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    43f6331 View commit details
    Browse the repository at this point in the history

  8. feat: Fix get_formatter function in datalake/utils.py

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    0e863aa View commit details
    Browse the repository at this point in the history

  9. Fix get_formatter function in datalake/utils.py

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    9f23398 View commit details
    Browse the repository at this point in the history

  10. feat: compatibility with historic vitacare data

    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    bbceae3 View commit details
    Browse the repository at this point in the history

  11. Merge pull request #159 from prefeitura-rio/feat/improved-datalake-up… 

    …load

Feat/improved datalake upload
    @TanookiVerde
    TanookiVerde committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    53f8e7d View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. fix: improvements and general fixes

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    bb0a13f View commit details
    Browse the repository at this point in the history

  2. feat: Add error handling and cleanup logic to DatalakeUploader

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    76d30ba View commit details
    Browse the repository at this point in the history

  3. feat: Improve cleanup logic in DatalakeUploader

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    c86422d View commit details
    Browse the repository at this point in the history

  4. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    a6ac1eb View commit details
    Browse the repository at this point in the history

  5. Merge pull request #160 from prefeitura-rio/feat/improved-datalake-up… 

    …load

Feat/improved datalake upload
    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    f77ba96 View commit details
    Browse the repository at this point in the history

  6. fix(lint): minor lint issues

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    93b0491 View commit details
    Browse the repository at this point in the history

  7. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    0b5bc34 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #161 from prefeitura-rio/feat/improved-datalake-up… 

    …load

fix(lint): minor lint issues
    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    5598367 View commit details
    Browse the repository at this point in the history

  9. fix: folders outside "app" are not available in container

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    e5140f4 View commit details
    Browse the repository at this point in the history

  10. Merge pull request #162 from prefeitura-rio/feat/improved-datalake-up… 

    …load

fix: folders outside "app" are not available in container
    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    38ffceb View commit details
    Browse the repository at this point in the history

  11. feat: Add VitacareProcedimentosClinicos model and update formatters

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    c50da07 View commit details
    Browse the repository at this point in the history

  12. feat: Enable upload to datalake in create_raw_data endpoint

    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    2ce258f View commit details
    Browse the repository at this point in the history

  13. Merge pull request #163 from prefeitura-rio/feat/updating-vitacare-en… 

    …counter-model

feat: Add VitacareProcedimentosClinicos model and update formatters
    @TanookiVerde
    TanookiVerde committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    bb54dd8 View commit details
    Browse the repository at this point in the history