Feature/orchestration v0 #1006
Merged
Feature/orchestration v0 #1006
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * removes an unnecessary abstraction * sync changes --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]>
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to 277a6ae in 2 minutes and 33 seconds
More details
- Looked at
6518lines of code in93files - Skipped
1files when reviewing. - Skipped posting
4drafted comments based on config settings.
1. py/core/base/abstractions/document.py:8
- Draft comment:
DocumentStatusreplaced withIngestionStatusandRestructureStatus. Verify this change aligns with intended logic and does not introduce bugs. - Reason this comment was not posted:
Confidence changes required:50%
The code inpy/core/base/abstractions/document.pyhas a change whereDocumentStatusis replaced withIngestionStatusandRestructureStatus. This change should be verified to ensure it aligns with the intended logic and does not introduce any bugs.
2. py/core/main/api/auth_router.py:38
- Draft comment:
Replacedself.enginewithself.service. Verify this change aligns with intended logic and does not introduce bugs. - Reason this comment was not posted:
Confidence changes required:50%
The code inpy/core/main/api/auth_router.pyreplacesself.enginewithself.service. This change should be verified to ensure it aligns with the intended logic and does not introduce any bugs.
3. py/core/main/api/base_router.py:12
- Draft comment:
Replacedself.enginewithself.service. Verify this change aligns with intended logic and does not introduce bugs. - Reason this comment was not posted:
Confidence changes required:50%
The code inpy/core/main/api/base_router.pyreplacesself.enginewithself.service. This change should be verified to ensure it aligns with the intended logic and does not introduce any bugs.
4. py/core/main/api/management_router.py:26
- Draft comment:
Replacedself.enginewithself.service. Verify this change aligns with intended logic and does not introduce bugs. - Reason this comment was not posted:
Confidence changes required:50%
The code inpy/core/main/api/management_router.pyreplacesself.enginewithself.service. This change should be verified to ensure it aligns with the intended logic and does not introduce any bugs.
Workflow ID: wflow_SFkzIawPtGj6maMk
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
emrgnt-cmplxty
added a commit
that referenced
this pull request
Sep 6, 2024
* Feature/orchestration v0 (#1006) * Feature/remove extra r2r abstraction (#996) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * removes an unnecessary abstraction * sync changes --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * first commit * move towards orchestration * tweaks * check in working ingestion * move * kg enrichment * update future, postgres compose * hatchetize ingestion pipeline * ready for prime time * finish --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Feature/add update files workflow (#1010) * add update files workflow * rm ingestion pipeline * Feature/add enrichment flow (#1013) * add update files workflow * rm ingestion pipeline * v0 restructure orch * Feature/merged enrichment flow (#1016) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * Rm graspologic (#1034) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter * Patch/ollama base cli (#992) * Dev (#990) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * fix ollama cli --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * Ingestion refactor (#991) * fix test (#993) * Increase Neo4j memory limits, add GDS plugin, and update LLM concurrency limit to 256. * Update ingestion sample file, disable KG node extraction pipe, add community processing in clustering, and enhance graph clustering queries. * Update runners (#1007) * Refactor KG clustering process to simplify community processing and enhance entity-triple retrieval from Neo4j. * Refactor Neo4j configuration for memory settings and update graph clustering logic in the KG provider. * Fix pipeline by enabling node extraction and refactor community processing logic in KGClusteringPipe. * hatchet works * throw error if you run global search before enrichment * Fix communities in local search * turn off node desc embedding * fix rag endpoint * Increase hatchet msg size * Update ingestion.py * Refactor and clean up code formatting * modified workflow * Add graph creation functionality * Refactor KG parameters and logging. * review * up --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]> * Feature/add hatchet api key setup rebased (#1040) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * add hatchet api key setup * cleanup * add hatchet api key setup (#1037) * add hatchet api key setup * cleanup * fix merge * cleanups * Feature/nolan logs refactored (#1041) * Update runners (#1007) * Check in logs --------- Co-authored-by: Nolan Tremelling <[email protected]> * Pull open PRs into dev (#1042) * Pull in subnet and graph PR * Add in templates * Add python files for templates in cli (#1043) * working hatchet integration (#1046) * Update local_llm_neo4j_kg.toml * Unstructured fixes (#1048) * dockerfile * Update ingestion file with new sample URL and enhance unstructured chunking configuration and error handling. * clean up * clean up dockerfile * up * Update sample file and clean code * Add hatchet-sdk dependency in project. * Update providers to include local option. * Introduce File Provider (#1044) * Draft of file provider * Some cleanup * Regenearte lock * Stream it * Use document_id as primary key * Pydantic v2 * File provider finished * Make 7272 the default port (#1045) * Fix poetry.lock * Precommit * Enhance Dockerfile and add telemetry events (#1049) * Fix File Provider (#1050) * Fix * Fix parsing pipeline * working * Feature/improve docs (#1051) * improve documentation * fix unstr * add ingestion * fix compose * Add unstructured chunking configuration updates * Revert "Add unstructured chunking configuration updates" This reverts commit bae8c0b. * Separate File Provider and Relational Database Provider (#1054) * Move to self.execute_query * Check in push * Check in * Get file provider running * Actually use file provider * Final touches * undo changes in compose * Patch/fix unstructured config rebased (#1059) * fix unstr err * tweak * by_title default * cleanups * checkin * merge * Graph docs (#1058) * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * up * Remove duplicate UnstructuredChunkingConfig entry. * cleanup docs * up --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Graph docs (#1060) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Remove duplicate method (#1061) * update docs (#1064) * rm extra prints * fix img * Fallback logic (#1062) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. * Implement fallback parsing mechanism * Fallback parser * Refactor code for readability and formatting * Refactor and enhance media parsers * Update response types in router. * Remove telemetry and add logging * Refactor logging format in parsers * Refactor image and movie parsers * Fix formatting in movie_parser.py * Remove debug logging statements * Remove debug logging for chunking config * Rename debug option to build. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Refactor response models for clarity * Refactor response types in router. * Feature/fix agent (#1065) * ready for merge * fix agent * Patch/fix 123 (#1066) * ready for merge * fix agent * fix import * Feature/add orchestration draft (#1067) * ready for merge * fix agent * fix import * Fix some of the tests (#1068) * Fix fallback parsing (#1069) * Fix fallback parsing * Fix * Compose * up * Feature/iterate on docs (#1070) * add orchestration docs * docs iteration * iterate * add images * add images * Fix restructuring enum (#1071) * Feature/formatting cleanup (#1072) * add orchestration docs * docs iteration * iterate * add images * add images * run pre-commit * reclean --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]>
shreyaspimpalgaonkar
added a commit
that referenced
this pull request
Sep 6, 2024
* Feature/orchestration v0 (#1006) * Feature/remove extra r2r abstraction (#996) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * removes an unnecessary abstraction * sync changes --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * first commit * move towards orchestration * tweaks * check in working ingestion * move * kg enrichment * update future, postgres compose * hatchetize ingestion pipeline * ready for prime time * finish --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Feature/add update files workflow (#1010) * add update files workflow * rm ingestion pipeline * Feature/add enrichment flow (#1013) * add update files workflow * rm ingestion pipeline * v0 restructure orch * Feature/merged enrichment flow (#1016) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * Refactor and update GraphRAG documentation * Rm graspologic (#1034) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter * Patch/ollama base cli (#992) * Dev (#990) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * fix ollama cli --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * Ingestion refactor (#991) * fix test (#993) * Increase Neo4j memory limits, add GDS plugin, and update LLM concurrency limit to 256. * Update ingestion sample file, disable KG node extraction pipe, add community processing in clustering, and enhance graph clustering queries. * Update runners (#1007) * Refactor KG clustering process to simplify community processing and enhance entity-triple retrieval from Neo4j. * Refactor Neo4j configuration for memory settings and update graph clustering logic in the KG provider. * Fix pipeline by enabling node extraction and refactor community processing logic in KGClusteringPipe. * hatchet works * throw error if you run global search before enrichment * Fix communities in local search * turn off node desc embedding * fix rag endpoint * Increase hatchet msg size * Update ingestion.py * Refactor and clean up code formatting * modified workflow * Add graph creation functionality * Refactor KG parameters and logging. * review * up --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]> * Feature/add hatchet api key setup rebased (#1040) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * add hatchet api key setup * cleanup * add hatchet api key setup (#1037) * add hatchet api key setup * cleanup * fix merge * cleanups * Feature/nolan logs refactored (#1041) * Update runners (#1007) * Check in logs --------- Co-authored-by: Nolan Tremelling <[email protected]> * Pull open PRs into dev (#1042) * Pull in subnet and graph PR * Add in templates * Add python files for templates in cli (#1043) * working hatchet integration (#1046) * Update local_llm_neo4j_kg.toml * Unstructured fixes (#1048) * dockerfile * Update ingestion file with new sample URL and enhance unstructured chunking configuration and error handling. * clean up * clean up dockerfile * up * Update sample file and clean code * Add hatchet-sdk dependency in project. * Update providers to include local option. * Introduce File Provider (#1044) * Draft of file provider * Some cleanup * Regenearte lock * Stream it * Use document_id as primary key * Pydantic v2 * File provider finished * Make 7272 the default port (#1045) * Fix poetry.lock * Precommit * Enhance Dockerfile and add telemetry events (#1049) * Fix File Provider (#1050) * Fix * Fix parsing pipeline * working * Feature/improve docs (#1051) * improve documentation * fix unstr * add ingestion * fix compose * Add unstructured chunking configuration updates * Revert "Add unstructured chunking configuration updates" This reverts commit bae8c0b. * Separate File Provider and Relational Database Provider (#1054) * Move to self.execute_query * Check in push * Check in * Get file provider running * Actually use file provider * Final touches * undo changes in compose * Patch/fix unstructured config rebased (#1059) * fix unstr err * tweak * by_title default * cleanups * checkin * merge * Graph docs (#1058) * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * up * Remove duplicate UnstructuredChunkingConfig entry. * cleanup docs * up --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Graph docs (#1060) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Remove duplicate method (#1061) * update docs (#1064) * rm extra prints * fix img * Fallback logic (#1062) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. * Implement fallback parsing mechanism * Fallback parser * Refactor code for readability and formatting * Refactor and enhance media parsers * Update response types in router. * Remove telemetry and add logging * Refactor logging format in parsers * Refactor image and movie parsers * Fix formatting in movie_parser.py * Remove debug logging statements * Remove debug logging for chunking config * Rename debug option to build. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Refactor response models for clarity * Refactor response types in router. * Feature/fix agent (#1065) * ready for merge * fix agent * Patch/fix 123 (#1066) * ready for merge * fix agent * fix import * Feature/add orchestration draft (#1067) * ready for merge * fix agent * fix import * bump (#1075) * Enhance KG search capabilities and examples. * Fix formatting and update documentation. * Remove debug print statements in parsing. * Fix content and search level values * Update OpenAPI spec and responses. * Add customizable RAG assistant example * Update API documentation and search models. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR introduces Hatchet-based orchestration, refactors ingestion and retrieval services, updates the database schema, and enhances the ingestion pipeline, along with adding new tests.
Key points:
HatchetOrchestrationProviderinpy/core/providers/orchestration/hatchet.pyfor workflow orchestration.py/core/main/services/ingestion_service.pyandpy/core/main/services/retrieval_service.py.py/core/pipelines/ingestion_pipeline.pyto support new orchestration.py/core/providers/database/document.pyand related files.py/tests/test_end_to_end.pyfor ingestion and retrieval functionalities.py/pyproject.tomlto require Python 3.10 and added new dependencies.Generated with ❤️ by ellipsis.dev