Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Remove user_inputs field from template and set values directly for each workflow #105

Closed
Tracked by #88
joshpalis opened this issue Oct 18, 2023 · 0 comments
Labels
enhancement New feature or request untriaged

Comments

@joshpalis
Copy link
Member

joshpalis commented Oct 18, 2023

The current use case template includes a user_inputs field, whose values can be used throughout the template in cases where this particular value may be used multiple times. For instance, the model_id or index_name can potentially be used in several areas.

This comes with a few issues regarding parsing, and ascertaining the values of a given field such as user_inputs.index_name during the execution of a WorkflowStep.

To reduce complexity, it's necessary to remove the user_inputs section of the use case template and just set those values directly.

For instance, the previous use case template to create an ingest pipeline and index (with created ingest pipeline set as the default ingest pipeline) :

{
  "name": "semantic-search",
  "description": "My semantic search use case",
  "use_case": "SEMANTIC_SEARCH",
  "operations": [
    "PROVISION"
  ],
  "version": {
    "template": "1.0.0",
    "compatibility": [
      "2.9.0",
      "3.0.0"
    ]
  },
  "user_inputs": {
    "index_name": "my-knn-index",
    "pipeline_id": "my-ingest-pipeline"
  },
  "workflows": {
    "provision": {
      "nodes": [
        {
          "id": "workflow_step_1",
          "type": "create_ingest_pipeline",
          "inputs": {
            "id": "user_inputs.pipeline_id",
            "description": "some description",
            "processors": [
              {
                "type": "text_embedding",
                "params": {
                  "model_id": "my-existing-model-id",
                  "input_field_name": "text_passage",
                  "output_field_name": "text_embedding"
                }
              }
            ]
          }
        },
        {
          "id": "workflow_step_2",
          "type": "create_index",
          "inputs": {
            "name": "user_inputs.index_name",
            "settings": "{\"index.knn\":true,\"default_pipeline\":\"user_inputs.pipeline_id\"}"
          }
        }
      ],
      "edges": [
        {
          "source": "create_ingest_pipeline",
          "dest": "create_index"
        }
      ]
    }
  }
}

Proposed use case template (without user inputs) :

{
  "name": "semantic-search",
  "description": "My semantic search use case",
  "use_case": "SEMANTIC_SEARCH",
  "operations": [
    "PROVISION"
  ],
  "version": {
    "template": "1.0.0",
    "compatibility": [
      "2.9.0",
      "3.0.0"
    ]
  },
  "workflows": {
    "provision": {
      "nodes": [
        {
          "id": "workflow_step_1",
          "type": "create_ingest_pipeline",
          "inputs": {
            "id": "my-ingest-pipeline",
            "description": "some description",
            "processors": [
              {
                "type": "text_embedding",
                "params": {
                  "model_id": "my-existing-model-id",
                  "input_field_name": "text_passage",
                  "output_field_name": "text_embedding"
                }
              }
            ]
          }
        },
        {
          "id": "workflow_step_2",
          "type": "create_index",
          "inputs": {
            "name": "my-knn-index",
            "settings": "{\"index.knn\":true,\"default_pipeline\":\"my-ingest-pipeline\"}"
          }
        }
      ],
      "edges": [
        {
          "source": "create_ingest_pipeline",
          "dest": "create_index"
        }
      ]
    }
  }
}
@joshpalis joshpalis changed the title Remove user_inputs field from template and set values directly for each workflow [FEATURE] Remove user_inputs field from template and set values directly for each workflow Oct 18, 2023
@joshpalis joshpalis added the enhancement New feature or request label Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request untriaged
Projects
None yet
Development

No branches or pull requests

1 participant