[FEATURE] Search API to fetch all the workflows present

[owaiskazi19]

Is your feature request related to a problem?

Currently, we don't have a way to track all the workflow IDs being created through execute API. An API is needed to fetch the list of workflow IDs created along with the name of it. Later, status API can be used to get the related information of the specific workflow.

What solution would you like?

Proposed APIs:
GET _plugins/_flow_framework/workflow/_search
POST _plugins/_flow_framework/workflow/_search

Request:

curl -XPOST --insecure -u 'admin:admin' -H "Content-Type: application/json" 'http://localhost:9200/_plugins/_flow_framework/workflow/_search?pretty
' -d '
{
  "query": {
    "match_all": {}
  }
}'

Response:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 30,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".plugins-ai-global-context",
        "_id" : "6C4XbosB7JmuZyDIA8Pc",
        "_version" : 1,
        "_seq_no" : 3,
        "_primary_term" : 1,
        "_score" : 1.0,
        "_source" : {
          "name" : "deploy-register-model",
          "description" : "test case",
          "use_case" : "SEMANTIC_SEARCH",
          "version" : {
            "template" : "1.0.0",
            "compatibility" : [
              "2.12.0",
              "3.0.0"
            ]
          },
          "workflows" : {
            "provision" : {
              "user_params" : { },
              "nodes" : [
                {
                  "id" : "workflow_step_1",
                  "type" : "create_connector",
                  "inputs" : {
                    "actions" : [
                      {
                        "method" : "POST",
                        "request_body" : "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }",
                        "action_type" : "predict",
                        "url" : "https://${parameters.endpoint}/v1/chat/completions"
                      }
                    ],
                    "credential" : {
                      "openAI_key" : "..."
                    },
                    "parameters" : {
                      "endpoint" : "api.openai.com",
                      "model" : "gpt-3.5-turbo"
                    },
                    "version" : "1",
                    "name" : "OpenAI Chat Connector",
                    "protocol" : "http",
                    "description" : "The connector to public OpenAI model service for GPT 3.5"
                  }
                },
                {
                  "id" : "workflow_step_2",
                  "type" : "deploy_model",
                  "inputs" : {
                    "model_id" : { }
                  }
                }
              ],
              "edges" : [
                {
                  "source" : "workflow_step_1",
                  "dest" : "workflow_step_2"
                }
              ]
            }
          }
        }
      },
      {

[joshpalis]

Perhaps an array of objects instead a collection of arrays for the get response:

{
  "workflows": [
    {
      "workflow_id": "....",
      "name": "..."
    },
    {
      "workflow_id": "....",
      "name": "..."
    },
    {
      "workflow_id": "....",
      "name": "..."
    }
  ]
}

[amitgalitz]

I have seen many other plugins follow the pattern here:
A get api for specific workflows like:
GET _plugins/_flow_framework/workflows/<workflow_id>
and a search api across all workflows
GET _plugins/_flow_framework/workflows/_search
POST _plugins/_flow_framework/workflows/_search

The response should be our entire workflow/template object which is in the GC. this is easy to do as we already have the data model for this object, we just return an array of those. User can add a filter potentially on there search if they just want to get the id's back. Are there other cases in opensearch where just IDs are returned?

There are also plugins like security who have an explicitly get all api which would equate to this
GET _plugins/_flow_framework/workflows/ and returns all workflows (not sure what is there default size limit here). which is basically a wrapper for doing:

{
  "query": {
    "match_all": {}
  }
}

[owaiskazi19]

@amitgalitz based on your comment above. Created 2 APIs for search:

GET _plugins/_flow_framework/workflows/_search
POST _plugins/_flow_framework/workflows/_search

The request and response has been updated in the description
Let me know WDYT?

[amitgalitz]

@amitgalitz based on your comment above. Created 2 APIs for search:

GET _plugins/_flow_framework/workflows/_search
POST _plugins/_flow_framework/workflows/_search

The request and response has been updated in the description Let me know WDYT?

This looks good to me, we should make sure to remember to test this when we implement the encryption on credentials so they are encrypted on search results back.

Additionally do you think having a GET API for a specific workflow would be a different issue. For example:
GET _plugins/_flow_framework/workflows/<workflow_id>

[joshpalis]

The URI that our API's use currently is _plugins/_flow_framework/workflow/, so lets change it to :

• GET _plugins/_flow_framework/workflow/_search
• POST _plugins/_flow_framework/workflow/_search

[owaiskazi19]

@joshpalis http://localhost:9200/_plugins/_flow_framework/workflow/_search this is how it is, right? In the issue description.
Oh I see. I missed workflow while typing but that's how it is if you see the request.

[joshpalis]

yeah just to avoid confusion if anyone reads the issue :)