[BUG] Non-registered Writeables can't be used in Stream collection methods

[dbwiddis]

What is the bug?

On some multi-node clusters, provisioning a workflow fails with an error log including this message:

[2024-01-04T02:11:48,542][WARN ][o.o.t.OutboundHandler    ] [opensearch-cluster-master-2] send message failed [channel: Netty4TcpChannel{localAddress=/192.168.8.228:54358, remoteAddress=192.168.38.120/192.168.38.120:9300}]
java.lang.IllegalArgumentException: can not write type [class org.opensearch.flowframework.model.ResourceCreated]
[2024-01-04T02:11:48,544][ERROR][o.o.f.w.AbstractRetryableWorkflowStep] [opensearch-cluster-master-2] Failed to update new created resource

How can one reproduce the bug?

Run latest main branch code on a multi-node cluster and provision something that updates the resource state.

What is the expected behavior?

Resources are provisioned.

Do you have any additional context?

The bug is in this line:

flow-framework/src/main/java/org/opensearch/flowframework/model/WorkflowState.java

Line 307 in efdbcb0

output.writeList(resourcesCreated);

This line is also a bug, but the failure to write means it never got here to read:

flow-framework/src/main/java/org/opensearch/flowframework/model/WorkflowState.java

Line 114 in efdbcb0

this.resourcesCreated = input.readList(ResourceCreated::new);

The collection-writing methods in StreamInput and StreamOutput rely on readGenericValue() and writeGenericValue() methods, but these only work on well-known types which have been registered in the WriteableRegistry.

These lists need to be iterated individually.