Spring Cloud Dataflow samples with Pub/Sub

[saturnism]

Is your feature request related to a problem? Please describe.
From user:

The standard apps for spring cloud data flow have implementations for apache kafka and RabbitMQ. Google PubSub is supported, but no premade apps is made with Google PubSub.

Describe the solution you'd like
A sample w/ Spring Cloud Dataflow w/ Pub/Sub

Describe alternatives you've considered

Additional context
User feedback from SpringOne

[nklausen]

Binders for Google PubSub exists https://cloud.spring.io/spring-cloud-stream/#binder-implementations

But there are no premade versions of the standard apps https://cloud.spring.io/spring-cloud-stream-app-starters/ for google pubsub - Only kafka and rabbitmq versions exists

It would be help adoption, if it was possible add PubSub version by just adding something like https://dataflow.spring.io/kafka-docker-latest but for pubsub, perhaps https://dataflow.spring.io/pubsub-docker-latest

[meltsufin]

This is definitely a gap in our suite of samples. We should draw a bit more attention to the availability of the Binder.
Contributions are welcome!
cc/ @dmitry-s

[nklausen]

As I understand it from https://youtu.be/AbeOdrNPfYg?t=3455 the https://github.com/spring-cloud-stream-app-starters/sftp/blob/master/spring-cloud-starter-stream-source-sftp/pom.xml just needs to add org.springframework.cloud/spring-cloud-gcp-pubsub-stream-binder and maybe

spring.cloud.stream.bindings.input.binder=pubsub
spring.cloud.stream.bindings.output.binder=pubsub

to the applications.properties

A different way could be to add a Google PubSub module in https://github.com/spring-cloud-stream-app-starters/tasklauncher-dataflow/tree/master/apps with POM like https://github.com/spring-cloud-stream-app-starters/tasklauncher-dataflow/blob/master/apps/task-launcher-dataflow-sink-kafka/pom.xml, but with the org.springframework.cloud/spring-cloud-gcp-pubsub-stream-binder instead of kafka.

I'm not sure where org.springframework.cloud.stream.app/sftp-app-starters-build is located?

[meltsufin]

You're right that the binders should be generally interchangeable. Yet, we still should test it and document it. I wasn't aware of the spring-cloud-stream-app-starters project. That might be a good place for Pub/Sub starter or maybe we would just keep it in spring-cloud-gcp. It's something to investigate further.

[nklausen]

So you prefer to make a new project spring-cloud-gcp-stream-app-starters under the spring-cloud-gcp umbrella.

I guess, that you don't want to fork the upstream projects, as that will result in a pretty high maintains for each release?!?

So we need to some how make it depend on the upstream project and then add the org.springframework.cloud/spring-cloud-gcp-pubsub-stream-binder or how do you see it?

[meltsufin]

If we're only providing a simple example, we can copy whatever we need from the upstream project into spring-cloud-gcp and it shouldn't add too much maintenance burden. However, I would like to understand a bit more about the the whole https://github.com/spring-cloud-stream-app-starters org, and see if it makes more sense to have a Pub/Sub sample there.
cc/ @artembilan @sobychacko

[artembilan]

No, it's not.
That project is obsolete and replaced with this: https://github.com/spring-cloud/stream-applications

See more info in Blog Post series : https://spring.io/blog/2020/07/13/introducing-java-functions-for-spring-cloud-stream-applications-part-0

So, what you are asking almost does not make sense since that project is not about "samples". It represents some function implementations and generated Spring Cloud Stream applications with Kafka and Rabbit binder bundle.

We probably can't provide Pub/Sub binder-based generation because we don't manage this GCP project, but you definitely can do something as a part of your project to provide generated artifacts with Pub/Sub binder based on those our out-of-the-box functions.

WDYT @sobychacko ?

[nklausen]

It's not a sample, it's a standard component, the headline is misleading

[nklausen]

I was not aware of the deprecation of https://github.com/spring-cloud-stream-app-starters, it should then be marked as deprecated

[nklausen]

@artembilan do you see the build of of https://github.com/spring-cloud/stream-applications with the org.springframework.cloud/spring-cloud-gcp-pubsub-stream-binder being done? should it be part of the https://github.com/spring-cloud/stream-applications, as that is intended to be a monorepo? or should it be done as part of https://github.com/spring-cloud/spring-cloud-gcp?

I sessions on Spring Cloud Dataflow @mminella have said, that the Spring team will not do a cartesian product of the steam application with all the binders in https://spring.io/projects/spring-cloud-stream, he said only premade RabbitMQ and Kafka will made

[artembilan]

No, I don't see because Spring team just don't support that binder and its release lifecycle might not be aligned with what we release in that stream-applications project.

I think @mminella's announcement is the same what I'm telling you. So, if you would like to provide an out-of-the-box stream applications bundled with the Pub/Sub binder, you should do that in this project or a new separate one.

And that's why I'm pointing to @sobychacko . He is a lead of stream-applications project and he can advice how to proceed with the proper final artifact generation.

[ttomsu]

@sobychacko Any thoughts here?

[nklausen]

@sobychacko how do you prefer, that the build of the out-of-the-box stream will be done, using the Google PubSub binder?

[meltsufin]

@nklausen I would suggest creating something under our spring-cloud-gcp-samples module in the new repo, and we can always move it somewhere else later if need be.