Allow custom subscription name for Pub/Sub healthcheck

[elefeint]

Issue ported from spring-attic/spring-cloud-gcp#2630.

Currently, the Pub/Sub healthcheck attempts a pull from a nonexistent subscription. This has two negative consequences:

1. The pull takes longer
2. Error logs get polluted

Allowing a pull from a known existing subscription would ameliorate issues like spring-attic/spring-cloud-gcp#2628 and spring-attic/spring-cloud-gcp#2570.

[elefeint]

Here are the detailed steps that will need to be taken for this task:

1. Introduce a new properties class GcpPubSubHealthIndicatorProperties in the com.google.cloud.spring.autoconfigure.pubsub.health package. You can model the class after GcpPubSubProperties, but it will be much simpler, only containing one property.
2. Introduce a new wrapper type that will contain PubSubTemplate and the healthcheck subscription name from the properties. PubSubHealthIndicatorAutoConfiguration will need to use the new type as the second parameterized type, and it will need to create/provide a bean of that type.
3. Update PubSubHealthIndicator class to accept the new wrapper type instead of PubSubTemplate.
4. Update/add tests for any lines of code affected.
5. Update the healthcheck section in Pub/Sub documentation

[elefeint]

cc/ @ThusithaDJ

[ThusithaDJ]

@elefeint, thanks a lot for the information and the opportunity. I'll let you know when I'm having some doubts.

[ThusithaDJ]

Hey @elefeint, could you please let me know is it possible to test this implementation locally. I mean, should I build the project, get the actuator lib, add it into a sample project and try to do a health check?

[elefeint]

You can do it easiest with a Pub/Sub emulator (follow instructions here to start up emulator, and set environment variable spring.cloud.gcp.pubsub.emulator-host to the host:port of the running emulator).

As far as code, yes, you add org.springframework.boot:spring-boot-starter-actuator dependency and test http://localhost:8080/actuator/health endpoint.

[dzou]

Following @patpe 's line of thinking in spring-attic/spring-cloud-gcp#2628 (comment)

What if the PubSubHealthIndicator had a timestamp indicating the last successful Pub/Sub operation. Then we say something like if currentTimestamp - timestamp < 5 seconds, return true; else do a healthcheck and set the timestamp = currentTimestamp. Then have the PubSubTemplate set the current timestamp on the PubSubHealthIndicator as well each time a successful pull goes through.

[patpe]

@elefeint I am investigating possible ways to resolve this and started to think about your proposal to allow for custom subscription names to pull from. The main issue with going down this route, I think, is that this health check will have side effects.

Compare with the JDBC health checks which performs a simple query (e.g. SELECT 1) which has no side effect in the database or the application other than consuming a connection for a short period of time.

If we allow for a custom subscription to health check from, users are expected to configure a subscription that are part of a business flow, e.g. a subscription on a business domain topic. What will the health check then do with the message it gets? Obviously it will nack() it in which case GCP PubSub will increment the nack count and we might trigger the newly implemented DLQ functionality, especially if the application is batch oriented and pulls messages on a schedule using the PubSubTemplate. Now the health check has side effects....

@dzou I will give it a try to see if I can find a way to represent the internal state of the message subscription and use this for health check scenarios.

[meltsufin]

@patpe I believe @elefeint was thinking of dedicating a separate subscription for the health check, since the current method of pulling from a non-existing subscription has its own issues. So, there wouldn't really need to be any side effect at all if it's a non-functional subscription to a non-functional topic or something like that.

Either way, I think the idea of using the ongoing processing as a side-effect for health state is a valuable avenue to explore.

[elefeint]

Checking ongoing processing would also allow per-subscription health tracking, which would help with Spring Cloud Stream granular checks (spring-attic/spring-cloud-gcp#2631).

My thinking about an "existing subscription" was that it would be safe to ack messages if received -- the subscription would either always remain empty or have non-business-logic no-op messages.

[thayhoang]

@elefeint I'm having the same issue. Kubernetes kills my application under heavy load because of the liveness probe. Do you know when this fix are ready?

[elefeint]

@thayhoang You mainly need spring-attic/spring-cloud-gcp#2628, not this issue. As a workaround, you could disable the Pub/Sub healthcheck with management.health.pubsub.enabled=false or increase the number of threads to a higher number with spring.cloud.gcp.pubsub.subscriber.executor-threads=N.

[thayhoang]

@elefeint I understand that this issue (236) is a part of the work that you guys trying to fix it, Am I right? If not, what is it for?
Increasing spring.cloud.gcp.pubsub.subscriber.executor-threads is not an option for me because my application require at most a N number of parallel processes. I'm looking for a better way.