feat: Implemented ability to dynamically reload TLS certificates

[cdoucy]

Fixes #856

[cdoucy]

Hey @jeevatkm !

As discussed, here is the PR.

If you are happy with the changes, I can update doc.

[jeevatkm]

@cdoucy Thanks for creating a PR. Please take a look at my feedback.

• Make use of https://pkg.go.dev/time#Ticker to execute the function to reload based on a timer
• Do not use OnBeforeRequest, instead keep the loading directly into the client instance TLS config
• Do not combine RootCA and ClientCA certs pool; keep them separate by nature
• 1 minute seems very frequent; certificate refresh only happens occasionally in a practical scenario. For now, let's keep 24 hours; then user can decide on it
• Keep pem refresher as part of client struct. Once it becomes part of the client, then we do not need lastChecked, log, debug, and certPool as part of the watcher

[cdoucy]

Hi @jeevatkm thanks for the feedback!

Make use of https://pkg.go.dev/time#Ticker to execute the function to reload based on a timer

yes, but that means that ticker.Stop() needs to be called at some point.
For this, SetRootCertificateWatcher may accept a context.Context, and ticker.Stop() will be called when the context is called.
What do you think?

keep the loading directly into the client instance TLS config

Note sure to understand what you mean here.
To avoid using OnBeforeRequest, I can implement an http.RoundTripper that would get the refreshed Certificate and pass it to the TLSClientConfig of the underlying http.Transporter.
What do you think of this approach?

P.S. : I wish I could leverage tls.Config.GetCertificate, but it only works on the server side.

[jeevatkm]

@cdoucy I'm not 100% sure, do we really need a context for this use case?

I may try to explain an approach based on what I learned from your PR and currently what is coming to my mind. This is not perfect; I am just dumping my thoughts.

type CertWatcherOptions struct {
	// Default interval is 24 hours
	PoolInterval time.Duration
}

func (c *Client) SetRootCertificateWatcher(certPath string, options *CertWatcherOptions) *Client {
	c.SetRootCertificate(certPath)
	c.initRootCertWatcher(options)
 	return c
}

func (c *Client) SetClientRootCertificateWatcher(certPath string, options *CertWatcherOptions) *Client {
	c.SetClientRootCertificate(certPath)
	c.initClientRootCertWatcher(options)
 	return c
}

func (c *Client) initRootCertWatcher() {
	c.rootCAWatcher = &pemWatcher{
		// ...
	}

	// ...
}

func (c *Client) initClientRootCertWatcher() {
	c.clientRootCAWatcher = &pemWatcher{
		// ...
	}

	// ...	
}

So, that user could call SetRootCertificateWatcher method to set N number cert files, and those get added to the watch list
...
Let's assume there is a method c.refreshRootCerts() and c.refreshClientRootCerts() gets triggered by time.Ticker and reloads the certs into c.tlsConfig().RootCAs and c.tlsConfig().ClientCAs, respectively.

We can add a method to the Client Stop cert watcher ticker and add documentation for users to call that method from their shutdown hook scenarios.

What do you think?