Prometheus can monitor peers that are added to the network after startup

[williambanfield]

Currently, the prometheus configuration is generated and loaded when the network is first created. To enable nodes to be added to the network, we will need to allow the prometheus monitoring application to receive metrics from new nodes.

Possible options for this include:

1. Using prometheus in a push based way so that new nodes push metrics to our monitor
2. Recreating the prometheus configuration when each node is instantiated.

[thanethomson]

I see now that the way we did this in GoT was to deploy Telegraf instances on all of the validators to collect all metrics and push them directly to a central InfluxDB instance, to which we connected our Grafana dashboard for querying/monitoring.