Allow configuration of aws_autoscaling_group name_prefix that is not dependent on the cluster_name in nomad_cluster module

[ryandens]

Background

My goal is to be able to force the recreation of the ASG created in this module when the launch configuration changes. This is a configuration parameter supported by the terraform-aws-autoscaling module. See configuration parameter recreate_asg_when_lc_changes. This would enable terraform to detect when the launch configuration has a new AMI and associated with it, forcing the replacement of every ec2 instance in the ASG with the new AMI. This makes the question "how do you roll out updates"? Something that can be completely handled by Terraform.

Suggested changes

I propose we add an optional variable called asg_name_prefix to the nomad-cluster module in this repository. I suggest we change the assignment of the name_prefix variable on the autoscaling_group resource to asg_name_prefix if it is not provided, we can fall back to the existing cluster_name variable using the coalesce function.

In addition, I propose we enhance the "how do you roll out updates" documentation to show how to configure this module to recreate your ASG when your launch configuration changes. This documentation would contain the following example code snippet.

resource "random_pet" "consul_asg_prefix" {

  separator = "-"
  length    = 2

  keepers = {
    # Generate a new pet name each time we switch launch configuration
    consul_servers_lc_name = module.consul_servers.launch_config_name
  }
}

module "consul_servers" {
  source = "github.com/hashicorp/terraform-aws-consul.git//modules/consul-cluster?ref=v0.7.4"
  asg_name_prefix = "consul-server-${element(concat(random_pet.consul_asg_prefix.*.id, [""]), 0)}"
}

Note that this concept of using the random_pet resource to force the recreation of the ASG is 100% taken from the terraform-aws-autoscaling module. We could also consider modifying this module to use the terraform-aws-autoscaling instead of creating the aws_autoscaling_group resource directly.

Other notes:

Note that introducing a variable for the ASG name is not unprecedented in the Hashicorp terraform module ecosystem. The nomad-cluster module provides an optional asg_name variable which enables this deployment strategy.

In addition, it's important to recognize that the sample code I supplied in the above section is currently not possible without this configuration variable. This is because the cluster_name variable is used in both launch_configuration's and aws_autoscaling_group's name_prefix variables. As a result, introducing a random_pet whose keeper is the launch_configuration's name and is used in the cluster_name variable introduces a cyclic dependency.

Curious to hear your thoughts. I'd be happy to take on this work immediately if we reach on consensus on a path forward. If there is another way to accomplish this with Terraform, let me know. I think this would simply the deployments of some systems which don't need 0 downtime and enable better use of this module without the need for scripts/lambdas which manage the consul server

[brikis98]

Thanks for filing the issue! I'm super booked this week, but will try to respond in the next few days.

[brikis98]

This makes the question "how do you roll out updates"? Something that can be completely handled by Terraform.

I'm not sure this is true, as you need each Consul instance, one at a time, to gracefully leave the cluster before its EC2 instance is terminated... And then be replaced by a fresh, health instance before you repeat with the next one.

Note that this concept of using the random_pet resource to force the recreation of the ASG is 100% taken from the terraform-aws-autoscaling module.

I think you can accomplish the same thing simpler by setting the ASG name to depend on the launch config name, as the launch config is recreated with a new name each time any of its params change. You can combine that with a load balancer, create_before_destroy, and min_elb_capacity to do a pure-Terraform rolling deployment.

Curious to hear your thoughts. I'd be happy to take on this work immediately if we reach on consensus on a path forward.

The big gotcha is that you need to update one Consul node at a time. Not sure that will work with pure Terraform.