Win a copy of Beginning Java 17 Fundamentals: Object-Oriented Programming in Java 17 this week in the Java in General forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Ron McLeod
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Junilu Lacar
  • Rob Spoor
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Jesse Silverman
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Frits Walraven

Terraform in Action: Module Source Pattern

 
Ranch Hand
Posts: 55
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hey Scott,

I have been using Terraform for a while now, eventually transitioning my codebase to TF Modules. As part of this, I'm using a mono-repo structure on Github to host all my TF Modules (for AWS, Datadog, etc) but I have also seen some folks keep one Module per repo to better manage the evolution of the individual resource and also create opinionated Modules fit for their purpose alone (often). I have read a couple of doc pages from Hashicorp which was left open ended.

Does your book cover any discussion around such patterns, especially from a long-term maintenance and TF version upgrade perspective? For instance, I am using TF 1.0.X now but when 2.0.X becomes GA sometime in the future, I would like to seamlessly make new Modules available that conform to the standards and patterns at that time, while still supporting the last known stable version of 1.0.X for a while. One of the thoughts I had around this was branches for specific versions and tags (possibly) but would be keen to see if there are better patterns for this.

 
Author
Posts: 15
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't see any problem with the "monorepo" style of project, this is normally how I start new projects. I understand where you are coming from with wanting to write reusable code and make it easier to version control.

Theres basically two kinds of modules: locally sourced and remote sourced. Locally sourced modules are basically like how you have been doing things, all the source code is referenced within the local directory structure. Remote modules are when you reference modules stored in other repos. You don't actually have to create a separate repo per modules. Take a look at this repo which actually has 4 different modules in it: https://github.com/terraform-in-action/terraform-cloud-vm.

And here it is in the registry:
https://registry.terraform.io/modules/scottwinkler/vm/cloud/latest

HashiCorp is actually opinionated on this matter, at least when it comes to their own platform (Terraform Cloud / Terraform Enterprise). They like having one module per repo, and then using git tagging for versioning.

What I would do if i were you is ask yourself "is this module going to be used in more than one place?". If the answer is yes, then it makes sense to do the work of breaking it out into a separate repo and tagging it and so forth. You also get the advantage of being able to write automated tests for that one module in isolation. If the answer is no then why make your life more difficult than it needs to be? Keep it as you have it.
 
Rahul Dayal Sharma
Ranch Hand
Posts: 55
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Just a follow-up question on the use of Modules.

What sort of a testing strategy would you recommend when building Modules?

I've seen some folks rely purely on a terraform plan being eyeballed and saved into a plan file and using that file to apply so that any unexpected outcomes (like stale plans) are not applied. There are also other folks who use something like terratest as part of their Module build and test strategy. Are there any patterns/tools that you recommend for testing TF Modules? Especially as an automation focused process (via CICD).
 
Scott Winkler
Author
Posts: 15
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

What sort of a testing strategy would you recommend when building Modules?



Terratest is one option, and indeed a popular option. However, I think it is a bit dated these days. You can achieve the same functionality, but with better feature parity for Terraform using HashiCorps terraform-exec library: https://github.com/hashicorp/terraform-exec. There is a chapter in my book that discusses how to use this to test a module. Automating this process just necessitates using a GitHub action or equivalent CI triggered workflow.

But we don't always have modules that make sense to write tests for. Sometimes it hard to even answer the question of what it is that you are testing. If you just want to validate that resources are created that should be created, well Terraform providers already do that for you. Integration and e2e testing is perhaps more valuable for Terraform modules.

A lot of times its sufficient to have CI triggers for validating and formatting Terraform configuration on PR, and maybe doing a test apply and destroy, since 99% of the time any bugs will show up during an apply/destroy anyhow. If this isn't satisfactory, then you could apply/destroy various test modules with combinations of input variables for popular use cases.
 
reply
    Bookmark Topic Watch Topic
  • New Topic