Should I use multiple Terraform projects or only one?

Question

I am starting to build a Terraform GIT project to build resources for azure. Currently my idea is to have a project that is taking care of identity and access topics & to have a project that is taking care of infrastructure topics.

Is it a good idea to have multiple terraform projects for different topics, or would it be better to just have one terraform project that is creating all resources for my organisation?

Answer 1

First of all, I propose a change in terminology. What you're calling a "project" should be referred to as a "module".

An environment is built out of a composition of modules.

This is a design decision. A key part of architecting a cloud environment is to design its components in a such a way that you can maintain them independently. In terraform, these come in the form of modules:

You can use modules to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.

Instead of having a huge lump of resources, you have a set of building blocks which express higher levels of abstraction. An example might be "compute cluster". This will be comprised of the instances, the networks, block storage, object storage buckets, access permissions, etc, all packaged in a module. If this concept (compute cluster) is a useful one for you, then you should design a Terraform module which encapsulates them, so that you can reuse the concept.

There is a tradeoff to be made between tiny modules (ie single resource), and mega-modules (ie, all the resources).

Consider your architecture, try to identify which components are re-usable and build modules out of those. To make an analogy with construction: a window and a door are both re-usable components. A wall is a useful concept, not a brick.

Answer 2

All three ways will work (modules mentioned in answers as third answer), in some cases better, in some - well.

But the exercise you have to do is you have to think, how your infrastructure will look in the nearest future. This is, as it was mentioned, a design and architecture decision. For sure you have some goals to achieve and this should be a base for your design.

You have to find the answers to couple of questions:

• how your infrastructure will be multiplied?
• do you allow to re-use of specific elements of the infrastructure (for example VPC)?
• are you going to serve these templates through Platform Engineering teams - self-service solutions?
• how do you deal with multiple environments?
• how do you manage the state?
• how do you orchestrate different modules/templates/environments?
• how do you want to manage the whole SDLC of your infrastructure?

These questions are the starting point for your design. I know, it sounds like I complicate things, but I try to think about big picture. And believe me, it will be very important later.

For example: if my solution is simple and I will run it generally only once, I will go with the simplest solution. If I know I will re-use some components, I will go with modules. If I will re-use my modules in wider way (many projects, env, etc), I will go with modules stored in registry. If I go even further, it is time to think about dedicated tools, like Spacelift (disclosure: I am Developer Advocate there). But if you want to orchestrate very complicated setup, with multiple teams, projects, accounts (or subscriptions), modules, dependencies, you should, really, look at tools like this.

Answer 3

[I agree with everything in the two other answers, but I'd like to add something else to the discussion.]

Terraform challenges

I've had a number of cases with terraform where the number of resources in a terraform module/directory become so huge that terraform takes an unreasonable amount of time to converge on a plan. This often seems to happen with for_each loops or other data-driven approaches where a few lines in a data file lead to lots of resources and work for terraform. With these huge directories it can takes hours for terraform to think about it. This is worse on Intel MacBook Pros than ARM/M2-based ones, but I've seen a plan that took hours on an Intel CPU take 15-30 minutes on an ARM processor.

Another issue is provider upgrades. These can lead to needing to do state transformations and often require debugging and iteration to get right. Do you want this debugging to be happening on all of your infrastructure? Probably not.

Solutions

• As others have said, using modules to keep your code organized is a very good habit to get into with terraform.
• Use environments - you can have different values for variables per environment and adding environments doesn't entail writing more code.
• Keep logical things together in the same directory.
• Beyond common providers that you need everywhere (like vault), minimize the number of different providers in each directory. I wouldn't develop something for AWS in the same directory where I'm managing datadog or another cloud service.
• Use remote state references to share data internally. Terraform modules that are just locals that get turned into outputs are good for centralizing information like domain names, IP lists, and other things so you're not copying and pasting this information all over the place.