Terraform: How to find root cause of the error “Error: Forbidden: Forbidden”

Hi!

Recently, while working with Terraform, I encountered a situation in which I received an error related to the lack of access to one of the objects in the AWS cloud. However, this error did not show which object the problem was with. The error was in this format:

Error: Forbidden: Forbidden
        status code: 403, request id: XXXXX, host id: XXXXX=

Due to the fact that the infrastructure managed by Terraform contained a lot of objects, it was difficult to find exactly where the error was here.

In this case, you can deal with this error using additional settings:

• toggle error logging mode
• run Terraform in only one thread

Error logging mode

Terraform allows you to set the error logging level to one of the following values:

• TRACE
• DEBUG
• INFO
• WARN
• ERROR

In our case, the most useful will be the debug level, in which we can preview not only errors but also communication between Terraform and API. To set the DEBUG level, set the TF_LOG environment variable:

export TF_LOG=DEBUG

Running Terraform with only one thread

By default the apply and plan commands take advantage of multithreading to speed up your work. In our case, this may cause a situation in which our logs will be heavily mixed up. Therefore, it is worth setting up only one thread here, which will cause the log to show information about subsequent operations in sequence.

To run these commands in this mode, use the parallelism parameter.

terraform plan -parallelism=1

Or:

terraform apply -parallelism=1 

What’s next?

Once we have managed to display all logs, it is now worth looking for where the problem is. Here you should look in the logs 403 error, such as:

HTTP/1.1 403 Forbidden

In my case, the error was related to the lack of permissions to one of the S3 Bucket objects.