Importing Azure resources to the Terraform State with Devops

Terraform is fabulous at creating and managing infrastructure via code. It creates repeatable documented infrastructure. Sometimes however, due to timeouts or bugs in the community driven platform, it goes wrong and you can be left a successful plan phase, and a failing apply phase that requires some additional steps to bring you configuration back under Terraform management.;

with the dreaded message 'A resource with the ID ## already exists - to be managed via terraform this resource needs to be imported to the state'.

We recently had an issue with a new feature in the AzureRM module which left 28 file share backups in the Terraform configuration, in Azure but not in the Terraform state file. This meant every time the release was run, it would error out and report the message above.

How exactly do we do this with an Azure Devops CI/CD Pipeline? If we've set this up correctly, we don't actually have access to the state file as it's accessed via access keys or a service principal from the pipeline itself and it's best to keep it that way so we can't import using VS code, via locally hosted terraform or manually using a cloud shell without granting extra permissions to the container where the state file exists which should be avoided.

Let's just go through what terraform import can and can't do. Terraform import can import Azure resources into the terraform state file, Terraform import can't generate your terraform config files (although that's coming soon). This means you need to create your terraform configuration, then run terraform import if you'd like to get your Azure resources under Terraform configuration control.

If you have the specific error above, you already have the resource in your configuration, and Azure - you just need them in the state file. 

To update the state, we first need to examine the pipeline, if you've followed one of the popular guides, you'll have something like the below;

1. An artifact drop to collect the tf files and move them to the execution folder

2. A Terraform plan phase

3. A Terraform apply phase that is only executed following approval

When we get this error, the plan phase executes successfully with additions being required, but the apply doesn't because the resource already exists in Azure. If the terraform state file was up to date, there would be no changes required.

 The fix

To fix this, we have to apply a temporary step the to apply stage, we need to use the import function here and target the resources one by one to get them imported into the state.

Firstly we need to collect two pieces of information from the apply error;

1. The terraform resource address and type

2. The Azure resource ID

Your log should look something like this;

The two pieces of information are the terraform address and type;

2021-02-02T09:17:25.6514372Z   on backup-file-shares.tf line 1, in resource "azurerm_backup_protected_file_share" "bk-share-smb-001":

and the Azure fully qualified resource ID;

2021-02-02T09:17:25.6508837Z Error: A resource with the ID "/Subscriptions/3c71dbd7-9c5b-48aa-a2e8-9864e470b873/resourceGroups/rg-afs-1337-01/providers/Microsoft.RecoveryServices/vaults/rsv-afs-001/backupFabrics/Azure/protectionContainers/StorageContainer;storage;rg-afs-1337-01;stzrsafs001/protectedItems/AzureFileShare;945A517E0F234226BCBDC5C1CDE756979CD79DDE288CD3035A55AA0B2858A0CC" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_backup_protected_file_share" for more information.

Copy these into a notepad or even better, notepad++ :D and save them for later. Next, do the following;

1. Go to your release pipeline and click edit at the top right

 2. Click 'view stage tasks' under the Apply phase

 3. Your apply phase should be similar to the below;

4. Right click the 'terraform apply' task and click 'clone task'

5. Disable the original terraform apply task, and rename the cloned task 'terraform import'

6. Change the terraform command to 'import'

7. Two new text fields are available - 'resource address' and 'resource ID'.

8. Go back to your notepad++ and select the Terraform Resource Address, the format needs to be <resource type>.<resource address>, so in our case it would be;

azurerm_backup_protected_file_share.bk-share-smb-001

9. Format the Resource ID, this step is important - note the lower case s in subscriptions, if you just copy it verbatim, you'll get an error reporting that the subscription ID cannot be found.

/subscriptions/3c71dbd7-9c5b-48aa-a2e8-9864e470b873/resourceGroups/rg-afs-1337-01/providers/Microsoft.RecoveryServices/vaults/rsv-afs-001/backupFabrics/Azure/protectionContainers/StorageContainer;storage;rg-afs-1337-01;stzrsafs001/protectedItems/AzureFileShare;945A517E0F234226BCBDC5C1CDE756979CD79DDE288CD3035A55AA0B2858A0CC

10. enter the values into the import arguments as above;

11. Click save in the top right, and go back to your CI/CD pipeline and initiate a manual run

12. Go back to the release, and watch the plan succeed - approve the apply phase, and the import should be successful;

13. Check the logs from the import phase - they have a section as the below;

13. Now go re-edit your release again, and disable the import task, and re-enable the apply task;

14. Go back to your CI/CD Pipeline and initiate another manual run, you should see that the plan phase now doesn't find any additions to your environment - it may require a change - in this instance, writing the correct ID (in this instance, there was a bug in the terraform code which meant the release was never finished correctly);

15. Approve the change and your releases are back to a healthy state.

If you have a number of resources, they unfortunately have to be completed individually. Any questions - please get in touch.