When an Azure runbook does not recognize your cmdlets…

Automating a Data Factory pipeline using a runbook seemed like an easy last step in a Azure DWH setup I developed lately. To automate the DWH load, I started to create a PowerShell script runbook in my Azure Automation Account. The goal of this runbook was to start the Azure DWH component, run the pipelines in Azure Data Factory, each containing a different step in the process enabling me to load the available source data, transform it into DWH structures and copy the resulting Datamart into an Azure DB, and finally, once it completed, stop the DWH component again.

The part in my script responsible to start and stop the DWH ran without problems, but then I wanted to call the Data Factory V2 pipeline to execute it.

This part immediately threw the following error.

The  ‘Check the spelling of the name’ part of the error was my first clue, but after investigating, this was not the case unfortunately. After some further investigation it seemed that the Automation Account did not include the AzureRM.DataFactoryV2 module inside my automation runtime.

To get an overview of the available PowerShell modules inside the runtime, you can run the following command:

Get-Module -ListAvailable Azure* | Select-Object Name, Version, Path

This confirmed that indeed the mentioned DataFactoryV2 module did not exist in this account, as it contained only the default modules.

If you want to use other modules, you need to import them into the runtime environment.

There are several ways to accomplish this

Importing using a Fully Qualified path

The easiest way to accomplish this is by importing the module using a fully qualified path:

Import-Module "$env:ProgramFiles(x86)\Microsoft SDKs\Azure\PowerShell\AzureRM.psd1"

(Source: https://docs.microsoft.com/en-us/powershell/azure/other-install?view=azurermps-6.0.0)

But sometimes this does not work, because there are some conditions that have to be fulfilled.

Some common reasons that a module might not successfully import to Azure Automation are:

• The structure does not match the structure that Automation needs it to be in.
• The module is dependent on another module that has not been deployed to your Automation account.
• The module is missing its dependencies in the folder.
• The New-AzureRmAutomationModule cmdlet is being used to upload the module, and you have not given the full storage path or have not loaded the module by using a publicly accessible URL
• …

I was not able to get this work, probably related to the fact I had no prior experience in PowerShell.

Fortunately, I found another method to import a new PowerShell Module.

Using the Azure Portal

Open your Automation Account in the Azure portal, scroll down to ‘Shared resources’ and choose ‘Modules’. The list of available modules in this account are shown.

When you want to start loading new modules, run the ‘Update Azure Module’ so you have the latest version of the standard modules.

Next step is to import the modules you need.

There are 2 different ways to do this:

Use the ‘Browse gallery’ to import the AzureRM.DataFactoryV2 module.

Unfortunately, it was not working in my case. It was throwing error because he expected an AzureRM.profile version > 5.0.0 (even though I updated the Azure modules before, he still had a 4.6.0 version).

In the gallery I searched for AzureRM.profile. I found a version > 5.0.0 and imported this one. I received a succeeded message but he still gave a 4.6.0 version in the modules tab.

So, both the ‘Update Azure Module’ and ‘Browse gallery’ were not able to load the required AzureRM.profile version required to load the AzureRm.DataFactoryV2 Module.

This is probably the same reason why my first attempts failed.

Add a PowerShell module from your local machine directly

The other option provided is to add a PowerShell module from your local machine.

Make sure that you updated your local PowerShell modules (https://docs.microsoft.com/en-us/powershell/azure/other-install?view=azurermps-6.0.0) prior to uploading them and then import the module as a zip file using the ‘add a module’ option.

Note: Even though a Microsoft blog (https://azure.microsoft.com/nl-nl/blog/announcing-azure-resource-manager-support-azure-automation-runbooks/) stated the below, I only had to import the module I need to make it work.

Please note the new guidance is that if the latest version of any Azure/AzureRM module is imported as a user module to an automation account, the latest versions of ALL Azure/AzureRM modules (not just the ones that ship out of the box in Azure Automation) should be imported to the automation account, to avoid any version mismatch issues that could occur now or in the future if Azure Automation later ships any additional (or newer version) Azure/AzureRM modules as global modules.

After the import from my local machine using the UI on the Azure Portal he imported the required module, and my runbook worked like a charm.