Table of Contents
Upload files to A storage blob using powerShell
We are configuring an Azure Automation environment. This is all done through Azure DevOps CI/CD.
We won’t use a static files server for files like html content or zip files, but we will upload them to an Azure Storage Blob instead.
We been working on uploading files to the Azure Storage Blob in a creative way since Azure DevOps won’t get access to the Storage Container. Our StorageContainer is limited by network & Azure DevOps is not on the trusted services list yet.
I’ve been working on a module for the Azure Rest API for some time now. This makes it easier for me to deal with throttling, batching, refreshing the AccessToken and so on.
The module is the easiest to use, but I understand that people don’t always want a third party module on their company servers.
I have written a tutorial for uploading zip files with the module and manually.
I always opt for the minimum use of modules because then you become dependent. If you need one file, I wouldn’t go for the Optimized.Aza module.
Do you have a series of files to upload from an Azure Storage Blob? Then I would look at the module to make sure throttling & RefreshToken won’t be an issue.
The easy way: Optimized.Aza
For more about Optimized.Aza:
PowerShellGallery.com
Github.com
Let’s start uploading files to a storage with PowerShell?
If you haven’t yet, install Optimized.Aza via:
Install-Module Optimized.Aza -Scope CurrentUser
Logon to the Azure Rest API:
$ConnectAzaSplatting = @{
ClientSecret = 'XXXXXXX'
Resource = 'https://XXXXXXX.blob.core.windows.net/'
ApplicationId = 'XXXXXXXX'
Tenant = 'XXXXXXX'
}
Connect-Aza @ConnectAzaSplatting
Get the bytes content from your file & create a CustomHeader.
Do not forget to change the path in [System.IO.File]::ReadAllBytes
.
$Bytes = [System.IO.File]::ReadAllBytes('C:\temp\temp.zip')
$CustomHeader = @{
'x-ms-version' = '2020-10-02'
'x-ms-blob-type' = 'BlockBlob'
'x-ms-type' = 'file'
'Content-Type' = 'application/octet-stream'
}
Now run the cmdlet with the correct Url & filename (in the Url):
$PutAzaSplatting = @{
CustomHeader = $CustomHeader
InputObject = $Bytes
KeepFormat = $true
Url = 'https://baswijdenes2019.blob.core.windows.net/moduleblob/temp.zip'
}
Put-Aza @PutAzaSplatting
The file should now be uploaded into the Azure Storage Container. Check if the file has been uploaded correctly.
I’d like to upload files without a third party module
Understandable. There are many companies that would rather not use a third party module because you simply don’t know what it does unless you go through the source code.
The disadvantage of doing this manually is that you will have to build in certain things yourself, such as: automatically refreshing the AccessToken, throttling issues, etcetera.
We need an OauthToken before we can continue
In short: we need an Oauth Token to identify us in header of the request that we invoke to Azure Rest API.
There are several ways to obtain the Access Token, but the most common are the following Identities: User, Service Principal, and Managed Identity.
For more about the Oauth protocol:
OAUTH 2.0 authentication with Azure Active Directory | Microsoft Docs
As an User or Service Principal identity
You can use Get-AzAccessToken
to get the OauthToken from your Connect-AzAccount
login.
Connect-AzAccount
$TokenCache = (Get-AzAccessToken).Token
$Headers = @{Authorization = "Bearer $TokenCache"}
Unfortunately this often doesn’t work because you’re missing permissions to actually download a file from the blob.
What you can try as well is the below, but you will need a Service Principal for this with the permission set for the Storage Blob endpoint.
$ApplicationID = '3aa17b12-ea2f-4c99-817c-XXXXXXX'
$loginURI = "https://login.microsoft.com"
$resource = 'https://STORAGEACCOUNT.blob.core.windows.net/'
$tenant = 'XXXXXXXX.onmicrosoft.com'
$Username = 'XXXXXXXXXXXXXXXXXXX'
$Password = 'XXXXXXXXXXXXXXXXXXX'
$Body = @{
grant_type = 'password'
resource = $($Resource)
username = $Username
password = $Password
client_id = $ApplicationID
}
$TokenCache = Invoke-RestMethod -Method Post -Uri $loginURI/$Tenant/oauth2/token?api-version=1.0 -Body $Body -UseBasicParsing
The downside of doing it this way is that your OauthToken won’t be refreshed automatically unless you use a Az cmdlet again.
For more about Get-AzAccessToken please check:
Get-AzAccessToken (Az.Accounts) | Microsoft Docs
As a Managed Identity
As a managed Identity you can use a function I created for a Customer of us
You can get the function from my Github: New-managedIdentityAccessToken.ps1
And then run this:
$Headers = New-ManagedIdentityAccessToken -Resource 'https://management.azure.com/'
Now that we’ve got our headers, let’s upload a file from Azure Storage Blob with PowerShell
Before we can start uploading to the Azure storage Container we will need to modify the headers a bit.
$Headers = @{
Authorization = "Bearer $($TokenCache.access_token)"
'x-ms-version' = '2020-10-02'
'x-ms-blob-type' = 'BlockBlob'
'x-ms-type' = 'file'
'Content-Type' = 'application/octet-stream'
}
We can now create the Body by Reading the file bytes.
Do not forget to change the path in [System.IO.File]::ReadAllBytes
.
$Bytes = [System.IO.File]::ReadAllBytes('C:\temp\temp.zip')
Last but not least we can invoke our request to Azure.
Do not forget to change the Uri StorageAccount & file
$InvokeRestMethodSplatting = @{
method = 'Put'
Uri = 'https://XXXXXXXXX.blob.core.windows.net/moduleblob/XXXXX.ext'
Headers = $Headers
Body = $Bytes
}
Invoke-RestMethod @InvokeRestMethodSplatting
The file should now be uploaded into the Azure Storage Container. Check if the file has been uploaded correctly. You now know how to upload a file to an Azure Storage Blob with PowerShell.
Where do you get your information from?
I use the official Azure Rest API documentation to compose the URL and headers.
The official docs for getting Blobs:
Put Blob (REST API) – Azure Storage | Microsoft Docs