Showing posts with label devops. Show all posts
Showing posts with label devops. Show all posts

Thursday, May 21, 2020

Part II - Integrating Automated Reverse Azure Database Migration PowerShell Script into Azure DevOps


In my last post, we wrote a handy PowerShell script that takes the latest Master and Web SQL Databases from a Production-level Azure Resource Group and imports them into a Staging/UAT/Dev Azure Resource Group for a seamless reverse database promotion process.  

The original script, however, relies on a developer to run the script manually on a local machine and authenticate their credentials in order to utilize the AzureRm commands:

We can take this script a step further and integrate it as a new stage in the existing Azure DevOps Release Pipeline, or as a new dedicated Release Pipeline that can be executed independently.

In this example, we will create a new Azure DevOps Release Pipeline.  We'll assume a Service Principle connection already exists (which is likely if you're deploying to your App Services using Azure DevOps already) and you have the proper administrator permissions to create pipelines in Azure DevOps.   We'll also be working with an Inline Azure PowerShell script job instead of including a script file from an artifact.  Steps will slightly differ if you want to go that route, but the concept would remain the same. 

Release Pipeline Setup


Head over to the Pipelines > Release dashboard, click the New dropdown and select New release pipeline.


In the 'Select a template' menu, click 'Empty job'.

Modify the Pipeline name, then click on Stage 1 and click the plus sign on Agent job to add a new agent.  Search for 'powershell', find Azure PowerShell task and click the Add button


Set the Azure Subscription to the appropriate service principle, set the Script Type to Inline Script, and set the Azure PowerShell Version to Latest installed version


Save the pipeline and navigate to the Variables section

Variable Setup

Here, we'll add all the variables that we'll consume in the script - allowing for future modification without touching the script code itself.  

In our case, our script calls for the following variables: 
  • - sourceResourceGroupName
  • - sourceSqlServerName
  • - sourceMasterDbName
  • - sourceWebDbName

  • - targetResourceGroupName
  • - targetSqlServerName
  • - targetSqlServerAdminUserName
  • - targetSqlServerAdminUserPassword
  • - targetMasterDbName
  • - targetMasterSqlUserPassword
  • - targetWebDbName
  • - targetWebSqlUserPassword
  • - targetCdServerName
  • - targetCmServerName


Script Modifications


Luckily, our original script doesn't need too much tinkering! Just a bit 😉 

First, we'll want to remove the Login-AzureRmAccount command altogether since the Azure PowerShell task in the pipeline will authenticate off of the service principle.
 
We'll then replace any hardcoded variables with their new corresponding variables we previously configured throughout the script using the $env:someVariableName format:

We'll finish this off by placing the modified script in the Inline Script field of our Azure PowerShell task.




Tuesday, April 21, 2020

Automate Reverse Azure Database Migrations using PowerShell



Working with Production-level content in lower environments (eg. DEV or UAT) is important for ongoing development and testing.  Depending on your item serialization/source control approach, keeping content in sync can be a challenge.

Using Unicorn or TDS for templates and layouts is common, but source-controlling all content (specifically media items) can bring a lot of weight to the project.  In lieu of utilizing serialization technologies or a synchronization tool such as Razl to synchronize content (which I've seen take hours to complete depending on the content load), a common approach is to periodically restore the Master/Web Databases from a Production environment down to lower environments.

In an Azure PaaS setup, without any automation or scripting, this manual process may look like this:

1) Log in to Azure Portal

2) Navigate to the source (production) SQL server instance's Master/Web database

3) Click the copy button and set up the database copy operation configuration (target database name, target server, and pricing tier)

5) Execute the copy operation and wait for the copied database to become available.

6) Log into the target SQL Server instance using SQL Server Management Studio or use the  SQL Database Query Editor built into Azure Portal, and execute an ALTER USER query to reset the login password to match the original database passwords

7) Rename the currently connected Master/Web database to include a suffix in the name (eg. _OLD)

8) Rename the copied Master/Web database to use the original Master/Web database name

9) Restart the server

Obviously, this process can vary and is generally a tedious and time-consuming process.

Luckily, Azure resources can be managed using the suite of PowerShell commands without ever needing to access the UI.  With the right script, the strain of manually executing these steps can be alleviated.

To use these commands, the Azure PowerShell Module must be installed.

For our scenario, let's assume the following:
1) PROD environment is in a separate Resource Group than the NON-PROD environment

2) While the Master database should suffice, we'll also copy down the Web database to avoid requiring a publishing operation after the script has completed.

3) ConnectionString.config value should not require modification.

4) A short "outage" of the NON-PROD environment will occur during the process since the connected database will be renamed to make room for the copy.

Let's Script It

Step 1 - Define Target and Source Variables

We need to define our target and source variables including source/target Resource Group Names, SQL server names, database names, and NON-PROD environment SQL Admin Credentials.


Step 2 -  Invoke Azure Login Process

This command will invoke the login process to a specific subscription ID. The user will be prompted to log in.


Step 3 -  Rename the currently connected database to make room for the copied database

Since the name of the database on the NON-PROD environment should remain constant, this command will rename the existing NON-PROD database to include a unique dated suffix. Note that this database will not be removed automatically and can be used as a backup in the case that the NON-PROD environment contained content that was not accounted for or backed-up prior to the migration.  Removing it will be up to you.


Step 4 -  Initialize the database copy operation

Once the name of the database is available on the target SQL server, the following command will execute the database copy process.

Step 5 - Execute the ALTER USER query

Since the database login from the source database comes with the process of a direct copy, an ALTER LOGIN query must be executed against the database to reset the [masteruser] or [webuser] passwords to match what's in the NON-PROD ConnectionStrings.config.


Step 6 - Restart the App Service

When the copy operation is completed, restarting both App Services will ensure a fresh connection to the databases is established.


Final Script




Wednesday, June 12, 2019

Azure App Service Deployment Error: There is not enough space on the disk.

I recently came across an error during a deployment to one of our Content Delivery App Service instances - stopping the deployment process dead in its tracks. 

The error occurred when deploying to the CD Slot:


This occurred each time we attempted to deploy this step.

The error in full does indicate troubleshooting codes and links (oddly enough, the Microsoft link did not have any trace of the error code):
Failed to deploy web package to App Service. Error Code: ERROR_NOT_ENOUGH_DISK_SPACE More Information: Web Deploy detected insufficient space on disk. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_NOT_ENOUGH_DISK_SPACE. Error: The error code was 0x80070070.

I found it especially strange given that we saw no indication that we've hit any limit on storage space in Azure Portal.  Initially, I thought it may have something to do with storage on the build server, but was able to quickly rule out this theory by verifying that there was plenty of disk space remaining on that machine.

The error was also evident when I attempted to upload any file directly via FTP:

I began a look for some giant file(s) that may be preventing any additional files from being uploaded (since that's what made the most sense given the context of the error message itself).

By logging into the App Service via FTP, I was able to identify two IIS memory dumps located in the /LogFiles directory.  These both appear to have been taken the month prior - and simply never removed. 

After deleting both memory dump directories, I was able to restart the deployment step - which completed without errors. Direct FTP uploads were also restored.

What's up with that?

Well, your Azure App Services is tied to a particular pricing tier that dictates storage, memory, ACU, etc.  In our case, this particular App Services is configured to use the Standard standard tier - which has a 50GB limit.




By leaving the remnants of these IIS memory dumps on the App Service's storage, we must have surpassed that storage limit.

This brings up some interesting questions:

First, based on the configured pricing tier, is there a limit on how much Sitecore can store in App_Data/MediaCache or /temp folders before hitting that cap?

I assume the answer to that is yes - if your application is not setup to periodically clean stale files (which it should by default), it's possible to reach that storage limit and cause this error to surface.  In that case, the quick fix to get your deployment out would be to remove some or all temp files in the application.

Second, how can we monitor this storage limit of App Services in Azure Portal?

I'll have to circle back on this one as I don't quite have the answer to it.  It may even already exist, and I just haven't spotted it yet.   I'll update this post if I do figure that one out, but please let me know if you have the answer in the comments!


Monday, April 29, 2019

Sitecore Azure Kudu Tools PowerShell Module



I've managed to start several blog post drafts with the intention of sharing a few of my PowerShell scripts - but haven't got around to finishing/posting any of them.

This actually led to an epiphany: the scripts I wanted to share all had an underlying theme: obtaining files using the Kudu Rest API using PowerShell.


Huh? Kudu?

Aw, a baby Kudu!

If you don't already know, Kudu is the "engine behind git deployments" in Azure App Service - but can also be used as a built-in diagnostics tool within Azure.   Any time you have an Azure website, you automatically get a 'companion' Kudu (aka SCM) site accessible via the following URL format:  https://{ResourceName}.scm.azurewebsites.net.  

For example, if your CM App Service name is
'mysitecoresite-xp2-small-prd1-cm',

the corresponding Kudu site would be:
https://mysitecoresite-xp2-small-prd1-cm.scm.azurewebsites.net/ 

If you've ever had to Rebuild the xDB index in Azure Search, Sitecore's documentation walks you through how to do so using the Kudu Debug console.

Kudu REST API

One of my favorite feature of Kudu is the Kudu REST API since any files you can access via FTP are also accessible via the Kudu REST API.   You can download files, upload new ones, etc.

Typically, all you need are:
  1. Subscription ID
  2. Resource Group Name
  3. Resource Name
In Azure Portal - you can copy these values from a resource in the overview panel:


Those values are the key to interact with the API using PowerShell's Invoke-RestMethod.
There are a couple prerequisites:
  1. Azure RM module installed and ConnectAccount has been executed.
  2. Valid Azure credentials (same ones used to log into Azure Portal) to invoke Login-AzureRmAccount (converted to Base64)



Sitecore Azure Kudu Tools

I'd never written a PowerShell Module, but I figured this would be a great segway to learning how. I got to reading -- and tinkering -- and refactoring.  

Before long I had written my first PowerShell Module: Sitecore Azure Kudu Tools.

Sitecore Azure Kudu Tools is a collection of functions (three at the time of this post) built to help you get information/files from Sitecore instances on hosted on Azure PaaS using the Kudu Rest API.

It's available on the PowerShell Gallery or check out the GitHub repository.


Get-SitecoreSupportPackage

Allows you to remotely generate a Sitecore Support Package.  The function will download files defined for Sitecore Support Packages into a specified path, obfuscate sensitive data from ConnectionStrings.config, and compress the contents:

\App_Config\* Global.asax
\Logs\* license.xml
eventlog.xml sitecore.version.xml
Web.config


Why?

While there's a built-in way to obtain Sitecore Support Packages in Sitecore's admin page, being able to remotely obtain this information is a nice alternative.  This is useful not only for providing the required information for Sitecore Support tickets - but also for your own diagnostics.

Usage

Invoke this function using the following syntax:


Output


Alternatively, if you just run Get-SitecoreSupportPackage without any parameters, you'll be prompted to provide each required parameter.



Invoke-SitecoreThumbprintValidation

Provides a way to verify that certificate thumbprints match across Sitecore Azure PaaS Resource Group.  The function will download ConnectionStrings.config and AppSettings.config files from all App Services in a given Resource Group, then display any certificate thumbprints discrepancies.

Why?

If you've ever had to replace thumbprint values across an Azure PaaS Sitecore topology, you know that it can be a bit of a pain identifying all the configs where the old thumbprint needs to be replaced.

Bram Stoop wrote a great post identifying all the places you'll need to modify those thumbprints.  If you're looking to semi-automate this process, you can utilize the Kudu REST API to get all the configurations, then identify any mismatched thumbprint values using PowerShell.

Usage

Invoke this function using the following syntax:


Output




Get-SitecoreFileBackup

Allows you to a full copy of an App Service's website file contents.  This function will download all files from in a given ResourceName.

The following folders are excluded:
_DEV sitecore modules
App_Data sitecore_files
App_Browsers temp
sitecore upload
xsl

Why?

Sometimes I just want everything. If I need to determine discrepancies between environments for any reason, this helps.

Usage

Invoke this function using the following syntax:

Output




What's Next?

Now that the module is established, I've created a few GitHub Issues which I hope to get through quickly.  I also plan to add more functions in the future and am 100% open to contributions. 

Is it perfect? Far from it! It's a work in progress. Thrilled to share anyway.

If you're interested in contributing, please check out the contribute section on the Github repo.

via GIPHY

😊