Wednesday, July 22, 2020

Approaches to Dockerizing Existing Sitecore Solutions for Local Development


As a developer at a digital agency working in Managed Services, I work with multiple customers spanning multiple versions of Sitecore. The client sites, more often than not, are inherited from vendors outside of reach - each with a unique set of onboarding steps and requirements.

Even with well-defined onboarding steps for local solution setups, depending on the complexity of the solution and local infrastructure required, this process could take new developers days to complete.

In terms of software, we may be required to maintain multiple versions of SQL server or Solr to accommodate the demands of the varying versions of Sitecore. Common tasks necessary to set up include, but are not limited to, database backup files to download and restore complex Sitecore installations to run, host binding updates, configuration changes, module installations, publishing profile setup, and configuration transforms. Just when you think you're finally done, you've got conflicting binaries, or misconfiguration causing YSOD.

I've seen developer's local environments break mid-day, stopping them in their tracks for hours. What about a scenario where a developer is only temporarily joining a team for coverage - or, if a developer simply needs a sandbox of a functioning client site to verify or execute a proof-of-concept?  The time it takes to onboard their local environment can be daunting. 

Of course, creative solutions to this have been developed and tried with varying results (pre-built VM with Visual Studio and a site on a USB drive, anyone?). Until, Docker.

Many modern Sitecore developers hare actively including Docker into their workflow for new development projects using the latest version of Sitecore (9.3 at the time of this post). These startup projects heavily rely on item serialization techniques (TDS, Unicorn, etc.) to deploy onto clean Sitecore images. Existing solutions, unfortunately, may not be able to benefit from this modern developer experience.

That doesn't mean we can't adjust to make Docker a viable option (or at least as a supplement). It's no question now that you will likely be working with Docker in some capacity as the Sitecore development landscape continues to evolve.

If you're like me, you've been "playing around" with Docker for many months now.  After spending time understanding the patterns and mechanics behind the Sitecore Docker Images repository, it becomes increasingly clear how Docker can be used in a real-life scenario.

The challenge, of course, is that the Sitecore Docker Images repository assumes your solution is using a serialization mechanism like TDS or Unicorn that will update the database will all base-level templates, layouts, etc.  While this is most often true when building Sitecore solutions from the ground up, it is less likely true for existing solutions that are already on a maintenance model.  

To take an existing solution and "Dockerize" it, we'll want to account for some site-specific customizations.  This includes:
1) Custom SQL Databases (Master, Web, and Core - at minimum)
2) Custom/Prebuilt Solr Cores 
3) Baseline files that make up the website (assuming you have a working local instance already)
4) Custom hostnames that can map to the containers. 

We'll rely heavily on the Sitecore Docker Images repository as a baseline example, following existing patterns while customizing the build.json, Dockerfile, and script files to include new processes that don't already exist. 
It's important to note that the Sitecore Docker Images repository changes frequently. Any examples I use are subject to differences in structure and content, but the concepts should generally remain the same.  

Goals


We'll review common Docker image customizations with repeatable patterns, which I've documented while creating unique Docker images for consumption by developers on a client project.

We'll also identify ways we can improve a developers' local environment onboarding speed without the need for extensive knowledge about container technology. Installing Sitecore locally would no longer be a requirement.

I'll share some self-descriptive PowerShell scripts in which any developer who has the client solution pulled from git (and Docker prerequisites installed) would be able to execute to spin up their environment.  Developers can expect a functional local copy of a unique Sitecore customer's website with full Remote Debugging capabilities (Visual Studio 2017 and 2019, depending on each solution's specific requirements).

I consider this to be a viable supplement (or full replacement) to traditional local environments - allowing you to spin up a local, debuggable copy of a client site in much less time than it would take to install and configure Sitecore locally.

This Example

For this exercise, I'll be using a real client site running on Sitecore 9.0.2.  To keep things simple and lightweight, we'll focus on setting up an XM topology over an XP topology with our baseline website files, custom Master, Web, and Core databases, and custom Solr cores. I'll assume you have a local instance of the site you want to Dockerize.  

Common Tools

Other Prerequisites


Let's start by creating a directory called 'Docker' at the same level as our solution's project directory.  Within this folder, we'll create a 'setup' directory and a 'deploy' directory.  The 'setup' directory will contain a smaller, less beefy version of the Sitecore Docker Images repository containing only what we need for the project.  The 'deploy' directory will act as the target directory for publishing the solution from Visual Studio (when the containers are running, a file watcher transfers the files from the 'deploy' directory to the 'inetpub' directory running in the container).  

The Sitecore Docker Images repository is chock-full of folders and files which allow users to build out clean Sitecore Docker images in various topologies.  Since our customized images won't require a majority for these directories, we can take only what we need to give us a starting point. 

In our case, we'll take the following folders and files from the cloned repository and place them into our 'setup' directory:
  • The modules folder (contains the SitecoreImageBuilder PowerShell module)
  • The windows folder 
  • Build.ps1
  • sitecore-packages.json



Inside the 'windows' folder, we'll find a series of folders, two of which stand out among the rest:
dependencies and tests
 
The 'dependencies' folder contains conventional images used across a majority of Sitecore flavors, so we'll just keep that where it is.  

The 'windows/tests' folder contains some crucial files we'll need to compose Docker up and down.  We'll copy the following files into the 'Docker' directory :
  • \windows\tests\9.x.x\.gitignore
  • \windows\tests\9.2.x\Clean-Data.ps1
  • \windows\tests\9.2.x\docker-compose.xm.yml 
    • Rename this to docker-compose.yml
  • \windows\tests\9.2.x\.env
👇

The remaining folders in the windows directory might seem confusing at first, so let's trim it up by removing all folders except the ones that pertain to version 9.0.2:


Let's take a look at the '9.0.2' directory:

The two folders here represent two self-descriptive images; 'sitecore-assets' and 'sitecore-xm' - both of which we will keep as a base.  The 'sitecore-assets' image is used to store entry point PowerShell scripts (all of which are located in the inner tools folder) and referenced in the images' Dockerfile.  You can think of it as a repository of assets that exist solely to store files that will be transferred to other images during the image building process.

The 'sitecore-xm' folder contains a Dockerfile and build.json (among a few other folders/files), which together defines the XM CM and CD images.

Within the 9.0.x directory, we'll find a lone 'sitecore-xm-solr' folder.  This folder - and the nested files within it - will be used to build our Solr instance.

Finally, in the 9.x.x directory, we'll find a series of folders that contain various available image topologies.  We can remove all folders except for 'sitecore-xm-sqldev':


To eliminate complexity for our folder structure, we'll copy over each inner folder (those in 9.0.2, 9.0.x, and 9.x.x) directly into the 'windows' folder (IMO: consolidating just makes it easier to navigate). This is optional, and really about preference.

The final folder structure looks like this:
  


As mentioned in the Preface, there are three specific assets we want to share with other Dockerfiles:

Databases

First: site-specific Master, Web, and Core databases.  Most of the websites I manage are hosted in Azure, and as a standard practice, database backups are periodically taken and stored in an Azure Storage Account.  These databases backups files are formated using '.bacpac.' 

If you dig into the existing Sitecore Docker Images Dockerfiles, you'll find that they utilize '.scwdp.zip' packages that contain everything needed to install a specific version Sitecore (the same files used during the days of Sitecore Installation Framework) across the several images that make up a Sitecore website.  The '.scwdp.zip' files contain '.dacpac' files that are used to deploy clean Sitecore Databases in the 'sitecore-xm-sqldev' image.  

By including '.bacpac' files, we'll be overriding the default behavior to upload and import our custom databases instead.  

Solr Cores

Second: custom or prebuilt Solr Cores to ensure Search capabilities work for whoever happens to grab the images and run the containers - without requiring index rebuild.  

This one may vary depending on your solution, but Solr is most commonly present in some capacity.  Whether you use the default Sitecore indexes for search or have custom Solr indexes, we can inject the pre-built indexes into the 'sitecore-xm-solr' image.  I admit now that this may be a bit overkill since a first-time index rebuild is a super common practice when setting up a local Sitecore environment. However, it is possible, and I've done it, so I'll share it anyway.   

In my case, my local Solr indexes can be packaged from my machines \server\solr location into a .zip file called 'CustomSolrCores.zip.'

Baseline Website Files

Third: baseline files that make up the website.  I found that publishing the solution to a folder from Visual Studio is a good starting point.  The file repository should exclude the ConnectionString.config as it will be generated to appropriately point to the appropriate Docker Solr and SQL containers in the 'sitecore-xm' image. 

This folder should also include any specific files that are not in source control - like Sitecore modules you may have installed that require particular files.  For example, if you have Sitecore PowerShell Extensions installed, you'll want to include the data from the 'sitecore modules' folder (the Core database we're uploading will have everything else we need for the module to function).  

Assets Image Customization

The 'sitecore-assets' folder we obtained in Part I can be used for this. In this folder, we'll create a folder called 'custom':


All custom assets can be copied into the 'custom 'folder:

  • XXXXX-master-db.bacpac
  • XXXXX-web-db.bacpac
  • XXXXX-core-db.bacpac
  • CustomSolrCores.zip
  • CustomBaselineFiles.zip

Note: I don't recommend zipping the  .bacpac files up due to Windows extraction size limitations I've experienced first-hand.

Customize the build.json 

The build.json file customizations are pretty limited in the 'sitecore-assets' folder.  We simply want to ensure the tag and source parameters are corrected for the Sitecore version we're setting this up for. 

Customize the Dockerfile 

We want to load the assets in the 'custom' folder into the 'sitecore-assets' image.  This can be done by adding a new COPY instruction in the existing '#copy local assets' section:



This command copies the folder into the c:\custom location of the Docker context during the build.
 
Near the end of the Dockerfile, we'll include another COPY instruction:


 The 'sitecore-assets' image now contains our custom .bacpac database files, which can be referenced and utilized on all other Dockerfiles - including 'sitecore-xm-sqldev.'  

Customize the build.json

The build.json configuration file for the 'sitecore-xm-sqldev' image contains several version-specific tags by default. 

 

To simplify this for our solution, we can remove all but the 9.0.2 tag.  We'll also include three additional build arguments which serve as path references to each custom database file within the C:\custom folder in the 'sitecore-assets' image. 

 

Customize the Dockerfile

In the default Dockerfile, we can see how the .scwdp.zip file (containing all Sitecore installation files for 9.0.2) is consumed from the 'sitecore-assets' image.

The ASSETS_IMAGE argument in the build.json file is referenced at the top using an ARG instruction. A FROM instruction specifies the 'sitecore-assets' image as a 'Parent Image.' An INSTALL_PATH environment variable is set to a C:/install folder in the 'sitecore-xm-sqldev' image. Finally, a COPY instruction copies the file from the 'sitecore-assets' to the defined INSTALL_PATH location on the 'sitecore-xm-sqldev' image. 

 

This file is then further processed (extraction by Extract-Database.ps1 and installation by Install-Database.ps1) using PowerShell commands via the RUN instruction.

We can follow these same patterns to copy our three custom database .bacpac files from the 'sitecore-assets' image to the 'sitecore-xm-sqldev' image:

Customize Database Installation Script

The extraction and installation PowerShell scripts accept the INSTALL_PATH variable where the .scwdp.zip Sitecore installation file and our three custom .bacpac files now reside. Since we don't need to extract anything new, we can skip any customizations to Extract-Databases.ps1. 

The default Install-Databases.ps1 script is responsible for executing a SqlPackage.exe command for each available .dacpac file (extracted from the .scwdp.zip file) in the $InstallPath location: