Showing posts with label query. Show all posts
Showing posts with label query. Show all posts

Wednesday, February 14, 2024

Sitecore ADM: Resolving Stalled Tasks and Restoring Task Processing

My team is currently in the process of purging millions of historical anonymous xDB contact records and associated data using the ADM module for a client whose xDB shard database sizes have been approaching max storage capacity for the Azure tier.  Because xDB is a crucial portion of the client site's operations, our options for reducing the DB size have been somewhat limiting due to complex custom external integrations with xDB. 

In our approach, we opted to use ADM to purge historical anonymous contacts in batches. We prepare ~300k contact records per shard for each batch, which are manually retrieved via SQL query. Once we've created the temporary table in the shard DB, we prepare the data by generating a comma-delimited list of contacts and then kick off the purge process via ADM. 

When ADM populates its Tasks table, each queued record is subsequently processed by ADM and removed from the Tasks table as it completes processing that record.  The ADM task execution is a generally slow process (1 contact processed every 2-3 seconds); we closely monitor the progress with a SQL query:



With this approach (in addition to SHRINK and REINDEX operations between batches), we have seen the necessary disk size reduction of both xDB shard DBs after running a cadence of several batches.  

However, we ran into a snag in a recent batch, which resulted in the entire ADM task processing halting entirely.  The issue appeared to directly correlate with general Azure Maintenance operations, which had occurred over the weekend while the batch was mid-process.  Azure Maintenance updates typically happen without any advanced notice or warning.  Usually, Azure Maintenance operations have minimal adverse effects, but this round seemed to have caused much of the infrastructure to spiral.  We observed that the ADM tasks were no longer processing when all was said and done.  

Attempts to re-start the job via ADM kept resulting in the same error:

"[ADM] Response from xConnect did not indicate success. Status code: BadRequest, Message: {\"Message\":\"The remove task can't be started while another one is running.\"}"

Upon initial analysis, we noted that the ADM tasks table was still populated with IDs that had yet to be processed when the operation was cut off. I began dissecting the ADM binary files for clues - specifically in search of the message "The remove task can't be started while another one is running".  

I learned that the StartContactsDataRemoving method queries an IsRunning method to determine if any other tasks are in progress. If there are, it throws a BadRequest
response and returns the "The remove task can't be started while another one is running.
message" message. 



Digging deeper led me to this ClearRemoveDataSettings method - called in the StopRunningTasksAndClearStorage method.  Deeper in, there are references to a PropertiesRepository class and an object name of "RemoveDataSettings" used to store task information:


This, in turn, finally led me to a PropertyValueQuery method in a PropertiesRepositoryQueries class, which contained a SQL command used as part of the process:




We reviewed the current state of the ADM Properties table within the ADM DB and found three entries, including RemoveDataSettings:


The RemoveDataSettings record's value appeared to be a JSON representation of ADM's last ADM removal task run.  However, the JSON representation was cut off after a few hundred characters.  With this state of the present value, ADM was convinced that the task was not completed.  

Following the approach used in the code (mimicking what should occur when an ADM removal task is completed), we ran the following command:

We also entirely cleared the remaining IDs and Tasks table and re-initialized the process.  With these steps, our ADM tasks were back to processing as expected.

I hope this one helps anyone in a similar situation!


Friday, September 8, 2023

Building Better Reports: 5 Sitecore PowerShell Extensions Functions for Your Toolbelt


When navigating the Sitecore ecosystem effectively, PowerShell extensions aren't just a helpful tool; they're practically a prerequisite. As someone who has spent a substantial amount of time in this space, I have distilled a set of functions that I've found myself using time and time again. 

Join me as I unpack a curated list of five functions that have become staples in my Sitecore toolkit.

Get-IsPublished

The "Get-IsPublished" function checks if a specific item, identified by its ID, is published on a "web" database. It takes an item as input and queries for it against the "web" database using its ID. If it finds the item in the database, it returns "TRUE," indicating that it is published. If it doesn't find the item (i.e., if the item is null), it returns "FALSE", indicating that it is not published.

This is particularly useful for displaying a column in your SPE report to denote if the item is published to the web database:


Get-ItemSitecoreCELink

The "Get-ItemSitecoreLink" function generates a URL to open a specific item in the Sitecore Content Editor. It takes a Sitecore item as input and uses various properties of that item (like its ID, version number, and language) to construct a URL. This URL, when accessed, will open the Sitecore Content Editor with that specific item loaded, allowing for easy navigation directly to the editing interface for that item. 

Please note that the base of the URL ("https://yourcmurl.com/") in this function is a placeholder that you'd replace with your actual Sitecore Content Management URL.

This is useful if the report is exported to Excel or CSV as it provides the direct link to reach the item without manually traversing the tree or searching for the item by GUID.


Get-LinkFieldUrl

The "Get-LinkFieldUrl" function retrieves the URL from a Sitecore item's link field. It takes a Sitecore item as its input and utilizes a regular expression to extract and return the URL stored in a "My Link Field" field. The regular expression is designed to find and capture the URL stored as a value in a link HTML element's URL attribute.  

If the "My Link Field" is not empty and matches the pattern specified by the regular expression, the URL is retrieved and returned. If the field is empty or doesn't match the pattern, the function returns nothing, essentially returning a null value.

This function couples well for reports if you need to extract the URL out of a Sitecore Link Field value to display in your report:

Assert-HasLayout

The "Assert-HasLayout" function checks if a given Sitecore item has a "final layout" defined. It accepts a Sitecore item as its input and uses the built-in "Get-Layout" function to retrieve the final layout details of the item.  If the item has a final layout (meaning the $layout variable is not null or empty), the function returns a "TRUE" string, indicating that a layout is present.

If no layout is found for the item (meaning the $layout variable is null or empty), it returns a "FALSE" string.

Usage example


Get-FormattedDate

The "Get-FormattedDate" function takes a raw date string as its input and attempts to turn it into a more user-friendly date format.

The raw date string is expected to follow a particular "yyyyMMddTHHmmssZ" pattern ("20230908T123456Z" representing September 8, 2023, 12:34:56 PM in Coordinated Universal Time, for example) - which is precisely how Sitecore typically stored DateTime fields in the database.

The function reads this string and converts it to a date format that is more commonly used, which includes the month, day, and year (like "09/08/2023"). If, for any reason, it can't convert the input into a date (maybe because the input doesn't follow the expected pattern), it will simply return an empty string. This way, even if it receives unexpected inputs, it won't crash and will still produce a result, even if that result is just an empty string.

By passing in the raw DateTime field value into the Get-FormattedDate function, the report will convert it to a readable string value when exporting the report to Excel, CSV, or JSON.




Whether you're a seasoned Sitecore developer or just starting out, these 5 functions can become essential tools in your developer toolkit, helping you navigate the complexities of Sitecore with greater ease and efficiency ✌.

Tuesday, February 28, 2023

Sitecore xDB Contact Lookup Utility

My client's lead flow includes capturing form data into a custom database. Each record contains an xDB Contact ID to correlate the data with the data captured in xDB directly.  We had a scenario where we had hundreds of records in the custom database where the xDB contact ID was missing, likely due to a networking issue between the CD and xConnect.

I needed a way to match users captured in the custom database against the data in xDB to obtain an existing xDB contact ID.  Luckily, Sitecore's documentation provides some sample code to help get me started: xConnect Client API (C#) (sitecore.com)

Thus was born, the...

🧰 xDB Contact Lookup Utility


The utility is your standard .aspx file that can be dropped into the/sitecore/admin folder or at the root of CM instance. It combines a series of xConnect API calls to search for contact information in the xDB.  Applicable parameters include email address, first name, last name, or xDB contact ID.

It can be used in two ways:

First, through the UI (as shown above), you have the ability to populate text boxes to obtain parameters and view the result list on the page.

Secondly as a simple rest endpoint, with the ability to pass parameters into the URL and return a JSON response containing a matching xDB contact ID.

For example: 
/xDB-Contact-Lookup.aspx?email=test@test.com&firstName=test&lastName=test




👨‍💻 Code Breakdown

Page_Load

The Page_Load event checks if a query parameter named "email," "lastName," or "firstName" are present in the URL. 

If at least the "email" parameter is present, the method calls the SearchForContactIdByEmail .If the lastName and firstName parameters accompany the email parameter, the SearchForContactIdByName method is also used to search for a contact. The search results are returned as a JSON response which contains an xDB contact ID of the first match or "NoMatch" if no match was found. The JSON string is written to the HTTP response, ending the response. If the "email" query parameter does not exist, the method does nothing and loads the UI.

btnLookup_Click

This button click handler retrieves user input from four text boxes for known contact ID, email, last name, and first name. If the user input is for a known contact ID, it calls the SearchForContactById method to retrieve the contact's information and displays it in an HTML table. 

If there is no matching contact, it displays a message saying that no contact was found. 

If the user input is for email, last name, and/or first name, it calls SearchForContactIdByEmail and/or SearchForContactIdByName methods to retrieve contact information and displays it in an HTML table. 

If there are no matching contacts, it displays a message saying that no contact was found.

There are three methods implemented for using the xConnect API:

SearchForContactById

This method searches for a Contact object in xDB using a known contact ID. The method SearchForContactById takes a string parameter knownContactId and returns a Contact object. The method first converts the knownContactId string to a Guid, creates an instance of Sitecore.XConnect.Client.XConnectClient, and uses the Get method of the client object to retrieve a Contact object using a Sitecore.XConnect.ContactReference object created from the Guid

It also uses a Sitecore.XConnect.ContactExpandOptions object to specify which facets of the Contact object to retrieve. 

If the Get method throws an XdbExecutionException, the method catches the exception and returns null.

SearchForContactIdByName

This method searches for a list of Contact objects by their LastName and FirstName facets of PersonalInformation

SearchForContactIdByName takes two string parameters, lastName and firstName, and returns a list of Contact objects. 

Inside the Sitecore.XConnect.Client.XConnectClient using block, the method creates an IAsyncQueryable object named queryableLastName that represents a xConnect API query to xDB. The query retrieves all contacts where the LastName and FirstName facets of PersonalInformation match the values of the lastName and firstName parameters. 

The query then sorts the results in descending order of LastModified. The WithExpandOptions method is used to specify which facets of the Contact object to retrieve. The method then retrieves the query results using a batch enumerator, which is added to a List<Contact> object named idsList. 

If the query throws an XdbExecutionException, the method catches the exception and returns null.

SearchForContactIdByEmail

This method searches for a list of Contact objects by their EmailAddress and optionally their LastName and FirstName facets of PersonalInformation

SearchForContactIdByEmail takes three string parameters, email, lastName, and firstName, and returns a list of Contact objects. 

Inside the using block, the method creates an IAsyncQueryable object named queryable that represents a xConnect API query to the xDB. The query retrieves all contacts where the PreferredEmail facet of EmailAddressList matches the email parameter value. 

The query then sorts the results in descending order of LastModified. The WithExpandOptions method is used to specify which facets of the Contact object to retrieve. The method then retrieves the query results using a batch enumerator, which is added to a List<Contact> object named idsList

If the firstName and lastName parameters are not empty, the method checks if the FirstName facet of PersonalInformation and LastName facet of PersonalInformation match the values of the firstName and lastName parameters respectively for each Contact object retrieved. 

If both match, the Contact object is added to idsList. If the firstName and lastName parameters are empty, all Contact objects retrieved are added to the idsList

If the query throws an XdbExecutionException, the method catches the exception and returns null.

Final Result

The full code can be copied here. Add the content to a .aspx file and place it where you need it.


As always, feel free to modify or build on top of this to satisfy your own requirements.

🚀


Tuesday, April 21, 2020

Automate Reverse Azure Database Migrations using PowerShell



Working with Production-level content in lower environments (eg. DEV or UAT) is important for ongoing development and testing.  Depending on your item serialization/source control approach, keeping content in sync can be a challenge.

Using Unicorn or TDS for templates and layouts is common, but source-controlling all content (specifically media items) can bring a lot of weight to the project.  In lieu of utilizing serialization technologies or a synchronization tool such as Razl to synchronize content (which I've seen take hours to complete depending on the content load), a common approach is to periodically restore the Master/Web Databases from a Production environment down to lower environments.

In an Azure PaaS setup, without any automation or scripting, this manual process may look like this:

1) Log in to Azure Portal

2) Navigate to the source (production) SQL server instance's Master/Web database

3) Click the copy button and set up the database copy operation configuration (target database name, target server, and pricing tier)

5) Execute the copy operation and wait for the copied database to become available.

6) Log into the target SQL Server instance using SQL Server Management Studio or use the  SQL Database Query Editor built into Azure Portal, and execute an ALTER USER query to reset the login password to match the original database passwords

7) Rename the currently connected Master/Web database to include a suffix in the name (eg. _OLD)

8) Rename the copied Master/Web database to use the original Master/Web database name

9) Restart the server

Obviously, this process can vary and is generally a tedious and time-consuming process.

Luckily, Azure resources can be managed using the suite of PowerShell commands without ever needing to access the UI.  With the right script, the strain of manually executing these steps can be alleviated.

To use these commands, the Azure PowerShell Module must be installed.

For our scenario, let's assume the following:
1) PROD environment is in a separate Resource Group than the NON-PROD environment

2) While the Master database should suffice, we'll also copy down the Web database to avoid requiring a publishing operation after the script has completed.

3) ConnectionString.config value should not require modification.

4) A short "outage" of the NON-PROD environment will occur during the process since the connected database will be renamed to make room for the copy.

Let's Script It

Step 1 - Define Target and Source Variables

We need to define our target and source variables including source/target Resource Group Names, SQL server names, database names, and NON-PROD environment SQL Admin Credentials.


Step 2 -  Invoke Azure Login Process

This command will invoke the login process to a specific subscription ID. The user will be prompted to log in.


Step 3 -  Rename the currently connected database to make room for the copied database

Since the name of the database on the NON-PROD environment should remain constant, this command will rename the existing NON-PROD database to include a unique dated suffix. Note that this database will not be removed automatically and can be used as a backup in the case that the NON-PROD environment contained content that was not accounted for or backed-up prior to the migration.  Removing it will be up to you.


Step 4 -  Initialize the database copy operation

Once the name of the database is available on the target SQL server, the following command will execute the database copy process.

Step 5 - Execute the ALTER USER query

Since the database login from the source database comes with the process of a direct copy, an ALTER LOGIN query must be executed against the database to reset the [masteruser] or [webuser] passwords to match what's in the NON-PROD ConnectionStrings.config.


Step 6 - Restart the App Service

When the copy operation is completed, restarting both App Services will ensure a fresh connection to the databases is established.


Final Script




Monday, August 28, 2017

Sitecore Text Searcher - The Ultimate SQL Script

"Where can I edit this text in Sitecore?"
"Can you find out where this content can be changed in Sitecore? 
"I tried changing this content in Sitecore here, but it's not reflecting on the front-end after I publish." 

I hear that a lot.  

When maintaining multiple Sitecore solutions - knowing every template, rendering, and how the data ties in to what you see on the front-end - is nearly impossible.   Prior to Helix, developers were free to develop using various patterns. Some are built in MVC, others are WebForms.  Some components rely on datasource items, others rely on data from the items template. Regardless of architecture, we need to be able to help Content Authors change content if they can't find what they're looking to change.

There are typically a few options to obtain the information for the Content Author's request:

     - If you have access to the solution - inspect the page elements, grab a common class or id, do a Entire Solution search, find some files and follow the code to determine how it populates the component.

     - Start with the page item in Sitecore and determine whether the content is located on the item's template, or in within a data source item.

Those options are both fine - and eventually effective - but there is a faster way - IF YOU HAVE ACCESS TO SQL (which is normally the case if you have a recent DB locally or access to the DB server).

Simply start a new query in SQL MS on the Sitecore DB you want to search in (Web or Master),  modify the @SEARCHTERM variable (include the percent symbols eg. '%content goes here%'), and execute this handy script:



DECLARE @SITECOREROOTID NVARCHAR(MAX), @FULLPATHSEARCH NVARCHAR(MAX), @SEARCHTERM NVARCHAR(MAX)

SET @SITECOREROOTID = '11111111-1111-1111-1111-111111111111';
SET @FULLPATHSEARCH = 'sitecore/content/%';
SET @SEARCHTERM = '%search term%';

WITH FullPathItems (ID, NAME, ITEMPATH, TEMPLATEID)
    
AS (SELECT ID,
                
NAME,
                
CAST(NAME AS NVARCHAR(MAX)) AS itempath,
                
TEMPLATEID
        
FROM   [dbo].ITEMS
        
WHERE  ID = @SITECOREROOTID
        
UNION ALL
        
SELECT i.ID,
                
i.NAME,
                
CAST(( ITEMPATH + '/' + i.NAME ) AS NVARCHAR(MAX)) AS itempath,
                
i.TEMPLATEID
        
FROM   [dbo].ITEMS i
                
INNER JOIN FullPathItems a
                        
ON i.PARENTID = a.ID)
                        
SELECT fieldsX.ITEMID,
      
fpi.NAME AS ITEMNAME,
      
fieldsX.VALUE,
      
fpi.TEMPLATEID AS TEMPLATEID,
      
templatesX.NAME AS TEMPLATENAME,
      
ITEMPATH,
      
fieldsX.FIELDID,
      
itemsX.NAME AS FIELDNAME,
      
fieldsX.LANGUAGE  AS ITEMLANGUAGE 

       FROM   [dbo].[FIELDS] fieldsX
      
INNER JOIN FullPathItems fpi
              
ON fpi.ID = fieldsX.ITEMID
      
INNER JOIN [dbo].ITEMS itemsX
              
ON itemsX.ID = fieldsX.FIELDID
      
INNER JOIN [dbo].ITEMS templatesX
              
ON templatesX.ID = fpi.TEMPLATEID

WHERE fieldsX.VALUE LIKE @SEARCHTERM 
AND fpi.ITEMPATH LIKE @FULLPATHSEARCH

/*INCLUDE  THE FOLLOWING LINE TO INCLUDE ITEM ID CONDITION */
-- AND fieldsX.ItemId = 'B3F15B8C-BAE5-40F4-A139-BA7B3EC6E1ED'

/*INCLUDE THE FOLLOWING LINE TO INCLUDE ITEM NAME CONDITION */
-- AND fpi.Name LIKE '%My Item%'

/* INCLUDE THE FOLLOWING LINE TO INCLUDE TEMPLATE NAME CONDITION */
--AND templatesX.Name = 'My Template'

/* INCLUDE THE FOLLOWING LINE TO ADD LANGUAGE CONDITION */
-- AND fieldsX.Language = 'en-US'


ORDER  BY fpi.ITEMPATH



The result includes the following columns:

Sometimes, you'll be presented with several instances of a specific keyword, in which case you can uncomment various lines near the bottom of the script to drill down and filter the results even further.

I find myself using this script on a daily basis and hope that sharing this script helps others track down content quickly and effectively, too.