Showing posts with label xdb. Show all posts
Showing posts with label xdb. Show all posts

Wednesday, February 14, 2024

Sitecore ADM: Resolving Stalled Tasks and Restoring Task Processing

My team is currently in the process of purging millions of historical anonymous xDB contact records and associated data using the ADM module for a client whose xDB shard database sizes have been approaching max storage capacity for the Azure tier.  Because xDB is a crucial portion of the client site's operations, our options for reducing the DB size have been somewhat limiting due to complex custom external integrations with xDB. 

In our approach, we opted to use ADM to purge historical anonymous contacts in batches. We prepare ~300k contact records per shard for each batch, which are manually retrieved via SQL query. Once we've created the temporary table in the shard DB, we prepare the data by generating a comma-delimited list of contacts and then kick off the purge process via ADM. 

When ADM populates its Tasks table, each queued record is subsequently processed by ADM and removed from the Tasks table as it completes processing that record.  The ADM task execution is a generally slow process (1 contact processed every 2-3 seconds); we closely monitor the progress with a SQL query:



With this approach (in addition to SHRINK and REINDEX operations between batches), we have seen the necessary disk size reduction of both xDB shard DBs after running a cadence of several batches.  

However, we ran into a snag in a recent batch, which resulted in the entire ADM task processing halting entirely.  The issue appeared to directly correlate with general Azure Maintenance operations, which had occurred over the weekend while the batch was mid-process.  Azure Maintenance updates typically happen without any advanced notice or warning.  Usually, Azure Maintenance operations have minimal adverse effects, but this round seemed to have caused much of the infrastructure to spiral.  We observed that the ADM tasks were no longer processing when all was said and done.  

Attempts to re-start the job via ADM kept resulting in the same error:

"[ADM] Response from xConnect did not indicate success. Status code: BadRequest, Message: {\"Message\":\"The remove task can't be started while another one is running.\"}"

Upon initial analysis, we noted that the ADM tasks table was still populated with IDs that had yet to be processed when the operation was cut off. I began dissecting the ADM binary files for clues - specifically in search of the message "The remove task can't be started while another one is running".  

I learned that the StartContactsDataRemoving method queries an IsRunning method to determine if any other tasks are in progress. If there are, it throws a BadRequest
response and returns the "The remove task can't be started while another one is running.
message" message. 



Digging deeper led me to this ClearRemoveDataSettings method - called in the StopRunningTasksAndClearStorage method.  Deeper in, there are references to a PropertiesRepository class and an object name of "RemoveDataSettings" used to store task information:


This, in turn, finally led me to a PropertyValueQuery method in a PropertiesRepositoryQueries class, which contained a SQL command used as part of the process:




We reviewed the current state of the ADM Properties table within the ADM DB and found three entries, including RemoveDataSettings:


The RemoveDataSettings record's value appeared to be a JSON representation of ADM's last ADM removal task run.  However, the JSON representation was cut off after a few hundred characters.  With this state of the present value, ADM was convinced that the task was not completed.  

Following the approach used in the code (mimicking what should occur when an ADM removal task is completed), we ran the following command:

We also entirely cleared the remaining IDs and Tasks table and re-initialized the process.  With these steps, our ADM tasks were back to processing as expected.

I hope this one helps anyone in a similar situation!


Tuesday, February 28, 2023

Sitecore xDB Contact Lookup Utility

My client's lead flow includes capturing form data into a custom database. Each record contains an xDB Contact ID to correlate the data with the data captured in xDB directly.  We had a scenario where we had hundreds of records in the custom database where the xDB contact ID was missing, likely due to a networking issue between the CD and xConnect.

I needed a way to match users captured in the custom database against the data in xDB to obtain an existing xDB contact ID.  Luckily, Sitecore's documentation provides some sample code to help get me started: xConnect Client API (C#) (sitecore.com)

Thus was born, the...

🧰 xDB Contact Lookup Utility


The utility is your standard .aspx file that can be dropped into the/sitecore/admin folder or at the root of CM instance. It combines a series of xConnect API calls to search for contact information in the xDB.  Applicable parameters include email address, first name, last name, or xDB contact ID.

It can be used in two ways:

First, through the UI (as shown above), you have the ability to populate text boxes to obtain parameters and view the result list on the page.

Secondly as a simple rest endpoint, with the ability to pass parameters into the URL and return a JSON response containing a matching xDB contact ID.

For example: 
/xDB-Contact-Lookup.aspx?email=test@test.com&firstName=test&lastName=test




👨‍💻 Code Breakdown

Page_Load

The Page_Load event checks if a query parameter named "email," "lastName," or "firstName" are present in the URL. 

If at least the "email" parameter is present, the method calls the SearchForContactIdByEmail .If the lastName and firstName parameters accompany the email parameter, the SearchForContactIdByName method is also used to search for a contact. The search results are returned as a JSON response which contains an xDB contact ID of the first match or "NoMatch" if no match was found. The JSON string is written to the HTTP response, ending the response. If the "email" query parameter does not exist, the method does nothing and loads the UI.

btnLookup_Click

This button click handler retrieves user input from four text boxes for known contact ID, email, last name, and first name. If the user input is for a known contact ID, it calls the SearchForContactById method to retrieve the contact's information and displays it in an HTML table. 

If there is no matching contact, it displays a message saying that no contact was found. 

If the user input is for email, last name, and/or first name, it calls SearchForContactIdByEmail and/or SearchForContactIdByName methods to retrieve contact information and displays it in an HTML table. 

If there are no matching contacts, it displays a message saying that no contact was found.

There are three methods implemented for using the xConnect API:

SearchForContactById

This method searches for a Contact object in xDB using a known contact ID. The method SearchForContactById takes a string parameter knownContactId and returns a Contact object. The method first converts the knownContactId string to a Guid, creates an instance of Sitecore.XConnect.Client.XConnectClient, and uses the Get method of the client object to retrieve a Contact object using a Sitecore.XConnect.ContactReference object created from the Guid

It also uses a Sitecore.XConnect.ContactExpandOptions object to specify which facets of the Contact object to retrieve. 

If the Get method throws an XdbExecutionException, the method catches the exception and returns null.

SearchForContactIdByName

This method searches for a list of Contact objects by their LastName and FirstName facets of PersonalInformation

SearchForContactIdByName takes two string parameters, lastName and firstName, and returns a list of Contact objects. 

Inside the Sitecore.XConnect.Client.XConnectClient using block, the method creates an IAsyncQueryable object named queryableLastName that represents a xConnect API query to xDB. The query retrieves all contacts where the LastName and FirstName facets of PersonalInformation match the values of the lastName and firstName parameters. 

The query then sorts the results in descending order of LastModified. The WithExpandOptions method is used to specify which facets of the Contact object to retrieve. The method then retrieves the query results using a batch enumerator, which is added to a List<Contact> object named idsList. 

If the query throws an XdbExecutionException, the method catches the exception and returns null.

SearchForContactIdByEmail

This method searches for a list of Contact objects by their EmailAddress and optionally their LastName and FirstName facets of PersonalInformation

SearchForContactIdByEmail takes three string parameters, email, lastName, and firstName, and returns a list of Contact objects. 

Inside the using block, the method creates an IAsyncQueryable object named queryable that represents a xConnect API query to the xDB. The query retrieves all contacts where the PreferredEmail facet of EmailAddressList matches the email parameter value. 

The query then sorts the results in descending order of LastModified. The WithExpandOptions method is used to specify which facets of the Contact object to retrieve. The method then retrieves the query results using a batch enumerator, which is added to a List<Contact> object named idsList

If the firstName and lastName parameters are not empty, the method checks if the FirstName facet of PersonalInformation and LastName facet of PersonalInformation match the values of the firstName and lastName parameters respectively for each Contact object retrieved. 

If both match, the Contact object is added to idsList. If the firstName and lastName parameters are empty, all Contact objects retrieved are added to the idsList

If the query throws an XdbExecutionException, the method catches the exception and returns null.

Final Result

The full code can be copied here. Add the content to a .aspx file and place it where you need it.


As always, feel free to modify or build on top of this to satisfy your own requirements.

🚀


Tuesday, December 13, 2022

Sitecore Analytics Database Manager (ADM) Contact Outliers - No Contact IDs Displaying

I recently installed the Analytics Database Manager (ADM) v10.0.2 module on a Sitecore 10.0.0 site. After completing the setup, I noticed that the xDB Contact Outliers report failed to render the Contact ID column in the table.


When I checked the browser's console, I saw that the data was indeed present in the dataset. The challenge was that the front-end code was closed-source and compiled in Angular, making it difficult to modify.

Since the Analytics Database Manager is distributed "as is" and not supported in the scope of the Sitecore Support Program, I had to try to resolve this myself. 

I searched for the `.NumberOfInteractions` property in the compiled ADM code since it was a unique enough property related to the data set that should point me in the right direction.

Luckily, I found a hit at `\sitecore\shell\client\Applications\ADM\main.577d3292f13f80d19c29.bundle.js`.

Next to it, I found what appeared to be the definition for the Contact ID column. However, the property `.id` did not match the data object's `Id` as seen in the console (note the capitalization). 



The table populated as expected by capitalizing the `i' in `Id`.



I hope this helps anyone who may encounter the same issue.



UPDATE
Sitecore released new versions of the module that address this issue on February 01, 2023:
https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0232559