Managing Items as Resources in Sitecore xmCloud

As you may know for a while Sitecore uses Items as resources for deployment (IAR). Sitecore uses Protobuf to generate static read-only serialized items.

Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can learn more about it in protobuf's documentation. You may also want to take a look at protobuf-net which is the .net implementation of the Protobuf, and I must say it is quite fast.

Protobuf (IARs) are deployed inside "App_Data/items" and "sitecore modules/items" folders. The Sitecore uses CompositeDataProvider to load IARs items from the disk:

<database id="core" type="Sitecore.Data.DefaultDatabase, Sitecore.Kernel" singleInstance="true">
    <param desc="name">$(id)</param>
    <connectionStringName>$(id)</connectionStringName>
    <icon>Images/database_core.png</icon>
    <dataProviders hint="list:AddDataProvider">
        <dataProvider type="Sitecore.Data.DataProviders.CompositeDataProvider, Sitecore.Kernel">
            <param hint="list" desc="readOnlyDataProviders">
                <protobufItems
                    type="Sitecore.Data.DataProviders.ReadOnly.Protobuf.ProtobufDataProvider, Sitecore.Kernel">
                    <filePaths hint="list">
                        <filePath>/App_Data/items/$(id)</filePath>
                        <modulesFilePath>/sitecore modules/items/$(id)</modulesFilePath>
                    </filePaths>
                </protobufItems>
            </param>
        </dataProvider>
    </dataProviders>
</database>

IARs have their own benefits, especially for the Upgrade process or deploying the Sitecore hotfixes.

The Item load priorities are the following:

• Main databases from "App_Data/items" folder
• Modules from "sitecore modules/items"
• Your serialized deployed items from "App_Data/items" folder
• Items saved directly in the Database

And now comes the challenges.

Challenges of IAR deployment

You may face a few Challenges with the IARs, few examples are:

• You don't know what is inside the IAR
• You may face a racing issue, if you have overwritten another IAR item
• The changes in the database may affect your deployment.

For the first challenge, I have provided a tool that allows you to browse the IAR items and see what is inside, in the Next blog post I will introduce it to you.

We had the second challenge in the xmCloud with our serialized items and Sitecore fixed it. Keep in mind in some cases it should be treated as a red flag for you, maybe you need to rethink your work. Modules usually should not be overridden.

The third challenge is the main topic of this post, so let us talk about it.

Resetting IAR items

We have always the issue that someone changes some of the IAR items on for example, the Staging or Production environment, what happens is, that item, for example, rendering/template or partial design will be saved in the database.
The next deployment will update the IAR items on the disk but because the Items from the database have higher load priority, basically the changes from deployed IAR items will be ignored. So we need a way to reset these items or remove them from the database.

There are a few ways to do this:

• If possible delete the item directly using content-editor, this won't work if the item has dependencies, for example, a Template that has pages based on it.
• In the early days of the xmCloud we did it using a SQL script to remove the IAR items from Database, and I do not recommend it.
• Sitecore added the reset functionality to the CLI
• Using Powershell

Deleting in Content Editor

In some cases, for resetting IAR items you can simply use the content editor:

and it is important to leave the links as they are, because you are not really deleting the item, you are just removing it from the database.
After the deletion, you will see the item is restored from resources:

** Note: Make sure the item is an IAR item (Use Powershell reports from this blog or check your serializations). Be specially careful if you have a subitems which are not serialized.

Reset using Sitecore CLI

Sitecore cli itemres API has a cleanup command which can be used with --what-if to show you an overview of what is going to happen:

dotnet sitecore itemres cleanup --what-if --force -n staging

Running it will clean up the changes in the database, you may need to be careful if you have customer contents in your serialization (ex. Home item).

** Downside: you can see or clean only the overridden items included in the serializations. If someone has changes O.O.B items, you may not notice them.

Reporting and Reseting using Powershell

The Sitecore API's have some classes which help you to figure out the Item locations (IAR or SQL).

Using those API, I have created some Powershell functions which help you along the way to manage the IAR on your instances.

The full report is slow and very resource-consuming, i may refactor it later to use solr search. It will check these paths:

• /Templates
• /System
• /Layout
• /Content

And based on the result it will generate the Report:

And Subtree Report will check the overridden items on the current subtree.

Clean-up scripts and challenges

As mentioned before clean-up could get tricky. In the case of the templates, if they have pages created based on them, the Sitecore API doesn't allow you to just delete them.

The only way I could find, was to delete them directly from the database:

# Remove given Item From Database
function Invoke-CleanupIARItem {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory = $true, Position = 0)]
        [Item]$Item
    )
    begin {
        Write-Verbose "Cmdlet Invoke-CleanupIARItem - Begin"
        Import-Function Test-InDelegatedArea
        Import-Function Test-IARItem
    }
    process {
        Write-Verbose "Cmdlet Invoke-CleanupIARItem - Process"
        if (Test-IARItem -Item $Item) {
            $sourceCode = @"
    namespace SitecoreDatabaseCleanup
    {
        using System;
        using System.Data.SqlClient;
        public class CleanDatabase
        {
            public static string DeleteItemFromDB(Sitecore.Data.Items.Item item)
            {
                string connectionString = Sitecore.Configuration.Settings.GetConnectionString("master");
                string itemId = item.ID.Guid.ToString();
                var result = DeleteFields(itemId, connectionString);
    
                if (string.IsNullOrWhiteSpace(result))
                {
                    result = DeleteItem(itemId, connectionString);
                }
    
                return string.IsNullOrWhiteSpace(result) ? "Success" : result;
            }
            private static string DeleteFields(string itemId, string connectionString)
            {
                var result = Delete(connectionString, GetFieldDeleteCommand("VersionedFields", itemId));
                if (!string.IsNullOrWhiteSpace(result))
                {
                    return result;
                }
                result = Delete(connectionString, GetFieldDeleteCommand("UnversionedFields", itemId));
                if (!string.IsNullOrWhiteSpace(result))
                {
                    return result;
                }
                result = Delete(connectionString, GetFieldDeleteCommand("SharedFields", itemId));
                if (!string.IsNullOrWhiteSpace(result))
                {
                    return result;
                }
    
                return string.Empty;
            }
    
            private static string DeleteItem(string itemId, string connectionString)
            {
                return Delete(connectionString, string.Concat("DELETE FROM [dbo].[Items] where [Id]='", itemId, "'"));
            }
    
            private static string GetFieldDeleteCommand(string tableName, string itemId)
            {
                return string.Concat("DELETE FROM [dbo].[", tableName, "] where [ItemId]='", itemId, "'");
            }
            private static string Delete(string connectionString, string deleteCommand)
            {
    
                try
                {
                    using (var cnn = new SqlConnection(connectionString))
                    using (SqlCommand cmd = new SqlCommand(deleteCommand, cnn))
                    {
                        cnn.Open();
                        cmd.ExecuteNonQuery();
                        cnn.Close();
                    }
                }
                catch (Exception e)
                {
                    return string.Concat(e.Message, "\r\n", e.StackTrace);
                }
    
                return string.Empty;
            }
        }
    }
"@
    
            Add-Type -TypeDefinition $SourceCode -Language CSharp -ReferencedAssemblies System.Data, Sitecore.Kernel  -PassThru | Out-Null
            $result = [SitecoreDatabaseCleanup.CleanDatabase]::DeleteItemFromDB($item)
            if ($result -ne "Success") {
                throw $result
            }
        }
    }
    
    end {
        Write-Verbose "Cmdlet Invoke-CleanupIARItem - End"
    }
}

The script will check if the item is IAR, it will delete it using the Sql-Command.
If you use Sitecore API to delete an Item, it will also delete the children, which could cause a content loss.
Read the source on github, test it, and use it at your own risk!

Conclusion

Sitecore IAR has resolved quite a few problems and made deployment much easier, the Sitecore CLI is also actively evolving and will get more features in the future.

Pwershell features are not xmCloud specific and can be used with every Sitecore version. Even if you do not want to use clean-up scripts, the reporting will help you to figure out which O.O.B or custom IAR items are overridden.

The Source code and Sitecore package are on Github .