Steps to update Sitecore SSL certificates (Sitecore XP 9.3)

 It's a general requirement to update the security certificates for the website.

It's more enjoyable when we work on the Sitecore, consider multiple endpoints like CMS, xConnect, Identity servers and XC Roles. I again got a requirement to update the certificates and wanted to share this with the community so everyone can quickly do this without any issue or hurdles.

So let's get started the journey to update the certificate :) 

Steps  1 - Open the mmc.exe

 Àdd certification--> computer account

Install your certificate and you will see your certificate here-

Copy the thumbprint of the installed certificate 

Copy the thumbprint of the installed certificate and keep in a file.

Now, got to certificate and add all application pools identifier.

Make sure you enter the name in   IIS AppPool\App Pool identifier name format otherwise you wouldn't get the identifier.

Now, Search for the existing thumbprint on the site, You will find in below files.

1. Identityserver\Config\production\Sitecore.IdentityServer.Host.xml
2. CMsite\App_Config\ConnectionStrings.config
3. xconnect\App_Config\AppSettings.config
4. xconnect\App_Data\jobs\continuous\ProcessingEngine\App_Config\ConnectionStrings.config
5. xconnect\App_Data\jobs\continuous\AutomationEngine\App_Config\ConnectionStrings.config

After replacing the file, got to IIS site and choose the correct certificate and restart the IIS.

that's it, It's very straightforward.

Troubleshooting - 

1. I got this error after updating the SSL certificate.

Fix - I missed to include the identifier role in the certificate, After including that role this issue got resolved,

Creating a new self-signed certificate for xDB (on-prem)

 So the SSL certificate for your xDB instance has expired, and you need to create a new one. Here’s some setup and troubleshooting steps (this is specific to On-Prem). The steps aren’t difficult but it took me a lot of searching and about five different articles to figure out everything I needed to do, so I’ve compiled the steps here.

NOTE: This pertains to lower environments. According to Sitecore documentation, self-signed certificates must NOT be used in production for security.

Use Powershell to create a new certificate with makecert.exe

There are a number of ways to generate a self-signed certificate, but it is important to make sure that 1) it has the correct common name (CN) so that it will be valid for your xconnect domain; and 2) that it has a private key so that you can install it on your CD instance.

Here is the Sitecore documentation for generating a certificate. However, the certificate I generated with this method did not have a Subject Alternative Name, so it came up as invalid in Chrome after I bound it to my xconnect instance. Instead, I used this command in Powershell:

1	New-SelfSignedCertificate -DnsName {DOMAIN} -CertStoreLocation cert:\LocalMachine\My

Get the thumbprint of the certificate you just created (Get-ChildItem -Path cert:\LocalMachine\My), and use that thumbprint to export a pfx file with the following command (replace $thumbprint with your thumbprint):

1 2 3 4 5

$certificateFilePath = "D:\Temp\$thumbprint.pfx" Export-PfxCertificate `     -cert cert:\LocalMachine\MY\$thumbprint `     -FilePath "$certificateFilePath" `     -Password (Read-Host -Prompt "Enter password that would protect the certificate" -AsSecureString)

If you have a separate CD server

1. Copy the exported pfx file to your CD server
2. Double click to run the installation wizard
3. Place the certificate in the Personal store
4. Check that the certificate is correctly installed on CD by opening Powershell and running Get-ChildItem -Path cert:\localmachine\my\$thumbprint

Add certificate to the Trusted Root Authority

If you see an invalid certificate warning, saying “CA Root certificate is not trusted”. Here is helpful documentation for that, but I’ll summarize the steps below:

1. Browse to the location you exported your certificate to (e.g. D:\Temp) and double click the pfx file.
2. Install -> Local Machine -> Place all certificate in the following store -> Trusted Root Certification Authorities
3. Repeat and place certificate in the Personal store

Update thumbprints in your Sitecore instances

You need to update all of the certificate thumbprints in your xconnect app_settings and in each site that uses xconnect. You can see the full details for updating thumbprints here; I’ve summarized below:

1. Get the thumbprint of your new certificate (on the machine that’s running xconnect). You can do this in Powershell using the command Get-ChildItem -Path Cert:\LocalMachine\My\ -DnsName "your.client.cert.dns*" . Copy the thumbprint.
2. In your xconnect instance, update the thumbprint in \App_Config\AppSettings.config
3. In all Sitecore XP roles, update the thumbprint in the connectionstrings
4. If applicable, update the thumbprint in \App_Data\jobs\continuous\AutomationEngine\App_Config\ConnectionStrings.config and \App_Data\jobs\continuous\ProcessingEngine\App_Config\ConnectionStrings.config
5. Ensure that the AllowInvalidClientCertificates app setting is true in xconnect and on all Sitecore XP roles (this is necessary for self-signed certificates)

Install the certificate on XP servers

If your Sitecore instances are hosted on a different server than xconnect (e.g. CD), you need to install the certificate on each of those servers as well.

1. Browse to xconnect.
2. Click on the Not Secure warning next to the domain and click on Certificate
3. Go to Details -> Copy to Files -> Base 64 Encode
4. Find the certificate in the location you copied it to and double click to begin the import wizard.

Note: if your connectionstrings are looking at “StoreName=My;StoreLocation=LocalMachine”, this corresponds to the “Personal” store

Additional Troubleshooting

Copy the following into a new file and save it as CertTestOnPremise.aspx; put this file in the root of your website:

<%@ Page Language="c#" Inherits="System.Web.UI.Page" CodePage="65001" %>

<%@ OutputCache Location="None" VaryByParam="none" %>
<%@ Import Namespace="Sitecore" %>
<%@ Import Namespace="Sitecore.Xdb.Common.Web" %>
<%@ Import Namespace="System.Security.Cryptography.X509Certificates" %>
<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">

<head runat="server">
  <title>Welcome to Sitecore</title>
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <meta name="CODE_LANGUAGE" content="C#" />
  <meta name="vs_defaultClientScript" content="JavaScript" />
  <meta name="vs_targetSchema" content="http://schemas.microsoft.com/intellisense/ie5" />
  <link href="/default.css" rel="stylesheet" />
</head>
<script runat="server">
  protected void Button1_Click(object sender, EventArgs e)
  {
    var thumbprint = TextBox1.Text;
    X509Certificate x509Certificate = FindClientCertificate(StoreName.My, StoreLocation.LocalMachine, X509FindType.FindByThumbprint, thumbprint, false);
    if (x509Certificate != null)
    {
      Label1.Text = x509Certificate.Subject;
    } else
    {
      Label1.Text = "Cert not found";
    }
  }

  protected void Button2_Click(object sender, System.EventArgs e)
  {
    var thumbprint = TextBox1.Text;
    X509Certificate x509Certificate = FindClientCertificate(StoreName.My, StoreLocation.LocalMachine, X509FindType.FindByThumbprint, thumbprint, true);
    if (x509Certificate != null)
    {
      Label2.Text = x509Certificate.Subject;
    } else
    {
      Label2.Text = "Cert not found";
    }
  }

  private static X509Certificate FindClientCertificate(StoreName storeName, StoreLocation storeLocation, X509FindType findType, object findValue, bool allowInvalidClientCertificates)
  {
    X509Store x509Store = new X509Store(storeName, storeLocation);
    try
    {
      x509Store.Open(OpenFlags.ReadOnly);
      X509Certificate2Collection x509Certificate2Collection = x509Store.Certificates.Find(findType, findValue, !allowInvalidClientCertificates);
      return (x509Certificate2Collection.Count > 0) ? x509Certificate2Collection[0] : null;
    }
    finally
    {
      x509Store.Close();
    }
  }


</script>
<body>
  <form id="mainform" method="post" runat="server">
    <div id="MainPanel">
      <asp:TextBox ID="TextBox1" runat="server"></asp:TextBox>
      <br />
      <asp:Button ID="Button1" runat="server" OnClick="Button1_Click" Text="Get valid cert" />
      <asp:Label ID="Label1" runat="server" Text="Valid cert"></asp:Label>
      <br />
      <asp:Button ID="Button2" runat="server" OnClick="Button2_Click" Text="Get any cert" />
      <asp:Label ID="Label2" runat="server" Text="Any cert"></asp:Label>
    </div>
  </form>
</body>
</html>

Browse to yoursite/CertTestOnPremise.aspx. Put in the thumbprint and click “Get Valid Cert”; this will tell you if the certificate is valid or not found.

If the cert is not found…

1. Verify that the user running your Sitecore instance has proper permissions (see https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0719199 and jump to “2.1”)
2. Run this command in Powershell to determine if the certificate is correctly installed:

Get-ChildItem -Path cert:\localmachine\my\{thumbprint}

If that command says the certificate does not exist, verify the following:

1. start -> run -> mmc -> File -> Add/Remove Snap-in -> Certificates -> Add -> Computer Account -> Local Computer
2. Double click into Certificates -> Personal
3. Verify the certificate is there
4. Double click the certificate -> Details
5. Verify Thumbprint is the same as you were using in the previous steps

Incompatibilities of xConnect and Client Certificates

  

Almost near to the end of a major Sitecore as well as infrastructure upgrade from Sitecore version 7.2 to 9.0.2. Thought of penning my upgrade story which becomes more spicier with lots of mysterious twists by having xConnect in the lead role. 

Just like all my previous Sitecore upgrades this was almost similar apart from adding Sitecore Official Nuget, CI/CD using Octopus and most importantly the tedious patch up between xConnect and the client certificates issued by organizational authorities. Using Sitecore official Nuget for latest assembly references and Express migration tool for Database migration, the upgrade was a bit smoother without any critical errors/hiccups. But when we were at the stage to test the complete ecosystem in XP9 platform from hitting the website and generating the related reporting graph on the Experience Analytics Dashboard, resolving the issues with xConnect and non-self-signed Client Certificates was such a bumpy ride. 

We faced a lot of issues and it was troublesome to find root cause behind the incompatibility between xConnect and Client Certificates. I also get a chance to chat with some of my Sitecore community friends over Slack and almost everyone who implemented Sitecore 9 for the very first time, sailed the same boat. Though as always I found a lot of excellent blogs and questions on SSE with similar problem and relevant answers. But for us the culprit was something else but not Certificates hence thought of blogging a consolidated post with all the issues we faced and our approach towards Nirvana!!!

So we have a scaled Sitecore 9.0.2 environment with

1. One Instance for combined Content Management, Processing and Reporting Roles
2. Scaled Instances for each the xConnect roles
   • xConnect Collection
   • xConnect Collection Search
   • xDb Reference Data
   • Marketing Automation Operation
   • Marketing Automation Reporting
3. Two Load balanced Instances for Content Delivery Roles
4. Two Solr Instances – Master and Slave
5. Two SQL Server Instances

Please have a look at the Sitecore Network Topology Diagram. The CM and few of the databases were on the corporate (internal) network whereas the xConnect, Solr, SQL and the CD Roles were on DMZ behind F5.

Following are the series of exceptions we faced one after another when we were applying the fixes during our research and debugging.

---

Series of Incompatibility Exceptions

Invalid Certificate

FATAL [Experience Analytics]: Failed to synchronize segments. Message: Ensure definition type did not complete successfully. StatusCode: 401, ReasonPhrase: 'Invalid certificate', Version: 1.1, Content: System.Net.Http.StreamContent, Headers: 

Forbidden Access

FATAL [Experience Analytics]: Failed to synchronize segments. Message: Ensure definition type did not complete successfully. StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers: 

Unauthorized Access

An unhandled exception of type 'Sitecore.XConnect.XdbCollectionUnavailableException' occurred in mscorlib.dll The HTTP response was not successful: Unauthorized 

xDB Unavailable with Time Out Exception

Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web   
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)   
at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)   
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)   
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)   
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)   
at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)   
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)   
at Sitecore.PathAnalyzer.Processing.Agents.TreeAggregatorAgent.Execute()   at Sitecore.Analytics.Core.BackgroundService.Run() 

Exception: Sitecore.Xdb.Common.Web.ConnectionTimeoutException
Message: A task was canceled.Source: Sitecore.Xdb.Common.Web   
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---   
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   
at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---   
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   
at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext() 

Could not create SSL/TLS secure channel

System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel

So we tried multiple solutions to establish the connection between xConnect and other roles, especially with CM while we were trying to generate the graphs on the Experience Analytics Dashboard. Though I will try my best to elaborate the approach of our debugging and research, but if you have any questions please free to drop a comment on this blog or reach out/DM me on slack or social channels.

---

Probable Root Causes of Certificate Incompatibility:

At the very beginning we had the Invalid Certificate exception on both CM as well as CD role, hence the traffic on CD was not even being recorded in the Shard databases. We did some research and found following suggestions on few blogs and SSE about the known root causes behind certificate incompatibilities with xConnect:

Note: This question on Sitecore Stack Exchange was the source of all the possible root cause analysis mentioned below.

Certificate not installed or Thumbprint missing/incorrect

There are possibilities that either the certificates are not installed on server/client or the thumbprint is missing/incorrect in the required configuration files.

Result: We verified multiple times and everything was perfect with this aspect.

Untrusted certificates in ‘Trusted Root Certification Authorities’

This PowerShell command will identify non-self-signed certificates:

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}

Move these non-self-signed certificates into the Intermediate Certification Authorities (i.e. CA) store

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Move-Item -Destination Cert:\LocalMachine\CA

Result: We had NO untrusted certificates in the trusted root authorities, hence this was not our case as well.

SQL Script Execution as Post Installation Step

Once the vanilla installation is done as Post Installation Steps we need to execute a SQL script which grants required permissions to collectionuser on the Shard Databases.

Result: We are on version 9.0.2 and the post installation script was for initial release only, I guess they fixed it for the later releases and the permissions are now granted during the installation itself. Though we verified on the database level, the collectionuser had all the permissions mentioned in the SQL script.

Invalid SSL Certificate on IIS Level

Verify if a valid Server certificate is not assigned in IIS to the respective instances.

Result: Valid SSL certificate is installed and assigned on all the IIS Instances.

SSL Setting in IIS accepts the Client Certificates

Verify the SSL Setting in IIS is configured to Accept the Client certificates for all the xConnect instances. 

Result: It was already selected as ACCEPT.

Application doesn’t have access to the certificate

Make sure the Network Service, IIS User and App Pool have full access to the respective client certificate. 

Result: We provided all the required access to the certificates.

No luck so far, we cross verified all the reasons mentioned above and problem still persists. Hence we decided to dig deeper.

---

Further Troubleshooting:

Since we verified almost everything related to certificates hence we decided to troubleshoot other areas of the topology.

Enabled to Allow Invalid Client Certificates:

We decided to give it a shot by allowing invalid certificates.

1. Set AllowInvalidClientCertificates to true in web.config on CM and CD Roles.
2. Set AllowInvalidClientCertificates to true in appsetting.config on xConnect Roles.
3. Comment out the validateCertificateThumbprint in appsetting.config on xConnect Roles.
4. Reset the app pools and give it a shot.

Result: Surprisingly the errors were gone by allowing the invalid certificates and we had cleaner log files. Then we generated some traffic and guess what, the data populated in the Shard DBs. For testing we reduced the Session Time Out on CDs to 2 minutes. After couple on minutes data populated in Reporting Database and we see the reports on the Analytics Dashboard. Looks like the entire cycle is up and running now. BUT WHY, WHAT ARE WE MISSING WITH CERTIFICATES?

Now we are confirmed that there is something definitely wrong either with the certificates or any related configuration which is not allowing the communication to take place via SSL.

Pro tip: In such disastrous situation make sure the Server Technologist or Info Sec person is your friend and I find myself very lucky here. 

So we reverted everything back to the previous state to disallow invalid certificates. And the 401: Invalid Certificates exceptions are back. Worked closely with the security team and here are the steps we followed for further debugging:

Allowed Direct Traffic bypassing the F5:

If you have a look at the topology diagram above the xConnect and CD instances are behind F5 whereas the CM is not cause it is on an internal network. Hence to avoid the possibilities of something misconfigured at F5 level, we removed the SSL profiles from the VIP. But this was not sufficient to resolve the issue therefore we temporarily allowed a bypass of the F5 altogether by putting in a temporary firewall rule to allow CM and xConnect to communicate directly. 

When we configured this the 401: Invalid Certificate exceptions were gone. And we start getting the Exception #3 above regarding Unauthorized Access.

Obviously we can’t bypass the F5 as a permanent fix hence re-visited the configurations on F5. Later we figured out that the HTTP Profile on the VIP was selected as HTTP. We changed the HTTP Profile on the VIP from “HTTP” to “None”and removed the temp firewall rule to make sure everything is kosher from a firewall perspective.

Important: HTTP profiles are incompatible with encrypted pass-through traffic, such as Secure Sockets Layer (SSL), and require a Client SSL profile to decrypt the traffic for L7 HTTP inspection. If the virtual server processing the encrypted traffic is configured with an HTTP profile and no Client SSL profile, the connection will fail.

Certification Revocation List was the next Culprit:

In our case since the xConnect boxes are in DMZ behind F5 and the URL for the Distribution Point Name for the CRL check was internal. But the traffic from external network to internal network was blocked. Due to this the CRL Check was not taking place and we were getting the Could not create SSL/TLS secure channel exception. If you face the similar issues, visit the CRL Distribution Points on your certificates.

CRL Distribution Point
     Distribution Point Name:
          Full Name:

As a temporary solution we decided to disable the CRL check for the certificates. To achieve this we added a registry entry DefaultSslCertCheckMode at HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo for every Role which need client certificate authentication i.e. all the xConnect Roles. Please have a look at this wonderful blog about how to Disable Client Certificate Revocation List Check on IIS.

Note: Though as a permanent fix the security team is revisiting the current configuration they have for the CRL checks. Once that is fixed we will be enabling the CRL Check again on IIS.

VOILA!!! Everything was up and running using the same set of certificates. No exceptions in logs and latest data on the Analytics Dashboard. The Ultimate Nirvana!!!

---

Conclusion: 

Every problem is an opportunity to learn something new. For us reason behind the incompatibility between xConnect and Certificates was NOT Certificates but the F5 and CRL Check. Hence when you are having fun with xConnect for the very first time make sure to inspect every single aspect of your entire topology. Good Luck!!!