This article aims to serve two purposes.  Firstly, as a reminder to always keep Lync Phone Edition (LPE) devices updated and not to grow lackadaisical regarding the routine administrative tasks of frequently downloading these periodic cumulative updates and applying them to the Lync servers.

Secondly, if the previous advice was not taken to heart then some Lync administrators may already be experiencing a specific problem directly related to running older device firmware versions in a current Lync environment.  There is also the potential for newly purchased devices coming from existing stockpiles to have older firmware preinstalled from the factory which can exhibit this same issue, by no fault of the Lync administrator.

The problem referred to is that Lync Phone Edition devices may fail to successfully sign-in to a Lync Server due to an untrusted certificate being presented by the server itself during the initial secure TLS network connection setup.  This scenario can occur when an LPE device is running on an older version of firmware which still contains an outdated public Certificate Authority (CA) store.  In the event that the Lync Server is using a server certificate from a public CA which was signed and issued by a newer CA than the phone is aware of it will fail to connect because it will treat the certificate presented by the Lync server as untrusted.

Be aware that this issue should be limited to the USB-tethered provisioning approach which uses a different method  to download root CA certificates then when using the PIN Authentication method.  Depending on the exact issue it can be possible to leverage the DHCP Option 43 configuration to allow a PIN Authentication attempt to download the proper public CA certificate directly from the Lync Server Certificate Provisioning Web Service.

But to support USB-tethered provisioning a workaround may be required to get the updated public CA certificate installed into an internally connected phone using an LDAP connection to Active Directory so it can then trust the certificate passed by the Lync server to establish a successful TLS connection.  There are two ways to accomplish this, the easy way and the hard way.

• The preferred method is simply the guidance stated in the opening paragraph and is a preventative step.  If the issue in this article is not a current problem in an environment but the ingredients sound familiar (public certificates on internal servers and LPE not running recent firmware versions) then heed this warning and go download and update those devices today before it is too late.  As usual test the newer version out on a few test phone to validate that no unrelated problems appear before rolling it out to all phones in the environment.

• On the other hand if this article came up as a result of an administrator frantically searching for a resolution to this problem then unfortunately the preventative route cannot be taken at this point as that ship has sailed.  Luckily there are a few options available to get past this ‘catch-22’ scenario of needing to update the firmware to get TLS working again but requiring TLS in order to facilitate an update.

Background

Because Lync utilizes TLS for all signaling traffic this secure connection must be established before any additional steps like the actual user registration or a device update can take place.

A typical Lync environment will most often utilize an internal Windows Enterprise CA for the Lync server certificates, while a public certificate from a well-known third party certificate authority would be used for only the external side of any Edge Servers.  While this still is the most popular topology in use today it is slowly becoming common to see public certificates used on the internal Lync server roles instead of private certificates for a myriad of reasons.

Environments using public certificates on the internal Lync server roles present a unique problem to LPE devices that many other clients are not subject to.  There is no mechanism in the Windows CE-based LPE firmware to automatically refresh any preinstalled public root and intermediate CA certificates like other Windows-based operation systems.  In the event that a specific public CA makes a change to one of their root or issuing CAs then the new certificate for that CA would need to be propagated down to any clients or servers which currently trust the previous certificate.  Standard Windows operating systems utilize Windows Updates to refresh these types of changes, but LPE does not.  The only way these phones can be made aware of these external changes is by being upgraded to a newer firmware release in which Microsoft has specifically updated the certificate store.  Note that many other 3PIP Qualified phones for Lync can have the same manufacturer-updated behavior so make sure to keep those current as well for the same reasons.

Trusted Certificate Authorities

For optimized Lync Phone Edition devices from all manufacturers Microsoft manages the firmware and updates the root CA store periodically as public CA server certificates expire, are revoked, or are simply replaced.  The TechNet article Certificates for Lync Phone Edition contains a list of the various CA certificates stored in the firmware’s Trusted Authorities Cache but this table is not kept up-to-date.  As of the posting of this article that page still reflects the October 2013 Cumulative Release 10 (4.0.7577.4414) which is over a year old.

The following table lists all of the trusted root CA certificates utilized by LPE, both old and new:

From a brief glance at the above table a few things should be apparent:

• In versions older than CU10 nearly half of the cached certificate are expired or revoked.

• If the phone is not running at at least the January 2014 Cumulative Update 11 (4.0.7577.4420) then it is not aware of a host of new CA certificates.

Root Cause

Where the problem introduced earlier comes into play is when a Lync environment is using a public certificate issued from one of the vendors in the table above that have made changes to the originally trusted certificates.  Older server certificates assigned to Lync server might have been signed by the original CA, but as those certificates approach their expiration dates any good administrator will preventively update them prior to that date.  So when these new certificate requests are generated on the Lync server and sent out to the same public CA for signing they are coming back issued from a different CA chain, and one that the older phone firmware is not aware of.  When the servers are updated with these new certificates then phones running that older firmware will no longer be able to register to Lync.

If there are plans to update an public Lync server certificates in the near future and there are any Lync IP Phones in the environment then now would be a good time to get those phones on a newer release.

It is required to upgrade to at CU11 (4.0.7577.4420) at a minimum to avoid this problem, but it is always recommended to move to the most recent firmware currently available.

Workaround

If the Lync server certificate has already been changed, or new phones running older firmware have been introduced into the environment, and they cannot successfully register then the following workaround can be used to address this problem.

Note that the hardcoded pre-authentication download attempt against the ‘ucupdates-r2’ hostname which was carried over from the original Office Communications Server Tanjay devices is not applicable here.  The issue is not that the phone is failing to register to the Lync server, but that it cannot even get to that step due to the underlying TLS requirement not being available.  This out-of-the-box update process requires TLS for a portion of the communication, even though the internal URL provided by the device update service uses HTTP.  So without a working TLS connection the device update service is not reachable.

While the following approach is not new to Lync Phone Edition this guidance may not be something that newer Lync administrators are often aware of.  Prior to the introduction of the current Aries family of LPE devices designed for Lync 2010 the older Tanjay devices introduced with OCS 2007 did not support the simplified PIN Authentication based on DHCP Option 43 as that feature was introduced in Lync Server 2010.  In OCS if public certificates where assigned to internal server roles then Active Directory could be used as a central distribution point for the required public CA certificates.  Because the current LPE firmware is based on that older platform then this behavior is still active in all LPE versions.

The process of publishing third party certificates into Active Directory was originally covered by Rui Jose Silva and references to this process can be found in various other articles, some of which appear to have vanished over the years.

Throughout the directions in this article the example scenario is a Lync 2013 Server utilizing the following newer DigiCert SLL certificate for all three usages on the Front End Server:

To clarify this is not an issue related specifically to DigiCert certificates, but any public CA unknown to the phone’s older firmware.  DigiCert has a great program for active Microsoft MVPs which allows them to utilize public certificates in personal test labs for researching and writing articles like this.

Retrieve Root Certificates

Before any certificates can be imported into Active Directory they must first be saved into a Base-64 encoded X.509 file.

This can be accomplished by either locating and downloading the file(s) from the Internet or manually exporting them from the Lync server.  The fastest method can be to just get them directly from the vendor’s website, but most public CAs have multiple root and subordinate/issuing servers so finding the correct certificate chain can sometimes be a daunting task.

• If already familiar with this process and the exact certificates have been identified then locate and download them from the vendor.  Make sure to get all files in the certificate chain in the event that the Lync server certificate was not issued directly by a root server but was instead signed by a subordinate server.  (This is most common as third party CAs rarely have the all-important root CA online and available to sign requests directly.)

Alternatively to ensure that the correct certificates are retrieved it is recommended to validate the current server certificate and then manually export the chain.

• On the Lync Server use the Lync Server Management Shell to execute the follow cmdlet which will list the currently assigned server certificate(s).

PS C:\> Get-CsCertificate | fl Issuer,Use,Thumbprint

Focusing on the Default, WebServiceInternal, and WebServicesExternal usages (and ignoring the OAuthTokenIssuer certificate) a  typical configuration will have the same certificate assigned to all three client-facing usages.  This Lync Server is utilizing a single certificate issued by DigiCert as made evident by the same Thumbprint value being reported for each.

• On the same Lync Server open a Microsoft Management Console window (mmc.exe) and then add the Certificates snap-in for the local Computer Account.

• Browse to the Certificates (Local Computer) > Personal > Certificates store and then locate the certificate which is most likely the one referenced in the previous output.

• To confirm that the correct certificate was selected switch to the Details tab to compare the Thumbprint value with what was reported by the Get-CsCertificate cmdlet (e.g. 3DFB6A83366BAD70DC4427565A64A07A0733EA49).

• Change to the Certificate Path tab to view the certificate chain for this certificate.  Highlight the root server (e.g. DigiCert) at the top of the tree and click the View Certificate button.

• In the next Certificate window switch to the Details tab and then click the Copy To File button to export the certificate to a text file.

• In the Certificate Export Wizard select the Export File Format as Base-64 encoded X.509 (.CER).

• Save the file (e.g. C:\Temp\DigiCertRootCA.cer) to the local disk and complete the wizard.

At this point the root CA certificate, which is essentially the public key of the certificate, is now in a format suitable for importing into Active Directory.  Because this example includes a 2-tier CA chain the same steps must be repeated for the subordinate certificate.

• Return to the Certification Path tab of the original server certificate, highlight the subordinate CA entry (e.g. DigiCert SHA2 Secure Server CA), and click View Certificate.

• On the Details tab use the Copy To File button to perform the same export procedure as before, saving the file with a unique name (e.g. C:\Temp\DigiCertSubCA.cer).

Import Root Certificates into Active Directory

The next step is to import these files into the proper Active Directory store location so that the Lync Phone Edition devices will attempt to download them the next time they are powered-on.

Prior to importing these though files first that a look at where they will be stored in Active Directory.

• On the Lync server launch ADSI Edit (adsiedit.msc) and connect to the Configuration Naming Context and browse to Services > Public Key Services.

CN=Public Key Services,CN=Services,CN=Configuration, DC=schertz,DC=name

CN=AIA

CN=CertificationAuthorities

Notice that the only object currently stored in these container is for the existing Windows Enterprise Certificate Authority which is AD-integrated by default when deployed in the domain.  If there is no internal Enterprise CA in the environment then these stores will typically be empty.

To import the two DigiCert CA certificates into this store the certutil.exe Windows command can be used, which will place each certificate file in both the CN=CertificationAuthorities and CN=AIA stores located under the Public Key Services container.

• From the same domain-joined Lync server open a Windows Command Prompt window and issue the following command.

certutil -f -dspublish c:\Temp\DigiCertRootCA.cer RootCA

If the action is successful then the CA certificate should be reported as added to both DS store locations in the AD Configuration container.

• Repeat the step for any additional certificates in the chain.

certutil -f -dspublish c:\temp\DigiCertSubCA.cer RootCA

• Refresh the ADSI Edit window to verify that the new certificates are displayed in the stores as reported.

The final step is to attempt to sign-in on the phone using the USB-tethering approach so that the AD credentials can be supplied to the phone.  If any older CX700 models are in the environment they are not required to be tethered as the AD credentials can be entered directly on the phone.

If the phone is still unable to register at this point then try one or both of the following tips:

• Perform a soft reset on the phone to clear any cached credentials and certificates and reattempt signing in.

• When signing-in enter the User Name in the odd-looking format of the DNS Domain instead of the NetBIOS name in this format: domain.com\username (e.g. schertz.name\jeff).