As OCS 2007 R2 has officially supported running on a Windows Server 2008 R2 system for nearly a year now there are a growing number of deployments on this version. Additionally most Lync Server 2010 installations will be on Server 2008 R2 as well.
There is an important behavioral change introduced in Server 2008 R2 (and Windows 7) which can impact how clients and servers negotiate NTLM authentication: the minimum encryption level has been raised to 128-bit by default. For most Microsoft client and server operating systems and software this change is not even noticed as 128-bit is a supported level on both sides. (Additional details on this change can be found here in TechNet.)
But when attempting to authenticate third-party solutions this can sometimes cause problems as not all systems will currently negotiate above 40-bit or 56-bit levels.
Update: the Polycom HDX natively supports 128-bit encryption since version 3.0 was released and the configuration changes in this article are not required.
For example, when attempting to natively register some video conferencing systems to an OCS or Lync registrar which is running on a default configuration of Server 2008 R2 the following error will be displayed on the System Status window after configuration:
Upon further investigation the diagnostic logs will report the following error after the SIP registration attempts fail.
ms-diagnostics: 1000;reason=”Final handshake failed”;HRESULT=”0xC3E93EE4(SIP_E_AUTH_NTLMMISMATCH)”;source=”LYNC.schertz.local”
The 0xC3E93EE4 error code indicates an NTLM encryption level mismatch meaning that an unsupported setting is typically being enforced by one side of the conversation.
This can be resolved by lowering the minimum level required on the OCS/Lync registrar server.
- Open the Local Security Policy administrative tool (secpol.msc), expand Local Policies and select Security Options.
- Locate the policy Network Security: Minimum session security for NTLM SSP based (including secure RPC) servers and view the properties.
Notice the Security Setting is currently at Require 128-bit encryption. Also notice that there are two settings, one for servers and one for clients; the client value does not need to be modified as the server setting is what impacts the HDX client registration process.
- Clear the Require 128-bit encryption setting.
- Verify that the Security Setting is now displayed as No minimum.
- Restart the OCS or Lync Front-End service to immediately pick-up the policy change and then reattempt SIP registration of the endpoint. The following status screen from an HDX reports a successful registration.