This article is a brief update to the process previously documented for Lync Server 2010. In Lync Server 2013 the Device Update service is unchanged and the steps shown in that article are still applicable, so why the need for a new post on this subject?
Background
It is important to understand that although Lync Server 2013 brings a host of new 2013-branded clients (e.g. Windows, Mobility), the Lync Phone Edition (LPE) client was not updated in the same manner. Microsoft continues to release semi-quarterly Cumulative Updates of firmware for the various LPE models. In the round of releases in December 2012 and January 2013 (aka CU7) official support for Lync Server 2013 was added, as well dropping the ‘2010’ from the previous name of Lync 2010 Phone Edition. This signals the intent to provide a single LPE client for both Lync platforms and not have separate 2010 or 2013 clients with different interfaces.
What this all means is that to utilize any Lync Phone Edition devices with Lync Server 2013 the phone firmware may need updated to at least the CU7 release, which is version 4.0.7577.4363 / .4366 / .4372 for different device families. As of the writing of this article the most recent firmware release is from April 2013 (CU8 or 4.0.7577.4387).
Understand that although Microsoft officially supports Lync Phone Edition with Lync Server 2013 starting with this CU7 release it is often still possible to register directly to Lync Server 2013 environments with many older firmware versions. In these cases it will still be required to update Tanjay device in a two-step process to get to a supported version.
The oldest tested version which could successfully register to a Lync 2013 server is 3.5.6907.187. Attempts to use 1.0.522.x versions would always fail with SIP 401 Unauthorized errors regardless of the client version policy configuration.
As older versions of firmware may still be in use or come pre-installed on new devices then there may exist a scenario where the devices are unable to successfully sign-in to a Lync Server 2013 environment (specifically OCS 2007 RTM versions). And because the only way to update the Lync Phone Edition firmware is by the device requesting and downloading the package from the Lync Server then this appears to be a cyclic problem. How can a device which cannot register to the server download an update from the server when it needs to first register to be provided the device update server information? This might mean that the device would instead need to be connected to and registered to Lync 2010 server first, then updated to CU7 or higher, no?
Actually, no. There is one alternative which would allow any previous version of Lync Phone Edition to download an update package directly from a Lync 2013 server without even the ability to first successfully register to it.
This little-known behavior is hardcoded into all firmware releases. The phones are programmed to look for a specific hostname which is not typically included in the Lync topology, but if it is added manually to the environment then phones will be able to update without signing in. Normally device updates are triggered after signing in to the Lync Server as described in detail in the previous article.
But by adding the suggested configuration to a Lync 2013 server as outlined in this article then the issue can be avoided and devices can be updated as needed.
Behavior
When a Lync Phone Edition device is powered up and receives a successful DHCP lease it will look for the commonly used DHCP Option 15 parameter which provides a single DNS Domain Name. It will also query for the optional DHCP Option 119 which provides a DNS Search List. In an environment that contains multiple domain names then Option 119 can be leveraged to provide more than one domain name to the client.
The device will then automatically perform a DNS query for the hostname ucupdates-r2.<domainname> for each domain name which may have been passed via those DHCP options.
Typically the certificate issued to the Lync Front End server is from an internal private Certificate Authority, which unfortunately means the device cannot automatically download the update without intervention. Lync Phone Edition will only attempt to retrieve the root certificate from the internal, Active Directory published CA during registration attempts, it was not programmed to perform this same action automatically upon bootup.
So in this case user-intervention is required by attempting to sign-in to the Lync Server. The registration does not need to actually succeed so if the phone is running a vey old version incapable of registering to the Lync server for some reason, or an invalid PIN is used simply to speed up the process it does not matter. Simply initiating the registration process is sufficient to trigger the root certificate retrieval process via DHCP Option 43.
In the event that a trusted public certificate is used this is not an issue, but this is a rare to impossible scenario as discussed in this blog article. A public Certificate Authority will no longer issue certificates which include names which are not valid FQDNs, so requests containing short hostnames and internal namespace utilizing an invalid Top Level Domain (TLD) like .local will be denied. Due to these factors it is realistically not possible to use a public certificate on the internal Lync server and support LPE upgrades prior to successful registration as covered in this article. An AD-integrated Enterprise CA must be used to issue the certificate to the Lync Server.
Requirements
The Device Out of Box section shown at the bottom of this page in TechNet lists the requirements to support this registration-less device process.
- The phone must be on the corporate network. This statement defines that the process will only work on devices connected internally which have direct access to the Lync Front End server. This process is not supported for devices connecting outside of the Edge Server and Reverse Proxy server.
- The UCUpdates-R2 host record must be created in DNS. Clearly the phone will need to correctly resolve the hardcoded DNS entry so a Host record needs to be defined in DNS. Only the FQDN needs to be defined, there is no need to deploy a WINS server solely for shortname resolution as the device only needs to resolve the FQDN. The example below (ucupdates-r2.schertz.name) shows that the Active Directory domain namespace, and not the SIP domain namespace is utilized in the event that these are not the same namespace. The phone will utilize the AD domain presented in DHCP (covered in a later step) to located the update FQDN, not the SIP domain.
- Device Update certificate must have SAN entries containing the hostname and FQDN. This means that the certificate installed on the Lync Front End server and assigned to the internal web services usage at a minimum which the device will connect to for the device updates must have both a Fully Qualified Domain Name entry (e.g. ucupdates-r2.schertz.name) as well as a short hostname (ucupdates-r2) defined in the Subject Alternative Name field. By default the Lync Certificate Wizard does not include either of these as options so typically they are not already defined. But a new certificate request can be generated with these entries added to the request manually in the wizard. The short hostname must be included in the certificate even though the device does not need to actually resolve this shortname (WINS is not a requirement for this solution). During the update process the phone will actually use the hardcoded shortname to connect to the Lync server even though the FQDN is used to resolve the server’s address. If the shortname is missing from the certificate then a TLS connection will fail to be established, preventing the device from accessing the update web service.
- Network Time Protocol (NTP) must be configured correctly for the device. Without an available time source the device will not be able to connect to and communicate with the Lync Server. If neither a Network Time Service Locator Record (SRV) nor DHCP Option 42 is available on the network then LPE will fall back to connecting to the hard-coded time.windows.com host to locate a time source.
- DHCP Options 15, 43, and 120 must be defined. The 43 and 120 options utilized by the PIN Authentication process need to be present in order to download the root certificate from the internal CA, as described in this previous article. Also make sure that Option 15 is configured to provide the AD domain name (e.g. schertz.name) so that the phone knows what domain namespace to search for the ucupdates-r2 FQDN in.
Update Process
- Connect a phone to the network which is currently installed with a firmware version older than what is approved on the Lync Server.
- Perform a soft-reset on the phone to wipe any cached client credentials or certificates.
The following text is from a network capture run on the DNS server which shows the device asking for the IP address of the ucupdates-r2<domainname> query for the only domain name provided by the DHCP Option 15 value.
DNS:QueryId = 0x3, QUERY (Standard query), Query for ucupdates-r2.schertz.name of type Host Addr on class Internet {DNS:517, UDP:516, IPv4:513}
A network capture run on the Lync server will show the following traffic indicating that the phone has requested to establish a TLS session with the Lync Server, which then passes its server certificate to the phone in the following message.
TLS:TLS Rec Layer-1 HandShake: Server Hello. Certificate. Server Hello Done. {TLS:43, SSLVersionSelector:41, TCP:40, IPv4:39}
Understand that this is the Lync server handing its assigned server certificate to the client, this is not the root CA certificate, so shown below by the certificate’s common name appearing in the trace data.
Traffic from the phone to the Lync server at this point will stop for a period of time as the connection cannot be established. Check the System Information menu on the phone to validate this failure by verifying that the Last Update Status code is reported as 0x2F0D/0 which resolves to the error message of ERROR_WINHTTP_SECURE_INVALID_CA as covered in this troubleshooting article. This confirms that the device does not inherently trust the certificate assigned to the Lync server so the root CA certificate which signed the Lync server certificate must be downloaded to the phone.
- To resolve this issue attempt to sign-in on the device using PIN Authentication to trigger a download of the root CA certificate. Simply enter a bogus phone number and let the process fail.
A network trace running on the Lync server will show the server response to the device request to retrieve the root certificate chain during the sign-in attempt.
SOAP SOAP:xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" {HTTP:13, TCP:12, IPv4:39}
That SOAP response contains the public key of the root CA certificate as shown below.
<GetRootCertChainsResponse ResponseClass="Success" xmlns="http://schemas.microsoft.com/OCS/AuthWebServices/">
<RootCertChains>
<Chain>
MIIDmgYJKoZIhvcNAQcCoIIDizCCA4DALBgkqxFzAVBgoJkiaJk/IsZAEZFzY2hlcn…<snipped>
</Chain>
</RootCertChains>
</GetRootCertChainsResponse>
At this point the device should now have the required root certicate in its store to support TLS communications with the Lync Server. All that is left now is to return the phone to main menu and leave it ide to do its thing.
It is important to not leave the phone at the failed Desk Phone Setup screen pictured above otherwise the update process will not begin.
- Select Backspace and then Back enough times to return to the initial Welcome screen and then leave the device alone. Seriously, don’t touch it.
The automatic device update process should kick-off within a few minutes, but there will be no indication on the device that this is happening. To validate that anything is actually happening then leverage the Lync server logs and fight the urge to check the phone’s menu, as any activity on the phone will reset the idle timer back to zero and just delay the pending update process.
- To confirm that the device has queried the server for any approved updates check the latest RequestHandlerAuditLog file located by default in the DeviceUpdateLogs folder.
\\LyncShare\1-WebServices-1\DeviceUpdateLogs\Server\Audit\imageUpdates
RequestHandlerAuditLog_lync_05312013.log
- Open the file and look for the most recent log entry and verify that the IP address matches the test device and that path to the image file (CPE.nbt) is passed to the device.
05/31/2013 17:08:10,,192.168.1.104,UCPhone
5/31/2013 3:08:07 PM,"0004F29531A2","0004f29531a2","POLYCOM","CX600","Rev-4","ENU",cpe.nbt; 4.0.7577.4366;12/18/2012 5:56:16 PM, http://lync.schertz.name/RequestHandler/Files/UCPhone/POLYCOM/CX600/
Rev-4/ENU/4.0.7577.4387/CPE/CPE.nbt; https://lyncweb.mslync.net/RequestHandlerExt/Files/UCPhone/POLYCOM/CX600/
Rev4/ENU/4.0.7577.4387/CPE/CPE.nbt;
4.0.7577.4387;4/10/2013 6:55:12 AM
Note that the first firmware version listed is what is currently on the device (.4366) while the later instances reflect the currently approved version (.4387) for this specific device model. The HTTP URL highlighted in red is the internal URL that the device will utilize to download the firmware package. (The second HTTPS URL is for externally registered devices which is not applicable for the purposes of this article.)
After a few additional minutes of idle time the phone will automatically reboot, activating the newly downloaded firmware version.
[…] http://blog.schertz.name/2013/05/updating-lync-phone-edition-devices-lync-2013/ […]
Upload Multiple Lync Phone Edition Firmware Updates to Single or Multiple Pools with New PowerShell Script…
If you’re having trouble staying on top of updates for Lync Phone Edition devices, this is your…
Thanks for yet another excellent and informative article! Just to note that it seems that your 2010 version of the article from Jan 11 has gone missing.
Thanks, I had an issue with WordPress that should now be resolved.
After moving myself to a 2013 pilot pool, I'm unable to search by name without having to dig down in the contacts list on the CX600 phones. This is both on firmware 4372 and 4387. Thoughts?
I haven't seen that; the address book search should still work the same. I suggest opening a support ticket on this.
Looks like it started working after updating to the most recent of 4397.
[…] http://blog.schertz.name/2013/05/updating-lync-phone-edition-devices-lync-2013/ […]
Hi Jeff, great article again! Only it's a bit unclear to me which domain to use. My Lync server is in an internal AD domain, which isn't the SIP domain the phones use. Do I have to add DNS & DHCP records for my AD domain where the Lync server is or do I need these records for the SIP domain (our public domain)?
The domain name you provide via DHCP is what needs to be used on the 'ucupdates-r2' FQDN as that is what the phone will append to the hardcoded hostname.
Thanks for all of this great info. What do I do if the path to the image file is not getting passed? In the log I see the phone's IP, MAC addr, etc, but no mention of a path.
This typically indicates that there is no newer applicable update for the phone, so no URL is passed to the phone.
Hi Jeff–Thank you for the breakdown of the sign in process!
We have our phones VLANed off on a separate network that prevents internet connectivity and for whatever reason, the phones are failing to sign-in (once internet connectivity is allowed, the phones sign in just fine). Upon further investigation, it turns out the phones are trying to contact something on our webserver via https (which is hosted outside of our network by a 3rd party/uses a public IP). Since we have a masked domain name (UPN) for mydomain.com instead of mydomain.local assigned to our users, when the phones somehow pulls the mydomain.com address (not sure how it gets this), it goes out and tries to talk to that server (which turns out to be our web server, using the public IP, which is blocked by the firewall on the phone VLAN). Do you know why the phones would be trying to contact mydomain.com over HTTPS and not talk directly to the Lync server the entire time? I am seeing this traffic occur after I type in the user's phone number and then pin. Once the sign-in process has been completed, I never see this traffic again either.
Have any ideas on why the phones would be trying to poll mydomain.com via HTTPS during the sign-in process?
Thanks!
-chubacca
This sounds like the phone's attempt to reach the Certificate Provisioning service via the web services FQDN. It appears you have your network configured in an unsupported method.
Nice post. One thing I have noticed in my environment is we were still able to sign in to our Lync 2013 environment using a Polycom CX600 with software version 4.0.7577.4100.
Yes, older versions can often still register but the scenario is not supported by Microsoft. The officially supported versions include numerous fixes specific to Lync 2013.
Hi Jeff – thanks for plenty of great stuff about Lync 🙂
We are migrating on to Lync 2013 and have gotten as far as getting phones to upgrade from 4100 to 4397 (Polycom CX600)
Hoewever after the Upgrade Exchange integration has stopped working. Now getting error message "Microsoft Exchange is unavailable" for Calendar and no Outlook contact lookup.
Phones are outside the Edge. Tried all usual stuff including soft resets. Are there changes to infrastructure requirements for LPE in this release?
The newer releases have been enforcing strict domain naming policies differently than past versions, so it's possible that our configuration is using different domain names between the autodiscover SRV and A records in Exchange. Otherwise I suggest opening a support ticket to identify why the Exchange integration is not working as there are a couple different issues with Exchange integration across nearly every different firmware release which can pop up in different environments.
Hi Jeff,
yet another great article regarding the Phone edition. I wonder though, that it needed 7 years to be documented by an external resource, as MS was unable to achieve this task. Nevermind, I would like to ask something related:
I noticed in the recent Lync 2013 firmwares, there is an option "Remote Log Access". I tried it on an ancient Tanjay phone (have no Aries currently), but it does not seem to do anything, after enabling it and clicking OK, I go back to the menu selector, and checking Remote Log Access again, its again disabled, no matter how many times I Enable it.
That option was added by Microsoft to allow Office 365 customers to send device logs to support personal.
Dear Jeff
I have a Polycom CX 3000 desk conferencing phone version is upgraded to ver 4.0.7577.4397.
We are trying to integrate with Office 365 Lync online.
When we did for the first time it did get integrated and we could make and receive calls from the CX 3000 to any online contacts after a few configuration steps.Subsequently we removed the POE connection and rebooted the unit and henceforth it comes to the point where it says "Authentication complete"Retrieving user information.After that it prompts us to put the six digit PIN but before we are able to thru with it; it says "Signing off"and in the computer which is connected with the USB cable with the CX 3000 unit gives the message "Lync cannot connect to your deskphone because you are not signed in.Make sure the phone is connected to the network and enter your sign in info on the desk phone if necessary"
For info we are signed in with the account credentials in the laptop having office 365 client.And onne we are connected to the CX 3000 with the USB cable a pop up comes which prompts to put the credentials for the CX 3000 to login.
Kindly can you suggest what may be the issue.The setup only worked once as mentioned above.
To use CX devices with Office 365 users you must have the proper licensing package (O365 Plan e4 or Lync Online Plan E3). these profiles will enable Enterprise Voice for the user account which is a requirement to use Lync Phone Edition devices. The immediate sign-out is typically an indication that the Lync user is still configured for Peer-to-Peer calling only.
Hi Jeff,
Thank you for your post.
We are having an issue with our external Lync devices getting any device update. The last device update status is listed as (0x2ee2/0) which is converted to code 12002 The request has timed out. In your post Troubleshooting Lync devices you mention in the IIS logs after the device has been given the path to download the update you should see shortly after the original POST message there should be a GET message showing the actual device requesting the file from the web server. I never see the GET message in the IIS logs. This device is a CX600_Rev-4 running ver. 4.0.7577.4387 with ver. 4.0.7577.4397 approved and 4.0.7577.4414 pending with my device as a test device. I do see in the request handler audit log my request for the .4397 file. However, I do not see the request for the pending file even though the test device is setup correctly with the correct MAC address of the device. Either way the device is not getting either file. Our devices are all external going through TMG for Web services. Any idea why we are not able to update or why we are not receiving the GET message after the post?
My only guess is that the device is not attempting to download the package. I've seen this before when the same version is already loaded into the inactive partition and in other erroneous cases. I would test resetting that device to the previous firmware version and testing again. If no devices are uable to update externally but work internally then it could be related to the reverse proxy rule.
Jeff, what's the best way to roll a handset back then without causing them all to roll back? Thanks!
Either manually (using the process in my latest article) or by utilizing the Test Device option after importing a different ‘pending’ version.
Jeff,
Should the ucupdates-r2.schertz.name be configured on the internal or external DNS? Or, both?
This approach is only supported for internal use. Typically a public certificate is used on the Access Edge service and public CAs will not issue certificates with requests for simple hostnames; they must be verifiable FQDNs.
On a related topic from a Lync novice in advance of an upcoming OCS/Lync migration. Can Lync 2013 be configured to use an alternative to DHCP option 43? E.g. if option 43 is already being used by a previously installed and separate live running ICT solution. Any advice and guidance would be much appreciated.
Option 43 is not actually DHCP option 43 but is a string of sub values which together created that vendor class ID. The exact configuration is used only for these Microsoft phone clients and when configured properly should not conflict with any other DHCP configuration you may have for other solutions.
We use a public CA for all of our Lync certs. Regarding the requirement for the short hostname ucupdates-r2, after November 1, 2015, internal names will no longer be supported in public certs. Is that definitely a requirement? Is there a workaround?
http://www.digicert.com/internal-names.htm
Yes, it is a requirement so if the entry is missing then I assume the updates will not function correctly. I have not tested with an incorrect certificate configuration in a very long time so I don't know what the exact behavior would be.
We've run into a similar issue here. We were finally able to get our CX600 devices connected, but our IP5000/7000 devices are still not working.
The new limitation of no short names on public certs seems to be a growing problem. I'm not exactly sure how short names play into Lync and if these names are 100% required for things to function. Perhaps MSFT needs to provide a working update to not include these SAN entries and support FQDNs.
I'm not sure what else we can do here. Perhaps you or Dave P can shed some light if he figures it out.
The only short hostname requirement in Lync is for unregistered Lync Phone Edition devices to automatically download firmware updates. Once an LPE device signs-in to Lync then the ucupdates-r2 names are not applicable. This has nothing to do with the IP 5000 or any other third party phone not running LPE. Also the IP 7000 is not supported for Lync, only the 5000 and Duo so you can give up on trying to get the 7000 to work with Lync. The 5000 is fully supported and should be working when using the Lync Base Profile.
You're right Jeff. We were finally able to get the devices to connect locally, but there seems to be some bugs with the Polycom software and how it selects which certificate to use. When there is one that is explicitly added to the phone, you must also explicitly select it as the only CA to use.
However, now our issue is that the same devices in our regional offices do not connect back across the WAN. Not sure if it is just timing out during the cert process or what. Will need to try to set up a packet trace to determine the issue.
Jeff, sorry for double posting. I posted a question today but ended up figuring it out.
Here's our environment. Two resilient lync pool in. We are using 95% vvx phones with the other 5% CX series (for common area location….cx3000 in conference rooms as we want the hotdesking features in these location….VVX does not support this at this time). The problem with VVX series phones is at this time they do not support pool fail-over and thus, in a pool failover situation, would require us to log into each DHCP scope, change the url to the lync environment (to now point to the backup pool FQDN) and then have the users reboot their phones to reconnect to the backup pool. As you can bet, this would be problematic. To get around this, instead of using the poolFQDN in our DHCP configs, we chose to use our Meeting URL (in a fail-over situation, we would adjust the internal DNS record for our meeting url to point to our backup pool). Here's what we found. If the CX3000 series phone is running .4372 version (this is what ours came shipped with), you could NOT login to the CX3000s when the URL dhcp is giving to the phones pointed to our meeting url (note the meeting url and the Pool FQDN point to the same IP). Firmware version .4372 requires the PoolFQDN. Simply changing it to the poolfqdn, reboot the CX series phone, and everything worked as expected. As I stated before, now VVX phones are up a creek when we have a pool failure. What we found is once you update the phones to .4420 the CX phones will use any FQDN on the Front End cert SAN entry to pull down the lync cert.
So what we're going to do is stand up a CX series provision VLAN. On this VLAN, we will use the PoolFQDN in our DHCP configs. 10 min after the CX series are online and not used, they will get updated to .4420 via lync device update. After the update, the CX3000's now running .4420 can then be deployed to their locations (and thus onto a different VLAN using our meeting URL in the DHCP configs).
Note that the download of the CPE.nbt file internally is http, NOT https. This fact actually kept downloads from working on my phones, as the HLB was not configured correctly for http, but was for https. I kept getting failures of 0x2f0d/200.
John, that is a good point that I probably should have called out specifically as it’s inferred based on the fact that the first URL returned by the server (for internally registered phones) is http: while the second URL is for external phones and uses https:. Connectivity over TCP80 is also a prerequisite for internal CA certificate download on the phones as covered here: http://blog.schertz.name/2012/12/http-utilized-lync-server
if you have a copy of the HP 4120 December 2012 cumulative update. 4.0.7577.4366 I would like to get a link to download it.
I need to roll back to that version in order to fix a nagging issue that was uncovered when I updated to the latest release.
I have a open case with Microsoft, but they do not offer previous versions of those cumulative updates.
only the most current one online.
I had 1 phone with the older version that does not have the issue, but I do not have the CU to import and roll back my phones.
if anyone has a copy, please reply with a link.
Hi Jeff,
I have CX600 which is installed in a remote location. Need to upgrade it to 7577.4512. Any recommendations on how to update external remote phones.
Internal CX600 phones are getting updated without any issues.
Regards
Terencve
External updates work the same as internal but the phones will connect to the external device update URL, via HTTPS, so make sure that is reachable outside your network.