Saturday, February 21, 2015

Software Defect Could Affect Custom IP Phone Service URLs

This is just a quick note on a software defect on Cisco 8800 series IP phones that could break normal operations for custom Cisco IP Phone Service URLs. The issue is documented in Cisco software defect CSCur13256 and may break IP Phone Services running co-resident on a web server (such as Microsoft IIS).

Background

A few months ago, my team (NetCraftsmen UC&C) ran into an issue with a custom Corporate Directory application that I built for a customer running Cisco Unified Communications Manager (UCM) 8.5. At the time of the original implementation the customer hadn't deployed any Cisco 8800 series phones (i.e. they weren't shipping yet). Over time, the customer procured and deployed several phones. Unbeknownst to us, the Corporate Directory application was not accessible from these phones.

We became aware of the problem after performing an upgraded of the UC 8.5 system to UC CSR 10.5. After every upgrade, NetCraftsmen runs a standard validation process where we test a full range of functionality on all systems. During this validation process, we found an issue with accessing our custom Corporate Directory application.

Environment

The environment where we encountered the issue:
  • CUCM: 10.5 (but this issue is independent of CUCM version)
  • 8800 Series Phone Firmware: 10.2(1.16)
  • Microsoft IIS (version not relevant)
  • IIS server hosted multiple web pages
Issue Details

When we provisioned the custom Corporate Directory, the customer had us deploy the web site on an existing IIS server that was hosting multiple sites. Our application was leveraging a binding based on the FQDN presented in the URL, for example:

http://ucmdirectory.domain.com/cncdirectory.aspx?name=SEPxxx&locale=en_us

So, in IIS, we created a binding on port 80 (default) to the "ucmdirectory.domain.com" URL. Whenever a client presents this URL in the HTTP GET request, IIS serves up the appropriate application. 

Unfortunately, the Corporate Directory would never render on the client device. Using IIS W3SVC logs, we were able to determine that the request was never making it to the IIS server. We know the logs were "good" because requests from other Cisco phones and clients were working and generating log entries.

The issue became clear when looking at the console messages on a Cisco 8800 series phone:


1893 NOT 19:14:36.524910 CVM-System P5-traceManager MQThread|HttpClientTask:? - Current State = 0
1894 NOT 19:14:36.526738 CVM-System P5-traceManager MQThread|cip.http.HttpClientConnection:? - Check if #DEVICENAME# or #EMCC# is present in the URL
1895 DEB 19:14:36.548370 Nov 11 01:14:36 dnsmasq[451]: query[A] ucmdirectory.domain.com from 127.0.0.1
1896 DEB 19:14:36.548681 Nov 11 01:14:36 dnsmasq[451]: cached ucmdirectory.domain.com is 
1897 DEB 19:14:36.550023 Nov 11 01:14:36 dnsmasq[451]: cached ns1.domain.com is 10.12.10.10
1898 NOT 19:14:36.560022 CVM-DNS LOOKUP http://ucmdirectory.domain.com/cncdirectory.aspx?name=SEPxxxxxxxxxxxx&locale=en_us|HttpClientTask:? - Current State = 5
1899 INF 19:14:36.683356 Nov 11 01:14:36 mtlog: _daisychain_cmd.c 243:register cmd: connected (1)
1900 INF 19:14:36.712840 Nov 11 01:14:36 mtlog: _daisychain_cmd.c 221:DCU cmd: disconnected (0)
1901 NOT 19:14:36.828542 CVM-HttpClientThread|cip.http.HttpClientConnection:? - listener.httpSucceed:  http://10.12.10.10/cncdirectory.aspx?name=SEPxxxxxxxxxxxx&locale=en_us



Most of this looks normal but the issue is clearly seen in the last line of the console log excerpt. The phone is using the cached IP address and not the actual FQDN (the way it is supposed to). This means that the bindings created on the IIS server won't engage and the web server will either render the wrong page or simply error out.

The Fix

I didn't post this issue when it happened because I got distracted by other things. I recently checked on the software defect and saw that there is a fix available via an Engineering Special (ES). So, I figured I might as well post my notes just in case it helps someone. Obviously, we actually believe that customer service is a "real thing". So, we didn't wait for a software fix on the phones. The workaround we applied was to modify the bindings in IIS.

We went ahead and preserved the FQDN binding. In addition, we added a customized port binding (e.g. 8080). First, we checked the the IIS server using netstat and a port scanner to find an available port. Then we configured IIS bindings to use this port. Finally, we configured the IP Phone Service in UCM to use the port (resetting all 8800 series phones in the process).




Thanks for reading. If you have time, post a comment!

1 comment: