The interesting thing about this particular issue is that if you weren't looking for it, you probably wouldn't catch it. Normal call setup was working fine. But when you called certain numbers you would receive ringing when you are expecting the call to be treated with an IVR.
There are a couple of ways of dealing with provisional SIP responses. The following covers some of the techniques I tested/used.
The deployment is fairly straightforward. We have a CUCM 9.1.2 system and Cisco 2900 ISRs running IOS 15.2(4)M acting as CUBE device. There is nothing really exotic with the configuration.
The ITSP we are using is TW Telecom and the integration guide is on the CUCM interoperability portal.
While walking through our validation we placed a test call to an AT&T customer service line (+18007272222) and after we dialed the number we hear normal ring back. Sounds good so far but we are expecting the call to be treated with an IVR. Call from a mobile phone? We hit the IVR immediately, with no ringing. That is what we in the biz call a clue.
Now, we knew what needed to be done but this blog entry would be a whole let less interesting if I just said "Apply this config to the profile and reset", right? Let's tinker under the hood and see what is going on.
The ITSP we are using is TW Telecom and the integration guide is on the CUCM interoperability portal.
While walking through our validation we placed a test call to an AT&T customer service line (+18007272222) and after we dialed the number we hear normal ring back. Sounds good so far but we are expecting the call to be treated with an IVR. Call from a mobile phone? We hit the IVR immediately, with no ringing. That is what we in the biz call a clue.
Now, we knew what needed to be done but this blog entry would be a whole let less interesting if I just said "Apply this config to the profile and reset", right? Let's tinker under the hood and see what is going on.
Go Go Gadget Debug
There are some processes/apps in the Cisco UC portfolio where debugging is something akin to oral surgery without local anesthesia. Fortunately, debugging SIP processes does not fall into this category. SIP debugging is very approachable and actually is a good learning tool.
Anyway, since I know that the call is making it through the CUBE, I opt to use debug ccsip mess on the IOS device. We see the following:
Message 1: We receive the Invite from CUCM
Message 2: We send an Invite from the CUBE to the ITSP
The next messages we see are a SIP "100 Trying" from CUBE to CUCM and another SIP 100 message from the ITSP to the CUBE. Following the SIP 100 Trying messages we see the following message from the ITSP to CUBE:
The CUBE sends a SIP 183 Session Progress message to the CUCM:
The tried and true "nail 'em up a MTP"! Talk about removing a splinter with a machete! Using this approach, we configure the SIP trunk associated with the CUBE as follows:
Anyway, since I know that the call is making it through the CUBE, I opt to use debug ccsip mess on the IOS device. We see the following:
Message 1: We receive the Invite from CUCM
*Dec 1 21:11:02.487: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg: Received: INVITE sip:+18007272222@11.11.11.1:5060 SIP/2.0 Via: SIP/2.0/TCP 10.8.8.73:5060;branch=z9hG4bK143a764402e75 From: <sip:+12025557499@10.8.8.73>;tag=474018~67b0b345-b765-4d66-9538-07198f89307d-37044307 To: <sip:+18007272222@11.11.11.1> Date: Sun, 01 Dec 2013 21:13:23 GMT Call-ID: 67ebe100-29b1a673-10850-49d0080a@10.8.8.73 Supported: timer,resource-priority,replaces Min-SE: 1800 User-Agent: Cisco-CUCM9.1 Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY CSeq: 101 INVITE Expires: 180 Allow-Events: presence Supported: X-cisco-srtp-fallback,X-cisco-original-called Cisco-Guid: 1743511808-0000065536-0000001556-1238370314 Session-Expires: 1800 P-Asserted-Identity: <sip:+12025557499@10.8.8.73> Remote-Party-ID: <sip:+12025557499@10.8.8.73>;party=calling;screen=yes;privacy=off Contact: <sip:+12025557499@10.8.8.73:5060;transport=tcp>;video;audio Max-Forwards: 69 Content-Length: 0
Message 2: We send an Invite from the CUBE to the ITSP
*Dec 1 21:11:02.495: //27373/67EBE1000000/SIP/Msg/ccsipDisplayMsg: Sent: INVITE sip:8007272222@11.11.11.62:5060 SIP/2.0 Via: SIP/2.0/UDP 11.11.11.61:5060;branch=z9hG4bK197B6E Remote-Party-ID: <sip:2025557499@11.11.11.61>;party=calling;screen=yes;privacy=off From: <sip:2025557499@11.11.11.61>;tag=3F0D535C-BFF To: <sip:8007272222@11.11.11.62> Date: Sun, 01 Dec 2013 21:11:02 GMT Call-ID: EAFAC152-5A0311E3-9ABCEF7C-85248138@11.11.11.61 Supported: timer,resource-priority,replaces,sdp-anat Min-SE: 1800 Cisco-Guid: 1743511808-0000065536-0000001556-1238370314 User-Agent: Cisco-SIPGateway/IOS-15.2.4.M5 CSeq: 101 INVITE Timestamp: 1385932262 Contact: <sip:2025557499@11.11.11.61:5060> Expires: 180 Allow-Events: telephone-event Max-Forwards: 68 Session-Expires: 1800 Content-Type: application/sdp Content-Disposition: session;handling=required Content-Length: 247 v=0 o=CiscoSystemsSIP-GW-UserAgent 9469 9873 IN IP4 11.11.11.61 s=SIP Call c=IN IP4 11.11.11.61 t=0 0 m=audio 16622 RTP/AVP 0 101 c=IN IP4 11.11.11.61 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20At this point, we learn a few things about the environment. The CUCM is not using Early Offer (EO) to the CUBE. So, this means that we haven't enable that option under the SIP profile nor have we "forced MTP". We can also see that the CUBE is using EO when initiating a session with the ITSP. Nothing really out of the ordinary here, per se. More on the EO thing in a bit.
The next messages we see are a SIP "100 Trying" from CUBE to CUCM and another SIP 100 message from the ITSP to the CUBE. Following the SIP 100 Trying messages we see the following message from the ITSP to CUBE:
*Dec 1 21:11:04.203: //27373/67EBE1000000/SIP/Msg/ccsipDisplayMsg: Received: SIP/2.0 183 Session Progress Via: SIP/2.0/UDP 11.11.11.61:5060;branch=z9hG4bK197B6E To: <sip:8007272222@11.11.11.62>;tag=MTRELAAEEB0010004ec7f88b58c41 From: <sip:2025557499@11.11.11.61>;tag=3F0D535C-BFF Call-ID: EAFAC152-5A0311E3-9ABCEF7C-85248138@11.11.11.61 CSeq: 101 INVITE Content-Type: application/sdp Content-Length: 184 v=0 o=fvx 1 1 IN IP4 someplace.com s=FWave c=IN IP4 11.11.11.62 t=0 0 m=audio 37680 RTP/AVP 0 101 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15
The CUBE sends a SIP 183 Session Progress message to the CUCM:
*Dec 1 21:11:04.207: //27372/67EBE1000000/SIP/Msg/ccsipDisplayMsg: Sent: SIP/2.0 183 Session Progress Via: SIP/2.0/TCP 10.8.8.73:5060;branch=z9hG4bK143a764402e75 From: <sip:+12025557499@10.8.8.73>;tag=474018~67b0b345-b765-4d66-9538-07198f89307d-37044307 To: <sip:+18007272222@11.11.11.1>;tag=3F0D5A0C-1069 Date: Sun, 01 Dec 2013 21:11:02 GMT Call-ID: 67ebe100-29b1a673-10850-49d0080a@10.8.8.73 CSeq: 101 INVITE Allow-Events: telephone-event Remote-Party-ID: <sip:8007272222@11.11.11.1>;party=called;screen=yes;privacy=off Contact: <sip:+18007272222@11.11.11.1:5060;transport=tcp> Supported: sdp-anat Server: Cisco-SIPGateway/IOS-15.2.4.M5 Content-Type: application/sdp Content-Disposition: session;handling=required Content-Length: 243 v=0 o=CiscoSystemsSIP-GW-UserAgent 649 1984 IN IP4 11.11.11.1 s=SIP Call c=IN IP4 11.11.11.1 t=0 0 m=audio 16620 RTP/AVP 0 101 c=IN IP4 11.11.11.1 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=ptime:20Where are we at this stage in the game? Well, we are listening to a nice, warm ringback tone. But we are supposed to be hearing an inordinately happy lady welcoming us to the customer service center. We see no other SIP messages related to this call.
Wha' Happen'
Essentially, what is happening is that the remote end is attempting to set up a media path to play the IVR menu prompts before it "answers" the call (i.e. sends a 200 OK). We are using Delayed Offer (DO) from CUCM. Therefore, we never see CUCM send a SDP to the CUBE and there is no media path established to the CUCM. In this scenario, the calling party (a phone registered to CUCM) only hears ringback tone.
The problem is a configuration issue and we have several methods that can be applied to remediate the issue. Let's look at a few of these.
Force MTP
The problem is a configuration issue and we have several methods that can be applied to remediate the issue. Let's look at a few of these.
Force MTP
The tried and true "nail 'em up a MTP"! Talk about removing a splinter with a machete! Using this approach, we configure the SIP trunk associated with the CUBE as follows:
- Media Termination Point Required: True
- Media Resource Group List: Assign a MRGL that has a valid MTP
- MTP Preferred Originating Codec: Set the CODEC to whatever the CUBE expects/requires
So, we nail up MTP and we place a call. What happens? Well, we hear the IVR (yay us!) and we see the CUCM send the following Invite (from the CUBE PoV):
This is an example of the CUCM using EO to establish a session. The SIP messages following this Invite are basically the same as they were in the original call. We receive the 183 Session Progress message from the ITSP and the CUBE relays the message to CUCM. However, since CUCM has already offered up some media parameters, we are able to establish a media path end to end.
That is a good thing, right? Well, yeah but we now have to nail up a MTP for 100% of calls (ingress and egress) when we only need this functionality on a small fraction of possible call scenarios. It is an inefficient use of resources.
There is another way to get the CUCM to use EO by way of the SIP Profile assigned to the SIP trunk. If you go this path then the first thing I like to do is copy the Standard SIP Profile to something new. For example: ITSP SIP Trunk Profile. Then you can edit the profile by going to the "Trunk Specific Configuration" and enabling the option: Early Offer support for voice and video calls (insert MTP if needed).
As with the Machete Termination Point, enabling EO on the SIP profile associated with the trunk is going to facilitate a successful call. The media path will get set up and the calling party hears the sweet sound of IVR success. Best of all, we don't need to nail up a MTP to get there. Bonus!
If you ever need to do EO from CUCM then you definitely want to leverage the SIP Profile approach. Nailing up a MTP on a SIP trunk should be avoided, unless you have no other options.
Note that Early Offer on CUCM was introduced with version 8.5(1).
SIP reliable provisional response was introduced to address issues exactly like the one I am describing in this article. The CUBE supports reliable response by default. You can change this under the "voice service voip" configuration stanza:
So, let's go back to our original problem. If the CUBE supports reliable provisional response by default, how would one run into the issue described in this article? Firstly, just because you have a CUBE that is configured to handle provisional response messages doesn't mean that CUCM is doing its part.
NOTE: For this section, assume that we haven't provisioned the SIP trunk to support EO.
In our scenario, we would also need the CUCM SIP trunk to be provisioned to support PRACK. This is done via the SIP profile configuration. Under Trunk Specific Configurations, there is an option called "SIP Re1XX Options". By default, this is disabled and CUCM won't respond with the prescribed PRACK when receiving the SIP 183 w/SDP. We basically have the same issue we illustrated at the beginning of this article.
We either need to enable PRACK for 1xx messages with SDP or we can enable PRACK for all 1xx messages. If we use one of these options and place our test call then we see some positive results.
The INVITE and SIP 100 trying message exchange between the three entities (CUCM, CUBE, and the ITSP) are the same as previously shown in this article. It starts to get interesting when we receive the SIP 183 message from the ITSP.
SIP 183 Message Exchange: The CUBE receives the SIP 183 Session Progress message from the ITSP and relays that to CUCM.
The CUCM responds with a provisional acknowledgement (PRACK) just like it was told:
Conclusion
The interesting part of my particular scenario was that the interoperability guide from Cisco specifically disabled the reliable response feature on the CUBE. So, even with PRACK enabled on CUCM, I still had the same issue.
We eventually worked things out with the ITSP to build and test a configuration. We wound up using EO on the CUCM SIP profile.
*Jan 10 05:53:45.104: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg: Received: INVITE sip:+18007272222@11.11.11.1:5060 SIP/2.0 Via: SIP/2.0/TCP 11.11.11.73:5060;branch=z9hG4bK4de786dcbedf1 From: <sip:+12025557499@11.11.11.73>;tag=2235268~67b0b345-b765-4d66-9538-07198f89307d-37136296 To: <sip:+18007272222@11.11.11.1> Date: Fri, 10 Jan 2014 05:55:39 GMT Call-ID: d3be4500-2cf18b5b-41800-49d0080a@11.11.11.73 Supported: timer,resource-priority,replaces Min-SE: 1800 User-Agent: Cisco-CUCM9.1 Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY CSeq: 101 INVITE Expires: 180 Allow-Events: presence Supported: X-cisco-srtp-fallback,X-cisco-original-called Cisco-Guid: 3552462080-0000065536-0000005848-1238370314 Session-Expires: 1800 P-Asserted-Identity: <sip:+12025557499@11.11.11.73> Remote-Party-ID: <sip:+12025557499@11.11.11.73>;party=calling;screen=yes;privacy=off Contact: <sip:+12025557499@11.11.11.73:5060;transport=tcp> Max-Forwards: 69 Content-Type: application/sdp Content-Length: 212 v=0 o=CiscoSystemsCCM-SIP 2235268 1 IN IP4 11.11.11.73 s=SIP Call c=IN IP4 11.11.11.1 t=0 0 m=audio 27964 RTP/AVP 0 101 a=rtpmap:0 PCMU/8000 a=ptime:20 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15
This is an example of the CUCM using EO to establish a session. The SIP messages following this Invite are basically the same as they were in the original call. We receive the 183 Session Progress message from the ITSP and the CUBE relays the message to CUCM. However, since CUCM has already offered up some media parameters, we are able to establish a media path end to end.
That is a good thing, right? Well, yeah but we now have to nail up a MTP for 100% of calls (ingress and egress) when we only need this functionality on a small fraction of possible call scenarios. It is an inefficient use of resources.
Another Way to Do Early Offer
There is another way to get the CUCM to use EO by way of the SIP Profile assigned to the SIP trunk. If you go this path then the first thing I like to do is copy the Standard SIP Profile to something new. For example: ITSP SIP Trunk Profile. Then you can edit the profile by going to the "Trunk Specific Configuration" and enabling the option: Early Offer support for voice and video calls (insert MTP if needed).
As with the Machete Termination Point, enabling EO on the SIP profile associated with the trunk is going to facilitate a successful call. The media path will get set up and the calling party hears the sweet sound of IVR success. Best of all, we don't need to nail up a MTP to get there. Bonus!
If you ever need to do EO from CUCM then you definitely want to leverage the SIP Profile approach. Nailing up a MTP on a SIP trunk should be avoided, unless you have no other options.
Note that Early Offer on CUCM was introduced with version 8.5(1).
SIP Reliable Provisional Response
SIP reliable provisional response was introduced to address issues exactly like the one I am describing in this article. The CUBE supports reliable response by default. You can change this under the "voice service voip" configuration stanza:
/*Reliable provisional response enabled (default)*/ voice service voip sip rel1xx supported 100rel /*Reliable provisional response disabled*/ voice service voip sip rel1xx disable /*Reliable provisionals response requiring the CUBE to wait for PRACK from UAC*/ voice service voip sip rel1xx require 100rel
So, let's go back to our original problem. If the CUBE supports reliable provisional response by default, how would one run into the issue described in this article? Firstly, just because you have a CUBE that is configured to handle provisional response messages doesn't mean that CUCM is doing its part.
NOTE: For this section, assume that we haven't provisioned the SIP trunk to support EO.
In our scenario, we would also need the CUCM SIP trunk to be provisioned to support PRACK. This is done via the SIP profile configuration. Under Trunk Specific Configurations, there is an option called "SIP Re1XX Options". By default, this is disabled and CUCM won't respond with the prescribed PRACK when receiving the SIP 183 w/SDP. We basically have the same issue we illustrated at the beginning of this article.
We either need to enable PRACK for 1xx messages with SDP or we can enable PRACK for all 1xx messages. If we use one of these options and place our test call then we see some positive results.
The INVITE and SIP 100 trying message exchange between the three entities (CUCM, CUBE, and the ITSP) are the same as previously shown in this article. It starts to get interesting when we receive the SIP 183 message from the ITSP.
SIP 183 Message Exchange: The CUBE receives the SIP 183 Session Progress message from the ITSP and relays that to CUCM.
*Jan 10 05:03:52.124: //90307/DA9589800000/SIP/Msg/ccsipDisplayMsg: Received: SIP/2.0 183 Session Progress Via: SIP/2.0/UDP 11.11.11.61:5060;branch=z9hG4bK29E61F56 To: <sip:8007272222@11.11.11.62>;tag=HSTNTXGIEB0010004ef96adce53a4 From: <sip:2025557499@11.11.11.61>;tag=9846424-1461 Call-ID: 6D6F0EA8-78EB11E3-A8C2BBD0-C8835D44@11.11.11.61 CSeq: 101 INVITE Content-Type: application/sdp Content-Length: 183 v=0 o=fvx 1 1 IN IP4 tekvizion.com s=FortisWave c=IN IP4 11.11.11.62 t=0 0 m=audio 34616 RTP/AVP 0 101 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 *Jan 10 05:03:52.136: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg: Sent: SIP/2.0 183 Session Progress Via: SIP/2.0/TCP 10.8.8.73:5060;branch=z9hG4bK4dc7b4e2ec16c From: <sip:+12025557499@10.8.8.73>;tag=2233355~67b0b345-b765-4d66-9538-07198f89307d-37136278 To: <sip:+18007272222@11.11.11.1>;tag=9846C7C-253A Date: Fri, 10 Jan 2014 05:03:49 GMT Call-ID: da958980-2cf17fa8-4160d-49d0080a@10.8.8.73 CSeq: 101 INVITE Require: 100rel RSeq: 7943 Allow-Events: telephone-event Remote-Party-ID: <sip:8007272222@11.11.11.1>;party=called;screen=yes;privacy=off Contact: <sip:+18007272222@11.11.11.1:5060;transport=tcp> Supported: sdp-anat Server: Cisco-SIPGateway/IOS-15.2.4.M5 Content-Type: application/sdp Content-Disposition: session;handling=required Content-Length: 241 v=0 o=CiscoSystemsSIP-GW-UserAgent 8310 9862 IN IP4 11.11.11.1 s=SIP Call c=IN IP4 11.11.11.1 t=0 0 m=audio 27928 RTP/AVP 0 101 c=IN IP4 11.11.11.1 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=ptime:20
The CUCM responds with a provisional acknowledgement (PRACK) just like it was told:
*Jan 10 05:03:52.140: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg: Received: PRACK sip:+18007272222@11.11.11.1:5060;transport=tcp SIP/2.0 Via: SIP/2.0/TCP 10.8.8.73:5060;branch=z9hG4bK4dc7c3dd907fd From: <sip:+12025557499@10.8.8.73>;tag=2233355~67b0b345-b765-4d66-9538-07198f89307d-37136278 To: <sip:+18007272222@11.11.11.1>;tag=9846C7C-253A Date: Fri, 10 Jan 2014 05:05:44 GMT Call-ID: da958980-2cf17fa8-4160d-49d0080a@10.8.8.73 CSeq: 102 PRACK RAck: 7943 101 INVITE Allow-Events: presence Max-Forwards: 70 Content-Type: application/sdp Content-Length: 236 v=0 o=CiscoSystemsCCM-SIP 2233355 1 IN IP4 10.8.8.73 s=SIP Call c=IN IP4 10.8.224.86 b=TIAS:64000 b=AS:64 t=0 0 m=audio 22474 RTP/AVP 0 101 a=rtpmap:0 PCMU/8000 a=ptime:20 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15At this point, we are able to hear our IVR and everything is working as desired. What is going on here is that the CUCM processes the SIP 183 and, because it is told to do so, it sends a PRACK with the appropriate SDP information so we can get the party started.
Conclusion
The interesting part of my particular scenario was that the interoperability guide from Cisco specifically disabled the reliable response feature on the CUBE. So, even with PRACK enabled on CUCM, I still had the same issue.
We eventually worked things out with the ITSP to build and test a configuration. We wound up using EO on the CUCM SIP profile.
Thanks for reading. If you have time, post a comment!
Excellent summary! I've had similar issues in the past but hadn't considered all the options you've presented, very helpful.
ReplyDeleteGreat Article. Facing the same issue and TAC mentioned the same fix but your article really gives great detail. Thanks and Kudos!
ReplyDeleteSuperb Explanation!
ReplyDeleteJust brilliant .. !
ReplyDeleteVery Brilliant article. I didn't have to finish the article to actually solve my problem.
ReplyDeleteGreat explanation.
Thanks I actually have an issue with an IVR but is with an SME cluster using DO on the sip trunks but for some reason when you get to the IVR options and press your selection, you don't hear a ringback tone, was thinking on disabling early media 183 and test...
ReplyDeleteExtremely well explained with the logs. Thanks !!
ReplyDeleteThank you for this simple and well explained article. It solved my problem real fast!!
ReplyDelete