Friday, December 7, 2012

CCIE-V "I Shoulda' Checked That" Tip #3: MTP Out To Lunch

This is the next installment in what I am calling the "I Shoulda' Checked That" series. The inspiration for this series is covered in the first installment. To save readers some time, I am going to jump right into things. On this go around, I am going to discuss the lazy MTP.

The Scenario...

I wasn't even planning on writing this tip. Fortunately, I was doing some testing this evening and I ran into an issue that I found to be somewhat interesting. I was playing around with troubleshooting and SIP setup to the imaginary ITSP in my lab. The question I was working on had the following requirements:

  • Set up a SIP trunk that uses UDP to communicate to the ITSP
  • Route all international calls for India to the ITSP
  • When placing a call to the ITSP you should use Early Offer
Easy enough... I copied the standard SIP security profile and set the outgoing transport type to UDP. I had already created a MRGL and Device Pool for the ITSP. I do this as part of my basic CUCM setup. I did have to edit my ITSP Device Pool and set it to use G711.

From here, I only need to create the SIP trunk. Set the MRGL, force the use of MTP, and make sure that the outbound CODEC is G711. Save, reset, and we are almost home.  Just a route group, route list, and an itty bitty route pattern to go. In a minute or two, I have everything setup.

I place the call to the ITSP and you know what happens? It works! I hear Anthrax (that's right) in my SubC wailing "I AM THE MAN!". Everything is good, right? Wrong. The call completes and that is good, but this is a troubleshooting question and I have to get some traces to prove  that I am the man. 

I have done this specific lab 2 or 3 times now. So, I open the trace file and do find for the special string ("INVITE sip:+91") and here is what I see:

12/05/2012 20:38:24.483 CCM|//SIP/SIPUdp/wait_SdlSPISignal: Outgoing SIP UDP message to[5060]:
INVITE sip:+916745731234@ SIP/2.0
Date: Thu, 06 Dec 2012 01:38:24 GMT
Call-Info: ;method="NOTIFY;Event=telephone-event;Duration=500"
From: "SiteA Phone 1" ;tag=d235cf7c-c70c-4304-998e-109e730ea978-48964600
Allow-Events: presence, kpml
P-Asserted-Identity: "SiteA Phone 1" 
Supported: timer,resource-priority,replaces
Min-SE:  1800
Remote-Party-ID: "SiteA Phone 1" ;party=calling;screen=yes;privacy=off
Content-Length: 0
User-Agent: Cisco-CUCM7.0
Expires: 180
Call-ID: 9e882d80-bf1f710-7-b78030a@
Via: SIP/2.0/UDP;branch=z9hG4bKb6d1b15b2
CSeq: 101 INVITE
Session-Expires:  1800
Max-Forwards: 70

Well, that is definitely not what I was hoping to see. Do you see what is wrong? 

I was asked to use Early Offer and this means that the SDP information from CUCM should be piggy backed on the SIP Invite. The "Content-Length: 0" tells me that there is no SDP message in the invite. With delayed offer, the "offer" is sent in a SDP message hitching a ride on a 200 OK from the remote call processing agent (the ITSP, in my case). 

My first thought is "I am so NOT the man". My second thought is that I must have missed something on the trunk. I jump over to the SIP trunk in CUCM and I scroll all up and down that page. Everything is good to go. OK, then maybe I need to reset the trunk. Do a reset, try again, same result. 

... Did I forget to set the region on the Device Pool? ...Nope
...... Did I forget to put the MRG into the MRGL?  ... Nope
.........Does the MRG have the wrong media resources (boy, would that be dumb)? ...Nope

Do the call again (for what purpose? I don't know, because that is what folks who have worked with Cisco voice products for ages do...). Check the MTP resource usage on CUCM:

admin:show perf query class "Cisco MTP Device"
==>query class :

 - Perf class (Cisco MTP Device) has instances and values:
    MTP_2           -> AllocatedResourceCannotOpenPort = 0
    MTP_2           -> OutOfResources                 = 0
    MTP_2           -> RequestsThrottled              = 0
    MTP_2           -> ResourceActive                 = 0
    MTP_2           -> ResourceAvailable              = 24
    MTP_2           -> ResourceTotal                  = 24
    MTP_3           -> AllocatedResourceCannotOpenPort = 0
    MTP_3           -> OutOfResources                 = 0
    MTP_3           -> RequestsThrottled              = 0
    MTP_3           -> ResourceActive                 = 0
    MTP_3           -> ResourceAvailable              = 24
    MTP_3           -> ResourceTotal                  = 24

Well, that confirms it. My MTP resources have gone out to lunch. Everything is configured and this should work. What is the 2nd thing that seasoned voice engineers do? That's right, reset that junk (the MTP, that is). Do another test, and everything is good to go (of course).

Well, why did the call succeed when MTP was required? I'll touch on that in a moment. For now, let's get back to the issue.

We configured everything correctly and we haven't looked at MTP since we first started to configure CUCM. I am still not sure exactly why I had to reset the MTP but I do have a hypothesis. In my lab approach, I rename CMGroup, Region, and Device pool to be Site A (HQ) specific. This effects MTP resource providers. If I forgot to reset things when I made the name changes (I usually do reset, but you never know) then maybe that caused the issue. 

The Take Away...

What I missed isn't as important as the the bigger lesson here: cosmetic proof is not enough. If your validation is only checking to see if a call succeeds then you are taking a gamble. I think we all do a thorough job of checking information elements on calls to/from the PSTN. You need to extend that to your other call scenarios, even if it means digging into CUCM trace or running some UCOS CLI commands. 

In my case, the call worked and if it wasn't a troubleshooting question (where I had to show how Early Offer works) then I may have never known that I was destined for a big fat zero. I guess I added another procedural step to my validation process. 

OK, So why did the call succeed...

There is a Call Manager service parameter named Fail SIP Trunk Call if MTP Allocation Fails. This is the SIP version of another parameter: Fail Call if MTP Allocation Fails. I think the function of these parameters is self-explanatory. 

The default values for these parameters is shown below:

admin:run sql select paramname,paramvalue from processconfig where paramname like '%MTPAlloc%'
paramname                            paramvalue
==================================== ==========
FailCallIfMTPAllocationFails         F
FailSIPTrunkCallIfMTPAllocationFails F

By default, our SIP parameter is set to false. So, the CUCM will try to allocate a MTP as instructed and, failing that, will still send the SIP invite. In a test config (where I removed the MTP MRG from my ITSP MRGL), the ITSP accepted the call and I was none the wiser. To confirm, I set the SIP MTP allocation parameter to true:

admin:run sql update processconfig set paramvalue='T' where paramname = 'FailSIPTrunkCallIfMTPAllocationFails'
Rows: 1

I attempt the call again and it fails. So, the service parameter works and I have all the proof I need.  

'nuff said.  

Thanks for reading. If you have time, post a comment!


  1. Service parameter changes via the CLI, now that is hardcore IE labbing! Great post!

  2. Awesome post Bill! Thank you and Merry Christmas.

  3. I cannot thank you enough Bill!! I have been pulling my hair for the whole weekend on SIP early offer. I even rebuilt UCM cluster for a fresh start. No luck until I finally found your post!!!

    Somehow software MTP_2 & MTP_3 just won't kick in. I had to add software MTP from HQ router to fix it. Now I see "Content-Type: application/sdp". 8-)

    1. Just a thought since I don't have enough info on your specific scenario, but CUCM MTP resources will only terminate G.711. Anything else will require an iOS sw resource with the codec required.

  4. I'm trying to run a sql query to get a list of all media resources dependencies (assigned to MRGs or not) since we're doing a lot of CUCM cleanup. can you please help with a sql query that does that? it would be highly appreciated.

    1. Saad,

      To get a count of MRGs in MRGLs:

      run sql select, Count(mrm.pkid) from mediaresourcegroup mrg left outer join mediaresourcelistmember mrm on mrg.pkid=mrm.fkmediaresourcegroup group by

      Getting mediaresource dependencies may be a little more involved and you will want to play with it some. The data is in device (that is where media resources are stored), mediaresourcegroup (that is the MRG), and mediaresourcegroupmember (maps devices to MRGs).

      Maybe something like:

      run sql select, Count(mrm.pkid) from mediaresourcegroup mrg left outer join mediaresourcelistmember mrm on mrg.pkid=mrm.fkmediaresourcegroup group by

      I don't have a system readily available to test. Syntactically, the above is correct. Functionally, I am not sure. I would need to test. The use of "inner join" would definitely show you devices in media resource groups. Using "outer join" the way I am here may not work as expected. What I am trying to do there is show resources NOT assigned to MRGs WITH resources that are.

      Explore the aforementioned tables and you should be able to get where you want to go. Otherwise, let me know. I will have to follow up when I have free cycles.


      -Bill (@ucguerrilla)