Monday, July 23, 2012

Side Effects of CCM-MANAGER Config

One of the lessons I learned this past week has to do with the logistics of shutting down and turning up my home lab gear. I just recently decided that my wallet may appreciate me more if I shut down the equipment rack I have in my lab on a nightly basis. Being as "nice" to my lab as possible, I shut everything down gracefully. I shut down all of my VMs first. A little later in the evening, I saved router/switch configs and powered off the rest of the gear. 

As I found out the next time I turned up the gear to do some studying, this was a mistake.


Background

So, it is Saturday morning and I am ready to do a 12 - 14 hour study session. I make breakfast for my beautiful wife and head down to the war room with a big cup of java. I do a preliminary check to make sure phones are registered, gateways and guest VMs are up. I immediately find an issue with my PRI connectivity on BR1-RTR. In the lab I am currently working through, BR1-RTR is a MGCP router, so I do some of the basic checks:

show ccm-manager
show mgcp end
show mgcp
show controller t1 x/y/z
show isdn stat

Everything looks good except for the ISDN stat:

BR1-RTR(config)#do sh isdn stat
Global ISDN Switchtype = primary-ni
ISDN Serial0/1/0:23 interface
        dsl 0, interface ISDN Switchtype = primary-ni
    Layer 1 Status:
        ACTIVE
    Layer 2 Status:
        TEI = 0, Ces = 1, SAPI = 0, State = TEI_ASSIGNED
    Layer 3 Status:
        0 Active Layer 3 Call(s)
    Active dsl 0 CCBs = 0
    The Free Channel Mask:  0x80000007
    Number of L2 Discards = 0, L2 Session ID = 2
    Total Allocated ISDN CCBs = 0


I then do a debug isdn q921 and this is what I see:
 

Jul 21 17:01:59.468: ISDN Se0/1/0:23 Q921: User RX <- SABMEp sapi=0 tei=0
Jul 21 17:01:59.468: ISDN Se0/1/0:23 **ERROR**: L2IF_SendPkt: idb is NULL
Jul 21 17:01:59.468: ISDN Se0/1/0:23 **ERROR**: process_rxdata:L2IF_SendPkt Failed

With the lack of a "CCM-MANAGER" statement in the show isdn stat output and the layer 2 issues announced via Q921, it is pretty apparent that I have a backhaul issue. Since show ccm-m shows a healthy registration to CUCM, I am beginning to suspect who the culprit may be. A quick check on configurations provides more confirmation.

BR1-RTR#sh run | s controller T1
controller T1 0/1/0
 framing esf
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-3,24 service mgcp
BR1-RTR#show run in se 0/1/0:23

interface Serial0/1/0:23
 no ip address
 encapsulation hdlc
 isdn switch-type primary-ni
 isdn incoming-voice voice
 no cdp enable
end

Do you see what I see?


...that's right, under the Serial 0/1/0:23 interface there is no layer 3 binding to CUCM. So, I add in the command:

interface Serial0/1/0:23
 isdn bind-l3 ccm-manager


Do a quick show isdn stat and Bob's your uncle, the ISDN is fat, dumb, and happy.

BR1-RTR#sh isdn stat
Global ISDN Switchtype = primary-ni

%Q.931 is backhauled to CCM MANAGER 0x0003 on DSL 0. Layer 3 output may not apply
ISDN Serial0/1/0:23 interface
dsl 0, interface ISDN Switchtype = primary-ni
L2 Protocol = Q.921 0x0000 L3 Protocol(s) = CCM MANAGER 0x0003
Layer 1 Status:
ACTIVE
Layer 2 Status:
TEI = 0, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED
Layer 3 Status:
0 Active Layer 3 Call(s)
Active dsl 0 CCBs = 0
The Free Channel Mask: 0x80000007
Number of L2 Discards = 0, L2 Session ID = 7
Total Allocated ISDN CCBs = 0

Let's roll it back a minute. What did happen here? Clearly, MGCP fallback is the culprit. You have probably seen this command before: ccm-manager fallback-mgcp. When you have MGCP fallback running, the IOS will detect when the gateway loses registration to CUCM. When this happens, a series of commands are ran on the gateway that essentially strips out the layer 3 binding on the PRI logical interface (e.g. 0/1/0:23). When connectivity to CUCM is restored, the same process puts things back the way they were.

What happened in my case was that I shut down the VMs, which meant I shut down my CUCM cluster nodes. This caused by gateway to transition to MGCP fallback mode. Later, I saved configurations and shutdown the gateway. Big Woops! Actually, more like a medium or medium-low woops.

OK, this isn't really all that interesting. The ccm-manager fallback-mgcp command was doing its job. What was interesting, to me anyway, is how that command was placed in the configuration. Usually, when I build MGCP gateways in production, I build the configuration files in advance and just paste them into the gateway. I have that kind of time to make sure I am only putting in the commands I want. If I have a SRST configuration, I will add ccm-manager fallback-mgcp. If I don't have a SRST config, I don't add in the command.

For the IE lab prep, I am working on speed so I am using the ccm-manager config server and ccm-manager config commands. That is where I got snagged. The configuration pushed down from the server must have added the ccm-manager fallback-mgcp command. Well, not much I can say about that. It is what it is. Now I know. When I encounter this issue in the future, instead of claiming ignorance, I can claim forgetfulness.

Note: Since I wrote this blog, I have had a hard time reproducing the issue. The behavior I observed was consistently reproducible at the time on a different IOS than I currently have in the lab. I was running 12.4.20T and now run 12.4.20T4 and 12.4.24T. 

Conclusion


The obvious conclusion (for me, anyway) is that details matter. Fortunately, I only spent 5 minutes figuring out and resolving the issue. Basically, if you have a MGCP gateway in your lab and you are using the default configs generated by ccm-manager config, save your router configs before you shut down your VMs (or CUCM servers, anyway). You could also remove the ccm-manager fallback-mgcp command, but only do this if your router isn't already configured for SRST.

I suppose this could have some real world implications for people who do MGCP in production. If you have MGCP gateways in production and you aren't running SRST on those gateways, then ditch the ccm-manager fallback-mgcp command. Actually, I would also ditch the ccm-manager config commands. I already decided to do that a long time ago (actually, a bunch of 12.4T bugs convinced me that ccm-manager config was hell spawn). At least, that is my conclusion.

In the words of the man Rakim:  \\Back to the lab...\\



Thanks for reading. If you have time, post a comment!

5 comments:

  1. I'm confused about one thing. What CallManager MGCP Gateway setting pushes down the 'ccm-manager fallback-mgcp' command. I'm looking at an 8.0.3 version, and none of the settings refer to fallback.

    ReplyDelete
  2. Hey William,
    For the lab I would strongly advice you to stay well away from the ccm-manager config server and ccm-manager config commands. It will not save you any time, quite the contrary actually. Learn how to do it manually, with only the needed commands to get it up and running. That will save you time in the end.

    My 2 cent on the matter :)

    ReplyDelete
    Replies
    1. Roger,

      I totally appreciate the feedback and POV. I have pondered that question myself recently and am already leaning in the direction of handling it the same way I do H323.

      Thanks again for taking the time to comment.

      -Bill (@ucguerrilla)

      Delete
  3. hi
    the output of sh isdn stat on my gateway keeps going from tei assigned to awaiting then multiple frame and finally timer recovery and so on. it's not stable and no calls can be done.
    sh isdn statu
    Global ISDN Switchtype = primary-net5
    %Q.931 is backhauled to CCM MANAGER 0x0003 on DSL 0. Layer 3 output may not apply

    ISDN Serial0/0/0:15 interface
    dsl 0, interface ISDN Switchtype = primary-net5
    L2 Protocol = Q.921 0x0000 L3 Protocol(s) = CCM MANAGER 0x0003
    Layer 1 Status:
    ACTIVE
    Layer 2 Status:
    TEI = 0, Ces = 1, SAPI = 0, State = TIMER_RECOVERY
    Layer 3 Status:
    0 Active Layer 3 Call(s)
    Active dsl 0 CCBs = 0
    The Free Channel Mask: 0xFFFF7FFF
    Number of L2 Discards = 0, L2 Session ID = 27
    Total Allocated ISDN CCBs = 0
    term mon shows the following when i try to make a call
    *Oct 18 10:43:39.191: %ISDN-6-LAYER2DOWN: Layer 2 for Interface Se0/0/0:15, TEI 0 changed to down
    *Oct 18 10:43:47.191: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0/0/0:15, TEI 0 changed to up
    can u please help me troubleshooting this issue?

    ReplyDelete
  4. I have a doubt sir, if you can help me out...
    In my lab, i have a CUCM and two MGCP gatways..... one is local gateway and one is remote gateway. I configured Fallback-mgcp in remote gateway. Both the gateways are connected with PSTN and WAN.
    i) I want all the calls go through PSTN but it goes through WAN, in my network signalling and calling goes through WAN.
    ii) When i disconnect/ Connect Wan, the active calls disconnect as the my E1 re-register using Fallback-Mgcp, But I break down the connectivity between the CUCM and my Local MGCP gatway then the active calls dont disconnect. why that so? This local gatway is also having a E1 port which is registered to CUCM so it needs to go down? so what can be the issue??

    ReplyDelete