Thursday, August 15, 2013

CUCM 9.1(2) and the Jump Upgrade

As we move further down the road with Cisco UCM releases we are gifted with more and more "upgrade" variants. First came the Bridge Upgrade, followed by the Refresh Upgrade and it's counterpart the L2 Upgrade. Now we have the Jump Upgrade. These "Upgrades" are breeding like crazy but I guess variety is the spice of life. 

All of these upgrade methods serve a function and while I think Cisco's upgrade methodologies are overly clunky, the Jump Upgrade is a step (not a leap) in the right direction.

Background

Let's define some of these upgrade options, shall we?

L2 Upgrade

I am starting with this one because I associate it to the OG upgrade method for the appliance models (CUCM 5.x and later). The "L2" name is short for "Linux to Linux" upgrade. This upgrade method installs a new version of the UC application to the inactive partition on a UCOS (UC Operating System) system. It has minimal impact on system availability because the upgrades are done in the background and a reboot can be scheduled in the future. 

The L2 upgrade methodology is applicable to most upgrade paths. It is leveraged for any service release or minor update and can be used for certain major revision upgrades. For instance, upgrading from 6.1(3) to 7.1(3) or from 8.6(1) to 9.1(1) all fall under the L2 Upgrade classification.

Refresh Upgrade

If memory serves, I didn't start hearing the "L2 Upgrade" terminology until the "Refresh Upgrade" came into being. The Refresh Upgrade covers situations where direct upgrades to the inactive partition are disallowed. The primary reason? A change in the underlying Red Hat kernel used by the "source" and "target" versions. This upgrade method came as a bonus with the CUCM 8.6(1) release. Which marked a transition to Red Hat release 5. 

With this upgrade methodology, you have to install a special COP file on your system prior to executing the UC application install. You also lose out on the option to schedule a reboot to the new UC application at a later time. This is due to the fact that the system needs to reboot, update the core OS, and then proceed with the normal UC application load. Another benefit [!] of this upgrade method is an additional 2 - 4+ hours where you can twiddle your thumbs while you wait for the system to upgrade. 

Bridge Upgrade

On a linear time line, the Bridge Upgrade methodology actually came before the Refresh Upgrade. This migration path is for customers who need to upgrade their CUCM solution but are currently running hardware that isn't supported with the target application version. The idea is that you can get the software upgraded and use DRS to archive your configuration. Meanwhile, you build a new cluster and bring it up to your target version. Once all of the dust settles, you use DRS to restore (or migrate) the old system.

If you follow the "text book" version of this upgrade. You need to perform the upgrade operations on each and every cluster node. Which is a huge PITA. More on that in a bit.

Jump Upgrade

With CUCM 9.1(2) (release July 29th or thereabouts) introduced a new upgrade methodology called the "Jump Upgrade". The primary objectives of this methodology is to provide customers with an upgrade path from pre-8x versions that minimizes the impact to production systems and reduces the number of "hops" needed to get from source to target versions. Boiled down to simple terms, the goal is to lower overall risk to production users.

The process goes like this:



  1. You need to get your production system to one of the following versions: 6.1.4, 6.1.5, 7.1.3, or 7.1.5  (process not shown above)
  2. Perform a DRS backup of your production system
  3. In VMware, build out a new cluster (all nodes) at the base version you are running in production
  4. Use DRS to restore the production backup -- now you have a copy of your production system
  5. Upgrade the source version cluster directly to 9.1.2 [You need to load a .COP file to execute a Refresh Upgrade first (not shown)]
  6. Use DRS to backup your interim 9.1.2 cluster
  7. Build a 9.1.2 cluster (from scratch) in VMware
  8. Use DRS to restore the interim 9.1.2 cluster files
  9. Now, do some migration magic and swing phones, apps, etc. from the source version to the target version
  10. Validate
  11. Go have a beer (or 10)

As you can probably deduce from the process steps, this upgrade method isn't necessarily "quick" but it does achieve its objective of minimizing impact to production users. Coming up with methods to optimize the process by doing things in parallel will help the time factor. It is still a black hole in regards to overall time. 

The upside? You can do most of this during normal working hours. So, stress is lower and I for one engineer all of my upgrade processes to reduce risk and stress. The latter can lead to bad decision making and, therefore, have a direct impact on the former.

To pull this upgrade off you do need to make sure you address several pre-requisites:


  • Media (in the form of .iso images). You want to minimize steps to build the interim clusters in vmware. So, get hold of the media. Cisco provides a way to do this: http://tools.cisco.com/gct/Upgrade/jsp/index.jsp
  • System integration dependencies. You need to segregate the VMware environment you are using to do the Jump Upgrade and you also need to account for the IP addresses of your system, NTP, DNS, and related services. IOW, you need to emulate your production environment (to a degree)
  • Licensing. You need to follow all of the Cisco guidance related to migrating licenses to CUCM 9.1 and the Enterprise License Manager (ELM)

To read more about the upgrade, I would check out the documentation that is posted here:

https://supportforums.cisco.com/docs/DOC-35163 

Additional Thoughts

In reality, many of us have already been employing a methodology that is very similar to the Jump Upgrade approach. Particularly when a customer is migrating to a virtualized UC environment. I can't speak for everyone, but in my approach I would focus on carrying forward just the publisher node to get at that precious DRS file. (note, there are trade offs with this this approach)

This method of just upgrading the publisher to get the new schema is not supported by Cisco. Consultants (like me) had to develop our own method because, well the Bridge Upgrade method is ridiculously inefficient. For that matter, any upgrade method where you have to drag the whole freakin' cluster forward just gulls me. 

While I am planning to use the Jump Upgrade and I will follow the "text book" approach, I don't like process inefficiencies. That said, the Jump Upgrade does deliver on its core objective and the idea of reducing overall risk is priority number one with anything I do. So, I can't knock it too much.

What would I like to see? Well, I'd like about a month off from "regular work" so that I can take some of the tools I have developed to streamline upgrades to the next level. Seeing that this is an unlikely scenario, I'll take a COBRAS-like solution for CUCM.




Thanks for reading. If you have time, post a comment!

22 comments:

  1. Can you elaborate on the tradeoffs you mentioned with the publisher upgrade approach? I haven't done that yet and since it's not an official process I haven't seen it documented.

    thanks,
    will

    ReplyDelete
  2. Will,

    No problem. I may incorporate this into the main blog but to avoid making you wait, here is a summary.

    1. MOH files. Any MOH files that you have uploaded to IPVMS nodes in the cluster need to be manually uploaded again. You have to be mindful of file names and you may need to retrieve originals from existing UCM nodes.

    2. TFTP files (backgrounds, ringtones, etc.). Basically, any custom file that you have uploaded to the TFTP nodes in the cluster would need to be manually dealt with.

    3. Certs (CTL). Mixed mode clusters. The DRS carries with it the various certs in the cluster. If you don't restore the subs then you don't get their certs.

    4. Certs (ITL). Security By Default. Same consideration as above.

    5. I have heard mention of the fact that when you add a new subscriber a new unique ID is created. Even if the IP addr and name is the same as in source version. This may result in some "ghosts in the shell". I recently read something on this (in a forum) and haven't been able to confirm. I have checked my lab systems and I see no such "ghosts". So, jury is still out (IMO) on this one.

    I think that sums it up.

    -Bill

    ReplyDelete
    Replies
    1. Why would you need to build out another 9.1 cluster? Could you not stop at number 5 and go live on that cluster?

      Delete
    2. I assume you are referring to the original article. After completing step 5, the VM guest OS is running on an unaligned disk partition. The reason being that the target source versions in the Jump Upgrade process are not optimized for virtualization.

      Running with unaligned disk partitions is not recommended due the impact on latency and I/O.

      HTH.
      -Bill (@ucguerrilla)

      Delete
  3. They actually covered the restore/rebuild the subscriber in today's VoE about jump upgrades. If you are a partner go check it out. Dan Keller gave a thorough response, which was essentially Bill's 1-4, plus some more info on #5. Apparently, if the new subscriber is the same IP/hostname, the feature services can come up as activated but won't actually run. They have to be deactivated and reactivated before they actually run because of some sort of markings somewhere. Also any trace file configurations are no longer kept. I think that was everything. They are also working on getting the steps for this documented to make it supported.

    ReplyDelete
  4. Good to know. Will check out the VoE recording once I get some cycles.

    Thanks again!

    -Bill (@ucguerrilla)

    ReplyDelete
  5. Great article Bill... TBH I think there is still a long way (hopefully in 10.x) to go before this lengthy process is truly commercially viable; the killer step for me is tearing the upgraded UCS 9.1.2 for a fresh install to get the DRS restore on (imagine rebuilding that 8 x Node cluster). Pretty sure we will just carry on grinding out upgrades using diligince on the existing sub/tftp/moh nodes and a single Pub Bridged upgrade..... though it looks like 9.1.2 isnt on the MCS Compatability matrix even for Bridged.

    ReplyDelete
  6. NICE WORK!!! Cisco notes for the upgrade are not very clear...wish you were able to post step by step instructions for upgrading cluster ENV. from 8.6 to 9.1 CUCM!!

    ReplyDelete
  7. We've been using this upgrade method since for the last two years. We have successfully upgraded CUCM from 6, and 7 versions to 8.5 and higher. Most of the time it includes changing the IP address of the CUCM cluster because you can't have two production systems running with the same IP's. Does Cisco now support running UC version 6 and 7 on VMware?

    ReplyDelete
    Replies
    1. No, Cisco does not support running UCM 6x/7x in vmware. What they support is a direct upgrade from specific versions of 6x/7x instead of forcing customers to follow a multi-step upgrade. For the purposes of the Jump Upgrade methodology, you can load the specific 6x/7x versions in vmware.

      As you noted, this is not a "new" way of thinking. I have been doing upgrades via vmware for a while, myself. What is new is that Cisco has provided a Tac supported process.

      HTH.

      -Bill

      Delete
    2. Well the main new thing here is correcting the licensing issue if you've ever seen the "Upgrades are prohibited during license grace period" message. Before, it would take licensing to issue a new license on your 6.x/7.x VM in order to proceed with the 8.x/9.x upgrade or to have TAC do a workaround via root (or do yourself via root). The main benefit of the new supported jump upgrade process is the refresh upgrade v1.3 Cop file which will allow you to bypass the licensing check in order to compete the upgrade.

      Delete
  8. Hey Bill.....always great to read your blog! Although not perfect, I do agree 100% that the Jump upgrade is a big step forward. I'd rather skip the upgrade altogether though and just go for the 10 beers with ya ;-)

    Cheers!
    Huff

    ReplyDelete
  9. here is my question: If I do have mcs pub and sub ver 8.0.3, can I just build 8.0.3 in VMware then load cop v1.3 then restore mcs pub backup to VMware then build fresh sub on VMware and let it replicate from pub rather than restoring original sub from mcs server

    ReplyDelete
  10. To :anonymous

    Yes you can but you will then need to rehost your license to your new Vmware Mac License address.
    Also like posted here above take care of all extra files are loaded to the new subscriber ( i.e. MOH, and such )

    ReplyDelete
  11. This is good stuff. However I"m running into a situation where I have to change the Hostname, and the IP's of the PUB/SUB during the upgrade process. I'm hoping that I don't get any database corruption. I have the PUB built with the old IP to get DRS to work properly. My issues is now I have to change the IP of the PUB to the new IP address that the customer gave me. All the while, I still have the old SUB IP address, and will be adding a new SUB server to the mix. I'm hoping I can escape this without breaking the database.

    ReplyDelete
    Replies
    1. If you are doing the jump upgrade then the recommendation is to stage the entire cluster in the virtual environment to do the upgrade. This is how I do it. I build a staging cluster in a separate staging environment (w/ DNS, NTP, etc. - enough to get through the install checks). Build the staging cluster with the same IP addresses/ hostnames.

      Then I do the DRS restore and run the jump upgrade process. Once the upgrade is complete, I change the IP address/hostname to the new settings. Do a DRS backup. In parallel, I have already built the new CUCM cluster with the correct IP addresses (changed addresses) and names. I will restore the last DRS backup and then expand the cluster.

      That is glossing over some details, but hopefully you get the gist.

      HTH

      -Bill (@ucguerrilla)

      Delete
  12. Thanks for the advice. I'll get you a round if we run into each other a live. It shouldn't be hard to recognize you.


    Thanks



    Terrence

    ReplyDelete
  13. Great post Bill. Thanks for saving me so much time reading numerous guides to get this infromation. Now for the 40 hour upgrade for my customer !!! As you say at least we can complete this in a lab environment.

    ReplyDelete
  14. Bill do you know of any way of doing the entire process in VMware. I have a 6.1.1 cluster that restores into VMware, but I can not patch to 6.1.3, 6.1.4 for the jump start.
    The issue isn't license or machine spec. the upgrade seems fails after the database transfer.

    Frank

    ReplyDelete
    Replies
    1. Frank,

      I haven't been able to successfully perform upgrades in VMware for pre-VM versions of CUCM. Well, I have been able to do it but not consistently. It fails more often than not.

      A while ago I just decided to use the MCS servers we have in our lab to stage this type of upgrade. You may not have that option. But, because I had that option I didn't need to work out the formula for getting a 6.1.1 system to 6.1.3.

      Maybe someone who comes across this thread has worked it out.

      HTH.

      -Bill (@ucguerrilla)

      Delete
  15. Great article, This helped me to migrate my 8.x CUCM environment to 9.1.2
    I have documented the notes and practices i considered during my upgrade http://letusexplain.blogspot.com/2015/08/cucm-upgrade-guide-8x-to-912.html

    ReplyDelete