Monday, February 23, 2015

Sometimes You Have to Use the Backdoor: Using CentOS to Access Cisco UCOS

It is the middle of the night and you are in the midst of a change control when you run into a brick wall. The kinda wall that can ruin your entire weekend. At a minimum, you have added at least a few hours to the process and boy you are not happy about that. 

Sometimes you just need more access than "the man" wants to give you and you don't want to wait for some tech support engineer to get on the phone to do something you can damn well handle on your own. Yes, sometimes you have to reach into the unconventional pocket of your tool belt and break off a little somethin'-somethin'. This series provides the necessary tools to get access to the Cisco UCOS root file system so that you can get the job done. 

Background

I won't pretend that I am the first one to find a way to "hack" into the Cisco UCOS because I am most certainly not that dude. I am also not the first guy to blog about the process I am getting ready to present. What I am trying to do in this series is consolidate information and present some of the scenarios where one could apply said information. 


The Issues

The solution we are going to discuss can solve several issues and I plan to go through a handful of actual "real life" scenarios to demonstrate the same. However, I like to state the "business issue" before I get all self-righteous about a solution. So, the issue (or need) is that sometimes you need to get access under the hood to fix a problem with your CUCM, Unity Connection, UCCX, etc.. 

There are times when the UC application portal (e.g. CCMAdmin, CMPlatform) is inadequate and the limited UCOS shell falls short of addressing your woes. If you have spent any time in the driver seat of a Cisco UC deployment then I don't need to explain the kind of problems that warrant bigger guns. 


The Disclaimer

I have to include a disclaimer here because I don't want anyone whining to me about a broken Cisco UCM cluster in the event they do something to shoot themselves in the foot. The disclaimer is: don't shoot yourself in the foot! More to the point, Cisco does not in any way approve of the methods I am going to document in this entry. Further, I am not in anyway claiming that everything I am going to present will work in every scenario. If you follow the processes and/or methods provided herein then you, and you alone, take responsibility for any issues that may arise. 

Look at the bright side - if you save the day, you can also take all of the credit. Yay you!

Seriously, be careful. I have used these methods on production systems but MOST of the time I use them on lab systems or temporary staging clusters and I am much more cavalier about those systems. If you have a problem, I can't guarantee I can help you out of it. 


The Process

We are going to cover the following:
  • Using CentOS to access the root file system of a UCOS application host
  • Some scenarios that demonstrate why you would do such a thing (separate, follow up entries throughout the week)

Using CentOS

CentOS (Community ENTerprise Operating System) in a community-supported Linux distribution derived from sources that Red Hat provides for Red Hat Enterprise Linux (RHEL). CentOS aims to be functionally compatible with RHEL and, since Cisco's UCOS is based on RHEL, it is the perfect distro for our purposes. 


Downloading CentOS

To do our do, we are going to download ISO images for the purpose of mounting them on virtual machines (VMs) in ESXi (4x/5x). The ISO images we need are located here:

http://wiki.centos.org/Download

Now, you will need to pay attention to the distribution versions. That is pretty important, particularly since the latest releases of the Cisco UC applications leverage RHEL x64 architecture. So, if you try to use a CentOS distribution that is built for i386 architectures on a UCM 10.x system (for example) then you'll be a little disappointed.

I have used the following:
  • CentOS 5.10: I have used this for UCM 6x, 7x, 8x, 9x
  • CentOS 7.0.1406: I have used this for UCM 10.0, 10.5
Preparation

Using the CentOS ISO means that we have to shutdown the running UCOS VM and then boot from the ISO. So, the first step is to clear the room. Just kidding, the actual first step is to make sure you schedule a proper outage before you do anything. Then, during the outage, you will shut down your VM.

If you are doing this on a production system then I would give serious consideration to taking a snapshot if you are uncomfortable with the process. Most of the time, I find that tasks where I have had to use this method usually coincide with a lab or off-production staging process. So, I don't bother with snapshots in those instances. 

You will want to download the CentOS ISO and then upload it to a SAN, NAS, NFS share, or DAS as appropriate for your environment. Make sure you use a datastore that has been added to your ESXi environment.

Finally, you will want to modify the settings of your VM guest so that you can mount the CentOS ISO on boot. Depending on your environment, you may need to:

  • Set the DVD vHardware to use the datastore ISO
  • Select the option to "Connect at Power On"
  • Modify the VM guest bios boot order to prefer DVD over vHDD



Booting Up CentOS 5.x

Once you have your environment set up, use the following process:

1. Power on the VM guest.

2. You will be greeted with the CentOS splash screen:

3. Type in "linux rescue" (without quotes) at the boot: prompt.

4. You are prompted to choose a language, select English and click OK.

5. You are prompted to specify your keyboard type, select us and click OK (with language and keyboard, use whatever works for you, I have only tested en_us).

6. You are asked whether you want to enable network interfaces or not. If you have a need to pull files off of the UCOS host then I recommend enabling the network. For example, later in the week we are going to discuss how to download the private keys (for the purpose of decoding communications to/from the UC host) and it is much easier to SCP/SFTP the files from the host. If you select No then move to Step 7.

6a. If you are configuring the network, you will be prompted to configure "eth0", select Yes

6b. At the network configuration screen, disabled IPv6 and enable IPv4 then select OK

6c. When prompted to specify the IP configuration, choose "Manual" and specify a usable IP address before clicking OK

6d. Set the default gateway and click OK

7. You will receive a dialog where the Rescuer says it is going to attempt to find your Linux installation and mount the file system. Click on Continue.

8. If the Rescuer was successful, it will mount your Linux installation under 'mnt/sysimage' and it will provide some instructions on how to access the UCOS file system. If an error occurs then you will receive an error message and, most likely, it will be completely uninformative! Click on OK if all is well. If not then go ahead and shut down the VM and work your way over to Google to do some research.

9. If all is well then you will be at a command prompt (e.g. sh-3.2#). Type in the command: chroot /mnt/sysimage and hit enter. The command prompt may change (to sh-<ver># or bash-<ver>#).

10. Use the command ls /mnt/sysimage/ to see if your UCOS file system has been mounted. If you see content in the output then you are in business.


Booting Up CentOS 7.x

If you are using RHEL with a x64 architecture then you will want to use CentOS 7x. The boot up process is different for this CentOS version. Once you have your environment set up, use the following process:

1. Power on the VM guest.

2. You will get the CentOS 7 boot menu. Be quick like a bunny here because the boot menu will time out (unlike 5x).

3. Select the "Troubleshooting" menu option and hit Enter on your keyboard.

4. Select the "Rescue a CentOS System" from the Troubleshooting menu and hit Enter.

There will be a pause and the system will ask you to hit Enter to start the installation DO NOT HIT Enter!! Just wait for the Rescue Dialog (step 5) to display.

5. You will receive a dialog where the Rescuer says it is going to attempt to find your Linux installation and mount the file system. Click on Continue.

6. If the Rescuer was successful, it will mount your Linux installation under 'mnt/sysimage' and it will provide some instructions on how to access the UCOS file system. If an error occurs then you will receive an error message and, most likely, it will be completely uninformative! Click on OK if all is well. 

7. At this point, you will be at a command prompt (e.g. sh-3.2#). Type in the command: chroot /mnt/sysimage and hit enter. The command prompt may change (to sh-<ver># or bash-<ver>#).



Where Do We Go From Here

There are several reasons that one would need to use CentOS to "hack" into the UCOS system. I wanted to provide a few examples but the scenarios will vary and change over time. The good news is that the issues that cause you to go down this road are uncommon. Unless, of course, you are an integrator then you may have to do this every couple of installs. 


To keep these blog entries from getting too long, I am going to provide the individual use cases over the course of the week. Hopefully, that isn't too annoying for anyone! I'll update the links as the articles are published.

Use Case #1: Modifying the License Mac on UCM

Use Case #2: License Expiry Issues during Jump Upgrade Process

Use Case #3: TFTP Custom Ring Tone Issues

Use Case #4: Fixing Errors with Hunt List Queuing Announcements

Use Case #5: Downloading the Tomcat Cert Private Key


Thanks for reading. If you have time, post a comment!

1 comment: