Installing ESXi onto a Cisco WAVE 594 WAN Optimisation Appliance

Installing ESXi on a Cisco Wide Area Virtualisation Engine Appliance

Why would you want to do this? No real reason, but we've been decommissioning some hardware, and it's pretty clear that Cisco WAVE Appliances are just a Compute Server, with some stuff like VGA Ports removed. Originally these Appliances were designed for CDN-like WanOp purposes, so they have extras like Cavium Crypto/Offload Cards onboard, and some SATA storage; so I thought I'd have a go at loading VMware ESXi Hypervisor onto them.

The box I have is a Cisco WAVE 594, with specifications as follows:

• Processor - Intel Xeon X3430 @ 2.4 GHz
• Memory - 8 GB DDR3 RAM
• Storage - 2x Hot-pluggable 500 GB SATA 7.2k Hard Drives
• Storage - 1x Internal 4 GB USB Flash Disk
• Network - 2x Intel 82574L 1 GbE Network Ports

The NICs are not detected by ESXi, even though they're on the VMware Hardware Compatibility List (HCL)

What have we got here, Captain?

Here's a few photos of what we've got to work with:

Inside, you'll notice an internal USB port, plugged into a 4 GB USB Flash Drive (by some company I've never heard of); outside, you'll notice I've plugged in a USB 3 Ethernet Adapter (that uses the Realtek RTL8152 Chipset).

Port-wise, all we have to play with is:

• 1x External USB Port
• 1x micro-USB Console Port
• 1x RJ45 Console Port (Serial Port)
• 2x RJ45 1 Gbps Network Ports

What you don't have is a VGA Port, or spare USB Port to plug a Keyboard into (as well as a USB Flash Disk for the ESXi HV/OS Volume), which will make it pretty hard to process the Next/Next/F11 sequence required to install ESXi.

Time to ask a friend

I was a bit flummoxed at this point, but handily a friend suggested that ESXi doesn't care about hardware changes after the fact - so I could stage all this by pre-installing ESXi onto the internal 4 GB USB Drive. Which is exactly what I did, so to do this, I:

1. Created a VMware Workstation (I know, it's a work machine - I'm normally a VirtualBox man) Virtual Machine called "USB Test" on my Laptop
   1. Allocate this at least 2x vCPUs with 2x Cores
   2. Allocate this at least 4 GB RAM
2. Followed this guide on How To USB Boot a VM in VMware Workstation 11
3. Downloaded ESXi 6.5.0 ISO from VMware vSphere Hypervisor (ESXi) 6.5
4. Inserted the 4 GB USB Drive
5. Opened Rufus Bootable USB Maker
6. Flashed VMware-VMvisor-Installer-6.5.0-4564106.x86_64.iso onto my 4 GB USB Drive
7. Booted my "USB Test" VM, which boots the 4 GB USB Drive
8. Followed the ESXi installation process and installed ESXi over the 4 GB USB Drive volume
9. Rebooted the "USB Test" VM, and attached a "Host-only" Network Adapter to it
10. Waited for ESXi to Boot, and receive a 192.168.85.x Host-only IP Address

Now I've got ESXi built onto the 4 GB USB, I need to tweak a few bits before I plug it into the Cisco WAVE 594. Using the Host-only NIC in VMware Workstation means I can locally navigate to https://192.168.85.x/ui/ on the same Laptop running VMware Workstation to jump onto ESXi vSphere and configure it ("Host-only" means it's a virtual network between just that VM and your Laptop's OS - Windows 7 for me - which sees it as a Virtual NIC).

Making it work without VGA

As well as any other ESXi settings - such as Hostname, vmk0 IP Address, Storage Volumes (although no point doing that until this is plugged into the Cisco WAVE 594 itself) - I'll need to tweak ESXi to output it's boot screen (VMware call this the Direct Console User Interface, or DCUI; I call it the "yellow and black ESXi boot screen", much catchier) somewhere other than VGA, as the WAVE 594 doesn't have a VGA Port.

Doing this is quite easy; what ends up happening is that a VGA-like output (i.e. the VMware DCUI) gets redirected to the Serial port, which in this case is the trusty old blue RJ45 Console port. To do this, follow the instructions on VMware's website Redirect the Direct Console to a Serial Port Using the vSphere Client:

1. Login to the vSphere HTML Client (i.e. https://192.168.85.x/ui/)
2. Click the Configuration tab
3. Click Host, then Advanced Settings
4. Search for parameter VMkernel.Boot.logPort
   1. Make sure it says default
5. Search for parameter VMkernel.Boot.gdbPort
   1. Make sure it says default
6. Search for VMkernel.Boot.tty2Port
   1. Set it to com1
7. Click OK

Job done, now we can simply insert the USB Drive into the internal USB slot, connect our trusty blue Console Cable and USB Adapter into the Console Port, and set PuTTY or Screen to 115200 Baud rate, and boot the Cisco WAVE, then wait for the ESXi Boot Messages and DCUI to flow...

If you want to see the WAVE BIOS boot messages, you'll have to set it to 9600 baud first, and then change it to 115200 when you get garbage characters on your screen output.

So close, but yet so far

Remember that note I wrote before, where VMware lie and say they support the Intel 82574L in their HCL? Well, they don't - and to save you time, they:

• Don't in ESXi 5.5
• Don't in ESXi 6.0
• Don't in ESXi 6.5
• Don't even when you mess around with custom and obsolete net1000e VIB driver packs

Now what, not much use having an ESXi Node with no Physical Networking on it! This is where the second brainwave clicks in; lets use that USB Ethernet Adapter we've got lying around! Luckily Jose Gomes has had exactly the same idea and created a lovely guide on using a USB Ethernet driver for ESXi 6.5 - so follow that. For me, this looked like:

1. Download the Driver VIB for the Realtek USB Adapter
2. Enable SSH Service in ESXi vSphere Web UI (the Service is called "tsm-ssh")
3. Use FileZilla to login as "root", and copy-paste the VIB to /tmp/
4. Follow VMware KB Article 2147650 to disable the newer USB Drivers
5. Install the custom Realtek VIB, from SSH this command should do it:
   1. esxcli software vib install -v /tmp/r8152-2.06.0-4_esxi65.vib
6. Reboot ESXi

Let's see what we get this time then, when we also plug our cheapo USB 3 Ethernet Adapter in to the front USB port (and ESXi 4 GB USB into the internal USB port):

Great Success!

There is a caveat here - I find that, on reboots, ESXi DCUI will uncheck the "Use vmnic32 for Management" box, so it won't be contactable from the Network/won't get a DHCP IP until you manually press F2 -> Login to DCUI -> Re-enable it, which isn't much use if it's remote and the power goes.

Apparently there's a fix for that here in Install ESXi on a server/laptop with only USB Ethernet with an aptly-named file called "weasel", but I've had stoat-all success in getting it to work, so it's a limitation I've just lived with.

As a side note, because we didn't run the interactive installer on ESXi while it was connected to the WAVE 594 Hardware, you'll need to manually use the ESXi Datastore -> Storage -> Adapter -> Delete Partition option to wipe the partitions of data on both the 2x 500 GB SATA Disks, and can then set them both up as "New Datastores", so they can be used to hold VMs as VMDK virtual hard drive files.

Here's a handy guide on How To Erase ESXi Disks With ESXi Host Client v3.

Have fun!