Migrate vSphere 4.1 to new host and fresher infra - follow up

Recently I moved an oldish 4.1 environment to a new base of operations. The process was fairly straight forward, but a few minor things I think are worth mentioning as a follow up.

Update Manager

So I did not disable the old Update Manager installation before moving the entire thing to a new host. I had already decided to go without UM for the new setup and had not spared a thought on the consequences. Furthermore when I had originally set up the old environment, I had failed to follow the best practices and had used IP addresses instead of FQDNs throughout the setup.

The result was a connection error because VCenter was continuously trying to connect to the old Update Manager installation. Searching around I could not find a good way to remove the UM binding so I decided to walk "The Windows Walk", installed UM to overwrite the previous registration in the process and uninstalled it properly afterwards.

Connection error resolved.

In retrospect what I could have done is to enable the old UM service again, let vCenter connect to it and then uninstall it properly.

Performance Statistic Rollup

The other thing that sticks out when I check vCenter Service Status is this message:

Performance Statistics rollup from Past xxx to Past xxx is not occurring in the database

In my specific setup the notification claims roll ups are not available for the following durations:

- Previous day to previous week
- Previous week to previous month
- Previous month to previous year

VMware KB2015763 describes this issue for 5.x installations and furthermore points out to enable statistic rollups in Administration > vCenter Server Settings > Statistics.

Regardless of whether I use vSphere Operations Manager to accumulate the data now or not I am not happy about these warnings, even though they may not affect operations as such. As you may guess historical performance data is not available in vSphere for now.

Both the 5 minutes and 30 minutes interval duration roll ups were already enabled, I added 2 hours and 1 day intervals as well, to no effect.

Digging around some more I find a few possible reasons and explanations for the behaviour:

1. Using SQL Express Edition the VPXD's internal scheduler handles statistic roll ups. My installation uses an Express Edition that was not bundled with the vSphere installer. Also, if you are using a full blown SQL server and the SQL Agent is not running, the installer supposedly reminds you to start the service, which in this case it did not.
2. Another issue might be KB1030819. I followed the instructions inscribed, as the datatype was reported to be "numeric" instead of "bigint". 

After some rather tedious mucking around and trying to work out a way on how to automate statistic rollups, running them by hand a few times in the past few days I have decided to migrate the environment to a full blown SQL 2008 server. I found out that our dev team has a fully licensed SQL server running that I may utilize. Our vSphere environemtn is small enough so that no mutual performance impacts are to be expected. Gladly this migration will be very easy and straight forward again and will pave the way to move vCenter itself back to its original home base.

Cannot delete Portgroup - works as designed

This morning when I start work I notice all those emails about a new ticket in our trouble ticket system. The owner added me among other colleagues as a monitor. The issue is the following:

There are numerous virtual port groups that do not have any VMs connected (after a lengthy and mostly automated migration to a new naming scheme) that cannot be deleted. They show up greyed out in the vSphere client and if you drill down each of those port groups will show its associated VMs and templates. If you check the config of any affected VM they will show the new port groups only, the summary pane will furthermore list the greyed out old port groups. The proposed problem solution is a restart of the vCenter service, as there is an OS update pending anyway.

Cause

As I have seen this behaviour before on several, albeit very rare occasions, I wanted to take my chances and investigate. I went ahead to the first affected port group and found a template and a running VM associated with it. The template was an easy and logical case. My colleagues had migrated all VMs but had failed to migrate the templates yet. Convert it to a VM, change the port group association and away it went. Convert back to template and everything is fine.

The running VM however is a different case. Its settings showed no binding to the old port group. Because its a customer's system I cannot just go and change things around wildly. I had a closer look and noticed an active snapshot. We have a policy that snapshots may not be kept longer than one week. However that policy is not being enforced by any automatism or audit trail. The Tasks & Events pane in my vSphere client was not able to tell me when the snapshot had been created...it was already beginning to mold and smell unpleasently. Same goes for the other affected VMs.

And it makes perfect sense. If I want to go back to my original point in time - when I took the snapshot - I expect the VM to be in the same network (thus virtual port group). Thinking I might be able to get a list of VMs and port groups via a simple PowerCLI script I went to work and came up with this slightly ugly code:

$Portgroups = Get-Datacenter $myDC | Get-VirtualPortGroup | Where {$_.Name -like $myPgFilter}
$PGNames = @()
foreach($PG in $Portgroups) {
    $PGNames += $PG.Name
}
$VM_with_Snapshots = @()

$VMs = Get-VM
$Snapshots = $VM | Get-Snapshot

foreach($SS in $Snapshots) {
    $NAs = $SS.VM.NetworkAdapters
    foreach($NetAdd in $NAs) {
        if($PGNames -contains $NetAdd.NetworkName) {
            $VM_with_Snapshots += $SS.VM
        }
    }
}

I'm sure there is massive potential for optimization, I'm not a coder nor do I have much practise at the moment (things are about to change in the near future though). However this approach did not yield the expected result as this script will not return the previous (and still caught in a snapshot) port group assignment, but the current one. Pointers towards the proper results would be greatly appreciated.

Among the affected VMs there are, however, some test systems and infrastructure systems as well that I have sufficient control, knowledge and privileges to test my snapshot theory. And of course, I was right.

I now want to refer to this very recent (re-)tweet and extend it to VCPs as well! :)

RT @josh_odgers: @grantorchard Business as usual, the VCDX comes in and cleans up the mess and ensures customer satisfaction :) > +1
— Michael Webster (@vcdxnz001) May 15, 2013

HP vPV - my own take

HP's Virtualization Performance Viewer, in short vPV, has been stirring up some dust in the past few weeks. I first learned about it from my good friend Amitabh, a passionate HP Infrastructure and Virtualization engineer with, what it seems to be a very broad view on current and up and coming technologies. When I first read his post I had just started playing around (and quite enjoying myself doing so too) with VMware's Operations Manager after hearing about it on several occasions such as the 1st Singapore VMUG meeting of 2013. I dug around some and setup a demo installation to let vPV and Operations Manager go head to head, talked to one of the managers at my employer and just got to know what it is and what it does.

I held a small presentation to some of my colleagues including the vPV aware manager at my company's headquarters recently. We had a look at both products side by side and talked about first impressions and ideas.

Just minutes ago I read Ben's take on vPV which prompted me to say a few words about the product and my experiences with it as well.

vPV - The ugly truth 

 

To understand, what vPV is, I think its quite helpful to know how it came to be. The afore mentioned manager had a chance to talk to one of the HP guys at the last GPC in Las Vegas and revealed an interesting detail: HP's vPV was initially developed as a helper for their internal operations, more or less "by accident". They realized the potential and showed it off, got a few interested visitors and decided to release it as a product.

vPV - What it does!

 

As both Amitabh and Ben pointed out the installation is dead simple. Download and deploy the virtual appliance, point it at your vCenter and you're ready to roll. You get an instant and, depending on your environment, rather colorful picture.

vPV supports Hyper-V (though not tested by any of the three of us, as far as I can tell) and vSphere. It offers a general overview and drill down capabilities and makes good use of HP's uCMDB to visualize the structures of your environment and qualities and properties of your managed objects. It allows you to access the very same real time metrics that can be found in the vSphere client, with the added bonus of 24 hours retention (for the free edition, 30 days for the licensed product) as opposed to just 1h. You get every last value (including the infamous CPU Co-Stop) in a cute little graph, can arrange them to correlate issues in the workbench or just rely on the dashboard to get the grand total.

To me its easier, and as Ben pointed out, a lot snappier than using vSphere client to access performance graphs of your vSphere environment. A major benefit is the ability to easily place a half dozen of graphs on one screen.

vPV - What it does not!

 

Same as the vSphere client itself, strictly speaking, its not a real time performance analysis tool (or rather visualizer). vPV retrieves its data from vCenter. vCenter retrieves its data from the connected ESXi hosts, which in turn retrieve data from the VMware tools in each VM and measure the individual VM's performance metrics from the hypervisor side. This chain alone adds multiple delays in itself, the overhead of drawing a fancy graph (and possibly using a lot of RMS to connected the dots without making the graphs bounce up and down like crazy) is not even included in that one yet. In order to visualize the data and make it humanly processable true real time is out of the question, as with most other tools and helpers in this field. You rather get the past 24 hours of everything that was going on, in nicely smoothed lines.

Unlike vCOPS, vPV does not process the data to generate, what I like to call a "management compatible dashboard". vCOPS, even the free edition, stands out for its health level visualization. It will correlate CPU ready times along the number of assigned vCPUs to a host's workload and generate dynamic thresholds where vPV will only show the individual values. Thus vPV will also not free you from analysing dozens of metrics on multiple objects to find the true source of an ongoing performance issue.

On the side there is one major issue with the free edition. It does not support any means of access control. By default it is open to anyone and everyone within your network. As it allows you to view the entire vSphere environment (up to 200 managed objects in the free edition, of course) it surpasses your vSphere permissions and provides an interesting source of information for the sneaky attacker in your own network. So be sure to at least work out some sort of .htaccess protection.

Unlike vCOPS, it does support Hyper-V in the free edition. However to my understanding the paid version does not integrate AWS (and potentially other Cloud vendors and OS'ses) at the moment.

At the end of the day ...

 

vPV is a great if you're serious about vSphere performance analysis and trouble shooting and a good helper when vSphere client itself does not provide the data you're looking for. I agree with Ben's resume that its a great addition to your preferred set of tools and being free for 200 managed objects should be on any aspiring VMware admin's list of favourites.

HP has done a great job in putting their modules together to aggregate already available data and visualize it in an attractive manner. However I have my doubts that this tool alone helps "to rapidly analyze bottlenecks", as they put it. In my very humble opinion you have to already know what you're looking at (and for) to make good use of vPV.

In that sense it is no real competition to vCOPS, it simply plays in a different field. Integrate it with BSM and I'm sure a trained and able HP consultant will be able to generate reports and dashboards that will blow your mind away. If you're looking for an all-in-one multi purpose solution, vCOPS might be the better choice as vPV itself may evolve somewhat, but as a stand alone tool it can and will only be one out of many to assist you in your virtual performance trouble shooting journey. It sure should come in handy when you train yourself to be a VCAP.

Migrate vSphere 4.1 to new host and fresher infra

For the past few weeks one of my vSphere installations has been seeing some major problems, that I have not been able to drill down into properly. As I set it up a few years ago and then handed it over to one of my coworkers for management it became over utilized, poorly managed and started acting weird. Recently it has been crashing every three days or so due to the transaction log file filling up. The infrastructure (Essentials License) runs on SQL Express 2005 (the one bundled with vSphere 4.1U2), the DB runs at its default settings (simple recovery model, one transaction log file, auto grow, max size 2GB) and there is a poorly configured backup (my bad, I was a bit of a greenhorn back then). Now what happens is a transaction is running on the DB that fills up the log file and then crashes the VPXD. Not ideal.

Because there are a few other things that bother me about this environment, I figured its about time I moved it to a separated VM and reinstalled it properly, leveraging all the experience of the past 3 years or so and taking into account the best practices of my vSphere Install, Configure and Manage course and certification last year.

The new setup

Firstly I rolled out a Windows 2008 R2 server, 64bit, configured hostname, dns domain, added it to our Samba domain (remember to configure this), installed all available updates and gave it a few reboots for good measure. Once the box was up and running smoothly I installed a fresh SQL Express 2008 with Management Tools from here (I actually installed a few other versions before that only to constantly find compatibility issues with W2K8R2 and so forth, but this one is fully supported). Once that was out of the way I imported an older backup of the production database, all went smoothly.

Next step was to create a data source. Following the documentation (which I did not need for this project, I might say so proudly) I created a System DSN that points to the newly created and populated database and tested it. The appropriate SQL Client libs has already been installed along with the SQL Server Express Edition. Keep in mind for a client server setup you need to install a supported SQL Client, the OS's preinstalled client has a beard longer than Santa Clause... well, you get the point. The Express Edition of SQL 2008 R2 seems to have TCP/IP enabled by default, a thing I seem to remember is different on the full blown servers. Also 2008 does install an SQL agent but don't bother trying to activate it, it will not work, very confusing.

With SQL Express and the data source working smoothly it was time to install the VCenter. At this point I was doing merely a test installation, so I would have to keep in mind to be cautious as to not have it connect to my ESX-hosts right away. The installation is pretty straight forward. Just choose your precreated dsn, let the installer use the existing database instead of wiping it and choose to manually connect your hosts and update the agents. I found, however, that the installation should be done as local admin, not domain admin. This might have something to do with our Samba domain, might be related to something else or might even be in the documentation. It did cost me a few hairs but eventually it wasn't anything that could not be conquered.

Once the installation was finished, I connected to the newly created vCenter and found every thing in place, hosts, vms, folder structures, resource pools, permissions and so on and so forth. Tests finished, ready to roll.

The day of the migration

During off hours this morning (a 6 hour time shift can be very helpful sometimes) I went away, stopped the production and newly created VPXDs, web services and so on, created a new backup of the database and moved it to the newly created environment. Confidently after restoring the database, I wanted to start the VPXD, only to find it crashing right away. What went wrong?

With the newly created installation I had also used the latest (and greatest?) release of 4.1, Update 3. The database of U3 is not compatible to U2, naturally. There might be a quicker way to solve this, I opted for a quick reinstallation of vSphere as this would update the database accordingly. After all, I had no reason to expect failure and, in the unlikely event something did actually go wrong, I could still fail back to the old and tested (and buggy) environment. Everything went as expected though.

Once the setup was finished I connected to the vCenter, reconnected my three hosts and found everything to be working just like it should. Playing around with the vSphere client I noticed only a few things to be off.

1. vSphere Service Status display was not working
2. Two of the plugins available for installation are not working as expected.

I had an idea what these issues might be related to. The old setup was not based on FQDNs, but everything was hardwired to IP addresses. The Service Status module would thus try to connect to my old environment, which has been shut
down. This post however quickly resolved issue number one. Just replace the mentioned variables' values with the correct ones, restart vCenter for good measure and reconnect with vSphere client. Now the status is working.

As for the plugins, one is the converter plugin. I tried installing through vSphere client and it pointed to my old IP address. Just install converter on your vSphere host and it will prompt connection data for the new environment thus registering and the installation properly. The second plugin is vcIntegrity, which as far as I can tell, is Update Manager related. I could just go and install the Update Manager on my new box, but I don't want that. For such a small environment, I opt to manage updates manually in the future, so I will have to have a look into how to fix that minor issue (is it really minor?). vcIntegrity also shows up as an error in vSphere Service Status.

Cleanup

I have a Operations Manager Appliance running, which wouldn't reconnect to the new environment. Running very short on time and really treating it as a fire and forget I didn't know how to access the admin UI. I guess I would have been able to rerun the setup wizard, I just redeployed the appliance. The HP Virtualization Performance Viewer however had already been using an FQDN and moving it to the new environment works seamlessly.

I'm happy to see that the migration was rather simple with very little caveats. Log file usage so far is good, will have to wait and see how it develops in the days to come. The DB itself is rather resilient, I found that it has self repairing capabilities recently when I was trying to work around the issues I had. All in all I'm quite happy and think I can go on vacation tonight and have sound nights of sleep without having to worry about a crashing vSphere environment any more.

ZFS

Very short article on brief ZFS testing. Initially I was running a Debian VM presenting a single lun as iSCSI target to my test host. Running on an ageing laptop the performance was not very good naturally. I then had a vision. What if I could stripe the traffic over multiple devices?

I have two fairly new USB drives lying around here, that would do the trick. I created two additional virtual disks, each on one of the USB drives, attached them to the Debian VM, setup ZFS and created a RAID0 pool striping across all three drives. A first dd gave me some promising results, I was writing zeroes at roughly 130MB/s. I'm not too familiar with ZFS and didn't want to waste any time reading a lot, so I just created a 400GB file using the above mentioned dd-method and exported it to my host as a iSCSI lun. Performance in the VMs however was not very good, IOMeter was seeing 80 to max 200 iops (80% sequential at 4kb). For a comparison I created a 1GB ramdisk and attached it to my test-vm as raw device. There I would see >3000 iops consistently. More so with the above described ZFS setup I would have serious issues when trying to power on a few VMs. The vCenter client would time out a lot as VCVA wouldn't be able to handle its basic functions. At one point I saw a Load of 40 inside the VCVA, while I was installing a VM from an ISO, nothing overly complicated.

Even with a hugely underpowered lab like mine I figured there is still quite some tuning I could do. However time being an issue I opted for a preconfigured NAS appliance. A quick look at Google made it pretty clear, I would use Nexenta Community Edition. I figured ZFS coming from Solaris, I would better use a Solaris based appliance rather than FreeNAS (for which my heart beats though as it's based on FreeBSD). So far what I'm seeing looks promising. I configured the ZFS pool and iSCSI target slightly different from my earlier deployment, using three virtual disks spread across my two USB drives and one internal HD at 100GB, created a pool and thus far only exported two 100GB luns out of the iSCSI target.

On the VMware side I created a SDRS cluster. No other reason than "Because I can".

Given the very low specs of my lab I'm quite happy with the results. IOMeter specs are 80% random IO at 4kb, 2/3 reading, 1/3 writing. At 100% sequential read is pretty consistent around 800 iops it peaks at a little over 900 iops. Heavy cache usage can be seen at 100% sequential writes, where IOmeter just bounces around from 13 to close to 2000 iops.

Unfortunately this setup is still not quite capable of handling more than one VM, especially when it comes to swapping.

Lab specs:

T61 laptop with C2D T8300 @ 2.4GHZ CPU, 3GB Ram, Nexenta CE VM has been assigned with 1.5GB (nowhere near enough for proper ZFS testing), internal drive WD1600BEVS-08RST2, externals are WD Elements 1023 and 10A2.

It runs Win7Pro64bit, VMware Workstation 9, there are no VMware tools installed in the NAS appliance (I assume that might kick things up a little more).

T400 running ESXi straight from a thumb drive, C2D P8700 @ 2.53GHZ, 8GB RAM.

Both laptops are connected directly via cross ethernet cable, the link's bandwidth is 1GBit/s.

Circular dependencies from hell - a poor man's lab part 2

So, after the embarrassing discovery, that I was trying to allocate and already taken IP address to the VCVA I dropped all W2K8 installation attempts (which went quite well, don't get me wrong, despite the fact that I was also having issues with the IP address, I really need to start writing things down) and deployed another VCVA. Everything went very smoothly, the VCVA is up and running and setting hostname, static IP and toggling SSL certificate recreation and a reboot everything is working smoothly now.

As I type this I am installing another Debian linux as a test VM to play around with Iometer. I may try to optimize the iSCSI link to utilize jumbo frames, but for now I will just leave everything as it is. I am very happy with the general performance of the VMs so far and gotta say this setup looks very promising, it may actually be a very capable lab setup.

Updates will follow tomorrow.

Circular dependencies from hell - a poor man's lab

Important update at the end of the article! It voids part of the article, will go back to deploying VCVA and then continue.

Recently I installed ESXi on my Thinkpads (1 and 2) using a USB thumb drive. The installation was very straight forward and great for a lab environment. However a host without a data store is no good and the internal drive is not to be messed with. The NAS in the living room is slow and only accessible via wifi (unless I want to carry the laptop around). Initially I just plugged in an old USB hdd to see if I could use it. Out of the box that doesn't work, most likely because the ESX does not support USB storage for data stores (USB can be passed through to VMs though, so it might be to prevent conflicts).

A virtual NAS might be a good way to make it all very portable and flexible I though. So I just grabbed the T61, installed Workstation 9 (and completed a Cloud Cred Task while I did so) and setup a Debian VM that would be my iSCSI target. The host has 3GB Ram, so I figured 2GB for the VM would be fair. That way its got a little bit of caching leverage. Pop in a cross over cable to connect both laptops and everyting should be fine... or so I thought.

I configured a local network among the two hosts, bridged the VM into it, fired up vCenter client to connect to the ESX and attached the iSCSI target... Here's me showing off my mad MSPaint drawing skills. Anyways you get the point.

Next I deployed VCVA onto the data store, which went surprisingly well...Boots up nicely still, but fails to configure the network via DHCP, as there is no DHCP server around. Confident I would be able to configure an IP manually I logged into the console and followed the instructions (/opt/vmware/shared/vami/vami_config_net) and configured a static IP address, only to find that I could not connect to the web config. I rebooted the VM countless times, checked resources inside the VM, everything looks good, but the web gui does not seem to come up. This is silly I thought, fired off a "aptitude install isc-dhcp-server" in the iSCSI VM and redeployed the VCVA. Yet again it did not get an IP address during boot. The DHCP logs say its been asigned, but the VM shows 0.0.0.0. I'm sensing resource contentions, but that's kind of the theme of the blog, isn't it? Logged into the VM again, reran dhclient, get an IP, login to web gui and configure the appliance straight forward (Eula, default SSO config). All is well in VCVA land, it takes a while but eventually all services start up fine. Next I configure a host name and static IP address, which becomes active instantly (at least ping works) but thats pretty much it. I get the certificate warning in the browser (and vCenter client) but cannot connect any further. As if the lighttp-proxy works, but the Tomcat handling the actual request, doesn't play along. Checking various VMware related services in /etc/init.d show all status running, however I notice once again that the hostname change is not reflected in the VM. I've seen that before, you change the hostname in the webconfig, reboot the appliance and are stuck with localhost.localdom again. That's a bit disappointing. I then configured hostname and corresponding /etc/hosts file manually on the console as I could still not access the webconfig and rebooted the VM. Unfortunately that doesn't seem to do the trick as now I was greeted with "Checking vami-sfcbd status........failed, restarting vami-sfcbd". That's where I'm getting really annoyed. As much as I like the idea of the appliance, a lot of times its more pain than gain. So as I type this I am installing a W2K8 server that will host my vCenter...

Summary

Another more or less road trip approach (as in "It's not the outcome that counts, but how you get there") that shows that with a little creativity you can actually set up a simple and very cheap lab at home, that will be more powerful (and especially more of a training experience) than installing ESXi in a VM. If you're (currently) limited by your hardware e.g. you do not own a home lab and neither of your accessible workstations/laptops etc. runs on at least a Intel Core i3, you're more or less stuck with this solution if you intend to run 64bit VMs inside your lab ESXi. And it also shows that operating the VCVA outside its recommended specs and/or in an unsupported environment is not always a good idea.

Update

Forget all the ranting about VCVA, it was my own fault. I had the T61 configured to the same IP as the VCVA ;).