Cannot delete Portgroup - works as designed

This morning when I start work I notice all those emails about a new ticket in our trouble ticket system. The owner added me among other colleagues as a monitor. The issue is the following:

There are numerous virtual port groups that do not have any VMs connected (after a lengthy and mostly automated migration to a new naming scheme) that cannot be deleted. They show up greyed out in the vSphere client and if you drill down each of those port groups will show its associated VMs and templates. If you check the config of any affected VM they will show the new port groups only, the summary pane will furthermore list the greyed out old port groups. The proposed problem solution is a restart of the vCenter service, as there is an OS update pending anyway.

Cause

As I have seen this behaviour before on several, albeit very rare occasions, I wanted to take my chances and investigate. I went ahead to the first affected port group and found a template and a running VM associated with it. The template was an easy and logical case. My colleagues had migrated all VMs but had failed to migrate the templates yet. Convert it to a VM, change the port group association and away it went. Convert back to template and everything is fine.

The running VM however is a different case. Its settings showed no binding to the old port group. Because its a customer's system I cannot just go and change things around wildly. I had a closer look and noticed an active snapshot. We have a policy that snapshots may not be kept longer than one week. However that policy is not being enforced by any automatism or audit trail. The Tasks & Events pane in my vSphere client was not able to tell me when the snapshot had been created...it was already beginning to mold and smell unpleasently. Same goes for the other affected VMs.

And it makes perfect sense. If I want to go back to my original point in time - when I took the snapshot - I expect the VM to be in the same network (thus virtual port group). Thinking I might be able to get a list of VMs and port groups via a simple PowerCLI script I went to work and came up with this slightly ugly code:

$Portgroups = Get-Datacenter $myDC | Get-VirtualPortGroup | Where {$_.Name -like $myPgFilter}
$PGNames = @()
foreach($PG in $Portgroups) {
    $PGNames += $PG.Name
}
$VM_with_Snapshots = @()

$VMs = Get-VM
$Snapshots = $VM | Get-Snapshot

foreach($SS in $Snapshots) {
    $NAs = $SS.VM.NetworkAdapters
    foreach($NetAdd in $NAs) {
        if($PGNames -contains $NetAdd.NetworkName) {
            $VM_with_Snapshots += $SS.VM
        }
    }
}

I'm sure there is massive potential for optimization, I'm not a coder nor do I have much practise at the moment (things are about to change in the near future though). However this approach did not yield the expected result as this script will not return the previous (and still caught in a snapshot) port group assignment, but the current one. Pointers towards the proper results would be greatly appreciated.

Among the affected VMs there are, however, some test systems and infrastructure systems as well that I have sufficient control, knowledge and privileges to test my snapshot theory. And of course, I was right.

I now want to refer to this very recent (re-)tweet and extend it to VCPs as well! :)

RT @josh_odgers: @grantorchard Business as usual, the VCDX comes in and cleans up the mess and ensures customer satisfaction :) > +1
— Michael Webster (@vcdxnz001) May 15, 2013