Mapping Windows Drive Letters to RDMs to SAN UIDs in VMWare

So here’s a VMWare pickle. You have a VM that has numerous RDMs. A big and busy file server for example. Those RDMs are, of course, mounted as drive letters on your Windows file server. Now one of those drives is approaching being out of space and you need to expand it. First you need to get your storage admin to extend that volume, but before you can ask them to do that you have to first be able to tell them what SAN UID you are talking about.. But before you can even do that you have to figure out which RDM you are looking to expand. Phew! Well here’s a step by step to get there.

On your Windows server bring up Disk Management. Locate the drive you need to expand and right click on the Disk and select Properties. Note that you need to right click on the Disk box (where it says Disk 1, for example) and not on the partition box. On the General tab of the disk properties you should see a Location label. We are interested in the Bus Number and Target ID, so note those numbers. For this example our disk will be Bus Number 0, Target ID 10, LUN 0.

Now in the VIC edit the settings of your VM. Select each Mapped Raw LUN Hard Disk and pay attention to the SCSI ID shown in the Virtual Device Node box. Note that Windows uses a 0 for the Bus Number where VMWare uses a 1. So in our example we are looking for a disk labeled SCSI (1:10). This is the RDM that matches your Windows drive. Once you find that then right disk click the Manage Paths button. Here you are interested in the Runtime Name for the disk. For example it will be vmhba1:C0:T6:L55. Note that you will likely have numerous runtime names, but that they should all end with the same L##. This is the LUN ID. If you have duplicate LUN IDs for volumes attached to your physical VMWare hosts then you have bigger issues to deal with than a drive running out of space.

Alright, so now we know which Windows drive letter corresponds with which RDM. We’re halfway home.. Now we need to find the UID to hand off to those storage folks in order to get the drive expanded. To do that we are going to ssh into the physical host where your VM is living. In most environments you can ssh into any host in your cluster as the storage mappings will be identical, but just in case you have some funkiness play it safe and ssh to the host where your VM currently resides. Once connected run the following command:

/usr/sbin/esxcfg-mpath -l

This will spit out all storage paths on the system with a lot of detailed information. If any reasonable sized environment this list will be quite large. If you find it overwhelming then pass it to a file:

/usr/sbin/esxcfg-mpath -l > pathInfo

Now we are going to search this output for your RDM’s Runtime name. If you passed the output to a file then simply open that file in vi and do a search by typing the following:

/vmhba1\:C0\:T6\:L55

Note the backspaces in front of the colons. You should now be looking at the Runtime Name for your RDM. Directly below it should be a line that starts with Device which is (ta-da) your UID. Copy it over to your storage folks and ask them to use their infinite pool of storage to make it bigger!

Automating Rebooting of Systems After Shavlik Patching

Yes, another Shavlik related post…

Shavlik is an odd beast indeed. The only thing consistent about it is how inconsistent it is. For example, there are numerous deployment methods after you perform a scan and all but one of them is setup in the deployment templates, and can be initiated as an action that is performed immediately after a scan or as part of a schedule. There is one deployment method, however, that for some reason is not available in the deployment templates and cannot be scheduled. That method is the deploy after reboot method. To use the deploy after reboot method you have to perform a scan of your machine group with no deployment options selected, view the Scan Summary, right click on the missing patches and select Deploy All Missing Patches, and then choose “Install at next reboot” under the Deploy When section.

I have no idea why you cannot select this option as part of a scheduled scan, but so be it. This deployment method is nice as it guarantees that the system is in a clean state prior to patch installation and allows for a really fine level of control as to when a system begins patching.. Deploy the patches ahead of time without a schedule and simply reboot it when you’re ready.

A strange omission from the Shavlik feature set though is a way of kicking off a reboot of your machines from the Shavlik console. Once you copy your patches over using the Install at Next Reboot option you’re on your own for actually getting your systems to reboot. If you choose to use this deployment method on a large machine group then this can be a rather annoying issue. Oh well, what Shavlik forgot to include we can always add with some PowerShell…

The following script will attempt to reboot either one system (using the -MachineName parameter) or all systems that are part of a Shavlik Machine Group (using the -MachineGroup parameter). It uses 3 mechanisms to try to perform the reboot: Restart-VMGuest, Restart-Computer, and PSShutdown in that order. Preference is given to Restart-VMGuest as it circumvents network ACLs. Successes and failures are logged to a specified file for later review.

Since this script could be very dangerous if aimed at the wrong Shavlik Machine Group there are some sanity check parameters added as well. The sanityCheck variable allows you to specify a threshold for the number of machines that may be rebooted without first warning the user. If you want to see what is going to happen regardless of the number of machines in the selected Machine Group, just set this variable to 0. The cMachines variable points to the location of a text file containing the hostnames of your critical machines. If those machines appear in a Machine Group the script will prompt you to make sure you are okay with rebooting them before continuing.

All variables that need set for your environment are at the top of the script.

Like my other Shavlik scripts, this script requires the PowerCLI and the MS SQL Server PowerShell extensions.

############################################
# shavlikReboot.ps1
#
# Script built to reboot machines after using
# the Deploy after Reboot option in Shavlik
#
# Script can either work with machines individually
# as specified by the machineName parameter or with
# all systems in a particular Shavlik MachineGroup
# as specified by the machineGroup parameter
#
# If a user specifies both a machineName and a
# machineGroup the script will only process the
# machineName.
#
# The script will use the following reboot methods in
# this order:
#
# 1. If the machine is a VM it will first attempt with
#    the PowerCLI restart-guestvm cmdlet
# 2. If the machine is physical or for some reason unable
#    to be rebooted with restart-guestvm the script will
#    attempt to reboot it with restart-computer cmdlet.
# 3. If restart-computer fails the script will attempt
#    to reboot the system using the PSShutdown command
#    from sysinternals
#
# Successes and failures are logged out.
###############################################

# Command Line Parameters
param(
	[string]$machineName = "", [string]$machineGroup = ""
)

##############
# Globals

# VMWare information
$vCenters = "server1","server2"
$credFolder = "C:\vmware"

# Create alias for PSShutdown
set-alias psshutdown "C:\pstools\psshutdown.exe"

# Log file
$scriptRun = get-date -format ddMMyyyyHHmms
$outLog = "logs\shavlikReboot$scriptRun.log"

# Sanity Check - Define the # of systems that must be returned from a machine group
# for the script to warn you before rebooting them.  For example, if you set $sanityCheck
# to 20 then attempt to reboot a Machine Group with 21 or more systems in it the
# script will prompt you before acting to make sure you are okay with what is about to
# happen
[int]$sanityCheck = 0

# Optional text file containing a list of hostnames of critical servers.  Servers listed here
# will be displayed in red text with a yellow background on a sanity check screen with the option
# to cancel execution
$criticalMachines = "cmachines.txt"

# Path to SQL Server Powershell dlls
$SQLPSPath = "C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\Microsoft.SqlServer.Management.PS*.dll"

# Shavlik SQL Server and Database information
$Server = "shavlikServer\SQLEXPRESS"
$Database = "ShavlikScans"

# Table and fields of concern
$Table = "ScanMachines"
$selectField = "smachName"
$whereField = "groupName"

##############
# Functions

Function connect-Vcenters {
	# Loop through vcenters array and connect
	Foreach ($vcenter in $vcenters) {
		$Creds = $credFolder += $vcenter
		Connect-VIServer $vcenter -User $Creds.User -Password $Creds.Password | out-null
	}
}

Function process-VM($system) {
	# Attempt to restart a virtual machine using the restart-vmguest powercli cmdlet

	$vm = get-vm $system -ev a -ea SilentlyContinue

	if (!$a) {
		Restart-VMGuest -VM $System -Confirm:$false -EA SilentlyContinue -EV a | out-null
		if ($a) {
			write-custom "Restart-VMGuest command failed for $system. Attempting to reboot it as a physical system." "error"
			# There was an error - Try rebooting with physical server tools
			process-Physical $system
		} else {
			write-custom "Successfully rebooted $system (VM)" "info"
		}
	} else {
		write-custom "Unable to get vm: $system.  Attempting reboot it as a physical system." "warn"
		# Unable to get-vm, treat it as a physical
		process-Physical $system
	}
}

Function process-Physical($system) {
	# Attempt to restart the system using the restart-computer cmdlet.  If this
	# fails try again with psshutdown

	# First check to see if we can even reach this machine
	if (test-connection -computername $system -quiet -count 1) {
		# This machine is unreachable.  We will still attempt to restart it just in case
		# a firewall is preventing ICMP.  Log the inability to ping for troubleshooting
		# purposes.
		write-custom "Unable to ping $machineName" "warn"
	}

	# Attempt reboot
	restart-computer -ComputerName $system -EA silentlycontinue -EV a | out-null
	if ($a) {
		# Unable to reboot with Powershell.  Try with PSTools as last resort
		& psshutdown \\$system -t 1 -r | out-null
		write-custom "Reboot of $system attempted with PSTools check results to verify success" "warn"
	} else {
		write-custom "Successfully rebooted $system (Physical)" "info"
	}
}

Function write-custom($msg, $level) {
	# Displays output to end user and logs to the $outLog file

	# color code output based on specify $level
	switch ($level) {
		"info" { $color = "green" }
		"warn" { $color = "yellow" }
		"err" { $color = "red" }
	}
	# time stamp for log file
	$time = get-date -format T 

	write-host $msg -Foregroundcolor $color
	add-content $outLog "$level - $time`: $msg"
}

#################
# Script Body

If ($machineName -ne "" -Or $machineGroup -ne "") {
	# command line parameters were given - Setup the VMware connections

	# Import VMWare modules
	Add-PSSnapin VMware.VimAutomation.Core -EA SilentlyContinue | out-null

	# Get the current VIServerMode setting so that we can set it back when the script ends
	$defaultVIServerMode = Get-PowerCLIConfiguration
	# Set mode to multiple
	Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false | out-null

	# Connect to virtual centers
	connect-Vcenters
}

if ($machineName -ne "") {
	# Process indivual machine name

	write-custom "Processing Machine Name: $machineName" "info"
	$vmtest = Get-VM -Name $machineName -ea silentlycontinue -ev a

	if (!$a) {
		# Machine is a VM, reboot with PowerCLI
		process-VM $machineName
	} else {
		# Unable to get vm, must be a physical system.
		process-Physical $machineName
	}

	# Set VIServerMode back to where it was when we started
	Set-PowerCLIConfiguration -DefaultVIServerMode $defaultVIServerMode.defaultVIServerMode -Confirm:$false | out-null
} elseif ($machineGroup -ne "") {
	# Process specified machineGroup

	# Import SQL Server Modules
	$PSFiles = Get-ChildItem $SQLPSPath
	foreach($PSFile in $PSFiles) {
		$PSFile | import-module | out-null
	}
	clear-host

	write-custom "Processing Machine Group $machineGroup" "info"

	# Setup connection to Shavlik DB
	$Conn = New-Object System.Data.SQLClient.SQLConnection
	$Conn.ConnectionString = "server=$Server;database=$Database;trusted_connection=true;"

	# Connect to the DB
	$Conn.Open()

	# Create new SQL Command object
	$Command = New-Object System.Data.SQLClient.SQLCommand
	$Command.Connection = $Conn

	$Command.CommandText = "SELECT $selectField FROM $Table WHERE $whereField = '$machineGroup'"

	# Execute Select Query
	$Reader = $Command.ExecuteReader()

	$Counter = $Reader.FieldCount

	$machineNames = @()

	# Loop through results and create array
	While ($Reader.Read()) {
		For ($i = 0; $i -lt $Counter; $i++) {
			$machineNames += $Reader.GetValue($i)
		}
	}

	# Close DB Connection
	$Conn.Close()

	# Remove Duplicate Values
	$machineNames = $machineNames | select -uniq

	# Attempt to reboot machines
	if ($machineNames) {

		# Sanity check

		# Feed in critical machine content
		$cMachines = Get-Content $criticalMachines

		if ($machineNames.count -gt $sanityCheck) {
			# Make sure the user is okay with the machines that are about
			# to be rebooted
			clear-host

			# Display machines that will be rebooted

			foreach ($machine in $machineNames) {
				if ($cMachines | Select-String $machine -quiet) {
					# Machine is indicated as a critical system, make it stand out
					write-host "$machine" -nonewline -foregroundcolor red -backgroundcolor yellow; write-host "`t" -nonewline
				} else {
					write-host "$machine`t" -nonewline
				}
			}
			write-host "`n"
			$confirm = read-host "You are about to reboot the above $($machineNames.count) systems.`nAre you sure you wish to continue? (y/n)"
			if ($confirm -like "n") {
				EXIT
			}
		} elseif ($cmachines | select-string $machineNames -quiet) {
			# The machine group contains machines listed in the critical machines text file.  Make sure
			# we really want to reboot these.

			clear-host

			write-host ($cmachines | select-string $machineNames) -foregroundcolor red -backgroundcolor yellow
			$confirm = read-host "You are about to reboot the above critical systems.`nAre you sure you wish to continue? (y/n)"
			if ($confirm -like "n") {
				EXIT
			}
		}

		[int]$i = 0
		foreach ($machineName in $machineNames) {
			# Progress bar
			$i++
			write-progress -id 1 -activity "Attempting to reboot $machineName" -Status "Percent Complete: "  -PercentComplete (($i / $machineNames.count) * 100)

			# determine if this is a VM
			$vmtest = get-vm -name $machineName -ea silentlycontinue -ev a
			if (!$a) {
				# machine is a VM, reboot with PowerCLI
				process-VM $machineName
			} else {
				# Machine must be physical
				process-Physical $machineName
			}
		}
	} else {
		clear-host
		write-custom "No results found for Machine Group: $machineGroup" "err"
	}

	# Set VIServerMode back to where it was when we started
	Set-PowerCLIConfiguration -DefaultVIServerMode $defaultVIServerMode.defaultVIServerMode -Confirm:$false | out-null

} else {
	clear-host
	write-host "Usage:"
	write-host "`".\shavlikReboot.ps1 -machineName <computer name>`" to Reboot one system"
	write-host "`".\shavlikReboot.ps1 -machineGroup <Shavlik Machine Group Name>`" to Reboot all systems in a machine group"
}

With enough scripting we just might manage to make Shavlik a usable product!

Creating Snapshots prior to Patching with Shavlik

Update – Changed the script to use get-vm to determine if a machine is a VM rather than using WMI. This is quicker and has a better chance of succeeding in secure environments with ACLs between VLANs.

Some more Shavlik (er.. VMWare vCenter Protect Essentials Plus Extreme Ultimate Super Duper Deluxe) joy…

Probably my biggest complaint with Shavlik is how it handles hosted virtual machines. In Shavlik you create Machine Groups containing hosts that you wish to patch using various patch groups, schedules, deployment templates, etc.. In a Physical world you have numerous mechanisms for adding hosts to Machine Groups including LDAP searches, importing from text files, searching subnets, typing in hostnames manually, etc… If, however, you want to add VMs from a clustered VMWare environment you have two options: Add all VMs connected to a specific VCenter or manually pick each VM from a list. What really stinks is that Shavlik for some unknown reason displays the VMs in a hierarchy view with each VM appearing under the Physical Host that it is currently running on. So if you want to be a bit more specific than selecting every VM in your VCenter for patching you have to figure out which physical host your VM is currently running on and then scroll through this cluttered view and add it. Imagine doing this in a large environment with thousands of VMs and dozens of physical hosts.

Now the immediate question is why not just treat your VMs as physical servers and skip all of this pain? Well the one benefit that Shavlik gives you if you select your VMs instead of treating them like physical servers is that you can specify in your Deployment Template that Shavlik should create a snapshot of your VMs prior to patching. The only way that Shavlik knows that the host it is patching is a VM and should be snapshotted is if you go through this awful host adding process.

I talked to Shavlik support as it seemed like this was too ridiculous to be right, but unfortunately this is just how the application works. Hopefully now that VMWare owns it in addition to adding to the word count of the product name they will also fix how VMs are handled. But until then….

I have created the below PowerCLI script that can be executed prior to patching a specific machine group. The script takes as input the name of the Machine Group that you will be patching. It then queries the Shavlik database for all hosts that are members of that Machine Group and then uses WMI to determine which of those hosts are VMs. It will then create a snapshot for each VM.

You will need the Microsoft Windows Powershell Extensions for SQL Server to run this script which can be found Here. You will also need PowerCLI from VMWare which can be found Here.

Modify the variables at the top of the script to match your environment and give it a go…


##############################################
# prePatchSnapshot.ps1
#
# Script determines if the system it is executing
# on is a VM and snapshots it if it is.
#
# Script meant to be used as a custom action in
# Shavlik.
##############################################

param([string]$machineGroup)

if ($machineGroup -eq "") {
	write-host "A machine group must be entered:"
	write-host ".\prePatchSnapshot.ps1 machineGroupName"
	EXIT
}

# Snapshot name and description
$snapDescription = get-date
$snapName = "Shavlik - $machineGroup"

# Path to SQL Server Powershell dlls
$SQLPSPath = "C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\Microsoft.SqlServer.Management.PS*.dll"

# Location of secure VMWare credentials file
# Script assumes that the file name for the login
# information is the same as the vcenter name as
# specified in the $vCenters array.
# Script will prompt for login information
# if location does not exist
$credFolder = "C:\VMWare\creds\"

# VCenters in the environment - Add or Subtract as appropriate
$vCenters = "server1","server2"

# Shavlik SQL Server and Database information
$Server = "shavlikServer\SQLEXPRESS"
$Database = "ShavlikScans"

# Table and fields of concern
$Table = "ScanMachines"
$selectField = "smachName"
$whereField = "groupName"

# Import VMWare modules
Add-PSSnapin VMware.VimAutomation.Core -EA SilentlyContinue

# Get the current VIServerMode setting so that we can set it back when the script ends
$defaultVIServerMode = Get-PowerCLIConfiguration
# Set mode to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false

# Connect to VCenters
$Creds = ""

Foreach ($vcenter in $vcenters) {
	$Creds = $credFolder += $vcenter
	Connect-VIServer $vcenter -User $Creds.User -Password $Creds.Password
}

# Import SQL Server Modules
$PSFiles = Get-ChildItem $SQLPSPath
foreach($PSFile in $PSFiles) {
	$PSFile | import-module
}

# Setup connection to Shavlik DB
$Conn = New-Object System.Data.SQLClient.SQLConnection
$Conn.ConnectionString = "server=$Server;database=$Database;trusted_connection=true;"

# Connect to the DB
$Conn.Open()

# Create new SQL Command object
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Conn

$Command.CommandText = "SELECT $selectField FROM $Table WHERE $whereField = '$machineGroup'"

# Execute Select Query
$Reader = $Command.ExecuteReader()

$Counter = $Reader.FieldCount

$results = @()

# Loop through results and create array
While ($Reader.Read()) {
	For ($i = 0; $i -lt $Counter; $i++) {

		$results += $Reader.GetValue($i)

	}
}

# Close DB Connection
$Conn.Close()

# Remove Duplicate Values
$results = $results | select -uniq

foreach ($result in $results) {

	# Check to see if the system is a VM
	$VM = get-vm -name $result -ea silentlycontinue -ev a

	# Check if the system is a VM
	if (!$a) {
		#Machine is a VM - Snapshot it
		new-snapshot -VM $VM -name $snapName -Description $snapDescription
		#new-snapshot -VM (Get-VM -name $result) -name $snapName -Description $snapDescription -WhatIf
	}
}

# Set VIServerMode back to where it was when we started
Set-PowerCLIConfiguration -DefaultVIServerMode $defaultVIServerMode.defaultVIServerMode -Confirm:$false

And there you go! I admit that this is not as nice as just letting Shavlik create the snapshots before each machine patch, but it sure beats scrolling through all of your physical hosts to add a hundred VMs out out of thousands to a machine group! Hopefully this will be a stop gap fix until VMWare brings some sanity to this process!

PowerCLI for adding new Datastores to your VMWare environment

Update – The Mighty Jeff Hicks (JeffHicks on Twitter) wrote in to call me out on not using the proper verb-noun Powershell naming standard for my functions. The script has been updated to fix this mistake. Thanks Jeff!

In larger environments adding new LUNs to VMWare can be tedious and messy. This is especially true if you have multiple Virtual Centers connected to the same SAN(s). Say, for example, that you have a corporate naming standard for your Datastores like the following:

IBM_Tier3_01

That’s SAN manufacturer, Tier level, and then a unique number. This is, in my experience, a pretty standard naming scheme. As the environment grows keeping track of those unique incremental numbers at the end and ensuring you are using the next one in line can be a bit annoying. If you have more than one Virtual Center then there is a real risk that you will re-use the same number twice. Say you have this scenario:

You have to vCenters: Cluster1 anc Cluster2. On Cluster1 the most recent Tier 1 datastore is named IBM_Tier1_21. On Cluster2 the most recent Tier 1 datastore is named IBM_Tier1_22. You need to add a new tier 1 Datastore to Cluster1 so you connect to the Cluster1 Vcenter, look at your current Tier1 Datastores and add one to the last Datastore in the list. If you forget to look at Cluster 2 first then you will be creating another IBM_Tier1_22 datastore. This calls for some automation…

The below PowerCLI script is useful for adding new Datastores in environment of one or more Virtual Centers. It performs the following operations:

  1. Connects to all Virtual Centers in your environment (Defined in the $VCenters array)
  2. Scans HBAs looking for new LUNs
  3. Asks the user what Tier storage the new Datastore should be part of
  4. Retrieves a list of all existing Datastores in that tier across all Virtual Centers and displays them to the user while suggesting the next datastore in line. It will not allow the user to re-use an existing store name.
  5. Generates the new Datastore name based on the company’s naming standards
  6. Displays a list of all LUNs presented to the VMWare environment which are available for being converted to a VMFS for the user to select
  7. Asks the user for what block size to use for the new datastore
  8. Displays the information for the new datastore to the user for confirmation and allows them to choose to continue with creation or not.
  9. Adds the new datastore!

I’ve tried to make this script easy to customize for various environments. The following settings at the top of the script will need to be modified for your environment.

  • $credFolder – Location of your stored credentials for your Virtual Center(s). This is necessary if you are not launching PowerCLI with a user account that has permission to sign on to Virtual Center. The script assumes that the filename for your credentials file is the name of the cluster, but this could be easily modified. If you do not specify this credential you will simply be prompted at the start of the script’s execution for your login information. See Storing your vCenter Credentials for instructions on creating a file containing your Virtual Center credentials
  • $tierNamePrefix & $tierNameSuffix – These variables are used to define your datastore naming standard. $tierNamePrefix is what will precede the Tier number and $tierNameSuffix is what will follow the Tier number and precede the Datastore’s unique number.
  • $vCenters – Array containing the host names of your Virtual Centers. If you only have one Virtual Center then simply specify that single Virtual Center. If you have more than one then add them all within quotes and comma separated.
  • $storageTiers – Array containing your storage Tier numbers. In most environments this is 1 – 3.

Additionally in the createNewDatastore function you can change the commenting on the New-Datastore commands to cause the script to not actually create the datastores for testing purposes.

Here’s the script… It is a fairly long one, but don’t let it intimidate you!

<####################################################################
newLUNAdder.ps1

 Automates the process of adding new LUNs to the VMWare clusters
 and provides datastore naming mechanism that helps to prevent overlapping names
 between the two clusters.

####################################################################>

#########################################
# GLOBALS

# Location of secure VMWare credentials file
# Script assumes that the file name for the login
# information is the same as the vcenter name as
# specified in the $vCenters array.
# Script will prompt for login information
# if location does not exist
$credFolder = "C:\VMWare\creds\"

# Prefix and suffix for the datastore names.  Datastore names
# will be built in the following format:
#
# tierNamePrevfix tierNumber tierNameSuffix volumeNumber
#
# For Example:
#
# SVC_Tier3_vStore42
#
# Where:
# tierNamePrefix = SVC_Tier
# tierNumber = 3
# tierNameSuffix = _vStore
# volumeNumber = 42
$tierNamePrefix = "SVC_Tier"
$tierNameSuffix = "_vStore"

# VCenters in the environment - Add or Subtract as appropriate
$vCenters = "cluster1","cluster2"

# Storage tiers, using the very typical 1, 2, 3..
$storageTiers = "1", "2", "3"

# Initialize global variables

$workingCluster
$workingHost
$workingDatastore
$workingTier
$workingStoreName
$totalVMHosts
$workingBlockSize

$physicalHosts = @();

# Datastore block sizes
$blockSize = @(("1", "2", "4", "8"),("256", "512", "1024", "2048"))

# Get the current VIServerMode setting so that we can set it back when the script ends
$defaultVIServerMode = Get-PowerCLIConfiguration
# Set mode to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false

clear-host
#########################################

# Functions

function Get-Cluster() {
	# Asks user which cluster to add the new datastore to.

	# Generate Menu

	$menuTest = $false
	do
	{
		clear-host
		$i = 0
		foreach ($vcenter in $vcenters) {
			Write-Custom "($i) $vcenter" "select"
			$i++
		}
		Write-Host ""
		$input = Read-Host "Which Cluster Would You Like to Add Storage To?" 

		# Check if the input is a number and less than $i
		if ($input -match '^\d+$' -and $input -lt $i) {
			set-variable -name workingCluster -value $vcenters[$input] -scope Global

			$menuTest = $true
		}
	} until ($menuTest -eq $true)
}

function Count-Hosts() {
	# Counts the total number of hosts across all clusters.  This is done for
	# for progess counters

	write-custom "Determining the total number of hosts across all clusters..." "info"

	$temphosts = get-vmhost
	$tempCount = $tempHosts.count

	set-variable -name totalVMHosts -value $tempCount -Scope Global
}

function Get-Host() {
	# Asks user which VMWare Host in the chosen cluster to use for initally adding the
	# new storage.  List of VMS is generated based on choice made in Get-Cluster

	# Get list of physical hosts in the cluster
	$physicalHosts = get-vmhost | where {$_.Uid -like "*$workingCluster*"} | select name | sort name

	$menuTest = $false

	# Generate Menu to display hosts
	do
	{
		[int]$i=0
		clear-host
		foreach ($physHost in $physicalHosts) {
			$hostname = $physHost.name
			write-Custom "($i) $hostname" "select"
			$i++
		}
		write-host "Working cluster $workingcluster"
		$input = Read-Host "Please select a physical host to map the datastore to first"

		# Convert $input to an integer in order to use the value in the $physicalHosts array
		$convInput = [int]$input

		if ($convInput -gt -1 -AND $convInput -lt $i) {

			# Set the workingHost variable to the chosen host globally
			set-variable -name workingHost -value $physicalHosts[$convInput].name -scope Global
			$menuTest = $true # Break out of loop on valid selection
		}

	} until ($menuTest -eq $true)
}

Function Scan-HBAs {
	# Scans HBAs on all hosts across all clusters

	$j = 0
	$totalHosts = get-vmhost

	foreach ($vm in get-vmhost) {
		$j++
		write-progress -id 1 -activity "Rescanning HBAs across all clusters" -Status "Percent Complete: " -PercentComplete (($j / $totalVMHosts) * 100)
		get-vmhoststorage -vmhost $vm.name -RescanAllHba
		clear-host
	}

	clear-host

	# Remove progress indicators
	write-progress -id 1 "Rescanning HBAs" "Complete" -Completed
}

Function write-custom ($msg, $type) {
	# Writes out using custom settings

	if ($type -eq "info") {
		write-host $msg -foregroundcolor black -backgroundcolor yellow
	}
	if ($type -eq "select") {
		write-host $msg -backgroundcolor darkgreen -foregroundcolor white
	}

}

function Get-StorageTier {
	# Lists standard tiers of storage and asks user to select
	# the appropriate tier for the new datastore.  This will be used to decide
	# the new name for the datastore

	$menutest = $false

	do {
		[int]$i = 0
		clear-host
		foreach ($tier in $storageTiers) {
			write-custom "($i) Tier: $tier" "select"
			$i++
		}

		$input = Read-Host "What is the tier of this new storage"

		$convInput = [int]$input

		if ($convInput -gt -1 -and $convInput -lt $i) {
			set-variable -name workingTier -value $storageTiers[$convInput] -scope Global

			$menutest = $true
		}
	}  until ($menutest -eq $true)
}

function Get-BlockSize {
	# Lists available block sizes for the new datastore and asks user to select
	# the desired size

	$menutest = $false

	do {
		clear-host

		# Loop through the array and display the block size options
		for ($i=0; $i -lt $blocksize[0].length; $i++) {
			$j = ($i + 1)

			# Select the values from the multidimensional array
			$blockSizeMB = $blocksize[0][$i]
			$maxFileSize = $blocksize[1][$i]
			write-custom "($i) Blocksize $blockSizeMB MB (Max file size $maxFileSize GB)" "select"
		}
		$input = read-host "What block size would you like to set for this new datastore"

		$convInput = [int]$input

		# Make sure selection is within range
		if ($convInput -gt -1 -and $convInput -lt ($blocksize[0].count)) {
			# Set the workingBlockSize variable to the chosen option
			set-variable -name workingBlockSize -value $blockSize[0][$convInput] -scope Global
			$menutest = $true
		}
	} until ($menutest -eq $true)
}

function Populate-TierArrays {
	# Builds an array containing the store numbers currently in use for the chosen Tier.

	# Build text to search for
	$tierToScan = "*$tierNamePrefix$workingTier*"

	$storeNumber = @();

	# Get the datastores that are part of the chosen tier
	$currentStores = get-datastore | where {$_.name -like $tierToScan} | sort Name

	# Build text to remove from results returned from get-datastore
	$tierText = "$tierNamePrefix$workingTier$tierNameSuffix"

	# Populate array with results
	foreach ($store in $currentStores) {
			$storeNumber += $Store.name.Replace($tierText, "")
	}

	return $storeNumber

}

function Show-CurrentDataStores {
	# Builds a sorted and filtered array of current datastores in the
	# chosen tier across all clusters.  This array is presented to the
	# user in the Name-NewStore function.

	# When creating the number array there will be errors for
	# data stores that do not follow naming convention and end
	# with something other than a number.  We can safely exclude
	# these items from the array.  We set error action to silently
	# continue to avoid ugly error msgs while building the array
	$ErrorActionPreference = "SilentlyContinue"

	# Populate aray numbers from all clusters
	write-custom "Getting List of datastores in selected tier" "info"

	$allStoreNumbers = Populate-TierArrays

	# Filter out duplicate entries
	$allStoreNumbers = $allStoreNumbers | select -uniq

	$intStoreNumber = @();

	# Convert array to integer for better sorting
	foreach ($num in $allStoreNumbers) {
		$intStoreNumber += [int]$num
	}

	# Sort integer array
	[Array]::Sort([array]$intStoreNumber)

	clear-host

	return $intStoreNumber
}

function Name-NewStore {
	# Displays list of current datastores in the chosen tier to the user
	# and asks the user to name the new datastore.  The next available datastore
	# integer is presented as a default option.  Should the user choose their
	# own integer the function will check that the number is not already in
	# use and will continue to prompt the user for a number until they choose
	# one that is available.

	# Build array of current datastores in the chosen tier
	$intStoreNumber = Show-CurrentDataStores

	# Find the last used datastore number and increment it to use as the default
	# new store number
	[int]$nextStoreNumber = ($intStoreNumber[-1] + 1)

	# Display current store numbers to user and request a new one
	$menutest = $false
	do {
		clear-host
		write-custom "Current Tier $workingTier numbers: " "info"
		foreach ($num in $intStoreNumber) {
			write-custom $num "info"
		}
		$input = read-host "Please enter new store number [$nextStoreNumber]"

		if ($input -eq "") {
			# User did not enter a number, use the default
			$input = $nextStoreNumber
			$menutest = $true
		} else {
			# Check to make sure user has not entered a number already in use
			foreach ($num in $intStoreNumber) {
				if ($num.ToString() -like $input) {
					$menutest = $false
					break
				} else {
					$menutest = $true
				}
			}

			if ($menutest) {

				# Build the newStoreName and set it to the global workingStoreName variable

				# I'm going to add a little hidden feature for the admins..  Test if the
				# input from the user is a number.  If so then create the new store name
				# using the standard.  If it isn't, then assume that the admin is breaking
				# the naming standard for a good reason and that they don't want their input
				# added to the naming convention.

				if ($input -match '^\d+$') {
					# it is a number, use the normal naming convention
					$newStoreName = "$tierNamePrefix$workingTier$tierNameSuffix$input"
				} else {
					# user's input contains a string.  Let's make sure they really mean
					# to do this

					$verifyInput = Read-Host "Your input is not a number, do you want to create a non-standard data store name? (y/n) "

					if ($verifyInput -like "y") {
						$newStoreName = $input
					} else {
						# set menutest back to false to repeat the menu
						$menutest = $false
					}
				}
			}
		}
	} until ($menutest -eq $true)

	set-variable -name workingStoreName -value $newStoreName -scope Global
}

function Get-AvailableDatastores {
	# Query chosen cluster to find datastores that are available to have VMFS added.
	# Display list to user for selection

	write-custom "Building list of available datastores..." "info"

	# Build list of available datastores
	$datastore = $null
	$hostView = get-vmhost -name $workingHost | get-view
	$dsView = get-view $hostView.ConfigManager.DatastoreSystem
	$unUsed = $dsView.QueryAvailabledisksForVMFS($datastore)

	$menuTest = $false

	# Generate Menu
	do
	{
		[int]$i=0
		clear-host
		foreach ($store in $unUsed) {
			$storeName = $store.CanonicalName
			write-custom "($i) $storeName" "select"
			$i++
		}
		$input = Read-Host "Please select a datastore to Add"
		$convInput = [int]$input

		# Make sure user selected a valid datastore option. Represent menu if they did not.
		if ($convInput -gt -1 -AND $convInput -lt $i) {

			set-variable -name workingDatastore -value $unUsed[$convInput].CanonicalName -scope Global
			$menuTest = $true
		}

	} until ($menuTest -eq $true)
}

Function Create-NewDatastore {
	# Present selection criteria to the user and make sure they want to move forward
	# with creating the new datastore.  If yes: Do it.  If not: Exit

	$menutest = $false

	do {
		clear-host
		write-custom "Creating new VMFS Datastore using the following criteria:" "info"
		write-custom "Cluster: $workingCluster" "info"
		write-custom "Host: $workingHost" "info"
		write-custom "Store: $workingDatastore" "info"
		Write-custom "Tier Level: $workingTier" "info"
		Write-custom "Block Size: $workingBlockSize" "info"
		Write-custom "New Store name: $workingStoreName" "info"
		$input = read-Host "Do you wish to continue (y/n)"

		if ($input -like "n") {
			# User chose not to create the datastore.  Return $input to the main program
			# and exit function
			return $input
		}

		if ($input -like "y") {
			$menutest = $true
		}

	} until ($menutest -eq $true)

	clear-host

	write-custom "Creating Datastore $workingstorename..." "info"

	# Create the datastore using the selected criteria.  Comment below line out if you are testing
	New-Datastore -Name $workingStoreName -VMHost $workingHost -Path $workingDatastore -VMFS -BlockSizeMB $workingBlockSize

	# Uncomment below line if you are testing
	#New-Datastore -Name $workingStoreName -VMHost $workingHost -Path $workingDatastore -VMFS -BlockSizeMB $workingBlockSize -WhatIf

	if ($?) {
		clear-host
		write-custom "Datastore Created.  Rescanning across all clusters" "info"
	}

	# Return input so script body will know if the datastore was created.
	return $input

}

Function Connect-VCenter {
	# Connect to the supplied virtual center

	write-custom "Connecting to VMWare environment" "info"

	$Creds = ""

	Foreach ($vcenter in $vcenters) {
		$Creds = $credFolder += $vcenter
		Connect-VIServer $vcenter -User $Creds.User -Password $Creds.Password
	}
}

######################################
# Main Script Body

#	Script launches each step of the add storage process via
#	sequential functions

# Connect to our VCenter(s)
Connect-VCenter
clear-host

# Get a count of all physical hosts across all clusters
Count-Hosts
clear-host

# Rescan HBAs so as to make sure VMWare is aware of new available datastores
Scan-HBAs

# Loop to allow user to add multiple new luns without having
# to restart the script and rescan HBAs each time
$input = "y"
do {
	clear-host

	# Determine which cluster to add the new datastore to
	Get-Cluster
	clear-host

	# Determine which host to initially add the datastore to.  This is required
	# for the new-datastore command
	Get-Host
	clear-host

	# Find available datastores that are eligible for VMFS
	Get-AvailableDatastores
	clear-host

	# Determine which tier this datastore should be a member of
	Get-StorageTier
	clear-host

	# Determine which block size to use with the new datastore
	Get-BlockSize
	clear-host

	# Build list of current datastore numbers in the chosen tier and determine
	# what the new store should be named
	Name-NewStore
	clear-host

	# Show user final selections and confirm the add storage operation.  Add the
	# storage if the user chooses yes
	$payload = Create-NewDatastore
	clear-host

	if ($payload -like "y") {
		write-custom "The new datastore has been added..." "info"
	}

	$input = read-Host "Would you like to add another LUN? (y/n)"
} until ($input -eq "n")

# Set VIServerMode back to where it was when we started
Set-PowerCLIConfiguration -DefaultVIServerMode $defaultVIServerMode.defaultVIServerMode -Confirm:$false

Getting a list of datastores available for VMFS in PowerCLI

Sometimes working with storage in PowerCLI can be surprisingly tricky. I wanted to produce a list of datastores that were available to be converted to VMFS in PowerCLI. This is essentially the list that is presented to you when you use the VIC to click Add Storage in the Storage configuration section and then select Disk/LUN. I wrongly assumed that there would be pre-built PowerCLI command to produce this list. After quite a bit of trial and error I finally came up with this solution:

$workingHost = "<VMWare Hostname>"
$datastore = $null
$hostView = get-vmhost -name $workingHost | get-view
$dsView = get-view $hostView.ConfigManager.DatastoreSystem
$AvailableForVMFS = $dsView.QueryAvailabledisksForVMFS($datastore)

Or… Since One Liners are all the rage.. If you love to make things ugly you could do it all in one line:

$AvailableForVMFS = (Get-View (Get-VMHost -Name "<VMWare Hostname>" | get-view).ConfigManager.DatastoreSystem).QueryAvailableDisksForVmfs($null)

Automatically Setting the Default Deploy Drive to the Largest Disk in Shavlik

Shavlik NetChk Pro (or the new VMWare name of VMWare vCenter Protect Essentials Plus Blah Blah Blah thanks for the value add VMWare marketing…) is an excellent package for deploying operating system and application patches across the enterprise. It offers a lot of advantages over WSUS/SCCM including automated patching of numerous 3rd party applications like the virus plagued Flash, Acrobat Reader, and JRE and the ability to launch targeted patches at a moment’s notice allowing for very fine tuned patching schedules.

One big problem with Shavlik is how it deploys patches. By default it deploys patches to the target server’s C drive and it requires the C drive to have as much as five times the total size of the patches to be installed free on the C drive. If you’re like a lot of environments you tend to run with very little free space on your server C drives.. Especially virtualized servers. While it seems like it would be trivial for Shavlik to determine the most appropriate drive to deploy patches to, it doesn’t, and instead it is up to the administrator to change the default drive manually for every single server. A rather painful task for a medium to large environment to be sure. Powershell to the rescue? You bet…

There are two steps to accomplish here: First, determine the most appropriate drive for each server and Second, update the Shavlik database to point the deployment target to that appropriate drive. We will accomplish these tasks with 2 separate scripts…

WHOA! WAITAMINUTE! Don’t forget step 0: BACKUP YOUR SHAVLIK DATABASE. Chances are it is a VM.. Snapshot it!

Script 1: largestDrive.ps1

This script queries Active Directory for all Server objects, connects to each one with WMI and queries their Hard Disk information, and outputs the largest drive for each server to a CSV file which will be accessed by our second script. I am making an assumption here that if your environment is large enough to justify the purchase of Shavlik then querying the server list of Active Directory is probably the most appropriate mechanism for snagging this data. However, if this is not the case it would be trivial to instead feed the data from a CSV or other mechanism. I also added in a second For loop in the script where you can perform some basic filtering of the servers returned from AD to exclude specific systems. In the script I provide an example for filtering out servers with “Test” in the hostname, but this can be changed to anything. In my production environment I used RegEx to remove numerous servers that were outside of our patching scope. Here’s the script!

#############################################
# largestDrive.ps1
#
# Script pulls all servers out of Active Directory
# and then loops through them using WMI to find the
# drive with the greatest amount of free space.
#
# Output is saved to a CSV file.
##############################################

#################
# Script Setup

# Import Active Directory module for AD query
Import-Module ActiveDirectory

# Location and filename for output CSV
$driveOut = "driveSpaceTest.csv"

# Header for drive space CSV
Add-Content $driveOut "hostname,drive_letter,free_space"

# Retrieve all server names listed in AD
$Servers = Get-ADComputer -Filter {operatingsystem -like "*server*"} -Property Name | Select-Object Name

<#
Reading input from Active Directory is just one of many ways
to feed servers into this script.  You could also use a CSV:

serverList.csv:

Name
Server1
Server2
Server3

$Servers = Import-CSV "serverList.csv"

Or you could just feed the $colComputers array directly:

$colComputers = New-Object System.Collections.ArrayList
$colComputers.Add("server1")
$colComputers.Add("server2")
$colComputers.Add("server3")

Or...

$colComputers = @()
$colComputers = $colComputers + "server1"
$colComputers = $colComputers + "server2"
$colComputers = $colComputers + "server3"

Or...

$colComputers = "server1","server2","server3"
#>

# New Array to hold server list
$colComputers = New-Object System.Collections.ArrayList

###################
# Script Body

ForEach ($Server in $Servers) {
	# Build our server array.  We could just process the $servers array
	# directly.  I am adding this loop to give an opportunity to perform
	# some simple filtering on the servers returned from AD.
	#
	# For example, we will remove any server with the word "test" in the
	# name

	$serverToAdd = $Server.Name

	if ($serverToAdd.Contains("test") -eq $False) {
		$colComputers.add($serverToAdd)
	}
}

# Loop through each system and get the Hard Disk information
ForEach ($strComputer in $colComputers) {

	# Zero out test variables
	$currentFreeSpace = 0
	$largestFreeSpace = 0

	# Use WMI to query disk informaiton
	$DiskData = gwmi Win32_LogicalDisk -Comp $StrComputer

	# Loop throuh returned data to find largest HD.  Will loop
	# through each server's HDs in order and allow the largest
	# to bubble up
	ForEach ($Disk in $DiskData) {

	# Set currentFreeSpace to the size in GB of the current Disk
		$currentFreeSpace = $Disk.freespace/1024/1024
		if ($currentFreespace -gt $largestFreeSpace) {
			# Current disk is larger than the previous, set the
			# variables to reflect this
			$largestFreeSpace = $currentFreeSpace
			$largestDisk = $disk.DeviceID
		}
	}

	# Variables should now hold the drive letter and size of the largest disk
	# for the current server.  Update our output file
	Add-Content $driveOut "$strComputer, $largestDisk, $largestFreeSpace"
}

Script 2: ShavlikDrive.ps1

This script reads in the CSV created by script 1, connects to the Shavlik database, and updates the mmDeploymentDrive field in the ManagedMachines table based on the values in the CSV. This script requires the SQL Powershell extensions which can be downloaded here. The script produces a log showing the results of each query. If you have some servers that fail to update then they are more than likely not in the Shavlik database. Try adding them to a machine group and scanning them. You may find some other error with that particular server like the remote registry service being disabled. And here’s the script!

#####################################################
# shavlikDeploymentDriveUpdate.ps1
#
# Script uses the drive space output from largestDrive.ps1
# to configure Shavlik to copy patches to the drive with
# the greatest amount of freespace per server in the
# environment
#
# Uses the pssql Powershell snapin.
#####################################################

##################
# Globals

# Shavlik SQL Server and Database information
$Server = "ShavlikServer\SQLEXPRESS"
$Database = "ShavlikScans"

# Table and fields of concern
$Table = "ManagedMachines"
$updateField = "mmDeploymentDrive"
$whereField = "name"

# File generated by largestDrive.ps1
$spaceFile = "driveSpace.csv"

# Log file
$outputLog = "results.log"

##################
# Script Setup

# Read in the CSV generated by largestDrive.ps1
$freeSpace = import-csv $spaceFile

# Setup connection to Shavlik DB
$Conn = New-Object System.Data.SQLClient.SQLConnection
$Conn.ConnectionString = "server=$Server;database=$Database;trusted_connection=true;"

$Conn.Open()

# Create new SQL Command object
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Conn

###################
# Script Body

# Loop through each server in the CSV file and update the Shavlik DB
ForEach ($server in $freeSpace) {

	# Skipping servers where the C drive is the largest
	# as that is Shavlik's default drive
	if ($server.drive_letter -ne "C:") {
		$newDrive = $server.drive_letter
		$serverName = $server.hostname

		# Build UPDATE query
		$Command.CommandText = "UPDATE $Table SET $updateField = '$newDrive' WHERE $whereField = '$serverName'"

		# Execute the Update query.  The query will return the number of rows that have been
		# changed.
		$result = $Command.ExecuteNonQuery()

		# If the query did not update at least one row then there was a problem.  If other servers
		# update correctly then chances are that the servers that did not update do not currently
		# exist in the Shavlik database.  Try manually adding servers that error out to a machine
		# group in Shavlik and perform a scan on them.

		# Update log file with success and failures.
		if ($result -lt 1) {
			write-host "Error updating $serverName" -foregroundcolor red
			add-content $outputLog "Error updating $serverName. Query: $($Command.CommandText)"
		} else {
			write-host "Updated $serverName" -foregroundcolor green
			add-content $outputLog "Successfully updated $serverName. Query: $($Command.CommandText)"
		}
	}

}

####################
# Clean Up
$Conn.Close()

Modify the various connection variables and paths in each script as appropriate for your environment and give it a whirl. These scripts certainly saved me numerous hours of effort!

Connecting to Multiple VCenters at Once

If you work in an environment that has multiple Virtual Centers you may have noticed that if you are connected to VCenter A in PowerCLI and try to connect to VCenter B you will first get booted from A. That is due to your Default VI Server Mode in PowerCLI and is easily changeable:

Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$False

The above command will change the VI Server Mode to allow you to connect to multiple VCenters at once. For obvious reasons you should be careful when running in this mode to ensure that the commands you are issuing are targeting only what you want to target. There is never a bad time to first try your commands using the “-WhatIf” parameter.

When writing scripts that need to interact with multiple Virtual Centers a good practice is to store system’s current DefaultVIServerMode prior to making changes and set it back when you’re done.

Start of script:

# First get the current setting so that we can set it back when the script ends
$defaultVIServerMode = Get-PowerCLIConfiguration
# Set mode to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false

End of script:

# Set the VI Server mode back to the way it was when the script started
Set-PowerCLIConfiguration -DefaultVIServerMode $defaultVIServerMode.defaultVIServerMode -Confirm:$false

Storing your vCenter credentials

So you just wrote a beautiful piece of PowerCLI that will automagically delete snapshots older than 30 days in your VMWare environment, and you’re thinking that it would be swell to run this script daily as a scheduled task or similar in order to keep your VMWare environment neat and tidy.  Ah, but your script always prompts for VCenter credentials when you execute the Connect-VIServer command.  Here’s how to store those credentials securely and reference them from your new script:

You will create a file that contains your login credentials in an encrypted fashion.  Note that this file will NOT be transferable between systems.  That is to say that the file will only work on the system it is created by.  If someone copies your credential file to their local system they will not be able to use it to connect to VCenter.  Similarly, if you are setting this up on your workstation as a test with the intention to transfer it to a production server later on, keep in mind that you will need to recreate this file on the production server.

Pick a location for your file and execute the following command:

New-VICredentialStoreItem -Host <VCenter Host> -User <admin username> -Password <admin password> -File <path to save xml file>

You now have a file containing your log on credentials.  Now in your script you can authenticate to your VCenter server by doing the following:

$Creds = <path to stored credential xml file>
Connect-VIServer <VCenter Host> -User $Creds.User -Password $Creds.Password

And that’s all the more there is to it!

One-Liner to change VM Network Mapping

Here’s a quick one liner that will take all VMs attached to a specific virtual network and move them to a new virtual network.  This is useful for VLAN changes and the like.  It is also useful for those Cisco Nexus 1000V customers (all five of them) that need to temporarily move to a VDS while upgrading ESXi versions.

Get-VM | Get-NetworkAdapter | ? {$_.NetworkName -eq "Old_Crummy_Network" } |  Set-NetworkAdapter -NetworkName New_Peachy_Network