Wednesday, November 27, 2013

Configuring Management VLAN in Hyper-V 2012 Core using Powershell

While configuring some Hyper-V 2012 core servers this week, I came across a bunch of issues trying to configure VLAN tagging on the management network. My goal was to configure the environment as follows:

Switch Port Configurations:
-2 ports to the Hyper-V host
-trunk ports
-native VLAN 1
-Management VLAN 28
-various other VLANs for VM traffic

Hyper-V Host Configurations:
-NIC Teaming
-Host IP on management VLAN
-1 VMswitch

Once the OS was installed and all the drivers were added, begin by teaming the NICs. Since there is no GUI in Hyper-V core, copy LbfoAdmin.exe and LbfoAdminLib.dll onto the server and run the executable to team the NICs.

Next, open up a command prompt. Identify the name of the virtual adapter for your NIC Team:

Create the VMswitch:
New-VMSwitch -AllowManagementOS 0 -Name "vSwitch" -NetAdapterName "<VirtualAdapterName>"

Create a management adapter and add it to the newly created VMswitch:
Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "vSwitch"

Tag the VLAN ID to the management adapter:
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Management" -Access -VlanId 28

Configure the IP of the management adapter:
New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress x.x.x.x -PrefixLength # -DefaultGateway x.x.x.x

The management interface should now be pingable.

Credits to Eric Siron.

Thursday, November 7, 2013

No visble datastores after storage failure

Last night there was a failure in our SAN in the QA/Test VMware environment. Upon bringing the SAN back online, I observed that all the datastores have been removed from the hosts. My first thought was that this was not an complicated issue - all I would have to do is restart the hosts and when all the paths have been re-established, do a rescan for the datastores and all would be well. Unfortunately after restarting the hosts, the datastores did not come up even though I can clearly see all the paths to the LUNs. Here is a summary of the symptoms:

  • Datastore do not show up after rescanning HBAs and VMFS even though all the paths are there
  • Trying to add a new datastore will show all the LUNs, but attempting to add the datastore while preserving VMFS signatures results in an error
  • VMs all show up as inaccessible
  • LUN numbers appear to have been changed on the storage

The last point regarding the LUN number change was the crucial finding. After a bit of googling, it appears that you would need to manually remount the datastores. Here's how to do it via vCLI.

First, get the host object for the affected host (you'll have to do it one at a time or combine the following commands in a script).

$esxcli = Get-EsxCli -VMHost "vm-host1.local"

Next, list the datastores it has detected that has been unmounted.


The list of datastores should now be displayed. Mount each of them using the following command.


Repeat this process for all the affected hosts and all your datastores should be properly mounted and working again.

Monday, October 21, 2013

Windows 2008 Standard KMS Activation Error: 0xC004F039 The computer could not be activated. The Key Management Service could not be reached.

While trying to activate a Windows 2008 Standard server today, I received the following error after trying to set the KMS server:

  • 0xC004F039 The computer could not be activated. The Key Management Service could not be reached

After double checking the connectivity to the KMS server (a successful ping response), I looked up the syntax for setting the KMS server again. From that point, it was pretty obvious I forgot to include the port number in setting the KMS server (the port number is not required in 2008R2 if using the default port). The correct syntax should be:

  • cscript C:\Windows\System32\slmgr.vbs -skms <FQDN of KMS>:1688
  • cscript C:\Windows\System32\slmgr.vbs -ato

Thursday, October 17, 2013

Configuring Multi-site SSO and vCenter Linked Mode in vSphere 5.1

This past week, I had to configure a secondary vCenter server at our US datacenter to manage the ESXi environment in that region. As always, I've documented the steps required for future reference. This post documents the steps required to configure a secondary vCenter server in linked mode with a multi-site SSO architecture.

Prior to building the VM for the vCenter server, I had to decide on how I wanted to configure the infrastructure to maximize redundancy while minimizing the different management points. Kendrick Coleman has an excellent post (here) regarding the different architectures while summarizing their benefits and tradeoffs. In the end, I decided to implement the secondary vCenter server in linked mode to have a single pane of management while using a multi-site SSO server approach due to our unreliable MPLS connection between the datacenters. The main steps to this approach are:
  1. Install Microsoft SQL Server
  2. Install SSO Server
  3. Install Web Client and Inventory Service
  4. Install vCenter Server
  5. Install VMware Update Manager

Without ado, here are the prerequisites:
  • Windows 2008 Standard R2 Server fully patched and added to the domain
  • admin@system-domain password for the main SSO server

1. Install Microsoft SQL Server

I chose to install Microsoft SQL Server Standard 2008R2 as it is fully supported by VMware. First off, launch the installer to begin a new installation.

Select the features required and the directory for installation and then continue the installation. This is what I generally choose.

Choose the instance name and default root directory. As this is a standalone SQL instance located on the same vCenter server, the defaults were sufficient.

Select the service account to be used.

Make sure to select mixed mode if you are using a SQL service account for the vCenter server. Also, add the necessary users to be SQL administrators. For me, this was just the account I was using to do the installation.

Configure the permissions for the Analysis services and then complete the install.

SQL Server is now installed. Next start up SQL Server Configuration Manager, start the SQL Server Agent and configure the Start mode to automatic. SQL Server Agent is required by vCenter Server to do the cleanup jobs.

2. Install SSO Server

Next, the SSO Server has to be installed. I chose to install it onto the same server for simplicity purposes. First thing that needs to be done is to create the SSO database and the necessary database users.

Open up Microsoft SQL Server Management Studio and connect to the SQL server, then select "New Query".

Luckily, VMware created scripts that would automatically create the SSO database and users. They are located in the "Single Sign On\DBScripts\SSOServer\schema\mssql" directory of vCenter installer ISO. The two scripts that need to be run are the rsaIMSLiteMSSQLSetupTablespaces and rsaIMSLiteMSSQLSetupUsers scripts. 

First open the rsaIMSLiteMSSQLSetupTablespaces script and change the locations for 'RSA_DATA', 'RSA_INDEX' and 'translog'. I've placed the first two in the default SQL data folder and the latter into the default log folder.

Press execute to create the SSO database. Make sure the command is completed successfully and a refresh of the database shows the RSA database on the left.

Open the second script (rsaIMSLiteMSSQLSetupUsers) and change the RSA_DBA and RSA_USER passwords. Make sure the passwords are noted as they will be used in the future. Execute the script once everything is ready.

The SSO database is prepared and the SSO server can now be installed. Launch the SSO server installer and proceed through the prompts.

Select the second option to join an existing SSO installation and then select multi-site in the next step. Enter the information for the first SSO server. The default port should be 7444 if it was not originally changed. The password field indicates the admin@system-domain password for the primary SSO server.

Select the password for the admin@system-domain account on the secondary SSO.

Select "use an existing supported database" and then enter the JDBC connection information for the SSO database that was created earlier.

Complete the rest of the setup.

The SSO server is now installed. Since the SSO servers do not replicate or synchronize with each other automatically, one crucial step is to do a manual replication of all the user accounts, application accounts and security privileges. This is required for linked mode to function properly. The steps to the manual replication are located here. Log in to the primary SSO server, then open up a command prompt and enter the following commands:

SET JAVA_HOME=C:\Program Files\VMware\Infrastructure\jrerepl_tool.cmd export -fc:\ssoexport_primary.dat -uadmin@System-Domain

Browse to the ssoexport_primary.dat file that was created on the C:\ and copy that onto the secondary SSO server/new vCenter. Run the following commands to import the SSO configuration.

SET JAVA_HOME=C:\Program Files\VMware\Infrastructure\jrerepl_tool.cmd import -fc:\ssoexport_primary.dat -uadmin@System-Domain

The primary SSO server configuration has now been replicated onto the secondary SSO server. Once the entire vCenter is installed, the secondary SSO server configuration will need to be replicated back onto the primary SSO server.

 3. Install Web Client and Inventory Service

 Launch the Web Client installer and proceed with the install. I selected the default options for everything.


Once the Web Client has been installed, we can verify whether SSO is working properly by accessing Web Client and logging in. On any desktop computer which has Adobe Flash installed, navigate to the Web Client at https://<new-vcenter>:9443/vsphere-client/. Login with the admin@system-domain account or any account which had access to the primary vCenter/SSO server (the configuration was imported earlier so this will work).

Navigate to administration > SSO Users and Groups > Application Users and verify that there are entries Web Client (two of them), vCenter Server, vCO, Inventory Service. If they are there, then the import of the primary vCenter SSO was successful.

Next, launch the installer for Inventory Service and proceed through the installer selecting the defaults and entering the required credentials.


4. Install vCenter Server

As with any vCenter Server installation, the database and the ODBC connections must be first created before running the installer. To create the database, start up Microsoft SQL Server Management Studio and create a new query. Copy the following script below to create the vCenter database and the SQL user (vpxadmin) that will be used to authenticate to this database.


use [master]


(NAME = N'vcdb', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\VCDB.mdf' , SIZE = 2000KB , FILEGROWTH = 10% )


(NAME = N'vcdb_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Log\VCDB.ldf' , SIZE = 1000KB , FILEGROWTH = 10%)

COLLATE SQL_Latin1_General_CP1_CI_AS

use VCDB

sp_addlogin @loginame=[vpxuser], @passwd=N'Passw0rd', @defdb='VCDB', @deflanguage='us_english'


CREATE USER [vpxuser] for LOGIN [vpxuser]

sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser'

use MSDB

CREATE USER [vpxuser] for LOGIN [vpxuser]

sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser'


Note the parts highlighted in red. You may need to change the directories to where the SQL server instance was installed. Execute the script and then verify that the VCDB database is listed in the object explorer on the left.

The next step is to create the ODBC connection. Begin by creating a new System DSN and connect it to the SQL server.

Enter the vpxuser credentials and then change the default database to VCDB. Test the connection to ensure that it is successful.

Now the vCenter Server is ready to be installed. Launch the installer and proceed with the prompts.

Select "Use an existing supported database" and then select the ODBC DSN that was created earlier.

Enter the vpxuser credentials and then proceed with using the SYSTEM account.

Select to join a Linked-Mode cluster and then point it to the primary vCenter Server.

Configure the ports and then the JVM Memory limits. The defaults are what was sufficient for me.

Enter the credentials to register this vCenter server with SSO.

Add the users/groups to be administrators on this vCenter server. I only added the admin@system-domain account right now since I will be editing the permissions using the vSphere client later on.

Continue with the rest of the installation by selecting the defaults.

vCenter Server is now fully installed. Next, the SSO configuration on the secondary vCenter server will need to be replicated back to the first vCenter server. Use the commands listed earlier to export the SSO configuration.

Copy the configuration file onto the primary vCenter server and then from that server run the import commands listed earlier to import the new SSO configuration. When I ran the import command, I received an error: "acquire token request rejected". Trying it a second time resulted in success.

Once that is done, both SSO servers will have the same configuration. You will also see the application service accounts for both servers.

5. Install VMware Update Manager

The last step in this process is to install the Update Manager. Just like the vCenter Server install, the database and the ODBC connection must be first created before the installer is run. For the last time in this post, start up Microsoft SQL Server Management Studio and create a new query. Copy the following script below to create the VUM database. Note: it is assumed that vpxuser created earlier is also used to authenticate to the VUM database.

use [master]


(NAME = N'vumdb', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\VUM.mdf', SIZE = 2000KB, FILEGROWTH = 10% )


(NAME = N'vumdb_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\VUM.ldf', SIZE = 1000KB, FILEGROWTH = 10%)

COLLATE SQL_Latin1_General_CP1_CI_AS


CREATE USER [vpxuser] for LOGIN [vpxuser]

sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser'


Once again, note the areas in red where changes may be necessary. After the script is executed and the database is created, create the ODBC connection. Unlike vCenter Server, VUM uses a 32 bit ODBC connection so it will have to be created using the 32 bit ODBC Data Source Administrator located at: C:\Windows\SysWOW64\odbcad32.exe

Create and test the ODBC connection and then run the VUM installer.

Once that is complete, you can now login to any vSphere client and there should now be both vCenters in the management pane (assuming you have access to both vCenters).