Back for 2020

It’s been a hot minute since I’ve concentrated on writing blog posts, after getting vExpert recognition I’ve let the content slide into non existence.

A busy work and family schedule meant I didn’t have the time but I’m hitting 2020 with renewed vigour and I hope these pages will again be filled with useful content.

Since the last posts (the majority of which were written at the beginning 2017!!) allot has changed, I’ve moved from the realm of infrastructure into infrastructure engineering and I’ve been exclusively working on vRealize Automation and Orchestrator and I’ve clocked up a ton of enterprise miles using the product on multiple projects.

I had a whole blog series written around how to “cloudify” vRealize 7.* including how to work CI/CD pipelines into your environment, coding practices, VM mobility and all that lovely stuff but with the release of vRA8 and the completely new design and extensibility engine I thought I’d start a fresh, the further I dive into infrastructure engineering the more I enjoy it and the more I drift into other product sets that are not VMware centric.

So going forward this blog will have a wider scope, it will likely have VMware product’s at its core but will branch off to look at alternatives, as the industry changes I want to fully transfer from being an “Infrastructure/VMware engineer” into being an automation engineer (DevOps engineer if you like)

With vRA8s new extensibility engine and native integration with Ansible, Azure, AWS etc this seems like the perfect time to do that.

With the above in mind, the first post will be about my new (ish) lab.

GUI Based VSAN Bootstrap VCSA Deployment

When deploying my 3 node NUC VSAN lab I got to try out the new bootstrap vcsa gui installer blogged about here…

https://blogs.vmware.com/virtualblocks/2017/04/25/bootstrap-vcsa-with-vsan-easy-install/#respond

Once you download and unpack the VCSA you’ll find a file called installer within the following path %filepath%\vcsa-ui-installer\win32\installer.exe (there’s a MAC & LINUX option too)

You need to have installed ESXi on your target host first, also if your networking is trunked to the ESXi host, make sure you tag the VM port group where the VCSA will reside before deploying.

By default this installer will enable the MGMT VMK for VSAN, this was fine with me as once the VCSA was deployed I retrospectively changed all the VSAN related host settings.

You will also need the VCSA to be resolvable by the DNS server you specify during the setup, I already had a Domain Controller deployed within workstation on my laptop which was routable from my lab network so I used that.

The Gui is relatively straight forward, ensure you select the option to install on a new virtual san cluster.

1

IP the VCSA

2

Select the disks on the host for the approproiate teir, if you want to enable Dedup do it now as the disk group will need to be evaucated to enable dedup and compression at a later date!

3

Do the traditional next next finish and it’ll start deploying. The VCSA deployment is now two step, once this stage is complete you need to log into the VCSA Mgmt page to complete the remainder of the setup.

4

Here is the proof of the pudding; my incredibly annoyingly named disks are now claimed by VSAN

5

The host is also now a member of a VSAN cluster.

6

A single host vsanDatastore

7

You can now complete the configuration through the vCenter

The Intel NUC\VSAN Home Lab

Having a home lab is a vital part of self-progression, I have always re-invested in myself, committing to self-study and home learning and it has helped me with career progression, so after my pizza box mistake I was in need of some toys to play with to continue that progression.

I looked into a number of options including Supermicro servers, HP Microservers and even building my own hosts in micro ATX cases but ultimately I don’t think anything gives you the same bang for their buck (or £ in my case) as the Intel NUC

The Intel nuc is the “official” unofficial VMware home lab!! There are posts all over the tinterweb from vSuperstars about why these bite size comps are ideal for a home lab, the downside being that they are not that cheap.

My BOM (bill of materials) doesn’t really differ from anyone else’s however if you’ve managed to stumble your way to my blog first, it goes like this

 

 

 

 

 

  • 3 x 5GB USB 3.0 thumb drives (To install ESXi on)

That lot cost me over £2400 (don’t tell the misses!!)

The above will build you a 3 Node all flash vSAN cluster (you will need a switch)

lab

The only thing that probably differs in my Lab to others who have chosen NUCs is that I’m using an M.2 for the capacity tier and an SSD HD for the caching tier. The only reason for this was I still had the 60gb SSDs from my pizza box mistake so I put them to good use!

The NUC only has one on board NIC so the USB to Dual port Gigabit adapter will give me three NICs per NUC (try saying that fast with a mouth full of Doritos!)

Two NICs for vSAN and NFS traffic my “Storage NICS” and 1 for everything else (vMotion, Data, MGMT)

I already have a Buffalo Terastation and a Cisco 3750, which I used to complete the lab setup, the 3750 is used as layer 2 only, I created non-routable subnets for vMotion, vSAN, NFS & VXLAN (NSX is in the lab, more on that later)

Since all devices are patched into the same 3750 switch, I didn’t have to worry about routing those subs outside the switch but carving them up in their own VLAN helps limit broadcast traffic.

Mgmt and VM data sit on the same subnet, it’s not ideal… but it’s a lab.

The Terastation is used to house ISO\OVAs and maybe eventually the occasional low I/O VM, it has two NICs. One NIC is on the unrouteable NFS sub. The other NIC is on my mgmt.\data subnet so I can manage it!

All NUC NICS (USB & ONBOARD) support an MTU of more than 1600, which means I can have NSX in my lab, however unfortunately they don’t quiet support Jumbo MTU, again it’s a lab but in real world scenarios VSAN, NFS and probably vMotion too should be across a JUMBO frame enabled NIC. vSAN should also be backed ideally by a 10gb SWITCH when using all flash, but the 1GB Switch in my case works just fine… it’s a lab…just tell yourself “I wont buy a 10gb switch… I don’t need a 10gb switch…”

So “back of a fag packet” logical & physical designs…

Physical

physical

Logical

logical

There are a few things to consider when building the NUC lab.

  1. Some versions of ESXi don’t support the NUC/USB NICS so you’ll need to create a custom ISO image with the required VIBs. That MAY have changed with versions later than 6.5 u1 (I installed 6.5 U1) I’ve not kept in the loop with what does and doesn’t work. William Lams blog or virten.net maybe able to help you on that front.

My details to create a custom ISO for 6.5 U1 can be found here

  1. If you’re planning to use VSAN as your sole storage resource, then you’ll need to “bootstrap” VSAN to deploy your vCenter, as VSAN configuration is predominately done in vCenter and since you cant deploy a vCenter without storage, you get stuck in a chicken and egg scenario. William LAM has some great tech blogs on how to do this manually, but since vCenter 6.5 u1 there’s an option within the VCSA deployment gui to configure VSAN as you deploy the VCSA. I’ve briefly blogged about that here.

The LAB performs really well, I’ll throw up a bench marking blog at some point, but for now I hope you found the above useful!

Pizza box mistake…

Since leaving EMC and subsequently losing access to a lab, I had been mulling over a home lab for some time. Ultimately, what I wanted to do was keep upfront costs down. I built a 3 node DL360 G6 vSAN lab with 3 x 10K SAS HDs and a 60gb SATA SSD per node, 24gb RAM per node, 2 x Intel Xeon procs, it didn’t perform to shabby and all for a few hundred quid, pretty good buy or so I thought!

I’m not stupid (honest) I’d read horror stories about large electricity bills and thought “meh, how bad can it be”. Luckily Scottish power (my energy provider) are on hand with their year on year electricity comparison graph to show just how bad it can be, from 15.05kWh on average (over 92 days) to 27.03kWh on average (over a shorter 78 days)

My capex saving is now an opex cost…

Take my advice, avoid the pizza boxes… learn from my mistake, like UK Top Gear, I have been ambitious… but rubbish!

Now to start planning my next lab…

By the way, I have three DL360 G6s going if anyone wants to buy them?

 

VMware vCenter Workflow Manager service not starting

I had an issue where a vCenter update to 6.0u2a failed and rolled back, after a restart we were unable to power on VMs there were in a powered off state.

On further inspection we found that the “VMware vCenter workflow manager” service wasn’t started and would not start.

In the Workflow-manager.log log located here C:\ProgramData\Vmware\vcenter\logs\workflow\ I could see error relating to the service not starting and “Error Creating bean with name ‘jmxconnectorstarter’”

1

Scroll a little further in the Workflow-manager.log and you should see the phrase “Caused by”

2

My error stated “Port value out of range: -1”

Browsing to the workflow.properties file located here C:\ProgramData\Vmware\vCenterServer\cfg\vmware-vpx-workflow\conf

3

In this file the workflow.jmx.port was set to -1, this was changed to 19999 by VMware support and hey presto, the service started!

Powering on all VMs was then possible within the VC.

Moral of this story… name your beans!

free-comedy-movies-5

vCenter 6u2a Upgrade failure with error 3010

I had an issue upgrading a VC to 6.0u2a recently, when running the upgrade I received the error “failed with error code 3010”

1

There’s a VMware KB here which explains the issue, however the resolution description is light on detail.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2149266

Check the pkgmgr-comp-msi.log to find out what DLL is locked; the log files will be in the ZIP file downloaded when the installation fails.

I found it useful to search for “is being held in use” within this log file to identify the dll being held.

2

In my case it was vmeventmsg.dll.

I stopped the vCenter Service (which in turn stops all other services that the VC is a decency of)

Using process explorer

Download – https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx

Open process explorer and select “show details for all processes”  under file.

3

Under view ensure “Show processes from all users” is selected.

4

Select  Find > find handle or ddl (ctrl+f) and seach for the dll identified in the pkgmgr-comp-msi.log, the PID will be listed (in my case 912)

5

In process explorer kill the process that’s holding the dll hostage, by right clicking and selecting “Kill Process”

6

NOTE – I first tried using taskkill to end the process however it didn’t work, even though it would be listed when running tasklist.

Double check in processes explorer that the DLL is no longer in use.

Re-run the VC upgrade, go to the Winchester, have a nice cold pint, and wait for all of this to blow over.

7

Support of 3rd party vSwitches removed

VMware has announced the discontinuation of the 3rd party vSwitch program in vSphere 6.5 u1, most notably the Nexus 1000v,

When I say they’re discontinuing the program, I mean they’re actually removing the APIs 3rd party switches interact with! [ref 1]

Customers who remain on 5.5 or 6.0 and stick with the 1k will still be entitled to support.

Full disclosure, I’m an ex VCE (Now Dell EMC CPSD) employee. I was an Escalation Engineer responsible for Vblock support, Vblocks come pre-packaged with the Nexus 1000v. You’d think that level of exposure would lead me down a path of whimsical nostalgia… “It had its place” or “it’ll be missed” maybe phrases you’d expect an advocate of the Vblock to spout against the back drop of such news, however…  I’m ecstatic

My feelings about the 1K are best summed up by Moff from human traffic and his hatred for Peter Andre…

The vDS is more than adequate to fit the use cases that previously could only be meet by the 1K. As we move further into the world of converged infrastructure and software defined the 1K represents a time where infrastructure teams had stricter silos, where network engineers wanted more control over the virtual network.

Not to mention the hundreds of hours of downtime caused by server engineers trying to find their way around a nexus switch for the first time. (if you ever missed the word “add” when adding VLANs to the Uplink Port Profiles you’ll understand my pain)

VMware have a 1000v to vDS automated migration tool available on their download site, however the documentation shows a support matrix that’s a bit thin on the ground when it comes to supported 1K versions [ref 2]

At time of writing, it lists only 2 supported 1k Versions which I assume translate as follows.

2.2.3 = SV2(2.3)

3.1.3 = SV3(1.3)

Table 1‑1 Support Matrix

ESXi Version Nexus 1000v Version
5.5 2.2.3
3.1.3
5.1 2.2.3
1.5.2b

If the above rings true, from a Vblock perspective only RCMs 5.0.5 > 5.0.8 are officially in scope/tested for this tool, although I’m sure the good people at CPSD have a plan\announcement pending for this relating to their own tool/guide/professional service to move away from the 1K.

Of course, it’s worth reiterating that this only becomes relevant if you intend to move to vSphere 6.5, an RCM based on 6.5 isn’t even GA yet…

Those that will stay on RCM 5.0.* or RCM 6.0 .* until tech refresh can sit back and relax knowing they can still get support.

The download also contains a manual guide to migrating away; the process is straight forward but could be lengthy if you have a large number of port-profiles to migrate to port groups.

For Vblock or for those using the 1k in a UCS environment, if you go down the manual migration route keep in mind what load balancing algorithm you can select. UCS only supports route based on virtual port ID or route based on source MAC hash.

Route based on IP Hash and route base on physical NIC Load are not supported in UCS [ref 3]

[ref1]

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2149722

[ref2]

https://my.vmware.com/web/vmware/details?downloadGroup=MIGRATION-TOOL&productId=614

[ref3]

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/200519-UCS-B-series-Teaming-Bonding-Options-wi.html