St. Louis UserCon

 

The St. Louis UserCon on March 1st 2023, was a great event this year attendance was up weather was great, overall I would rate it a 8 out of 10.

First off the night before the usercon we had tickets for speakers and distinguished guest to go to a St. Louis Blues Hockey Game. It was great to be able to interact with speakers and other VMUG members before the usercon in a casual environment. everyone enjoyed the game and thought doing something like that is a great idea.

The next day was an early day for the usercon to make it tot he convention center by 7 AM I had to leave my house at 6 AM.  Got there and went over the opening scripts and how we are doing the intro's of people that day. opened up the usercon to the largest crown we have seen since COVID started, and was great to see.

Chris Wolf VMware CIO was our keynote speaker, he is always great to listen to and did a great job. This year our charity of choice we Toys for Tots toy drive, and had a member of the marines Corps Reserve in attendance to talk about what they do.  At the end of the day we collected about 15 toys, and over $500 in monetary donations.

I had two speaking session at the usercon, first one was for the company I work for Lumen, talking about their Private Cloud offering and a second on getting back to the basic's with vSphere.

I cant wait till next year to see how much we grow now that COVID is over !!!

Cincinnati VMUG UserCON - Speaking Session

 Cincinnati VMUG UserCON

December 8th 2022
8am – 4:30pm
In-person at the Sharonville Convention Center

The Cincinnati VMUG community welcomes you back to the Sharonville Convention Center to network and engage in face-to-face conversations with your peers at the Cincinnati VMUG UserCon. Learn about the latest in IT innovations to help you stay on the cutting edge of technology. Advance your knowledge, become a better IT professional and grow as a strategic leader for your organization..

I had the privilege to be invited to speak at the  Cincinnati VMUG UserCON. I was invited last min. not sure if they had a cancelation, but I am grateful to be asked to speak and spread my knowledge when I can.

I used the same topic I have spoken to before.

This is a presentation that goes over some basics/ best practices when building your own ESX hosts that seems to get missed on builds as I have been supporting VMware. And this presentation is short, but i open it up for questions about how to configure ESX hosts and questions about the info I presented. Each time I get a half dozen questions about this topic, and people come to me after saying there were things in the presentation they did not know about.

vCloud Director: How to bypass SAML authentication for a tenant Org in the H5 UI

Customer turned on SAML Authentication for their Org. and then after that could not login using SAML and did not know how to use local authentication because now each time they logged in it asked for SAML. 

The only URL that was given to them was. https://(VCD FQDN)/tenant/(org)/vdcs 

So essentially, they were locked out, and did not know how to get back in. They did not know why SAML was not working either. So called for support. First, I gave them the URL to be able to login with local authentication; and verified they could get in while I looked into the SAML issue. 

 https://(VCD FQDN)/tenant/(org)/login 

After looking thru the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log 

I see errors pertaining to time sync issues: 

 

org.opensaml.common.SAMLException: Response issue time is either too old or with date in the future, skew 60, time 

Since this was a customer SML setup I did not have access to the Azure AD servers they were authenticating to. I asked them to check the time on the AD server. 

While waiting I restarted NTP service and forced a NTP time sync. Even though the local time looked fine in VCD. After that SAML started working.

NSX-T NV exam

Few weeks ago, I took the NSX-T exam and passed it with flying colors as people might say. this exam has haunted me for at least a year now. It was something I wanted to accomplish but me networking skills we a little amateur in my opinion. I had issues with understanding fully about routing and subnetting. https://www.vmware.com/education-services/certification/vcp-nv-nsxt-3-0-exam.html So a few months ago I was approved to take the test and get re-imbursed from my work, so I put my head down and started reading blogs, doing labs, etc.. anything I could find for free to do with NSX. I asked other vExperts about the test and see what they did to prep for the test. I was nervous to say the least… I got to a point where I figured I would give it a shot. I scheduled the test on afternoon so I could cram study in the morning. After a night and morning of cramming in as much info as I can I went to take the test. Concentrating on command line as someone mentioned that was on the test, and probably my weakest. Went in and took the test, not sure if it was the batch of questions I got there were only about 3 command line questions, YAY !!!! But still a pretty good test and covered all aspects of the topics listed in the exam guide. ( https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/certification/vmw-vcp-nv-exam-preparation-guide.pdf) I ended up passing the exam to my excitement… If you’re looking to learn more about Network Virtualization and want to test your knowledge I would highly recommend this test.

ESX Hosts disconnected from vCenter, Unable to connect them back.

I had a couple Hosts  disconnected from vCenter, 

First step will be to restart management agents.

 A. Restart Management agents in ESXi Using ESXi Shell or Secure Shell (SSH):

1. Log in to ESXi Shell or SSH as root.
   
   For Enabling ESXi Shell or SSH, see Using ESXi Shell in ESXi 5.x and 6.x (2004746).
    
2. Restart the ESXi host daemon and vCenter Agent services using these commands:
   
   /etc/init.d/hostd restart
   
   /etc/init.d/vpxa restart

B. To restart all management agents on a host:  (Please note the Cautions below)

• To restart all management agents on the host, run the command:
  
  services.sh restart

Caution:

• If LACP is enabled and configured, do not restart management services using services.sh command. Instead restart independent services using the /etc/init.d/module restart command.
• If the issue is not resolved, and you are restarting all the services that are a part of the services.sh script, take a downtime before proceeding to the script.
• If NSX is configured in the environment, do not run the /sbin/services.sh restart command because this will restart all services on the ESXi host. If you need to restart the management agents on the ESXi host, restart vpxa, host.d, and fdm individually. If you also need to run the /sbin/services.sh restart command because restarting each management agent does not work, then migrate all the VMs off the ESXi host and put the host in maintenance mode if possible.

 

If restarting Hostd and vpxa service does not work ( as in my case)  and you are on 6.7 u1, and while in SSH you try to do ESXCLI commands and get a error about connection refused.......   refer to this KB..   

https://kb.vmware.com/s/article/78124

Also run this command to check if there is an issue with libcimsvc:

cat /var/log/vobd.log | less

If you see this in the vobd.log over and over:   [UserWorldCorrelator] 995067852232us: [vob.uw.core.dumped] /bin/sfcbd(5856860) /var/core/sfcb-intelcim-zdump.000

Then you know the libcimsvc is failing even though hostd service says its running, and perform the workaround in the above VMware KB.

To workaround follow the below steps

1. /etc/init.d/hostd stop
2. edit the /etc/vmware/hostd/config.xml

Find the line:
     <cimsvc>
        <path>libcimsvc.so</path>
        <enabled>true</enabled>
     </cimsvc>

 set to: <enabled>false</enabled>

3. save the file
4. /etc/init.d/hostd start

 

-----------------------------------------------------------------------------------------------------------------------------------------------

 

Here are those other bugs for hosts getting disconnected from VC:

 

https://kb.vmware.com/s/article/70597 : 

ESXi 6.x host is disconnected from vCenter Server due to dcism exhausting inodes 

https://kb.vmware.com/s/article/74966 : 

ESXi 6.5/6.7 hangs during certain tasks like maintenance mode, connecting to vCenter, or after a reboot.

https://kb.vmware.com/s/article/67920 :

Multiple attempts to log in to an ESXi host with incorrect credentials might cause the hostd service to stop responding (CVE-2019-5528)

 

Dallas VMUG UserCon Speaking

 I had the pleasure of getting invited to have a speaking engagement at the Dallas VMUG UserCon back on August 11th , 2020.  I was asked to be a speaker on August 4th and they wanted everything done by the 7th.  that gave me only 3 days to put together a presentation and record a video. Needless to say it was rushed a little as I did not have a presentation ready.  HA.

https://www.vmug.com/events2/vmug-usercon/2020-dallas-usercon

The presentation I put together was named, " It's the basics that build a better foundation to a better environment".  This session we will be getting back to the basics of vSphere and going over some of the best practices and what they mean to your environment and how they can be deter mental to performance of your infrastructure.

-Cluster Configs

   Plan for server failure (HA)

       An important consideration for cluster design is planning for server failure, or planned                 maintenance.

•         N+1 Design

• Ref. https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/vcat/architecting-a-vmware-vsphere-compute-platform.pdf

   DRS to balance workloads

        Although it is not essential that you configure vSphere DRS , VMware recommends using this                 mechanism as a way of balancing workloads across hosts in the cluster for optimal performance.

• Ref. https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/vcat/architecting-a-vmware-vsphere-compute-platform.pdf

-Host General Settings

• Disconnect or disable any physical hardware devices that you will not be using. These might include devices such as:

• COM ports

• LPT ports

• USB controllers

• Floppy drives

• Optical drives (that is, CD or DVD drives)

• Network interfaces

• Storage controllers

• Disabling hardware devices (typically done in BIOS) can free interrupt resources

• DNS and NTP settings…

• Scratch Configs

• It's recommended by VMware to have a persistent scratch location for VMkernel logs in case that ESXi is installed on USB stick or SD card.

• Remember to redirect the scratch partition, if applicable. In a design that employs SD/USB or boot from SAN as the installation destination, the host installer does not allow for the creation of a scratch partition during the initial setup process. ( ramdisk )

• Separate vMotion traffic and management traffic onto separate NIC’s
• Keep hosts Firmware updated 
• Update ESX with Vendor ISO’s from Vmware
• Verify Firmware and drivers with Vmware HCL

-Host storage Settings

• General ESXi Storage Recommendations.

The number of LUNs in a storage array, and the way virtual machines are distributed across those LUNs, can affect performance: 

• Provisioning more LUNs, with fewer virtual machines on each one, can enable the ESXi servers to simultaneously present more I/O requests to the array. 

• On the other hand provisioning too many LUNs, especially when many ESXi servers are connected to a single array, can allow the ESXi hosts to simultaneously send so many I/O requests that they fill the array queue and the array returns QFULL/BUSY errors. This can reduce performance due to the need to retry the rejected I/O requests.

• Queue Depth for HBA’s  ( VMware KB 1267)

• If the performance of your (HBAs) is unsatisfactory, you can adjust your ESXi hosts' maximum queue depth values. 

• When you lower this value, it throttles the ESXi host's throughput and alleviates SAN 

• VMware recommends keeping the HBA Vendors uniform across all hosts participating in a cluster. 

• If you often observe QFULL/BUSY errors, enabling and configuring queue depth throttling might improve storage performance. 

• https://www.codyhosterman.com/2017/02/understanding-vmware-esxi-queuing-and-the-flasharray/

• iSCSI and NFS Recommendations

• For iSCSI and NFS it’s sometimes beneficial to create a VLAN, if the network infrastructure supports it. This minimizes network interference from other packet sources.

• Best practice is to have a dedicated LAN for iSCSI traffic and not share the network with other network traffic. It is also best practice not to oversubscribe the dedicated LAN.

• VMkernel NICs can be placed into teaming configurations. VMware recommendation is to use port binding rather than NIC teaming. iSCSI can leverage VMkernel multipath capabilities such as failover on SCSI errors and Round Robin path policy for performance.

• Aim to configure as few as possible hardware segments between the servers in a cluster. to limit single points of failure, which is best achieved through simplicity. In addition, too many network hops can cause networking packet delays for heartbeat traffic and increase the possible points of failure.

-Host Security Settings

• ESXi Host Hardening

To provide an ESXi security baseline, consider the requirements for hardening the hypervisor. 

VMware guidance on security hardening and the recommendation level depends on the rating that corresponds to the operational environment in which it is to be applied. Each service provider will need make their own determination as to the applicability of each level.

• Lockdown Mode

• Enable lockdown mode to increase security of ESXi hosts and to further mitigate the risk of unauthorized access to the ESXi console by limiting it to only the appropriate operational team through vCenter Server

• Ref. https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/vcat/architecting-a-vmware-vsphere-compute-platform.pdf

-VM Settings

• Remove unused and unnecessary virtual and/or physical hardware from the VM

• By disabling devices you are freeing up interrupt resources.  

You also will get an increase in performance by disabling devices that consume extra resources due to polling such as USB adapters and PCI devices which reserve blocks of memory for their operation.  

Also, when using Windows guests be sure to disable optical drives as Windows constantly polls them which can cause issues especially when multiple guests are doing so simultaneously.

• Keep VMware tools updated.

• The Paravirtual SCSI (PVSCSI) controller should be used on guest VMs for high performance.

• The Paravirtual SCSI (PVSCSI) controller should be used on guest VMs for high performance. It is important that users choose the correct PVSCSI controller, because choosing the wrong controller can adversely affect performance.

• Use multiple SCSI adapters on a VM ( max 4 ) and spread the I/O across those SCSI controllers.

• Try not to use Paravirtualized for the OS disks.

• If need more I/O than the default PVSCSI Queue Depth refer to this:

• Large-scale workloads with intensive I/O patterns might require queue depths significantly greater than Paravirtual SCSI default values (VMware KB 2053145)

• Snapshots ( are not backups !!  ☺ )

• Delete old backups

The session went well there were a few questions from viewers, so all in all I guess it went OK.. I have not received any negative feedback from VMUG or anything as of this post.

Well I went to AWS Re-Invent for my speaking session, and I must say that was an adventure. I have been to large conventions before, like VMworld, Dell , etc. This by far was the largest.
I am not sure how many hotels were involved, seemed like every hotel on the strip had something for Re-Invent, from sessions, meals, presentations, or dinning specials for attendee's. I believe there was an estimated 60,000 people in attendance, every session was full that I tried to attend so I spent time on waiting lists, or in line hoping for a spot.
As far as the session I was speaking at, there were allot of seats open, i invited everyone I know or met to attend the session. All in all we got about 25-30 people to attend the session on Zerto. And it let to some good discussions around VM's and protecting them in AWS.
I also tested for AWS Sysops certification, I first took a boot camp the day before and paid additional $225 for, which in my opinion was not worth the money. It did not teach more that I feel i could have gotten offline. The test was challenging not a typical certification teat I have taken before, very in depth and really need to understand how AWS works. Needless to say I did not pass the test I came close I believe within about 5 questions... I am going to study more and take the test again in the next few month's...
I dont think I will go back to Re-Invent unless asked again, it was to many people and could not get into any sessions I wanted to, I will stick o smaller conferences  :-)..