vCloud Director: How to bypass SAML authentication for a tenant Org in the H5 UI

Customer turned on SAML Authentication for their Org. and then after that could not login using SAML and did not know how to use local authentication because now each time they logged in it asked for SAML. 

The only URL that was given to them was. https://(VCD FQDN)/tenant/(org)/vdcs 

So essentially, they were locked out, and did not know how to get back in. They did not know why SAML was not working either. So called for support. First, I gave them the URL to be able to login with local authentication; and verified they could get in while I looked into the SAML issue. 

 https://(VCD FQDN)/tenant/(org)/login 

After looking thru the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log 

I see errors pertaining to time sync issues: 

 

org.opensaml.common.SAMLException: Response issue time is either too old or with date in the future, skew 60, time 

Since this was a customer SML setup I did not have access to the Azure AD servers they were authenticating to. I asked them to check the time on the AD server. 

While waiting I restarted NTP service and forced a NTP time sync. Even though the local time looked fine in VCD. After that SAML started working.

NSX-T NV exam

Few weeks ago, I took the NSX-T exam and passed it with flying colors as people might say. this exam has haunted me for at least a year now. It was something I wanted to accomplish but me networking skills we a little amateur in my opinion. I had issues with understanding fully about routing and subnetting. https://www.vmware.com/education-services/certification/vcp-nv-nsxt-3-0-exam.html So a few months ago I was approved to take the test and get re-imbursed from my work, so I put my head down and started reading blogs, doing labs, etc.. anything I could find for free to do with NSX. I asked other vExperts about the test and see what they did to prep for the test. I was nervous to say the least… I got to a point where I figured I would give it a shot. I scheduled the test on afternoon so I could cram study in the morning. After a night and morning of cramming in as much info as I can I went to take the test. Concentrating on command line as someone mentioned that was on the test, and probably my weakest. Went in and took the test, not sure if it was the batch of questions I got there were only about 3 command line questions, YAY !!!! But still a pretty good test and covered all aspects of the topics listed in the exam guide. ( https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/certification/vmw-vcp-nv-exam-preparation-guide.pdf) I ended up passing the exam to my excitement… If you’re looking to learn more about Network Virtualization and want to test your knowledge I would highly recommend this test.

ESX Hosts disconnected from vCenter, Unable to connect them back.

I had a couple Hosts  disconnected from vCenter, 

First step will be to restart management agents.

 A. Restart Management agents in ESXi Using ESXi Shell or Secure Shell (SSH):

1. Log in to ESXi Shell or SSH as root.
   
   For Enabling ESXi Shell or SSH, see Using ESXi Shell in ESXi 5.x and 6.x (2004746).
    
2. Restart the ESXi host daemon and vCenter Agent services using these commands:
   
   /etc/init.d/hostd restart
   
   /etc/init.d/vpxa restart

B. To restart all management agents on a host:  (Please note the Cautions below)

• To restart all management agents on the host, run the command:
  
  services.sh restart

Caution:

• If LACP is enabled and configured, do not restart management services using services.sh command. Instead restart independent services using the /etc/init.d/module restart command.
• If the issue is not resolved, and you are restarting all the services that are a part of the services.sh script, take a downtime before proceeding to the script.
• If NSX is configured in the environment, do not run the /sbin/services.sh restart command because this will restart all services on the ESXi host. If you need to restart the management agents on the ESXi host, restart vpxa, host.d, and fdm individually. If you also need to run the /sbin/services.sh restart command because restarting each management agent does not work, then migrate all the VMs off the ESXi host and put the host in maintenance mode if possible.

 

If restarting Hostd and vpxa service does not work ( as in my case)  and you are on 6.7 u1, and while in SSH you try to do ESXCLI commands and get a error about connection refused.......   refer to this KB..   

https://kb.vmware.com/s/article/78124

Also run this command to check if there is an issue with libcimsvc:

cat /var/log/vobd.log | less

If you see this in the vobd.log over and over:   [UserWorldCorrelator] 995067852232us: [vob.uw.core.dumped] /bin/sfcbd(5856860) /var/core/sfcb-intelcim-zdump.000

Then you know the libcimsvc is failing even though hostd service says its running, and perform the workaround in the above VMware KB.

To workaround follow the below steps

1. /etc/init.d/hostd stop
2. edit the /etc/vmware/hostd/config.xml

Find the line:
     <cimsvc>
        <path>libcimsvc.so</path>
        <enabled>true</enabled>
     </cimsvc>

 set to: <enabled>false</enabled>

3. save the file
4. /etc/init.d/hostd start

 

-----------------------------------------------------------------------------------------------------------------------------------------------

 

Here are those other bugs for hosts getting disconnected from VC:

 

https://kb.vmware.com/s/article/70597 : 

ESXi 6.x host is disconnected from vCenter Server due to dcism exhausting inodes 

https://kb.vmware.com/s/article/74966 : 

ESXi 6.5/6.7 hangs during certain tasks like maintenance mode, connecting to vCenter, or after a reboot.

https://kb.vmware.com/s/article/67920 :

Multiple attempts to log in to an ESXi host with incorrect credentials might cause the hostd service to stop responding (CVE-2019-5528)