NSX – New Compute Manager – FQDNs are not configured!
During my latest VCF lab deployment I encountered an issue that was not easily found both internally and external. Therefor I needed to troubleshoot the issue in detail which resulted in a working environment and now known root-causes
My homelab is old and started it’s life in the Windows 2012R2 era. This means I still use the DNS-services from that version. Up until now it worked flawlessly and I had no compatibility or integration issues. Other, not so wise, decision was to use a .local domain. We know now this domain is used for multicast DNS scenarios resulting in known issues on the Photon platform.
So, knowing the above lab-details, what’s the deal with deploying NSX? Below the error I got from the Cloudbuilder during the SDDC bringup phase where it tries to add a vCenter/Compute Manager as part of the Fabric.
The action fails with the following error (as found in the logs):
2024-04-09T15:01:50.306Z m01-nsx01a.vmw.local NSX 87186 FABRIC [nsx@6876 audit="true" comp="nsx-manager" <br>level="INFO" reqId="8d4d015a-1354-473a-a442-24da457d62db"
subcomp="cm-inventory" update="true" username="admin"] <br>UserName="admin", Src="m01-nsx01.vmw.local:443",
ModuleName="InventoryCmObj", Operation="AddComputeManager", <br>Operation status="failure",
New value=[{"server":"m01-vc01.vmw.local","origin_type":"vCenter","credential":<br>{"username":
"svc-m01-nsx01-m01-vc01@vsphere.local","thumbprint":"9A:03:E3:C3:56:B5:80:31:BA:91:F3:31:C6:FD:<br>12:E8:61:BD:70:4B:44:A1:DC:B4:CD:99:71:0C:<br>85:24:52:D7",
"credential_type":"UsernamePasswordLoginCredential"},"create_service_account":false,"set_as_oidc_pro<br>vider":
true,"access_level_for_oidc":"FULL","reverse_proxy_https_port":443,"multi_nsx":false,"display_name":
"m01-<br>vc01.vmw.local"}]<br><br>2024-04-09T15:01:50.323Z m01-nsx01a.vmw.local NSX 87186 SYSTEM
[nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="cm-inventory"] UserName:'admin'
ModuleName:'inventory-mgmt' Operation:'POST@/api/v1/fabric/compute-managers' Operation status: 'failure'
Error: FQDNs are not configured for nodes IPs [192.168.1.67, 192.168.1.66, 192.168.1.69, 192.168.1.68]The error we need to zoom into is: “FQDNs are not configured for nodes IPs“. This means NSX-Manager is having issues resolving the IP’s to a FQDN. In my lab the first reason for it to fail is the use of the .local domain. As mentioned, this is a known issue on (at least) PhotonOS and easily fixed by altering the resolving.conf file:
Alter your /etc/systemd/resolved.conf to the below containing your DNS-server and local domain-names
[Resolve]
DNS=192.168.1.220
Domains=local vmw.local
Restart the service with:
systemctl restart systemd-resolved
You should now be able to ping/nslookup fqdns with your .local domain!Now, for the more interesting part of the way NSX resolves ip-addresses/fqdns! NSX leverages dig within a wrapper to achieve that. In my lab using dig against the W2012-DNS fails to report the correct answer:
root@m01-nsx01c:~# dig @192.168.1.220 m01-vc01.vmw.local
; <<>> DiG 9.16.1-Ubuntu <<>> @192.168.1.220 m01-vc01.vmw.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 59770
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: aca1ad82a8aaf8b9 (echoed)
;; QUESTION SECTION:
;m01-vc01.vmw.local. IN A
;; Query time: 4 msec
;; SERVER: 192.168.1.220#53(192.168.1.220)
;; WHEN: Thu Apr 11 11:57:06 UTC 2024
;; MSG SIZE rcvd: 59After some investigation it was clear I needed to make use of some additional options to get dig to report correct answers. The options that make it work are +noedns and/or +nocookie:
root@m01-nsx01c:~# dig +noedns @192.168.1.220 m01-vc01.vmw.local
; <<>> DiG 9.16.1-Ubuntu <<>> +noedns @192.168.1.220 m01-vc01.vmw.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27174
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;m01-vc01.vmw.local. IN A
;; ANSWER SECTION:
m01-vc01.vmw.local. 3600 IN A 192.168.1.232
;; Query time: 4 msec
;; SERVER: 192.168.1.220#53(192.168.1.220)
;; WHEN: Thu Apr 11 11:57:40 UTC 2024
;; MSG SIZE rcvd: 52Great, dig is working now!! But, it’s still not the solution to register the new Compute Manager… The NSX-syslog provides us with the lead where to look.
... getFqdnFromIp(): invoked with Ip Address 192.168.1.66
... Executing script: /opt/vmware/nsx-common/python/nsx_utils/ip_utils.py -dx 192.168.1.66
... getFqdnFromIp(): script returned FQDN:As you can see NSX leverages the highlighted script to resolve ip addresses and fqdn’s. Within this script we find it uses dig to do the actual job. This means this script needs to output a valid ip address or fqdn for the registration of a Compute Manager to work. The above log-snippet shows we get an empty result and therefor the registration fails.
Now, in my homelab case, we need to implement/add the +noedns option in this script. Note the below changes to the script and the result of executing it.
At line 31 add a new option:
_NOEDNS_OPT = ['+noedns']
Add this to the code on lines 351, 354 and 378 as follows:
cmd = (cmd + _IPV6_OPT + [fqdn] + _NOEDNS_OPT + _SHORT_OPT +
_FORWARD_LOOKUP_IP6_FILTER_OPT)
The result:
root@m01-nsx01a:/opt/vmware/nsx-common/python/nsx_utils# /opt/vmware/nsx-common/python/nsx_utils/ip_utils.py -dx 192.168.1.66
m01-nsx01.vmw.localAs a result of the above changes the registration of a new Compute Manager in NSX will succeed!
This concludes this blog item. Hope you find it useful!
