Skip to main content

Troubleshooting: DNS Resolution Failed

· One min read
JenTing Hsiao

Applicable versions​

All Longhorn versions.

Symptoms​

The longhorn-driver-deployer or longhorn-csi-plugin or longhorn-ui Pods unable to access the longhorn manager backend http://longhorn-backend:9500/v1.

Reason​

The CoreDNS of the Kubernetes cluster is unable to resolve the longhorn-backend service, causing the DNS resolution to fail.

Solution​

  1. Check the longhorn-backend Service is available. 
    kubectl get service longhorn-backend -n longhorn-system
  2. Make sure the longhorn-manager Pod(s) is/are up and running. 
    kubectl get pod -l app=longhorn-manager -n longhorn-system
  3. Make sure the CoreDNS Pod(s) is/are up and running. 
    kubectl get pod -n kube-system
  4. SSH into one of the longhorn-manager Pod, and check the nslookup test result, to make sure the DNS resolution result. 
    kubectl exec -it <longhorn-manager-pod-name> -- nslookup longhorn-backend
  5. If you're using k3s or rke2, add 
    K3S_RESOLV_CONF=/etc/resolv.conf
    RKE2_RESOLV_CONF=/etc/resolv.conf
    to k3s or rke2 env files on all nodes, then restart the k3s or rke2 server/agent service.
  6. If you're not using k3s or rke2, check the Kubernetes DNS Resolution Known Issues to see how to set kubelet's --resolv-conf flag to point to the correct resolv.conf.