Skip to main content

Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume

· 2 min read
Chin-Ya Huang

Applicable versions​

All Longhorn versions.

Symptoms​

Recurring job does not create new jobs when the volume is attached after being detached for a long time.

According to Kubernetes CronJob limitations:

For every CronJob, the CronJob Controller checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error

Cannot determine if job needs to be started. Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

That means the duration of the attach/detach operation that the recurring job can tolerate is depending on the scheduled interval.

For example, if the recurring backup job is set to run every minute, then the toleration would be 100 minutes.

Troubleshooting: Use Traefik 2.x as ingress controller

· One min read
JenTing Hsiao

Applicable versions​

All Longhorn versions with Traefik 2.x as ingress controller.

Note that, the CORS problem would not happen with Traefik 1.x as ingress controller.

Symptoms​

The Longhorn GUI frontend API requests sometimes failed to reach longhorn-manager backend.

Reason​

The API requests would change the protocol between HTTPS/WSS, and the change would lead to a CORS problem.

Troubleshooting: Create Support Bundle with cURL

· 2 min read
Chin-Ya Huang

Applicable versions​

All Longhorn versions.

Symptoms​

Not able to create the support bundle with a web browser.

Solution​

  1. Expose Longhorn backend service. Below is an example using NodePort, you can also export via load balancer if one is set up. 
    ip-172-30-0-21:~ # kubectl -n longhorn-system     patch svc longhorn-backend -p '{"spec":    {"type":"NodePort"}}'
    service/longhorn-backend patched
    ip-172-30-0-21:~ # kubectl -n longhorn-system get     svc/longhorn-backend
    NAME               TYPE       CLUSTER-IP          EXTERNAL-IP   PORT(S)          AGE
    longhorn-backend   NodePort   10.43.136.157       <none>        9500:32595/TCP   156m