Skip to main content

Troubleshooting: Some old instance manager pods are still running after upgrade

· 3 min read
Derek Su

Applicable versions​

Longhorn ≥ v0.8.0.

Symptoms​

Some old instance manager pods are still running after upgrade.

Details​

This behavior is an expected behavior rather than a bug. In the following paragraphs, we will explain why.

Let us first take a look of the example. We created a pod with a volume backed by 3 replicas in a Kubernetes cluster with 1 master and 4 workers nodes. The running volume is associated with the engine instance manager pod instance-manager-e-ec3eb207.

Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted

· 4 min read
Chin-Ya Huang
Derek Su

Applicable versions​

All Longhorn versions.

Symptoms​

The pod using a longhorn volume with an ext4 filesystem stays in container Creating with errors in the log.

  Warning  FailedMount             30s (x7 over 63s)  kubelet                  MountVolume.SetUp failed for volume "pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d" : rpc error: code = Internal desc = 'fsck' found errors on device /dev/longhorn/pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d but could not correct them: fsck from util-linux 2.31.1
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/longhorn/pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d is mounted.
/dev/longhorn/pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d contains a file system with errors, check forced.
/dev/longhorn/pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d: Inodes that were part of a corrupted orphan linked list found.  

/dev/longhorn/pvc-bb8582d5-eaa4-479a-b4bf-328d1ef1785d: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
  (i.e., without -a or -p options)