AWS k8s pod CrashLoopBackOff 상태 시 체크 내용

AWS k8s pod CrashLoopBackOff 상태 시 체크 내용

- 갑자기 pod status가 CrashLoopBackOff 로 되어 비정상 동작

1. kubectl get pod 

* kubectl get pod --output=wide
* kubectl get pod node -o wide

xxx-6b474844f5-zsbxp                1/1     Running            0          4d6h
xxx1-7fdbb846b5-2ks98            0/1     CrashLoopBackOff   3          114s
xxx1-7fdbb846b5-ssmsj            0/1     CrashLoopBackOff   3          114s

 

2. kubectl exet -ii xxx1-7fdbb846b5-2ks98 //bin//sh

pod가 정상적으로 동작 안하기 때문에 접속 불가

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
error: unable to upgrade connection: container not found ("xxx1")

 

3. kubectl describe pod xxx1-7fdbb846b5-2ks98

pod 상태 확인

...
  Warning  BackOff    31s (x7 over 112s)  kubelet            Back-off restarting failed container

 

4. kubectl logs pod xxx1-7fdbb846b5-2ks98 -p

pod 로그 조회

===================================================================
2022-11-07 17:17:09 [INFO]: Listening on port 3031...
...
  code: 'ER_HOST_IS_BLOCKED',
  errno: 1129,
  sqlMessage: "Host 'xxx.xx.xx.xx' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'",
  sqlState: undefined,
  fatal: true

대략 DB 연결이 실패되는 것으로 판단

 

RDS 재구동해서 처리

 

끗!@#@!#%$!#