GitLab CI Timeout with Kaniko and EKS - gitlab

Question

I am trying to follow the GitLab example code for using kaniko as outlined here. The only thing I have changed is that I am using the v1.7.0-debug tag instead of simply debug.
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.7.0-debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
- >-
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
My build job is stalling out at the following line:
Running with gitlab-runner 14.4.0 (4b9e985a)
on gitlab-runner-gitlab-runner-84d476ff5c-mkt4s HMty8QBu
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.7.0-debug ...
Using attach strategy to execute scripts...
Preparing environment
00:03
Waiting for pod gitlab-runner/runner-hmty8qbu-project-31186441-concurrent-0bbt8x to be running, status is Pending
Running on runner-hmty8qbu-project-31186441-concurrent-0bbt8x via gitlab-runner-gitlab-runner-84d476ff5c-mkt4s...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/...
Created fresh repository.
Checking out 4d05d22b as ci...
Skipping Git submodules setup
Executing "step_script" stage of the job script
It just stops at Executing "step_script" and never moves on. I've researched all over and read through as much documentation as I can find but am unable to troubleshoot this issue.
Setup
Amazon EKS version 1.21
GitLab Runner Helm Chart version 0.34.0
kaniko executor image v1.7.0-debug

Answer 1

This ended up being an issue with how the Kubernetes runner itself was configured inside of the runner configuration toml. The default container image we were using for our runners required a modification to the PATH environment variable so we were using the environment configuration setting to do this as outlined here. It seems that this PATH variable did not include the busybox shell defined in the kaniko debug image. We have since moved that PATH change inside our Docker image where it should've been in the first place and things are working as expected.