Bitbucket self hosted runner failing during step setup - bitbucket-pipelines

Question

I tried posting this in Atlassian/Bitbucket community support, but my posts aren't posting over there. They aren't even showing up as pending/needing moderation/whatever.
I can't figure out why a step is failing during setup. I'm running a self-hosted runner (and giving 6 GB of memory to the runner container, so I don't think this is an OOM error) and I'm testing a fairly simple pipeline:
Step 1 - bump version as needed - super lightweight - passes fine.
Step 2 - build a go executable into a Docker image, publish to gcr.io - still pretty lightweight - fails during setup
Here is my bitbucket-pipelines.yml:
image:
name: gcr.io/novo00/bitbucket-pipelines:latest
username: _json_key
password: '$NOVO_GCR_IO_JSON_KEY'
pipelines:
branches:
master:
- step:
runs-on:
- self.hosted
script:
- build/bitbucket_pipelines/version.sh
- step:
runs-on:
- self.hosted
script:
- build/bitbucket_pipelines/build.sh
services:
- docker
Here is the output of "Build setup" in the pipeline dashboard:
+ umask 000
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="master" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN#bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIRCloning into '/opt/atlassian/pipelines/agent/build'...
Nothing after that, then it goes into "Build teardown".
Here is a snippet of my runner's stdout:
[2021-07-27 21:52:30,706] Starting container.
[2021-07-27 21:52:30,739] Adding container log: /var/lib/docker/containers/02152994b7d0485fa65a5dd8a4dd5254ea1c7a41b9b00f98ed5478245e3255da/02152994b7d0485fa65a5dd8a4dd5254ea1c7a41b9b00f98ed5478245e3255da-json.log
[2021-07-27 21:52:30,740] Waiting on container to exit.
[2021-07-27 21:52:30,741] Creating exec into container.
[2021-07-27 21:52:30,749] Starting exec into container and waiting for exec to exit.
[2021-07-27 21:52:30,949] Adding container log: /var/lib/docker/containers/1467fcbb959c9f17eacd2bc04967e4fe8042ba1fbc3bcb0a345d6da6db1717d0/1467fcbb959c9f17eacd2bc04967e4fe8042ba1fbc3bcb0a345d6da6db1717d0-json.log
[2021-07-27 21:52:30,949] Waiting on container to exit.
[2021-07-27 21:52:30,981] Adding container log: /var/lib/docker/containers/a62ece4849a5ee784f4f483d58eb87c3684f52ebca35308e6d485e014a17ac99/a62ece4849a5ee784f4f483d58eb87c3684f52ebca35308e6d485e014a17ac99-json.log
[2021-07-27 21:52:30,981] Waiting on container to exit.
[2021-07-27 21:52:31,051] Container has state (exitCode: Some(4), OOMKilled Some(false))
[2021-07-27 21:52:31,054] Removing container build
[2021-07-27 21:52:31,065] Not uploading caches. (numberOfCaches: 0, resultOrError: FAILED)
[2021-07-27 21:52:31,066] Not uploading artifacts. (numberOfArtifacts: 0, resultOrError: FAILED)
[2021-07-27 21:52:31,066] Updating step progress to PARSING_TEST_RESULTS.
[2021-07-27 21:52:31,401] Test report processing complete.
[2021-07-27 21:52:31,401] Removing container clone
[2021-07-27 21:52:31,562] Removing container clone
[2021-07-27 21:52:31,565] Removing container build
[2021-07-27 21:52:31,568] Removing container system-docker
[2021-07-27 21:52:31,583] Removing container system-auth-proxy
[2021-07-27 21:52:31,698] Removing container pause
[2021-07-27 21:52:31,749] Appending log line to log: {bc88f95e-9f7b-425a-adaf-e3060f3a53ee}.
[2021-07-27 21:52:31,754] Appending log line to main log.
[2021-07-27 21:52:31,861] Updating step progress to COMPLETING_LOGS.
[2021-07-27 21:52:31,972] Appending log line to log: {71708781-63f3-46de-8042-730f079fe9d6}.
[2021-07-27 21:52:32,009] Shutting down log uploader.
[2021-07-27 21:52:32,224] Tearing down directories.
[2021-07-27 21:52:32,227] Cancelling timeout
[2021-07-27 21:52:32,228] Completing step with result Result{status=FAILED, error=None}.
Notice the Container has state (exitCode: Some(4), OOMKilled Some(false)) Any help? Any other data I could give? I tried looking at some of the logs mentioned in the output, but they either were no longer there or didn't contain much useful information.
UPDATE: removing services: - docker from step 2 fixes the current issue, but I need docker services available to the step. Looking for a fix now.

Answer 1

I've got a couple of problems while using self-hosted runners. After 2 days of digging through internet, i know:
Self-hosted docker runners work well in most cases
They work well except docker-in-docker case. If you need to run docker inside your pipeline, you should use selected Linux types and versions. I've tested Debian 9, Debian 11, Ubuntu 20 LTS, CentOS8, CentOS 7.9.2009. Only the last one works correctly.
Even if you use compatible Linux installation, you can get problems during your docker build process. Different problems. Thanks to this post, I've fixed it.
If you use WSL2 and DockerDesktop, you won't have any logs from containers. That's due to another log files location in DD (and they're not available in executing WSL machines). If you need logs (i.e. for debugging) and you use Windows, just install VirtualBox with Linux, setup docker and additional runner inside it.