I fail to cross compile rust program with zmq. To reproduce easily (with docker) :
Write the following Dockerfile
FROM rust:latest
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y g++-mingw-w64-x86-64 libudev-dev libzmq3-dev
RUN rustup target add x86_64-pc-windows-gnu
RUN rustup toolchain install --force-non-host stable-x86_64-pc-windows-gnu
WORKDIR /app
# This permit to pkg-config to found zmq
ENV PKG_CONFIG_SYSROOT_DIR=/usr/x86_64-w64-mingw32/
CMD ["cargo", "build", "--target", "x86_64-pc-windows-gnu"]
Build docker image with docker build . -t mycrossimage.
Create Cargo.toml and src/main.rs files
echo "[package]\nname = \"demo\"\nversion = \"0.1.0\"\nedition = \"2021\"\n[dependencies]\nzmq = \"0.9.0\"" > Cargo.toml && mkdir -p src && echo 'use zmq; fn main() {}' > src/main.rs
Try to compile through docker with docker run --rm -v $(pwd):/app mycrossimage:latest
And fail on :
error: linking with `x86_64-w64-mingw32-gcc` failed: exit status:
[...]
= note: /usr/bin/x86_64-w64-mingw32-ld: cannot find -lzmq
collect2: error: ld returned 1 exit status
How to configure a linux system to be able to compile a Rust program using zmq ?
I have Node.js app that uses SQLite database. To use it on ARM architecture, I need to build sqlite3 binaries so I need some packages while building Docker image.
Here is my Dockerfile:
FROM node:14-alpine as builder
WORKDIR /app
COPY package*.json ./
RUN apk update \
&& apk --no-cache --virtual build-dependencies add python make g++ \
&& npm install --production
COPY . .
RUN mkdir -p ./public ./data \
&& cd ./client \
&& npm install --production \
&& npm run build \
&& cd .. \
&& mv ./client/build/* ./public \
&& rm -rf ./client \
&& apk del build-dependencies
FROM node:14-alpine
COPY --from=builder /app /app
WORKDIR /app
EXPOSE 5005
ENV NODE_ENV=production
CMD ["node", "server.js"]
I've been using it for 6 months and it was working fine but now it throws this error:
> [linux/amd64 builder 4/6] RUN apk update && apk --no-cache --virtual build-dependencies add python make g++ && npm install --production:
#10 0.166 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#10 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#10 1.141 v3.14.2-123-g010734651f [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]
#10 1.141 v3.14.2-120-g90167408c8 [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]
#10 1.141 OK: 14943 distinct packages available
#10 1.216 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#10 1.476 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#10 1.936 ERROR: unable to select packages:
#10 1.989 python (no such package):
#10 1.989 required by: build-dependencies-20211108.132318[python]
------
Dockerfile.multiarch:7
--------------------
6 |
7 | >>> RUN apk update \
8 | >>> && apk --no-cache --virtual build-dependencies add python make g++ \
9 | >>> && npm install --production
10 |
--------------------
error: failed to solve: process "/bin/sh -c apk update && apk --no-cache --virtual build-dependencies
add python make g++ && npm install --production"
did not complete successfully: exit code: 2
I tried to specific python version like so: ... add python3 make .... It passes this step but I'm getting this error while building sqlite3 binaries:
#20 392.8 make: Entering directory '/app/node_modules/sqlite3/build'
#20 392.8 CC(target) Release/obj.target/nothing/../node-addon-api/nothing.o
#20 393.6 AR(target) Release/obj.target/../node-addon-api/nothing.a
#20 393.8 COPY Release/nothing.a
#20 394.0 ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3340000/sqlite3.c
#20 394.1 /bin/sh: python: not found
#20 394.1 make: *** [deps/action_before_build.target.mk:13: Release/obj/gen/sqlite-autoconf-3340000/sqlite3.c] Error 127
#20 394.1 make: Leaving directory '/app/node_modules/sqlite3/build'
RUN apk add --no-cache --virtual .gyp python3 make g++
We hit the same error and in our case explicitly targeting python3 seems to fix it. I asked on IRC, did not find out yet why python is not working anymore, but was told that python2 still provides /usr/bin/python.
So maybe you actually need python2?
I am trying to build image for linux/arm64/v8 on linux/amd64 Gitlab runner. I run it with this command:
docker buildx create --use
docker buildx build --platform linux/arm64/v8 -f Dockerfile.dev -t registry.gitlab.com/xxx:yyy --build-arg ENV=dev --build-arg COMPOSER_AUTH_FILE=./auth.json .
My Dockerfile is fairly simple:
FROM alpine:latest
RUN apk update && \
apk add --no-cache composer git bash openssh-client patch && \
apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev
But it fails with:
#6 [2/2] RUN apk update && apk add --no-cache composer git bash openssh-client patch && apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev
#6 0.185 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
#6 0.832 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/aarch64/APKINDEX.tar.gz
#6 1.679 v3.14.2-119-g9c4e1aa60c [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]
#6 1.679 v3.14.2-120-g90167408c8 [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]
#6 1.679 OK: 14818 distinct packages available
#6 1.828 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
#6 2.263 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/aarch64/APKINDEX.tar.gz
#6 2.977 (1/31) Installing ncurses-terminfo-base (6.2_p20210612-r0)
#6 3.010 (2/31) Installing ncurses-libs (6.2_p20210612-r0)
#6 3.049 (3/31) Installing readline (8.1.0-r0)
#6 3.075 (4/31) Installing bash (5.1.4-r0)
#6 3.137 Executing bash-5.1.4-r0.post-install
#6 3.144 ERROR: bash-5.1.4-r0.post-install: script exited with error 1
#6 3.144 (5/31) Installing php7-common (7.4.25-r0)
#6 3.159 (6/31) Installing argon2-libs (20190702-r1)
#6 3.171 (7/31) Installing libedit (20210216.3.1-r0)
#6 3.190 (8/31) Installing pcre2 (10.36-r0)
#6 3.225 (9/31) Installing xz-libs (5.2.5-r0)
#6 3.244 (10/31) Installing libxml2 (2.9.12-r1)
#6 3.308 (11/31) Installing php7 (7.4.25-r0)
#6 3.503 (12/31) Installing php7-phar (7.4.25-r0)
#6 3.527 (13/31) Installing ca-certificates (20191127-r5)
#6 3.575 (14/31) Installing brotli-libs (1.0.9-r5)
#6 3.620 (15/31) Installing nghttp2-libs (1.43.0-r0)
#6 3.638 (16/31) Installing libcurl (7.79.1-r0)
#6 3.674 (17/31) Installing php7-curl (7.4.25-r0)
#6 3.690 (18/31) Installing php7-iconv (7.4.25-r0)
#6 3.702 (19/31) Installing php7-json (7.4.25-r0)
#6 3.716 (20/31) Installing oniguruma (6.9.7.1-r0)
#6 3.747 (21/31) Installing php7-mbstring (7.4.25-r0)
#6 3.815 (22/31) Installing php7-openssl (7.4.25-r0)
#6 3.833 (23/31) Installing libzip (1.7.3-r2)
#6 3.849 (24/31) Installing php7-zip (7.4.25-r0)
#6 3.863 (25/31) Installing composer (2.1.9-r0)
#6 3.939 (26/31) Installing expat (2.4.1-r0)
#6 3.958 (27/31) Installing git (2.32.0-r0)
#6 4.559 (28/31) Installing openssh-keygen (8.6_p1-r3)
#6 4.589 (29/31) Installing openssh-client-common (8.6_p1-r3)
#6 4.683 (30/31) Installing openssh-client-default (8.6_p1-r3)
#6 4.727 (31/31) Installing patch (2.7.6-r7)
#6 4.749 Executing busybox-1.33.1-r3.trigger
#6 4.754 ERROR: busybox-1.33.1-r3.trigger: script exited with error 1
#6 4.754 Executing ca-certificates-20191127-r5.trigger
#6 4.758 ERROR: ca-certificates-20191127-r5.trigger: script exited with error 1
#6 4.765 1 error; 36 MiB in 45 packages
#6 ERROR: process "/dev/.buildkit_qemu_emulator /bin/sh -c apk update && apk add --no-cache composer git bash openssh-client patch && apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev" did not complete successfully: exit code: 1
------
> [2/2] RUN apk update && apk add --no-cache composer git bash openssh-client patch && apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev:
#6 3.958 (27/31) Installing git (2.32.0-r0)
#6 4.559 (28/31) Installing openssh-keygen (8.6_p1-r3)
#6 4.589 (29/31) Installing openssh-client-common (8.6_p1-r3)
#6 4.683 (30/31) Installing openssh-client-default (8.6_p1-r3)
#6 4.727 (31/31) Installing patch (2.7.6-r7)
#6 4.749 Executing busybox-1.33.1-r3.trigger
#6 4.754 ERROR: busybox-1.33.1-r3.trigger: script exited with error 1
#6 4.754 Executing ca-certificates-20191127-r5.trigger
#6 4.758 ERROR: ca-certificates-20191127-r5.trigger: script exited with error 1
#6 4.765 1 error; 36 MiB in 45 packages
------
Dockerfile.dev:3
--------------------
2 |
3 | >>> RUN apk update && \
4 | >>> apk add --no-cache composer git bash openssh-client patch && \
5 | >>> apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev
6 |
--------------------
error: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apk update && apk add --no-cache composer git bash openssh-client patch && apk add --no-cache libzip-dev libpng-dev libxslt-dev curl-dev libxml2-dev icu-dev oniguruma-dev freetype-dev libjpeg-turbo-dev" did not complete successfully: exit code: 1
make: *** [Makefile:18: feature] Error 1
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 2
It's Gitlab runner version 13.4.1 and Docker executor docker:stable. What can I do about this issue?
There were three problems with my approach:
I needed to install buildx extension
- mkdir -p ~/.docker/cli-plugins
- wget -qO ~/.docker/cli-plugins/docker-buildx `wget -O - https://api.github.com/repos/docker/buildx/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4`
- chmod a+x ~/.docker/cli-plugins/docker-buildx
I had to install emulators through qemu
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
I was hosting the Gitlab runner on a CentOS 7 with Kernel 3.13, which is not supported by qemu. After update to Ubuntu 20.04 with Kernel version 5.4 it worked just fine.
In my dockerfile i'm installing cryptography python dependency however when building wheel from rootless docker i'm getting following error:
...
generating cffi module 'build/temp.linux-x86_64-3.7/_openssl.c'
running build_rust
Updating crates.io index
warning: spurious network error (2 tries remaining): failed to resolve address for github.com: Name does not resolve; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for github.com: Name does not resolve; class=Net (12)
error: failed to get `pyo3` as a dependency of package `cryptography-rust v0.1.0 (/tmp/pip-install-h22hlaqn/cryptography_a2037904c10949639ccf8a3b2519f187/src/rust)`
Caused by:
failed to fetch `https://github.com/rust-lang/crates.io-index`
Caused by:
network failure seems to have happened
if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
Caused by:
failed to resolve address for github.com: Name does not resolve; class=Net (12)
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed:
https://cryptography.io/en/latest/installation.html#rust
5) If you are experiencing issues with Rust for *this release only* you may
set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
=============================DEBUG ASSISTANCE=============================
...
The strangest thing that the same dockerfile compiles absolutely successfully when built from root user.
I'm using the most recent docker version on arch linux and have tried following instructions here: https://docs.docker.com/engine/security/rootless/ and installing AUR and in both cases had no luck.
That's my docker file:
FROM python:3.7-alpine
ENV LANG C.UTF-8
RUN apk update \
&& apk add --virtual build-deps gcc g++ musl-dev python3 python3-dev autoconf automake linux-headers make libffi-dev openssl-dev \
&& apk add --no-cache bash postgresql-dev libxml2-dev libxslt-dev jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev git openssh-client rust cargo
RUN mkdir -p /opt/carryall/app
RUN mkdir -p /opt/carryall/static
RUN mkdir -p /opt/carryall/media
WORKDIR /opt/carryall/app
RUN mkdir /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa_temp
RUN cat /root/.ssh/id_rsa_temp | tr -d '\r' > /root/.ssh/id_rsa
RUN chmod 400 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
ADD requirements.txt /opt/carryall/app/
RUN pip install --no-cache-dir -r /opt/carryall/app/requirements.txt
RUN apk del --purge build-deps \
&& rm -rf /root/.cache /tmp/*
ADD . /opt/carryall/app
I am building a Docker image with python 3.7.10 (Linux Alpine v3.13) but when building the image with docker build . the package hdf5 will fail during the installation. This is my Dockerfile:
FROM python:3.7.10-alpine3.13
RUN mkdir /app
WORKDIR /app
COPY requirements.txt /requirements.txt
ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1
RUN apk add --update --no-cache --virtual .tmp gcc libc-dev linux-headers
RUN apk add --no-cache jpeg-dev zlib-dev mariadb-dev libffi-dev openblas-dev libgfortran lapack-dev build-base openssl-dev
RUN apk add --no-cache hdf5 hdf5-dev
RUN pip install -r /requirements.txt
RUN apk --no-cache del build-base
ENV PYTHONUNBUFFERED 1
COPY . /app/
CMD ["uwsgi", "my_app"]
requirements.txt file:
h5py==2.10.0
numpy==1.19.2
I have tried with and without the --no-binary flag but still no luck. Was somebody able to install that library on this Alpine version?
Error logs:
17858 | __pyx_t_1 = H5Ovisit_by_name(__pyx_v_loc_id, __pyx_v_obj_name, __pyx_v_idx_type, __pyx_v_order, __pyx_v_op, __pyx_v_op_data, __pyx_v_lapl_id); if (unlikely(PyErr_Occurred())) __PYX_ERR(0, 1641, __pyx_L1_error)
| ^~~~~~~~~~~~~~~~
In file included from /usr/include/H5Apublic.h:22,
from /usr/include/hdf5.h:23,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/api_compat.h:27,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:654:
/usr/include/H5Opublic.h:213:15: note: declared here
213 | H5_DLL herr_t H5Ovisit_by_name3(hid_t loc_id, const char *obj_name,
| ^~~~~~~~~~~~~~~~~
/tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c: In function '__pyx_f_4h5py_4defs_H5Pget_driver_info':
/tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:21768:13: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
21768 | __pyx_t_1 = H5Pget_driver_info(__pyx_v_plist_id); if (unlikely(PyErr_Occurred())) __PYX_ERR(0, 2016, __pyx_L1_error)
| ^
In file included from /usr/include/H5public.h:32,
from /usr/include/hdf5.h:22,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/api_compat.h:27,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:654:
/tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c: In function '__pyx_f_4h5py_4defs_H5Sencode':
/tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:34528:15: error: too few arguments to function 'H5Sencode2'
34528 | __pyx_t_1 = H5Sencode(__pyx_v_obj_id, __pyx_v_buf, __pyx_v_nalloc); if (unlikely(PyErr_Occurred())) __PYX_ERR(0, 3303, __pyx_L1_error)
| ^~~~~~~~~
In file included from /usr/include/hdf5.h:38,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/api_compat.h:27,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:654:
/usr/include/H5Spublic.h:129:15: note: declared here
129 | H5_DLL herr_t H5Sencode2(hid_t obj_id, void *buf, size_t *nalloc, hid_t fapl);
| ^~~~~~~~~~
In file included from /usr/local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:21,
from /usr/local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/api_compat.h:26,
from /tmp/pip-install-fkianwj6/h5py_dfa9366c1fdb47e98e9e2e8de85ac21e/h5py/defs.c:654:
At top level:
/usr/local/lib/python3.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1464:1: warning: '_import_array' defined but not used [-Wunused-function]
1464 | _import_array(void)
| ^~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1
----------------------------------------
Seems like I have used a wrong and old h5py version (h5py==2.10.0). The following setup worked just fine when updating to h5py==3.2.1 within the requirements.txt.
FROM python:3.7.10-alpine3.13
RUN mkdir /app
WORKDIR /app
COPY requirements.txt /requirements.txt
ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1
RUN apk add --update --no-cache --virtual .tmp gcc libc-dev linux-headers
RUN apk add --no-cache jpeg-dev zlib-dev mariadb-dev libffi-dev openblas-dev libgfortran lapack-dev build-base openssl-dev
RUN apk add --no-cache hdf5-dev
RUN pip install -r /requirements.txt
RUN apk --no-cache del build-base
ENV PYTHONUNBUFFERED 1
COPY . /app/
CMD ["uwsgi", "my_app"]
requirements.txt file:
h5py==3.2.1
numpy==1.20.1
Thanks to #jordanvrtanoski and #valiano for the feedback.