bitbake do_fetch[nostamp] = "1" has no effect - linux

Question

I am trying to install the compiled firmware for a microprocessor in my yocto image. This firmware is sent to a microprocessor on startup. The compiler for this firmware only runs on windows, therefore it is not possible to just clone the git repository and compile during build. I get this firmware file from the artifacts of a GitLab CI CD pipeline.
For debugging purposes, i would like to download this firmware file from the master branch, everytime that I build the image.
I previously worked with a custom do_fetch task and had set do_fetch[nostamp] = "1". This worked, but I ran into trouble when curl just put the 404 Error into the file. I have now tried to switch to wget and SRC_URI, but even though the nostamp is still set, it generates a .done file. It now never downloads the file again.
Here is the Recipe file I use:
S = "${WORKDIR}/"
PRIVATE_TOKEN = "xxxxxxxxxxx" # redacted
PROJECT_ID = "224"
VERSION = "2.00.01"
TAG = "master"
DOWNLOAD_FILE_PATH = "Production/firmware.bin"
INSTALL_NAME = "my-firmware-file.bin"
VERSION_FILE = "version.txt"
URL = "https://git.mycompany.com/api/v4/projects/${PROJECT_ID}/jobs/artifacts/${TAG}/raw/${DOWNLOAD_FILE_PATH}?job=publish_executable"
FETCHCMD_wget = "/usr/bin/env wget --header "PRIVATE-TOKEN: ${PRIVATE_TOKEN}""
BB_STRICT_CHECKSUM = "0"
SRC_URI = "${URL};downloadfilename=${INSTALL_NAME}"
do_fetch[nostamp] = "1"
do_patch(){
echo ${VERSION} > ${VERSION_FILE}
}
do_install(){
install -d ${D}${libdir}/test-dir/${PN}
# Install binary from artifacts
install -m 644 ${S}${INSTALL_NAME} ${D}${libdir}/test-dir/${PN}/${INSTALL_NAME}
# Instal version file created from tag
install -m 644 ${S}${VERSION_FILE} ${D}${libdir}/test-dir/${PN}/${VERSION_FILE}
}
FILES_${PN} += "${libdir}/test-dir/${PN}"
After running the recipe (either bitbake <PN> or bitbake <PN> -c fetch -f) I can run ls -la build/download | grep my-firmware-file and it shows two entries:
-rw-rw-r-- 1 root root 613924 Nov 11 11:50 my-firmware-file.bin
-rw-rw-r-- 1 root root 463 Nov 14 14:26 my-firmware-file.bin.done
As you see, it generated a .done file, that is newer than the last download of the actual file. Even when changing the TAG variable, no new file is downloaded. However for some reason, it always puts the correct version into the version.txtfile.
What am I missing? I could run a cleanall before every build, but that does not seem like a permanent solution. Also, the endgoal is to version the recipe and use the package version to pull from the correct release, but currently it pulls from master, so this does not make much sense.
Update: I checked the logfiles, and it seems wget is not the problem, as the command is never even executed. The first time after a clean, the log for do_fetch looks like this:
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Direct dependencies are []
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: []
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: Trying Upstream
DEBUG: Fetching https://git.mycompany.com/api/v4/projects/224/jobs/artifacts/master/raw/Production/firmware.bin?job=publish_executable;downloadfilename=my-firmware-file.bin using command '/usr/bin/env wget -r --header "PRIVATE-TOKEN: xxxxxxxxxxxxxx" -O /home/.../build/downloads/my-firmware-file.bin.tmp -P /home/.../build/downloads 'https://git.mycompany.com/api/v4/projects/224/jobs/artifacts/master/raw/Production/firmware.bin?job=publish_executable''
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -r --header "PRIVATE-TOKEN: xxxxxxxxxxxxxx" -O /home/.../build/downloads/my-firmware-file.bin.tmp -P /home/.../build/downloads 'https://git.mycompany.com/api/v4/projects/224/jobs/artifacts/master/raw/Production/firmware.bin?job=publish_executable'
DEBUG: Running export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1005/bus"; export PATH="..."; export HOME="/home/..."; /usr/bin/env wget -r --header "PRIVATE-TOKEN: xxxxxxxxxxxxxx" -O /home/.../build/downloads/my-firmware-file.bin.tmp -P /home/.../build/downloads 'https://git.mycompany.com/api/v4/projects/224/jobs/artifacts/master/raw/Production/firmware.bin?job=publish_executable' --progress=dot -v
WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.
--2022-11-18 07:01:11-- https://git.mycompany.com/api/v4/projects/224/jobs/artifacts/master/raw/Production/firmware.bin?job=publish_executable
Resolving git.mycompany.com (git.mycompany.com)... <IP>
Connecting to git.mycompany.com (git.mycompany.com)|<IP>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 613924 (600K) [application/octet-stream]
Saving to: ‘/home/.../build/downloads/my-firmware-file.bin.tmp’
2022-11-18 07:01:11 (70.4 MB/s) - ‘/home/.../build/downloads/my-firmware-file.bin.tmp’ saved [613924/613924]
FINISHED --2022-11-18 07:01:11--
Total wall clock time: 0.5s
Downloaded: 1 files, 600K in 0.008s (70.4 MB/s)
WARNING: Missing checksum for '/home/.../build/downloads/my-firmware-file.bin', consider adding at least one to the recipe:
SRC_URI[sha256sum] = "83aa3c373228b48eea58964b4ffec7ad42226014351d18bed12cb9b5eb3d261e"
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
When I then run do_fetch again (even when changing the TAG variable, so SRC_URI changes), the log looks as follows:
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Direct dependencies are []
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: []
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
WARNING: Missing checksum for '/home/.../build/downloads/my-firmware-file.bin', consider adding at least one to the recipe:
SRC_URI[sha256sum] = "83aa3c373228b48eea58964b4ffec7ad42226014351d18bed12cb9b5eb3d261e"
WARNING: Missing checksum for '/home/.../build/downloads/my-firmware-file.bin', consider adding at least one to the recipe:
SRC_URI[sha256sum] = "83aa3c373228b48eea58964b4ffec7ad42226014351d18bed12cb9b5eb3d261e"
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
Notice the duplicated missing checksum warning in the second log. I also tried to delete only the ./downloads/my-firmware-file.bin.done file, but it still is not redownloaded, the log is the same as the second one.

Answer 1

try to add -r flag to force wget to re-download even if the file exists:
FETCHCMD_wget = "/usr/bin/env wget -r --header "PRIVATE-TOKEN: ${PRIVATE_TOKEN}""

Answer 2

I have tried a lot of things, and by now I feel like it is a bug in bitbake. For the moment I implemented this workaround, that seems to work fine. It deletes the downloaded file as well as the .done file before the execution of do_fetch.
# Deletes the old, cached version of the firmware file
do_fetch_prepend(){
import os
fw_file = f"{d.getVar('DL_DIR')}/{d.getVar('INSTALL_NAME')}"
done_file = f"{fw_file}.done"
if os.path.isfile(fw_file):
os.remove(fw_file)
if os.path.isfile(done_file):
os.remove(done_file)
}
I hope to soon be able to correctly version my recipe, so I don't have to rely on such hacks.