For documentary purpose,
I am looking for efficient ways to list the packages installed in a target rootfs built using oe-core.
The list of packages installed in your image is stored in the manifest file (besides of build history which is already mentioned).
Content of the manifest file looks like:
alsa-conf cortexa7hf-neon-vfpv4 1.1.2-r0.1
alsa-conf-base cortexa7hf-neon-vfpv4 1.1.2-r0.1
alsa-lib cortexa7hf-neon-vfpv4 1.1.2-r0.1
alsa-states cortexa7hf-neon-vfpv4 0.2.0-r5.1
alsa-utils-alsactl cortexa7hf-neon-vfpv4 1.1.2-r0.5
alsa-utils-alsamixer cortexa7hf-neon-vfpv4 1.1.2-r0.5
...
The list consists of the package name, architecture and a version.
That manifest is located in the deploy directory (i.e. deploy/images/${MACHINE}/). Here as an example of the directory listing (there are target images and the manifest file)
example-image-genericx86.ext3
example-image-genericx86.manifest
example-image-genericx86.tar.bz2
USER_CLASSES ?= "buildname image-mklibs image-prelink buildhistory"
ERROR: Error executing a python function in /opt/apps_proc/oe-core/meta/recipes-core/eglibc/eglibc_2.17.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'buildhistory_emit_pkghistory', lineno: 216, function:
0212:
0213: write_pkghistory(pkginfo, d)
0214:
0215:
***0216:buildhistory_emit_pkghistory(d)
0217:
File: 'buildhistory_emit_pkghistory', lineno: 207, function: buildhistory_emit_pkghistory
0203: filelist = []
0204: pkginfo.size = 0
0205: for f in pkgfiles[pkg]:
0206: relpth = os.path.relpath(f, pkgdestpkg)
***0207: fstat = os.lstat(f)
0208: pkginfo.size += fstat.st_size
0209: filelist.append(os.sep + relpth)
0210: filelist.sort()
0211: pkginfo.filelist = " ".join(filelist)
Exception: OSError: [Errno 2] No such file or directory: '/opt/apps_proc/oe-core/build/tmp-eglibc/work/armv7a-vfp-neon-oe-linux-gnueabi/eglibc/2.17-r3/packages-split/eglibc-thread-db/lib/libthread_db-1.0.so'
ERROR: Function failed: buildhistory_emit_pkghistory
Add build history to your USER_CLASSES variable in local.conf
USER_CLASSES ?= "buildhistory"
After you rerun the build look in build/buildhistory for more info.
You may need to force rebuilds to properly populate the directory.
Related
I have a project where a lot of the source files needs to be modified by a script before they are compiled.
The build process has 2 steps:
Run a script on the original sources to create intermediate sources.
Compile the intermediate sources.
It works fine when the original source files is modified or a new one is created.
In such cases SCons is able to build / rebuild the appropriate files.
However, when a source file is deleted, the corresponding intermediate file is not removed, which may end in successful build where it should fail due to missing source.
Example:
SConscript:
env = Environment()
source_files = ['main.cc.template', 'some-header.hh.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
env.Program('my-program', intermediates)
main.cc.template:
#include "some-header.hh"
int main() {
return get_number();
}
some-header.hh.template:
inline int get_number() {
return 0;
}
This compiles correctly but if you remove the file some-header.hh.template from the list and from the filesystem, it still compiles while it shouldn't.
You need to manually remove the intermediate file some-header.hh from the file system or else you'll get a false-positive result of build and subsequent tests.
I would like to automate the deletion process to prevent inevitable broken commits that will happen if I won't.
I've managed to create a dirty solution of the problem:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
old_intermediates = Glob('*.hh') + Glob('*.cc')
intermediates_to_delete = [x for x in old_intermediates if x not in intermediates]
for x in intermediates_to_delete:
x.remove()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This more or less works.
However, the files are removed too late and SCons seem to already be aware of their presence which causes the build error to origin from SCons and not the C++ compiler.
Because of that, the error message is less helpful. Also, I don't know if such operations are good for the stability of the SCons itself.
The error I'm getting is:
scons: *** [main.o] /home/piotr/tmp/some-header.hh: No such file or directory
Is there a clear way to delete outdated intermediate files?
Your approach is more or less correct.
SCons doesn't have any built-in mechanism to remove such dangling intermediate files; you need to write your own.
The error you're getting is caused by the fact you've used the SCons function Glob. It creates File nodes and makes SCons aware of existence of those files.
(Btw, the SCons function remove() is not designed to be called outside of a builder; you shouldn't do that.)
To avoid the problem, you need to delete the file before SCons has a chance to search for it.
You can just replace SCons function with standard Python library, like pathlib.
(It will require some tinkering to convert intermediates to pathlib objects too, but it won't be that much more code.)
A fixed solution:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
from pathlib import Path
old_intermediates = list(Path.cwd().glob('*.hh')) + list(Path.cwd().glob('*.cc'))
current_intermediates = [Path(x.get_path()).resolve() for x in intermediates]
intermediates_to_delete = [x for x in old_intermediates if x.resolve() not in current_intermediates]
for x in intermediates_to_delete:
print('unlink:', x)
x.unlink()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This gives the expected error message:
main.cc:1:10: fatal error: some-header.hh: No such file or directory
1 | #include "some-header.hh"
| ^~~~~~~~~~~~~~~~
compilation terminated.
scons: *** [main.o] Error 1
Building my workspace with colcon, some OSM files which are found in a directory "OSM" in a sub_package in the workspace are not found in built space. So when I go to the install space, the files are not there. I am not sure how to do this and if I should put it in the setup.py.
I tried putting this in the setup.py file in arguments of setup():
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
('.package_name/sub_package', glob('OSM_folder/*.osm')),
],
.
.
.
) # close setup()
but it did not work.
I am using ROS2 Galactic.
Directory structure:
package_name
┃
┣━━━━setup.py
┣━━━━package.xml
┣━━━━resource/
┣━━━━launch/
┗━━━━package_name
┗━sub_package_name
┗━OSM
┣━__init__.py
┗━some_osm_files.osm
I have the OSM directory in the built workspace but it has only the init.py file
I solved it. I am not sure if this is the right way of doing this or is there like another better/proper way or not, but here we go.
In the setup.py file, I added the line
(os.path.join('lib/python3.8/site-packages/package_name/sub_package/OSM'),glob(package_name+'/sub_package_name/OSM/*.osm')), in the data_files variable.
The first part of the new line which is os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM') determines the new location of the files in the install folder after building the workspace.
The second part which is glob(package_name+'/sub_package_name/OSM/*.osm') determines the files original location in the project workspace.
So the result is that it takes the files from the location mentioned in the second part and puts them in the location mentioned in the first part.
The resulting block is:
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
(os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM'), glob(package_name+'/sub_package_name/OSM/*.osm')),
],
.
.
.
)
I'm trying to make a python package and I have most of the things already setup by when I try to install the library from Github here, it installs everything except for the folder called champs and it's files
This is my File directory structure
LeagueYue
champs
-Lname_num.json
-Lname_Uname.json
-num_Uname.json
-__init__.py
-champion_files.py
-external.py
-match.py
-rank.py
-status.py
-summoner.py
-requirements.txt
-setup.py
All the files are installed except for the folder and the files inside champs
As this question answers:
There are 2 ways to add the static files:
1) Include_package_data=True + MANIFEST.in
A MANIFEST.in file in the same directory of setup.py, that looks like this:
include src/static/*
include src/Potato/*.txt
2) Package_data in setup.py
package_data = {
'static': ['*'],
'Potato': ['*.txt']
}
Specify the files inside the setup.py.
Two of the files could probably be derived at runtime from num_Uname.json, but that's fine.
I do not yet see a data_files directive in https://github.com/CharmingMother/LeagueLib/blob/async/setup.py
Thomas Cokelaer suggests using an expression like
datafiles = [(datadir, list(glob.glob(os.path.join(datadir, '*'))))]
and then
setup(
...
data_files = datafiles,
)
in http://thomas-cokelaer.info/blog/2012/03/how-to-embedded-data-files-in-python-using-setuptools/
In your case this could be as simple as:
data_files = [('', ['champs/num_Uname.json'])],
Martin Thoma explains you should access them using filepath = pkg_resources.resource_filename(__name__, path)
in How to read a (static) file from inside a Python package?
When I Read The Fine Manual, this setup.cfg alternative surfaces:
[options.data_files]
...
data = data/img/logo.png, data/svg/icon.svg
suggesting a line like . = champs/num_Uname.json or champs = num_Uname.json
I am using Scons for building an environment which include propagating a partial directory structure including hard links.
So it looks like this, I compile some sources into $BIN directory, some of the created targets are hard links, and in some other module I am scanning the $BIN directory and triggering a builder which either strip the the binaries to the new target, but maintain hard links. So if there are two files which are hard links in source, it copies the first and create hard link to the other target for the second.
Now the problem is if I try to run this with -j4 for example, a race condition could happen, because at the moment the builder tries to create a hard link, the target which is created by strip is not done yet.
Below is my builder:
def GenerateM3Strip(target, source, env):
if type(source) is list:
source = source[0]
if type(target) is list:
target = target[0]
src_stat = os.stat(source.abspath)
if (src_stat.st_nlink > 1
and src_stat.st_ino in GenerateM3Strip.inodes_dict):
previous_target = GenerateM3Strip.inodes_dict[src_stat.st_ino]
if target.get_dir().get_path() in previous_target:
source = previous_target[target.get_dir().get_path()]
print('LinkTarget: %s %s' % (source.get_path(), target.get_path()))
try:
os.unlink(target.get_path())
except Exception:
pass
os.link(source.get_path(), target.get_path())
return
# We have to keep a dictionary of target directory and target pairs, to
# support separate hard links for different directory targets, for example:
# fsdata/default and fsdata/bench
GenerateM3Strip.inodes_dict[src_stat.st_ino] = {
target.get_dir().get_path(): target}
action_str = env.subst(cross + 'strip -o $TARGET $SOURCE',
source=source, target=target)
env.Execute(env.Action(action_str))
GenerateM3Strip.inodes_dict = {}
I have a trouble of putting my initramfs.cpio in my kernel image by yocto.
I have two bb files, one is used to build an initramfs, and the other one is used to build a fitimage.
I successful to build the fitimage bundled with my initramfs image.
But it always failed to build a fitImage that has an initramfs.cpio.gz in the /usr directory in the fitImage.
( I mean, I want to see a file named initramfs.cpio in the /usr when I use my fitImage booting to console )
====================================================================
Here are my error message..
ERROR: linux-mine-1_4.9.27+gitAUTOINC+d87116e608-r0 do_package: QA Issue: linux-mine: Files/directories were installed but not shipped in any package:
/usr
/usr/initramfs-mine-qemu.cpio
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
linux-mine: 2 installed and not shipped files. [installed-vs-shipped]
ERROR: linux-mine-1_4.9.27+gitAUTOINC+d87116e608-r0 do_package: Fatal QA errors found, failing task.
ERROR: linux-mine-1_4.9.27+gitAUTOINC+d87116e608-r0 do_package: Function failed: do_package
ERROR: Logfile of failure stored in: /home/paul/projects/Test/yocto/build/tmp/work/mine-poky-linux-gnueabi/linux-mine/1_4.9.27+gitAUTOINC+d87116e608-r0/temp/log.do_package.26149
ERROR: Task (/home/paul/projects/Test/yocto/yocto-2.2/poky/../meta-mine/recipes-kernel/linux/linux-mine_4.9.bb:do_package) failed with exit code '1'
====================================================================
Here is my kernel image bb file
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}-${PV}:"
LINUX_VERSION ?= "4.9.27"
SRCREV = "d87116e608e94ad684b5e94d46c892e33b9e2d78"
SRC_URI = "git://local/kernel;protocol=ssh;branch=master"
#FILES_${PN} += "/usr /usr/initramfs-mine-${MACHINE_ARCH}.cpio"
#FILES_${PN}-${PV} += "/usr /usr/initramfs-mine-${MACHINE_ARCH}.cpio"
#IMAGE_INSTALL = "initramfs-mine"
do_install_append () {
echo "WangPaul : S=[${S}]"
echo "WangPaul : B=[${B}]"
echo "WangPaul : D=[${D}]"
install -d ${D}/usr/
install -m 0444 ${B}/usr/initramfs-mine-${MACHINE_ARCH}.cpio ${D}/usr/
}
====================================================================
Here is my initramfs bb file
LICENSE = "GPLv2"
PACKAGE_INSTALL = "initramfs-live-boot ${VIRTUAL-RUNTIME_base-utils} udev ${ROOTFS_BOOTSTRAP_INSTALL}"
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
inherit core-image
====================================================================
I have found similar questions:
Ship extra files in kernel module recipe and
An example of using FILES_${PN}
The way in aboves discussion are not work...
Any information would be appreciate !!
Thanks !!
The error seems to QA issues it means the source is compiled but not adding to rootfs. Add below line to yourkernel-image.bb. it will solve the issue.
FILES_${PN} += "${exec_prefix}/*"
Note: may be In your kernel.bb file you have given wrong format