I'm trying to build a flex-bison parser with SCons, using a custom decider. (You can learn more about the decider in this question, although, the details probably aren't too relevant to the current problem.)
Files:
My SConstruct:
env = DefaultEnvironment()
deciderEnv = env.Clone()
def source_and_target_decider(dependency, target, prev_ni, repo_node=None):
src_old_csig = prev_ni.csig if hasattr(prev_ni, 'csig') else None
src_new_csig = dependency.get_csig()
tgt_stored_info = target.get_stored_info()
tgt_old_csig = tgt_stored_info.ninfo.csig if hasattr(tgt_stored_info.ninfo, 'csig') else None
tgt_new_csig = target.get_csig()
return src_new_csig != src_old_csig or tgt_new_csig != tgt_old_csig
deciderEnv.Decider(source_and_target_decider)
deciderEnv['YACCFLAGS'] = ['-d']
deciderEnv['YACCHXXFILESUFFIX'] = '.hh'
bison_source = deciderEnv.CXXFile('src/test.yy.cc', 'src/test.yy')[0]
deciderEnv['LEXFLAGS'] = [f'--header-file=${{SOURCE}}.hh']
flex_source = deciderEnv.CXXFile('src/test.ll.cc', 'src/test.ll')[0]
Program('test', [bison_source, flex_source])
src/test.yy:
%language "c++"
%code {
int yylex(int*);
}
%token X
%%
everything: X {}
and src/test.ll:
%{
#include "./test.yy.hh"
%}
%option c++
%%
. { return yy::parser::token::X; }
Result:
I run a command scons ./src/test.ll.o.
The first time you run it, it works correctly.
flex --header-file=src/test.ll.hh -t src/test.ll > src/test.ll.cc
bison -d -o src/test.yy.cc src/test.yy
g++ -o src/test.ll.o -c src/test.ll.cc
SCons seems to understand that it needs to generate src/test.yy.hh using bison before it compiles the src/test.ll.o because src/test.ll.cc #includes that header.
However, if I then remove .sconsign.dblite (leaving the rest of the files intact), SCons seems to lose that understanding for the duration of the next build.
Running scons ./src/test.ll.o yields:
flex --header-file=src/test.ll.hh -t src/test.ll > src/test.ll.cc
g++ -o src/test.ll.o -c src/test.ll.cc
and only after calling it a second time it decides to build the src/test.yy.hh:
bison -d -o src/test.yy.cc src/test.yy
g++ -o src/test.ll.o -c src/test.ll.cc
Dependency tree:
Normally the dependency tree (scons --tree=all src/test.ll.o) looks like this:
+-src/test.ll.o
+-src/test.ll.cc
| +-src/test.ll
| +-/bin/flex
+-src/test.yy.hh
| +-src/test.yy
| +-/bin/bison
+-/bin/g++
However, right after deleting the .sconsign.dblite it is incomplete:
+-src/test.ll.o
+-src/test.ll.cc
| +-src/test.ll
| +-/bin/flex
+-/bin/g++
Further observations:
The last line of the custom decider seems to be the culprit.
Specifically, the part or tgt_new_csig != tgt_old_csig. If you remove this part, the problem no longer occurs.
However, I have no clue why this particular piece of code breaks the build.
Any idea how to fix this?
The sconsign is where SCons stores info about previous builds.
If you remove it, then SCons will not have that info and will rebuild everything.
Because in your decider you are setting:
tgt_old_csig = tgt_stored_info.ninfo.csig if hasattr(tgt_stored_info.ninfo, 'csig') else None
and then comparing None to the current csig, which will not be equal and require a rebuild.
You'd likely need to change as follows to work when you're explicitly removing the source of the tgt_stored_info (the sconsign file).
def source_and_target_decider(dependency, target, prev_ni, repo_node=None):
src_old_csig = prev_ni.csig if hasattr(prev_ni, 'csig') else None
src_new_csig = dependency.get_csig()
tgt_stored_info = target.get_stored_info()
tgt_old_csig = tgt_stored_info.ninfo.csig if hasattr(tgt_stored_info.ninfo, 'csig') else None
tgt_new_csig = target.get_csig()
if tgt_old_csig:
return src_new_csig != src_old_csig or tgt_new_csig != tgt_old_csig
else:
return src_new_csig != src_old_csig
Why are you removing .sconsign.dblite?
Enviroment:
Compiler: android-ndk-r21b, api 24
Ubuntu 20.04
Glib 2.66
I have built iconv, and have built glib 2.56 with make successfully, now i try to build glib 2.66 with meson, i build i run this script:
OUTDIR=$HOME/work/ndk/builddir/out
CC=arm-linux-androideabi-gcc CFLAGS="-L$OUTDIR/lib -I$OUTDIR/include" LDFLAGS="-L$OUTDIR/lib" LIBS="-liconv" CXXFLAGS="-L$OUTDIR/lib" meson release/ \
--cross-file cross_file.txt \
--prefix=$OUTDIR \
but compile error:
Header <iconv.h> has symbol "iconv_open" : YES
meson.build:1930:4: ERROR: C shared or static library 'iconv' not found
A full log can be found at /home/git/work/ndk/builddir/glib-2.66.4/reconfigure/meson-logs/meson-log.txt
cross_file.txt:
[host_machine]
system = 'android'
cpu_family = 'arm'
cpu = 'arm'
endian = 'little'
[constants]
android_ndk = '/home/git/work/ndk/toolchain/bin/'
toolchain = '/home/git/work/ndk/toolchain/bin/arm-linux-androideabi-'
outdir = '/home/git/work/ndk/builddir/out/'
[binaries]
c = android_ndk + 'arm-linux-androideabi-gcc'
cpp = android_ndk + 'arm-linux-androideabi-g++'
ar = android_ndk + 'arm-linux-androideabi-ar'
ld = android_ndk + 'arm-linux-androideabi-ld'
c_ld = android_ndk + 'arm-linux-androideabi-ld'
objcopy = android_ndk + 'arm-linux-androideabi-objcopy'
strip = android_ndk + 'arm-linux-androideabi-strip'
pkgconfig = '/usr/bin/pkg-config'
[built-in options]
c_std = 'c11'
libdir = '/home/git/work/ndk/builddir/out/lib'
prefix = '/home/git/work/ndk/builddir/out/'
c_args = ['-L/home/git/work/ndk/builddir/out/lib', '-I/home/git/work/ndk/builddir/out/include']
cpp_args = ['-L/home/git/work/ndk/builddir/out/lib', '-I/home/git/work/ndk/builddir/out/include']
c_link_args = ['-I/home/git/work/ndk/builddir/out/include', '-L/home/git/work/ndk/builddir/out/lib']
pkg_config_path = '/home/git/work/ndk/builddir/out/lib/pkgconfig'
I've got it. I shouldn't set the c_ld variable. And there is a bug in meson scripts of gio, so i must apt remove libelf-dev on ubuntu(or modify gio/meson.build).
Visit my github to get config file:
https://github.com/edaplayer/android-ndk-harfbuzz/blob/main/glib-2.66.4/build_glib_meson.sh
https://github.com/edaplayer/android-ndk-harfbuzz/blob/main/glib-2.66.4/cross_file.txt
I have created a recipe for the dlib python library
on the python3 setup.py install command the do_install task failed.
(recipe-sysroot-native/usr/lib/libnsl.so: error adding symbols: file in wrong format)
Please could anyone help?
Recipe Script:
SUMMARY = "A toolkit for making real world machine learning and data analysis applications"
HOMEPAGE = "https://github.com/davisking/dlib"
LICENSE = "Boost-Software"
LIC_FILES_CHKSUM = "file://dlib/LICENSE.txt;md5=2c7a3fa82e66676005cd4ee2608fd7d2 \
file://dlib/external/pybind11/LICENSE;md5=beb87117af69fd10fbf9fb14c22a2e62 \
file://dlib/external/libpng/LICENSE;md5=243135ddedf702158f9170807cbcfb66 \
file://docs/docs/license.xml;md5=2e6ff4080dcb217d4d300b90e9aabb5b \
file://examples/LICENSE_FOR_EXAMPLE_PROGRAMS.txt;md5=57eee82829ed297e23d84de5f905afee \
file://examples/video_frames/license.txt;md5=127ee508b60a6be9dea8af3b441993dc \
file://python_examples/LICENSE_FOR_EXAMPLE_PROGRAMS.txt;md5=064f53ab40ea2b6a4bba1324149e4fde"
SRC_URI = "git://github.com/davisking/dlib.git;protocol=https"
PV = "1.0+git${SRCPV}"
SRCREV = "3b794540baeabbcd033b544230401967106d5483"
S = "${WORKDIR}/git"
inherit setuptools3 cmake
FILES_${PN} += "/${base_prefix}"
do_install_append() {
cd ${S}/
python3 setup.py install
}
I would like to run my py_test with python 3 in Bazel.
py_library(
name = "foo",
srcs = ["foo.py"]
)
py_test(
name = "foo_test",
srcs = glob(["foo_test.py",]),
deps = [":foo"]
)
py_runtime(
name = "python-3.6.3",
files = [],
interpreter_path = "/usr/local/bin/python3",
)
I was able to achieve this using command
bazel test --python_top=//path/to/foo:python-3.6.3 foo_test
However, I would like to import python3 to bazel sandbox with new_http_archive and provide the interpreter_path for the py_runtime rule that points to that http_archive within bazel sandbox. So far I am not able to find what is the interpreter_path... Do I have to reference the http_archive label somewhere from the py_runtime or somewhere else?
new_http_archive(
name = "python_version",
urls = ["https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz"],
strip_prefix = "Python-3.6.3",
build_file_content = """
py_library(
name = "python_srcs",
srcs = glob(["Lib/*.py"]),
visibility = ["//visibility:public"]
)"""
)
The tgz that you're downloading doesn't contain an interpreter. It contains the source code for the interpreter. If you want to build the interpreter as part of your build, you could do something like this
new_http_archive(
name = "python_version",
urls = ["https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz"],
strip_prefix = "Python-3.6.3",
build_file_content = """
genrule(
name = "build_python",
srcs = glob(["**"]),
outs = ["python"],
cmd = "./external/python_version/configure && make && cp python $#",
visibility = ["//visibility:public"],
)""",
)
And then your py_runtime rule would set the interpreter attribute (not interpreter_path):
py_runtime(
name = "python-3.6.3",
files = [],
interpreter = "#python_version//:python",
)
I build a simple software in my Debian VM. It tries the download a file from a FTP server. For that I use the libcurl library(libcurl-openssl-dev).
Now I have a SAMA5D3x-Evalboard available. For self education purpose I like to build a Linux + Toolchain for it. I found the Yocto project. The meta layer concept seems to be good approach for managing different platforms - so let's try it!
I found some examples at the net. But no which covers the combination of "hello world" + bigger lib.
I want:
get the X-compile toolchain + Linux image(includes my hello world + the libcurl shared objects)
My approach:
meta-atmel: for the SAMA5D3x eval board
meta-debian: for the lib "libcurl-openssl-dev"
meta-mySoftware: for the hello world program
The constrains are:
main.c - name of the hello world program
libcurl - c lib name (I have chosen this lib, because it has a lot of dependencies)
How to setup it right? If there are better ideas they are welcome!
Installation & Configuration of Yocto
I.) install toolchain
I.1.) apt-get:
sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat libsdl1.2-dev xterm
I.2.) opt. configure git after proxy
git config --global core.gitproxy <gitproxy>
I.3.) download Toolchain + meta packages and checkout the "right" commit
git clone git://git.yoctoproject.org/poky
cd poky
git checkout jethro
git clone git://github.com/linux4sam/meta-atmel.git
cd meta-atmel
git checkout jethro
cd ..
git clone git://git.openembedded.org/meta-openembedded
cd meta-openembedded
git checkout jethro
cd ..
II.) build kernel+rootfs for SAMA5D3x
II.1) create new project
source oe-init-build-env
the new project is created at the "build" dir
modify layers "conf/bblayers.conf" via commandos
bitbake-layers add-layer "../meta-atmel"
bitbake-layers add-layer "../meta-openembedded/meta-oe"
bitbake-layers add-layer "../meta-openembedded/meta-networking"
bitbake-layers add-layer "../meta-openembedded/meta-python"
edit of "conf/local.conf"
MACHINE ??= "sama5d3xek"
DL_DIR ?= "${TOPDIR}/../downloads"
DISTRO ?= "poky-atmel"
the download directory is outside of the build environment. The downloaded sources can be used for different builds.
II.2) create image
bitbake -c clean core-image-minimal
bitbake core-image-minimal
the result files are created at "tmp/deploy/images/sama5d3xek"
II.3) flash image
The sam-ba tool from Atmel is needed. The Flash-Process can be automated by a script(we create a tcl file). The self written tool "buildAtmelImages.py" copies the needed image files for the SAMA5D35, creates the u-boot configuration and a Flash-Script. An Example:
python buildAtmelImages.py --help
python buildAtmelImages.py -s "/home/user/poky/build/tmp/deploy/images/sama5d3xek" -d "/home/user/images" -t "\\\\1.2.3.4\\admin\\images"
here comes the code for "buildAtmelImages.py":
#!/usr/bin/python
# encoding: utf-8
'''
buildAtmelImages -- make yocto build images ready of the sam-ba flash tool
buildAtmelImages is a description
- reverse engineered /meta-atmel/scripts/nandflash.tcl script
- works only for the SAMA5D35
- params:
-s "<dir>" = source dir of the images generated by yocto. like "/home/user/poky/build/tmp/deploy/images/sama5d3xek"
-d "<dir>" = destination dir
-t "<dir/url>" = tcl refence dir. The path to the images, used to flash, is hard coded in the tcl script. typical example "\\\\192.168.1.2\\admin\\images" for an samba based network
- way of operation:
# hints:
I.) the SAMA5D3x implements some ECC features. This means that in every NAND sector are bits for error correction reserved.
This acts into 2 directions:
1.) the binary files have to be flashed by takeing care of the extra ECC bits. This is done by the sam-ba flash tool
2.) the kernel needs to take care, that the data stored on the NAND drive has some extra handling. This is managed by the meta-atmel package used by the yocto toolchain.
II.) u boot configuration
1.) the u boot configuration is placed into the NAND. But the configuration is not generated by the yocto toolchain. Atmel delivers a flashing script for its demo boards. This was reversed engineered into this python script.
The layout of the boot config is a defined memory space which is guared by an crc32(litle endian). The boot configuration is placed as ANSII string parameters. These parameters are seperated by a zero.
2.) because of the ecc the binary device tree(dtb) and kernel needs to be copied to memory before been executed. For that it is needed that u-boot loads them into ram before starting the kernel.
# operation:
First the generated images are copied. After the configuartion is generated and placed into the output dir. At least the tcl script is generated and placed into the output dir, as well.
#author: Stefan Jaritz
#copyright:
#license: license
#contact:
#deffield updated: Updated
'''
import sys
import os
import shutil
import binascii
import struct
from argparse import ArgumentParser
from argparse import RawDescriptionHelpFormatter
__all__ = []
__version__ = 0.1
__date__ = '2016-04-19'
__updated__ = '2016-04-19'
class _memorySeg():
def __init__(self, add, sf, df):
self.add = add
self.sf = sf
self.df = df
gArch = 'sama5d3xek'
gTCLfilename = 'SEK4.tcl'
gMemSetup = {
'bootloader' : _memorySeg(0x00000000, 'at91bootstrap-sama5d3xek.bin', 'bootloader.bin'),
'uboot' : _memorySeg(0x00040000, 'u-boot-sama5d3xek.bin', 'uboot.bin'),
'ubootEnv' : _memorySeg(0x000c0000, None, 'ubootEnv.bin'),
'ubootEnvRed' : _memorySeg(0x00100000, None, 'ubootEnv.bin'),
'dtb' : _memorySeg(0x00180000,'zImage-sama5d35ek.dtb', 'kernel.dtb'),
'kernel' : _memorySeg(0x00200000,'zImage', 'kernel'),
'rootfs' : _memorySeg(0x00800000,'core-image-minimal-sama5d3xek.ubi', 'rootfs.ubi')
}
gSourceDir = ''
gDestDir = ''
gTCLfilepath = ''
gECCcnfg = {
'at91sam9x5ek' : 0xc0c00405,
'at91sam9n12ek' : 0xc0c00405,
'sama5d3xek' :0xc0902405,
'sama5d3_xplained' : 0xc0902405,
'sama5d4ek' : 0xc1e04e07,
'sama5d4_xplained' : 0xc1e04e07
}
def _copyFiles ():
''' copies image files from source to the destination'''
global gMemSetup
for (k, ms) in gMemSetup.items():
if ms.sf is not None:
print("copy {k}".format(k=k))
shutil.copy(os.path.join(gSourceDir, ms.sf), os.path.join(gDestDir, ms.df))
def _genUBootCfg ():
global gMemSetup
global gSourceDir
global gDestDir
print("generate u-boot config")
bootCfgStr = "bootcmd=nand read 0x{dtbLoadAddr:X} 0x{dtbAddr:X} {dtbSize}; nand read 0x{kernelLoadAddr:X} 0x{kernelAddr:X} {kernelSize}; bootz 0x{kernelLoadAddr:X} - 0x{dtbLoadAddr:X}".format(
dtbLoadAddr=0x21000000,
dtbAddr=gMemSetup['dtb'].add,
dtbSize=os.path.getsize(os.path.join(gSourceDir, gMemSetup['dtb'].sf)),
kernelLoadAddr=0x22000000,
kernelAddr=gMemSetup['kernel'].add,
kernelSize=os.path.getsize(os.path.join(gSourceDir, gMemSetup['kernel'].sf))
)
bootVars = (
'bootdelay=0',
'baudrate=115200',
'stdin=serial',
'stdout=serial',
'stderr=serial',
'bootargs=console=ttyS0,115200 mtdparts=atmel_nand:256k(bootstrap)ro,512k(uboot)ro,256k(env),256k(env_redundant),256k(spare),512k(dtb),6M(kernel)ro,-(rootfs) rootfstype=ubifs ubi.mtd=7 root=ubi0:rootfs rw',
bootCfgStr
)
cfgMemCfg = bytearray(0x20000-5)
l = len(cfgMemCfg)
indx = 0
for v in bootVars:
l = len(v)
cfgMemCfg[indx:indx+l] = v.encode('utf-8')
indx += l + 1
crc = binascii.crc32(cfgMemCfg)
with open(os.path.join(gDestDir, gMemSetup['ubootEnv'].df), mode='wb') as f:
# crc32 as little endian
f.write(struct.pack('<I',crc))
f.write(bytearray(1))
f.write(cfgMemCfg)
f.close()
def _helper_genTCLFilename (idName):
pstr = os.path.join(gTCLfilepath ,gMemSetup[idName].df)
pstr = pstr.replace('/', '\\')
return pstr.replace('\\', '\\\\')
def _genFlashScript():
global gMemSetup
global gECCcnfg
global gTCLfilename
global gDestDir
global gArch
print("generate tcl script for sam-ba")
tclStr = """puts "-I- start flashing"
NANDFLASH::Init
NANDFLASH::NandHeaderValue HEADER 0x{pmeccConfig:X}
NANDFLASH::EraseAllNandFlash
NANDFLASH::SendBootFilePmeccCmd "{bootstrapFile}"
send_file {{NandFlash}} "{ubootFile}" 0x{ubootAddr:X} 0
send_file {{NandFlash}} "{ubootEnvFile}" 0x{ubootEnvAddr:X} 0
send_file {{NandFlash}} "{ubootEnvFileRed}" 0x{ubootEnvAddrRed:X} 0
send_file {{NandFlash}} "{dtbFile}" 0x{dtbAddr:X} 0
send_file {{NandFlash}} "{kernelFile}" 0x{kernelAddr:X} 0
NANDFLASH::NandSetTrimffs 1
send_file {{NandFlash}} "{rootfsFile}" 0x{rootfsAddr:X} 0
puts "-I- finished flashing"
""".format(
pmeccConfig=gECCcnfg[gArch],
bootstrapFile=_helper_genTCLFilename('bootloader'),
ubootFile=_helper_genTCLFilename('uboot'),
ubootAddr=gMemSetup['uboot'].add,
ubootEnvFile=_helper_genTCLFilename('ubootEnv'),
ubootEnvAddr=gMemSetup['ubootEnv'].add,
ubootEnvFileRed=_helper_genTCLFilename('ubootEnvRed'),
ubootEnvAddrRed=gMemSetup['ubootEnvRed'].add,
dtbFile=_helper_genTCLFilename('dtb'),
dtbAddr=gMemSetup['dtb'].add,
kernelFile=_helper_genTCLFilename('kernel'),
kernelAddr=gMemSetup['kernel'].add,
rootfsFile=_helper_genTCLFilename('rootfs'),
rootfsAddr=gMemSetup['rootfs'].add
)
with open(os.path.join(gDestDir, gTCLfilename), mode='w') as f:
f.write(tclStr)
f.close()
class _CLIError(Exception):
'''Generic exception to raise and log different fatal errors.'''
def __init__(self, msg):
super(_CLIError).__init__(type(self))
self.msg = "E: %s" % msg
def __str__(self):
return self.msg
def __unicode__(self):
return self.msg
def main(argv=None): # IGNORE:C0111
'''Command line options.'''
global gSourceDir
global gDestDir
global gTCLfilepath
if argv is None:
argv = sys.argv
else:
sys.argv.extend(argv)
program_name = os.path.basename(sys.argv[0])
program_version = "v%s" % __version__
program_build_date = str(__updated__)
program_version_message = '%%(prog)s %s (%s)' % (program_version, program_build_date)
program_shortdesc = __import__('__main__').__doc__.split("\n")[1]
program_license = '''%s
Created by Stefan Jaritz on %s.
Copyright 2016 organization_name. All rights reserved.
Licensed under the Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
Distributed on an "AS IS" basis without warranties
or conditions of any kind, either express or implied.
USAGE
''' % (program_shortdesc, str(__date__))
try:
# Setup argument parser
parser = ArgumentParser(description=program_license, formatter_class=RawDescriptionHelpFormatter)
parser.add_argument("-s", "--source", dest="srcD", help="source path of the images [default: %(default)s]", default=os.getcwd())
parser.add_argument("-d", "--destination", dest="dstD", help="destination where the images and the flash script is copied [default: %(default)s]", default=os.getcwd() )
parser.add_argument("-t", "--tclFilePath", dest="tclD", help="TCL script path. The path where the TCL file gona be located [default: %(default)s]", default='' )
parser.add_argument('-V', '--version', action='version', version=program_version_message)
# Process arguments
args = parser.parse_args()
gSourceDir = args.srcD
gDestDir = args.dstD
gTCLfilepath = args.tclD
shutil.rmtree(gDestDir, ignore_errors=True)
os.makedirs(gDestDir)
print("start")
_copyFiles()
_genUBootCfg()
_genFlashScript()
print("finished")
return 0
except KeyboardInterrupt:
### handle keyboard interrupt ###
return -1
except Exception as e:
indent = len(program_name) * " "
sys.stderr.write(program_name + ": " + repr(e) + "\n")
sys.stderr.write(indent + " for help use --help")
return -2
if __name__ == "__main__":
sys.exit(main())
III.) Create own package
yocto-layer create own
bitbake-layers add-layer "../meta-own"
modify the example.bb
SRC_URI = "file://testFTP.c"
S = "${WORKDIR}"
DEPENDS ="curl"
do_compile() {
${CC} testFTP.c -o testFTP ${CFLAGS} -lcurl
}
do_install() {
install -d ${D}${bindir}
install -m 0755 testFTP ${D}${bindir}
}
now add in "layer.conf"
IMAGE_INSTALL_append = " vim testftp"
Now the vim editor and the testftp recipe, which includes the testFTP binary is installed at the "user/bin" at the rootfs.
After the flashing, start the the SAMA5D35 Evalboard. Login as "root". Type: "testFTP" or "vim" and enjoy the output.
Cheers!