tl;dr version:
Is it possible with CMake (>= 2.8) to generate zip files from some files and put the packed zip file in a specific location?
longer version:
I have a CMakeLists.txt that builds my project into a .exe file, and this exe file will read data from a zip file. The content to be packed in the zip file is in my git repository so that it can be edited, too. But, the program needs this data in a zip file. So it would be good if the CMake script could take the data, put it in a zip file, and place it next to the exe. I already heard of CPack, but I did not find any easy examples and am not sure if this is even the right tool for my task.
Is this possible? If yes, how?
Since version 3.2 CMake has the functionality to generate a zip file built-in. The CMake command-line mode sub-command tar supports both the creation of zip and 7zip archives.
For example, if the current CMake source directory contains the file testfile.txt and the directory testdir, you can use the following CMake commands to create a zip file containing both items:
add_custom_target(create_zip COMMAND
${CMAKE_COMMAND} -E tar "cfv" "archive.zip" --format=zip
"${CMAKE_CURRENT_SOURCE_DIR}/testfile.txt"
"${CMAKE_CURRENT_SOURCE_DIR}/testdir")
As a work-around for earlier CMake versions, you can use the jar command that is part of a standard Java JRE installation.
find_package(Java)
execute_process(
COMMAND
"${Java_JAR_EXECUTABLE}" "cfM" "archive.zip"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testfile.txt"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testdir"
RESULT_VARIABLE _result
)
The zip file will be generated in the current CMake binary dir (CMAKE_CURRENT_BINARY_DIR).
It's never late to show real answer:
function(create_zip output_file input_files working_dir)
add_custom_command(
COMMAND ${CMAKE_COMMAND} -E tar "cf" "${output_file}" --format=zip -- ${input_files}
WORKING_DIRECTORY "${working_dir}"
OUTPUT "${output_file}"
DEPENDS ${input_files}
COMMENT "Zipping to ${output_file}."
)
endfunction()
Use like
file(GLOB ZIP_FILES "${CMAKE_CURRENT_SOURCE_DIR}/zip/*")
create_zip("${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" "${ZIP_FILES}" "${CMAKE_CURRENT_SOURCE_DIR}/zip")
This will pack all files from zip/ subdirectory into native_data.zip (in build directory). Then either include your archive (path will differ in different CMakeLists.txt!) as source file or add it as target:
add_custom_target("project-data" ALL DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip")
Install will not differ a lot from usual:
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" DESTINATION ${DATADIR} RENAME "data000.zip") # Install our zip (as data000.zip)
I assume you already have a zip-tool installed (WinZip or 7z, etc.). You could write a find_zip-tool script which will search for WinZip, or 7Z, etc...
Snippet for WinZip:
FIND_PROGRAM(ZIP_EXECUTABLE wzzip PATHS "$ENV{ProgramFiles}/WinZip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" -P \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Snippet for 7-zip:
FIND_PROGRAM(ZIP_EXECUTABLE 7z PATHS "$ENV{ProgramFiles}/7-Zip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" a -tzip \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Take a look at the file
<cmake-install-dir>\share\cmake-2.8\Modules\CPackZIP.cmake
it shows how CPack searches for a Zip_Executable and prepares some "useful" default flags.
After that, I would suggest to execute_process, similar to sakra's answer
As of version 3.18, CMake now directly supports creating zip or archive files using the file() command with ARCHIVE_CREATE:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.zip
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT zip
)
Be sure to specify a full path for the OUTPUT zipped filename, or the file may not be generated. Also, the PATHS option accepts files or directories to be placed in the zip file, but it does not accept wildcards at the time of writing.
This command supports several archive formats and compression flavors. So, you can use the same command to create tarballs as well:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.tar.gz
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT gnutar
COMPRESSION GZip
)
Since this is the top search result for creating zip files with CMake, here is a CPack solution for completeness. The basic idea is that you make calls to install() and then tell it what to name the resulting zip file. It will be placed in the build directory, though there may be a way to change that. Then you can create the zip file with make package or cpack.
# Version 1: Subtractive
# Include everything in the project source directory.
# Put it at the top level of the zip via `DESTINATION .`
# Subtract things we don't want.
# The trailing slash after "${PROJECT_SOURCE_DIR}/" prevents
# an extra layer of directories.
install(DIRECTORY "${PROJECT_SOURCE_DIR}/"
DESTINATION .
PATTERN ".git*" EXCLUDE
PATTERN ".DS_Store" EXCLUDE
PATTERN "examples" EXCLUDE
PATTERN "docs" EXCLUDE
PATTERN "README.md" EXCLUDE
)
# Version 2: Additive
# Include only the list of things we specify.
# Put it at the top level of the zip via `DESTINATION .`
# install(FILES
# ${SRCS}
# "Notes.txt"
# DESTINATION .
# )
# Tell CPack to create a zip file.
set(CPACK_GENERATOR "ZIP")
# Tell CPack what to name the zip file. It will append `.zip`.
set(CPACK_PACKAGE_FILE_NAME "${CMAKE_PROJECT_NAME}")
# Tell CPack not to put everything inside an enclosing directory.
set(CPACK_INCLUDE_TOPLEVEL_DIRECTORY OFF)
# Apparently this should be always on but isn't for backwards compatibility.
set(CPACK_VERBATIM_VARIABLES YES)
include(CPack)
Essentially what I did was create custom target
add_custom_target(STAGE_FILES)
With this target I copy the files and directories to the CMAKE_CURRENT_BINARY_DIR
add_custom_command(
TARGET STAGE_FILES
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/video ${CMAKE_CURRENT_BINARY_DIR}/video
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/data ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_rules_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E rename ${CMAKE_CURRENT_BINARY_DIR}/data.zip ${CMAKE_CURRENT_BINARY_DIR}/data
)
The important line
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
inside my
to_zip.txt
I specify all the files I want to include in my zip
data/
video/
...
I can now execute the command
make STAGE_FILES
which will copy and zip everything i need
Related
I am compelled to use an older cmake version 2.8.12, in a Linux environment.
As a pre-build step, I must copy multiple header files from a source directory to a destination directory. I decided to use the add_custom_target clause. If that in itself is the Bad Idea, please let me know. For example:
add_custom_target( prebuild
COMMENT "Prebuild step: copy other headers"
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_SOURCE_DIR}/../other/include/alpha.h ${CMAKE_SOURCE_DIR}/include/other
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_SOURCE_DIR}/../other/include/bravo.h ${CMAKE_SOURCE_DIR}/include/other
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_SOURCE_DIR}/../other/include/charlie.h ${CMAKE_SOURCE_DIR}/include/other
)
add_executable( myapp main.cxx )
# My application depends on the pre-build step.
add_dependencies( myapp prebuild )
set_target_properties( myapp PROPERTIES COMPILE_FLAGS "-g" )
install( TARGETS myapp DESTINATION ${BIN_INSTALL_DIR} )
It would be tedious to list each header file. I know how to search for all of the header files and put them in a list variable. For example.
file( GLOB other_headers "${CMAKE_SOURCE_DIR}/../other/include/*.h" )
But, how do I put that list variable to use within the add_custom_target clause?
Is there a way to copy multiple files within the add_custom_target clause?
Is there a better way to copy multiple files as a pre-build step that can be a dependency for the build of my application?
Being restricted to an older version of cmake limits my options. The following are things I have tried without success.
If I had cmake version 3.5, then the copy_if_different command could take multiple source paths.
Adding COMMAND_EXPAND_LISTS to the add_custom_target clause has no effect. That must not be available until a newer cmake version.
Using a foreach loop within the add_custom_target clause does not work. In other examples I have seen, they always use a foreach loop, and the add_custom_target is inside that loop. But, if I do that, then I do not know how to make that a dependency for the build of my application.
Using a foreach loop within the add_custom_target clause does not work.
But using foreach you may create a variable with all required commands. Then use that variable in add_custom_target:
set(commands)
# Assume 'other_headers' contain list of files
foreach(header ${other_headers})
list(APPEND commands
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${header} ${CMAKE_SOURCE_DIR}/include/other)
endforeach()
add_custom_target( prebuild
COMMENT "Prebuild step: copy other headers"
${commands}
)
You could use foreach to add custom targets for each header file, but also pull the add_dependencies() call into the loop block:
add_executable( myapp main.cxx )
foreach(cur_header ${other_headers})
# Get the filename from the full path.
get_filename_component(my_header_name ${cur_header} NAME)
# Add a new custom target for the current header.
add_custom_target( prebuild_${my_header_name}
COMMENT "Prebuild step: copy other headers"
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${cur_header} ${CMAKE_SOURCE_DIR}/include/other
)
# My application depends on the each pre-build target.
add_dependencies( myapp prebuild_${my_header_name} )
endforeach()
I am new, yocto build at imx6q embedded system.
I want to overwrite linux system files after do_rootfs. For example, target system files are below.
/etc/network/interface
/etc/issue
/etc/init.d/rcS
/home/root/mytest.sh
so, i made custom layer and custom recipe.
helloworld binary is copy ok.
but, do_mytask function is not called.
what's wrong with my code?
or any other method for my purpose.
#
# This file was derived from the 'Hello World!' example recipe in the
# Yocto Project Development Manual.
#
SUMMARY = "Simple helloworld application"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
SRC_URI = "file://helloworld.c"
S = "${WORKDIR}"
do_compile() {
${CC} helloworld.c -o helloworld
}
do_install() {
install -d ${D}${bindir}
install -m 0755 helloworld ${D}${bindir}
}
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += " \
file://interfaces \
file://issue \
file://mytest.sh \
"
addtask mytask after do_rootfs before do_image
do_mytask() {
install -d ${D}/etc/network
cp -af ${WORKDIR}/interfaces ${D}/etc/network/interfaces
cp -af ${WORKDIR}/issue ${D}/etc/issue
}
You'll need to extend the recipes that provide the files you want to replace.
Using /etc/network/interfaces as an example, the first step is to figure out which recipe installs that file.
From the bitbake prompt:
$ oe-pkgdata-util find-path /etc/network/interfaces
init-ifupdown: /etc/network/interfaces
So this tells us that /etc/network/interfaces is installed by the init-ifupdown receipe.
A file search shows that init-ifupdown is part of poky:
$ find . -name init-ifupdown*.bb
./poky/meta/recipes-core/init-ifupdown/init-ifupdown_1.0.bb
Now, since you need to modify the output of init-ifupdown, you'll need to extend init-ifupdown by creating a similarly named .bbappend in your own layer.
You might create the new .bbappend at
my-layer/receipes-core/init-ifupdown/init-ifupdown_%.bbappend
The % is a wildcard that ensures the .bbappend will apply to all future versions of the init-ifupdown recipe, which is probably what you want.
Place your custom interfaces file in a folder below the .bbappend:
my-layer/receipes-core/init-ifupdown/files/interfaces
The .bbappend then only needs to contain a single line to enable bitbake to pick up the new interfaces file:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
Finally, repeat the above with each system file you'd like to replace.
It depends on the file to modify. For example, if you search 'interfaces' in poky directories, you'll find it in 'meta/recipes-core/init-ifupdown/init-ifupdown-${PV}/'. You just need to create a recipe named init-ifupdown-${PV}.bbappend in your meta, recreating the path seen in poky (recipes-core/init-ifupdown/). This recipe can contain a single line :
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
Then you create a 'files' folder with the 'interfaces' file you want to have.
For 'issue', like others found in the /etc directory (profile, fstab, ...), it's the same procedure, with the sources in poky/meta/recipes-core/base-files/.
For init.d scripts, use the 'update-rc' class.
You recipe is not "image recipe" (and it shouldn't be for hello world) thus you cannot use tasks do_rootfs and do_image in this case. A bit of clarification: image recipe is .bb file that you use to build image with bitbake or devtool (in your case some containing imx6q, you can find them with bitbake-layers show-recipes "*-image-*").
It looks like you are looking really is a way to override do_install of some recipe that installs that mentioned files. Then find what recipe installs those files and create bbappend file in your top layer. This bbappend file may contain do_install_append task where you can place your install <file> <dir> lines (note, using cp as not recommended, everything should be done with install tool).
Adding an extra comment based on Carsten Hansen original answer for folks working with Xilinx/Petalinux.
Under Petalinux environment we don't really have the command: oe-pkgdata-util, so the strategy is to do a search in the Xilinx SDK folder. You might have it installed on Linux under /opt according to the documentation. If you do a:
grep -r syslog-startup.conf .
you will see busybox recipe being the one that does the installation of the syslog-startup.conf.
You can create the override recipe called busybox_%.bbappend under:
../project-spec/meta-user/recipes-core/busybox/
Put the modified syslog-startup.conf file under:
../project-spec/meta-user/recipes-core/busybox/files/syslog-startup.conf
Rebuild via petalinux-build. You can also force the creation of the rootfs via petalinux-build -c rootfs and the system should populate your new file.
I have this structure:
release/folder1/file1
release/folder2/file2
...
release/folderN/fileN
I want to include all those folders (folder1, folder2 ... folderN) in a tar file.
The key is that I want these folders to be in the final tar within another directory named MYAPP so when you open the tar you can see this:
MYAPP/folder1/file1
MYAPP/folder2/file2
...
MYAPP/folderN/fileN
How can I achieve this without renaming the original "release" directory and/or creating new directories.
Is this possible to achive just in the tar process?
Thanks
Add
--transform=s#^release/#MYAPP/#
to your tar command line.
The argument of the --transform command line is a command that is passed to sed together with the file path before it is stored in the archive (use tar -tf to show the names of the files stored in the archive).
The command s#^release/#MYAPP/# tells sed to search (s) release/ at the beginning of the string (^) and replace it with MYAPP/.
The / at the end of the search and replace strings is needed to be sure the complete name of the component is release (to not replace release.txt). The # character is just a regex delimiter. Usually / is used as a regex delimiter but we prefer to use a different delimiter here to avoid the need to escape / (because it is used in the search and replace strings).
Read more in the documentation of tar and sed.
I have a pretty big file structure of a project which I need to convert into a multiplatform cmake project. Now it seams that cmake requires ever single cpp file be added individually to the executable. But is there a script that automates this? That snoopes through the file structure and just adds every source file automatically? Since the project will probably get a lot more source files and I probably wont be able to manually add every single one.
You could use execute_process() with a cmake -P script that uses globbing to recursively scan for source files which writes to an included file in your CMakeLists.txt i.e. something like:
"CMakeLists.txt":
execute_process(COMMAND ${CMAKE_COMMAND}
-D "RDIR=${CMAKE_CURRENT_SOURCE_DIR}"
-P "scansources.cmake"
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}")
include("sources.cmake")
add_executable(myexe ${sources})
"scansources.cmake" (generates "sources.cmake"):
file(GLOB_RECURSE sourcelist
*.c
*.cc
*.cpp
*.cxx)
string(REGEX REPLACE "${RDIR}/" "" relative_sources "${sourcelist}")
string(REPLACE ";" "\n" sources_string "${relative_sources}")
set(sources_string "set(sources\n${sources_string})")
file(WRITE sources.cmake "${sources_string}")
The reason why this works is because execute_process() occurs at configure time.
You could, of course, generate sources.cmake via some other tool or IDE then you wouldn't need scansources.cmake or execute_process().
I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure.
Determine the directory structure first,
then build and use a download description file:
aria2c -i uri.txt
where uri.txt might contain
http://serverA/file1.iso http://mirror-serverB/file1.iso
# parameters must begin with a space, otherwise it's treatened as url!
dir=/downloads/a
# not mandatory
out=file1.iso
http://serverA/file2.iso http://mirror-serverB/file2.iso
dir=/downloads/b
out=file2.iso
Keep in mind that aria2 is a download util - not an sync util, like rsync or lftp.
Referencing an rsync answer: https://stackoverflow.com/a/4147263/1163786
and an lftp answer: https://superuser.com/a/305236.