I have a directory where there are files -:
abc_002.txt
abc_003.txt
abc_004.txt
abc_005.txt
xyz_001.txt
for_ex.sh
abc_001.txt
I want to gzip only files starting with names abc, like abc*.
for file in abc*.txt
do
gzip $file
done
i need to recognize files with different extensions even when there is a combination of multiple extensions
so if my cwd has this files:
file-1 .zip
file-2 .tar
file-3 .tar.gz
file-4 .gz
file-5 .zip.tar
file-6 .tar.gz
file-7 .gz
i need to tell bash what to do when the extension (in this case) is:
zip
tar
zip.tar
tar.gz
gz
because for every extension i need to do different things, this implies that if the extension is .tar (only) or .gz (only) i need to do certain things, but if the extension is .tar.gz i need to run another snippet.
example:
if the filename has .tar extension i need to do
# stuff
tar xf filename.tar
# other stuff
if the filename has .zip.tar extension i need to run more complex code (but the code is not totally dependent on the extensions, my only objective is to get the full extension of the filename (filename.tar.gz should return .tar.gz instead of .gz or .tar)
Also, is there any way using gawk?
Use case:
case "$filename" in
*.tar.gz) code for .tar.gz ;;
*.gz) code for .gz ;;
*.zip.tar) code for .zip.tar ;;
*.tar) code for .tar ;;
...
esac
Just make sure you put the combined extensions before the single extensions that they contain, because case executes the statements for the first pattern that matches.
The file command is a good option to detect file types, then you can write a logic
file -i test.*
test.gz: application/x-gzip; charset=binary
test.tar: application/x-tar; charset=binary
test.tar.gz: application/x-gzip; charset=binary
test.zip: application/zip; charset=binary
I have a folder with multiple types of file ( mp4, mp4, jpg, wma .etc) and these files have either have no extension, or all messed up extensions extension such as mp3.mp3, mp3.jpg, or just file name. I was reading that exiftool or even python magic can be used to assign correct file extension on understanding filetype. I am looking for exiftool based solution where these file can be renamed with correct file extension.
eg
filename (this is mp3 file)
filename1.jpg ( this is again mp3 file, with jpg as file extension)
filename.mp3.mp3.mp3 (repetition of extension)
At the simplest, try this (change double quotes to single quotes if on Mac/Linux):
exiftool -ext "*" "-filename<$filename.$filetype" TargetDir
or
exiftool -ext "*" "-testname<%f.$filetype" TargetDir
That will simply add the extension all the files in TargetDir. To recurse, add -r. If there was already an extension, this will add the proper extension at the end of the false extension e.g. filename.mp3 would become filename.mp3.jpeg.
For a more complex version which strips away some of the previous, false extensions, you could try something like this:
exiftool -ext "*" "-filename<${filename;s/(\.(mp3|mp4|jpe?g|png|wma|mov))*($)//i}%-c.$filetype" TargetDir
which would strip away extensions that are in the center parens in the regex. The %-c will add a number if the resulting rename would be a duplicate e.g. filename.jpeg, filename-1.jpeg, … filename-n.jpeg.
Edit: added -ext option to deal with files without an extension.
I downloaded some files with extension .tar.bz2. I was able to untar these into folders containing .bz2 files. These should unzip as hdf5 files (Metadata said they were hdf5) , but they unzip into files with no extensions I have tried the following but didnt work:
untar("File.tar.bz2")
#Read lines of one of the files from the unzipped file
readLines(bzfile("File1.bz2"))
[1] "‰HDF" "\032"
library (rhdf5)
#Explore just as a bzip2 file
bzfile("File1.bz2")
description "File1.bz2"
class "bzfile"
mode "rb"
text "text"
opened "closed"
can read "yes"
can write "yes"
#Try to read as hdf5 using rhdf5 library
h5ls(bzfile("File1.bz2"))
Error in h5checktypeOrOpenLoc(). Argument neither of class H5IdComponent nor a character.
Is there some sort of encoding I need to do? What am I missing? What should I do?
tl;dr version:
Is it possible with CMake (>= 2.8) to generate zip files from some files and put the packed zip file in a specific location?
longer version:
I have a CMakeLists.txt that builds my project into a .exe file, and this exe file will read data from a zip file. The content to be packed in the zip file is in my git repository so that it can be edited, too. But, the program needs this data in a zip file. So it would be good if the CMake script could take the data, put it in a zip file, and place it next to the exe. I already heard of CPack, but I did not find any easy examples and am not sure if this is even the right tool for my task.
Is this possible? If yes, how?
Since version 3.2 CMake has the functionality to generate a zip file built-in. The CMake command-line mode sub-command tar supports both the creation of zip and 7zip archives.
For example, if the current CMake source directory contains the file testfile.txt and the directory testdir, you can use the following CMake commands to create a zip file containing both items:
add_custom_target(create_zip COMMAND
${CMAKE_COMMAND} -E tar "cfv" "archive.zip" --format=zip
"${CMAKE_CURRENT_SOURCE_DIR}/testfile.txt"
"${CMAKE_CURRENT_SOURCE_DIR}/testdir")
As a work-around for earlier CMake versions, you can use the jar command that is part of a standard Java JRE installation.
find_package(Java)
execute_process(
COMMAND
"${Java_JAR_EXECUTABLE}" "cfM" "archive.zip"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testfile.txt"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testdir"
RESULT_VARIABLE _result
)
The zip file will be generated in the current CMake binary dir (CMAKE_CURRENT_BINARY_DIR).
It's never late to show real answer:
function(create_zip output_file input_files working_dir)
add_custom_command(
COMMAND ${CMAKE_COMMAND} -E tar "cf" "${output_file}" --format=zip -- ${input_files}
WORKING_DIRECTORY "${working_dir}"
OUTPUT "${output_file}"
DEPENDS ${input_files}
COMMENT "Zipping to ${output_file}."
)
endfunction()
Use like
file(GLOB ZIP_FILES "${CMAKE_CURRENT_SOURCE_DIR}/zip/*")
create_zip("${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" "${ZIP_FILES}" "${CMAKE_CURRENT_SOURCE_DIR}/zip")
This will pack all files from zip/ subdirectory into native_data.zip (in build directory). Then either include your archive (path will differ in different CMakeLists.txt!) as source file or add it as target:
add_custom_target("project-data" ALL DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip")
Install will not differ a lot from usual:
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" DESTINATION ${DATADIR} RENAME "data000.zip") # Install our zip (as data000.zip)
I assume you already have a zip-tool installed (WinZip or 7z, etc.). You could write a find_zip-tool script which will search for WinZip, or 7Z, etc...
Snippet for WinZip:
FIND_PROGRAM(ZIP_EXECUTABLE wzzip PATHS "$ENV{ProgramFiles}/WinZip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" -P \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Snippet for 7-zip:
FIND_PROGRAM(ZIP_EXECUTABLE 7z PATHS "$ENV{ProgramFiles}/7-Zip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" a -tzip \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Take a look at the file
<cmake-install-dir>\share\cmake-2.8\Modules\CPackZIP.cmake
it shows how CPack searches for a Zip_Executable and prepares some "useful" default flags.
After that, I would suggest to execute_process, similar to sakra's answer
As of version 3.18, CMake now directly supports creating zip or archive files using the file() command with ARCHIVE_CREATE:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.zip
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT zip
)
Be sure to specify a full path for the OUTPUT zipped filename, or the file may not be generated. Also, the PATHS option accepts files or directories to be placed in the zip file, but it does not accept wildcards at the time of writing.
This command supports several archive formats and compression flavors. So, you can use the same command to create tarballs as well:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.tar.gz
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT gnutar
COMPRESSION GZip
)
Since this is the top search result for creating zip files with CMake, here is a CPack solution for completeness. The basic idea is that you make calls to install() and then tell it what to name the resulting zip file. It will be placed in the build directory, though there may be a way to change that. Then you can create the zip file with make package or cpack.
# Version 1: Subtractive
# Include everything in the project source directory.
# Put it at the top level of the zip via `DESTINATION .`
# Subtract things we don't want.
# The trailing slash after "${PROJECT_SOURCE_DIR}/" prevents
# an extra layer of directories.
install(DIRECTORY "${PROJECT_SOURCE_DIR}/"
DESTINATION .
PATTERN ".git*" EXCLUDE
PATTERN ".DS_Store" EXCLUDE
PATTERN "examples" EXCLUDE
PATTERN "docs" EXCLUDE
PATTERN "README.md" EXCLUDE
)
# Version 2: Additive
# Include only the list of things we specify.
# Put it at the top level of the zip via `DESTINATION .`
# install(FILES
# ${SRCS}
# "Notes.txt"
# DESTINATION .
# )
# Tell CPack to create a zip file.
set(CPACK_GENERATOR "ZIP")
# Tell CPack what to name the zip file. It will append `.zip`.
set(CPACK_PACKAGE_FILE_NAME "${CMAKE_PROJECT_NAME}")
# Tell CPack not to put everything inside an enclosing directory.
set(CPACK_INCLUDE_TOPLEVEL_DIRECTORY OFF)
# Apparently this should be always on but isn't for backwards compatibility.
set(CPACK_VERBATIM_VARIABLES YES)
include(CPack)
Essentially what I did was create custom target
add_custom_target(STAGE_FILES)
With this target I copy the files and directories to the CMAKE_CURRENT_BINARY_DIR
add_custom_command(
TARGET STAGE_FILES
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/video ${CMAKE_CURRENT_BINARY_DIR}/video
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/data ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_rules_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E rename ${CMAKE_CURRENT_BINARY_DIR}/data.zip ${CMAKE_CURRENT_BINARY_DIR}/data
)
The important line
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
inside my
to_zip.txt
I specify all the files I want to include in my zip
data/
video/
...
I can now execute the command
make STAGE_FILES
which will copy and zip everything i need