I am trying to write a bash script that takes a .cmpr compressed file and uncompress it.
I've tried running the following command in the shell to see if it works -
uncompress file.cmpr
but the file is not being uncompressed and I get the following message -
gzip: file.cmpr: unknown suffix -- ignored
I want to use the 'uncompress' command as specified in here -
http://man7.org/linux/man-pages/man1/uncompress.1p.html
how do I fix this? how to I add support for '.cmpr' files on my machine? or is it something else
uncompress expects files to have a .Z suffix. Rename it to file.Z and then run either uncompress file.Z or uncompress file—either one works.
Related
i need to recognize files with different extensions even when there is a combination of multiple extensions
so if my cwd has this files:
file-1 .zip
file-2 .tar
file-3 .tar.gz
file-4 .gz
file-5 .zip.tar
file-6 .tar.gz
file-7 .gz
i need to tell bash what to do when the extension (in this case) is:
zip
tar
zip.tar
tar.gz
gz
because for every extension i need to do different things, this implies that if the extension is .tar (only) or .gz (only) i need to do certain things, but if the extension is .tar.gz i need to run another snippet.
example:
if the filename has .tar extension i need to do
# stuff
tar xf filename.tar
# other stuff
if the filename has .zip.tar extension i need to run more complex code (but the code is not totally dependent on the extensions, my only objective is to get the full extension of the filename (filename.tar.gz should return .tar.gz instead of .gz or .tar)
Also, is there any way using gawk?
Use case:
case "$filename" in
*.tar.gz) code for .tar.gz ;;
*.gz) code for .gz ;;
*.zip.tar) code for .zip.tar ;;
*.tar) code for .tar ;;
...
esac
Just make sure you put the combined extensions before the single extensions that they contain, because case executes the statements for the first pattern that matches.
The file command is a good option to detect file types, then you can write a logic
file -i test.*
test.gz: application/x-gzip; charset=binary
test.tar: application/x-tar; charset=binary
test.tar.gz: application/x-gzip; charset=binary
test.zip: application/zip; charset=binary
I'm trying to decompress ~8GB .zip file piped from curl command. Everything I have tried is being interrupted at <1GB and returns a message:
... has more than one entry--rest ignored
I've tried: funzip, gunzip, gzip -d, zcat, ... also with different arguments - all end up in the above message.
The datafile is public, so it's easy to repro the issue:
curl -L https://archive.org/download/nycTaxiTripData2013/faredata2013.zip | funzip > datafile
Are you sure the mentioned file deflates to a single file? If it extracts to multiple files you unfortunately cannot unzip on the fly.
Zip is a container as well as compression format and it doesn't know where the new file begins. You'll have to download the whole file and unzip it.
I am using lz4mt multi-threaded version of lz4 and in my workflow I am sending thousands of large size files (620 MB) from client to server and when file reaches on server my rule will trigger and compress file using lz4mt and then remove uncompressed file. The problem is sometimes when I remove uncompressed file, I am not able to get compressed file of right size its because lz4mt returns immediately before sending output to disk.
So is there any way lz4mt will remove uncompressed file itself after compressing as done by bzip2.
Input: bzip2 uncompress_file
Output: Compressed file only
whereas
Input: lz4mt uncompress_file
Output: (Uncompressed + Compressed) file
Below script sync command also not working properly I think.
The script which execute as my rule triggers is:
script.sh
/bin/lz4mt uncompressed_file output_file
/bin/sync
/bin/rm uncompressed_file
Please tell me how to solve above issue.
Thanks a lot
Author here. You could try the following methods
Concatenate commands with && or ;.
Add lz4mt command line option -q (suppress prompt), and -f (force overwrite).
Try it with original lz4.
I am trying to zip a file using shell script command. I am using following command:
zip ./test/step1.zip $FILES
where $FILES contain all the input files. But I am getting a warning as follows
zip warning: name not matched: myfile.dat
and one more thing I observed that the file which is at last in the list of files in a folder has the above warning and that file is not getting zipped.
Can anyone explain me why this is happening? I am new to shell script world.
zip warning: name not matched: myfile.dat
This means the file myfile.dat does not exist.
You will get the same error if the file is a symlink pointing to a non-existent file.
As you say, whatever is the last file at the of $FILES, it will not be added to the zip along with the warning. So I think something's wrong with the way you create $FILES. Chances are there is a newline, carriage return, space, tab, or other invisible character at the end of the last filename, resulting in something that doesn't exist. Try this for example:
for f in $FILES; do echo :$f:; done
I bet the last line will be incorrect, for example:
:myfile.dat :
...or something like that instead of :myfile.dat: with no characters before the last :
UPDATE
If you say the script started working after running dos2unix on it, that confirms what everybody suspected already, that somehow there was a carriage-return at the end of your $FILES list.
od -c shows the \r carriage-return. Try echo $FILES | od -c
Another possible cause that can generate a zip warning: name not matched: error is having any of zip's environment variables set incorrectly.
From the man page:
ENVIRONMENT
The following environment variables are read and used by zip as described.
ZIPOPT
contains default options that will be used when running zip. The contents of this environment variable will get added to the command line just after the zip command.
ZIP
[Not on RISC OS and VMS] see ZIPOPT
Zip$Options
[RISC OS] see ZIPOPT
Zip$Exts
[RISC OS] contains extensions separated by a : that will cause native filenames with one of the specified extensions to be added to the zip file with basename and extension swapped.
ZIP_OPTS
[VMS] see ZIPOPT
In my case, I was using zip in a script and had the binary location in an environment variable ZIP so that we could change to a different zip binary easily without making tonnes of changes in the script.
Example:
ZIP=/usr/bin/zip
...
${ZIP} -r folder.zip folder
This is then processed as:
/usr/bin/zip /usr/bin/zip -r folder.zip folder
And generates the errors:
zip warning: name not matched: folder.zip
zip I/O error: Operation not permitted
zip error: Could not create output file (/usr/bin/zip.zip)
The first because it's now trying to add folder.zip to the archive instead of using it as the archive. The second and third because it's trying to use the file /usr/bin/zip.zip as the archive which is (fortunately) not writable by a normal user.
Note: This is a really old question, but I didn't find this answer anywhere, so I'm posting it to help future searchers (my future self included).
eebbesen hit the nail in his comment for my case (but i cannot vote for comment).
Another possible reason missed in the other comments is file exceeding the file size limit (4GB).
I converted my script for unix environment using dos2unix command and executed my script as ./myscript.sh instead bash myscript.sh.
I just discovered another potential cause for this. If the permissions of the directory/subdirectory don't allow the zip to find the file, it will report this error. Actually, if you run a chmod -R 444 on the directory, and then try to zip it, you will reproduce this error, and also have a "stored 0%" report, like this:
zip warning: name not matched: borrar/enviar
adding: borrar/ (stored 0%)
Hence, try changing the permissions of the file. If you are trying to send them through email, and those email filters (like Gmail's) invent silly filters of not sending executables, don't forget that making permissions very strict when making zip compression can be the cause of the error you are reporting, of "name not matched".
spaces are not allowed:
it would fail if there are more than one files(s) in $FILES unless you put them in loop
I also encountered this issue. In my case, the line separate is CRLF in my zip shell script which causes the problem. Using LF fixed it.
tl;dr version:
Is it possible with CMake (>= 2.8) to generate zip files from some files and put the packed zip file in a specific location?
longer version:
I have a CMakeLists.txt that builds my project into a .exe file, and this exe file will read data from a zip file. The content to be packed in the zip file is in my git repository so that it can be edited, too. But, the program needs this data in a zip file. So it would be good if the CMake script could take the data, put it in a zip file, and place it next to the exe. I already heard of CPack, but I did not find any easy examples and am not sure if this is even the right tool for my task.
Is this possible? If yes, how?
Since version 3.2 CMake has the functionality to generate a zip file built-in. The CMake command-line mode sub-command tar supports both the creation of zip and 7zip archives.
For example, if the current CMake source directory contains the file testfile.txt and the directory testdir, you can use the following CMake commands to create a zip file containing both items:
add_custom_target(create_zip COMMAND
${CMAKE_COMMAND} -E tar "cfv" "archive.zip" --format=zip
"${CMAKE_CURRENT_SOURCE_DIR}/testfile.txt"
"${CMAKE_CURRENT_SOURCE_DIR}/testdir")
As a work-around for earlier CMake versions, you can use the jar command that is part of a standard Java JRE installation.
find_package(Java)
execute_process(
COMMAND
"${Java_JAR_EXECUTABLE}" "cfM" "archive.zip"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testfile.txt"
"-C" "${CMAKE_CURRENT_SOURCE_DIR}" "testdir"
RESULT_VARIABLE _result
)
The zip file will be generated in the current CMake binary dir (CMAKE_CURRENT_BINARY_DIR).
It's never late to show real answer:
function(create_zip output_file input_files working_dir)
add_custom_command(
COMMAND ${CMAKE_COMMAND} -E tar "cf" "${output_file}" --format=zip -- ${input_files}
WORKING_DIRECTORY "${working_dir}"
OUTPUT "${output_file}"
DEPENDS ${input_files}
COMMENT "Zipping to ${output_file}."
)
endfunction()
Use like
file(GLOB ZIP_FILES "${CMAKE_CURRENT_SOURCE_DIR}/zip/*")
create_zip("${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" "${ZIP_FILES}" "${CMAKE_CURRENT_SOURCE_DIR}/zip")
This will pack all files from zip/ subdirectory into native_data.zip (in build directory). Then either include your archive (path will differ in different CMakeLists.txt!) as source file or add it as target:
add_custom_target("project-data" ALL DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip")
Install will not differ a lot from usual:
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/native_data.zip" DESTINATION ${DATADIR} RENAME "data000.zip") # Install our zip (as data000.zip)
I assume you already have a zip-tool installed (WinZip or 7z, etc.). You could write a find_zip-tool script which will search for WinZip, or 7Z, etc...
Snippet for WinZip:
FIND_PROGRAM(ZIP_EXECUTABLE wzzip PATHS "$ENV{ProgramFiles}/WinZip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" -P \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Snippet for 7-zip:
FIND_PROGRAM(ZIP_EXECUTABLE 7z PATHS "$ENV{ProgramFiles}/7-Zip")
IF(ZIP_EXECUTABLE)
SET(ZIP_COMMAND "\"${ZIP_EXECUTABLE}\" a -tzip \"<ARCHIVE>\" #<FILELIST>")
ENDIF(ZIP_EXECUTABLE)
Take a look at the file
<cmake-install-dir>\share\cmake-2.8\Modules\CPackZIP.cmake
it shows how CPack searches for a Zip_Executable and prepares some "useful" default flags.
After that, I would suggest to execute_process, similar to sakra's answer
As of version 3.18, CMake now directly supports creating zip or archive files using the file() command with ARCHIVE_CREATE:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.zip
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT zip
)
Be sure to specify a full path for the OUTPUT zipped filename, or the file may not be generated. Also, the PATHS option accepts files or directories to be placed in the zip file, but it does not accept wildcards at the time of writing.
This command supports several archive formats and compression flavors. So, you can use the same command to create tarballs as well:
file(ARCHIVE_CREATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/MyData.tar.gz
PATHS ${CMAKE_CURRENT_SOURCE_DIR}/data
FORMAT gnutar
COMPRESSION GZip
)
Since this is the top search result for creating zip files with CMake, here is a CPack solution for completeness. The basic idea is that you make calls to install() and then tell it what to name the resulting zip file. It will be placed in the build directory, though there may be a way to change that. Then you can create the zip file with make package or cpack.
# Version 1: Subtractive
# Include everything in the project source directory.
# Put it at the top level of the zip via `DESTINATION .`
# Subtract things we don't want.
# The trailing slash after "${PROJECT_SOURCE_DIR}/" prevents
# an extra layer of directories.
install(DIRECTORY "${PROJECT_SOURCE_DIR}/"
DESTINATION .
PATTERN ".git*" EXCLUDE
PATTERN ".DS_Store" EXCLUDE
PATTERN "examples" EXCLUDE
PATTERN "docs" EXCLUDE
PATTERN "README.md" EXCLUDE
)
# Version 2: Additive
# Include only the list of things we specify.
# Put it at the top level of the zip via `DESTINATION .`
# install(FILES
# ${SRCS}
# "Notes.txt"
# DESTINATION .
# )
# Tell CPack to create a zip file.
set(CPACK_GENERATOR "ZIP")
# Tell CPack what to name the zip file. It will append `.zip`.
set(CPACK_PACKAGE_FILE_NAME "${CMAKE_PROJECT_NAME}")
# Tell CPack not to put everything inside an enclosing directory.
set(CPACK_INCLUDE_TOPLEVEL_DIRECTORY OFF)
# Apparently this should be always on but isn't for backwards compatibility.
set(CPACK_VERBATIM_VARIABLES YES)
include(CPack)
Essentially what I did was create custom target
add_custom_target(STAGE_FILES)
With this target I copy the files and directories to the CMAKE_CURRENT_BINARY_DIR
add_custom_command(
TARGET STAGE_FILES
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/video ${CMAKE_CURRENT_BINARY_DIR}/video
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/assets/data ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/assets/strings_rules_en.csv ${CMAKE_CURRENT_BINARY_DIR}/
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/data
COMMAND ${CMAKE_COMMAND} -E rename ${CMAKE_CURRENT_BINARY_DIR}/data.zip ${CMAKE_CURRENT_BINARY_DIR}/data
)
The important line
COMMAND ${CMAKE_COMMAND} -E tar "cfv" "data.zip" --format=zip --files-from=${CMAKE_SOURCE_DIR}/assets/to_zip.txt
inside my
to_zip.txt
I specify all the files I want to include in my zip
data/
video/
...
I can now execute the command
make STAGE_FILES
which will copy and zip everything i need