I want to create a 100 Byte file of zeroes.
I used the script:
dd if=/dev/zero of=zero_file_100B bs=50 count=2
It works fine, but when I do cat zero_file_100B it does not print anything. What might be the reason for that?
The reason for that is the character with ASCII code 0 has no graphical representation in your terminal emulator, which is the expected behavior. You should't confuse it with the character '0' which has ASCII code 48.
To view binary data a portable way, you might use od:
od -v -t u1 zero
but when I do cat zero_file_100B it does not print anything
perhaps you were looking for this
$ printf "%0*d" 100 0 > 38650594
$ ls -l 38650594
-rw-rw-r-- 1 me me 100 Jul 29 10:32 38650594 # Size is 100 bytes.
What do you want with zero?
You already succeeded in \0, you can print '0' with
for i in {1..100}; do printf 0; done`
Slower than other solutions, but you can change this for other requirements:
# create file with size 100 only containing strings "zero"
for i in {1..25}; do printf "zero"; done
# create file with size 100 only containing strings "zeros"
for i in {1..20}; do printf "zeros"; done
Related
I'm trying to clean a binary file to delete all the NULL on it. The task is quite simple, but I found out a lot of files have a NULL at the end of the file and i dont know what. I'm dumping the hexadecimal value of each byte and i dont see the null anywhere, but if I do a hexdump of the file, I see a value 00 at the end and I dont know why.... Could be that it is a EOF, but it's weird becuase it doesnt appear in all files. This is the script I have, quite simpel one, it generates 100 random binary files, and then reads file per file, char per char. Following the premise that bash wont store NULL's on variables, rewritting the char after storing it on a variable would avoid the NULL's, but no....
#!/bin/bash
for i in $(seq 0 100)
do
echo "$i %"
time dd if=/dev/urandom of=$i bs=1 count=1000
while read -r -n 1 c;
do
echo -n "$c" >> temp
done < $i
mv temp $i
done
I also tried with:
tr '\000' <inFile > outfile
But same result.
This is how it looks the hexdump of one the files with this problem
00003c0 0b12 a42b cb50 2a90 1fd6 a4f9 89b4 ddb6
00003d0 3fa3 eb7e 00c4
c4 should be the last byte butas you can see, there's a 00 there ....
Any clue?
EDIT:
Forgot to mention that the machine where im running this is something similar like raspberry pi and the tools provided with it are quite limited.
Try these other commands:
od -tx1 inFile
xxd inFile
hexdump outputs 00 when the size is an odd number of bytes.
It seems hexdump without options is like -x, hexdump -h gives the list of options; hexdump -C may also help.
Suppose I have a file temp.txt with 100 lines. I would like to split into 10 parts.
I use following command
split a 1 -l 10 -d temp.txt temp_
But I got temp_0, temp_1, temp_2,...,temp_9. I want output like this temp_1,temp_2,..,temp_10.
From man split
I got
-d, --numeric-suffixes
use numeric suffixes instead of alphabetic
I tried to use
split -l 10 --suffix-length=1 --numeric-suffixes=1 Temp.txt temp_
It says split: option '--numeric-suffixes' doesn't allow an argument
Then, I tried to use
split -l 10 --suffix-length=1 --numeric-suffixes 1 Temp.txt temp_
It says
split: extra operandtemp_'`
The output of split --version is
split (GNU coreutils) 8.4
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Torbj�rn Granlund and Richard M. Stallman.
I tried to use split -a 1 -l 10 -d 1 Temp.txt temp_. But it shows error split: extra operand temp_' `
-d doesn't have an argument. It should be written as you originally tried;
split -a 1 -l 10 -d Temp.txt temp_
But, forgetting the syntax variations for a moment;
you're asking it to split a 100 line file into 10 parts, with a suffix length of 1, starting at 1.
^- This scenario is erroneous as it is asking the command to process 100 lines and giving it fixed parameters restricting it to processing only 90 lines.
If you're willing to extend your allowable suffix length to 2, then you will at least get a uniform two digit temp file starting at 01;
split -a 1 -l 10 --numeric-suffixes=1 -d Temp.txt temp_
Will create: temp_01 thru temp_10
You can actually negate the -a and -d argument altogether;
split -l 10 --numeric-suffixes=1 Temp.txt temp_
Will also create: temp_01 thru temp_10
If for some reason this was a fixed and absolute requirement or a permanent solution (i.e. integrating to something else you have no control of), and it was always going to be an exactly 100 line file, then you could always do it in two passes;
cat Temp.txt | head -n90 | split -a 1 -l 10 --numeric-suffixes=1 - temp_
cat Temp.txt | tail -n10 | split -a 2 -l 10 --numeric-suffixes=10 - temp_
Then you would get temp_1 thru temp_10
Just to throw out a possible alternative, you can accomplish this task manually by running a couple of loops. The outer loop iterates over the file chunks and the inner loop iterates over the lines within the chunk.
{
suf=1;
read -r; rc=$?;
while [[ $rc -eq 0 || -n "$REPLY" ]]; do
line=0;
while [[ ($rc -eq 0 || -n "$REPLY") && line -lt 10 ]]; do
printf '%s\n' "$REPLY";
read -r; rc=$?;
let ++line;
done >temp_$suf;
let ++suf;
done;
} <temp.txt;
Notes:
The test $rc -eq 0 || -n "$REPLY" is necessary to continue processing if either we've not yet reached end-of-file (in which case $rc eq 0 is true) or we have reached end-of-file but there was a non-empty final line in the input file (in which case -n "$REPLY" is true). It's good to try to support the case of a non-empty final line with no end-of-line delimiter, which sometimes happens. In this case read will return a failing status code but will still correctly set $REPLY to contain the non-empty final line content. I've tested the split utility and it correctly handles this case as well.
By calling read once prior to the outer loop and then once after each print, we ensure that we always test if the read was successful prior to printing the resulting line. A more naïve design might read and print in immediate succession with no check in between, which would be incorrect.
I've used the -r option of read to prevent backslash interpolation, which you probably don't want; I assume you want to preserve the contents of temp.txt verbatim.
Obviously there are tradeoffs in this solution. On the one hand, it demands a fair amount of complexity and code verbosity (13 lines the way I've written it). But the advantage is complete control over the behavior of the split operation; you can customize the script to your liking, such as dynamically changing the suffix based on the line number, using a prefix or infix or combination thereof, or even taking into account the contents of the individual file lines in $REPLY.
Looking for a command or set of commands that are readily available on Linux distributions that would allow me to create a script to generate a checksum for a file.
This checksum is generated by a build system I have no control over by summing every single byte in the file and then truncating that number to 4 bytes.
I know how to do do this using tools like node.js, perl, python, C/C++, etc, but I need to be able to do this on a bare bones Linux distribution running remotely that I can't modify (it's on a PLC).
Any ideas? I've been searching for awhile and haven't found anything that looks straightforward yet.
The solution for byte by byte summation and truncating that number to 4 bytes using much primitive shell commands.
#! /usr/bin/env bash
declare -a bytes
bytes=(`xxd -p -c 1 INPUT_FILE | tr '\n' ' '`)
total=0;
for(( i=0; i<${#bytes[#]}; i++));
do
total=$(($total + 0x${bytes[i]}))
if [ $total > 4294967295 ]; then
total=$(($total & 4294967295))
fi
done
echo "Checksum: " $total
If you just want to do byte by byte summation and truncating that number to 4 bytes then the following command can be used.
xxd -p -c 1 <Input file> | awk '{s+=$1; if(s > 4294967295) s = and(4294967295, s) } END {print s}'
The xxd command is used extract hexdump of the input file and each byte is added to compute the sum. If the sum exceeds 2^32-1 = 4294967295 value, then a bitwise and operation is performed to truncate the bits.
Have you tried cksum? I use it inside a few scripts. It's very simple to use.
http://linux.die.net/man/1/cksum
I have a sequence of files:
image001.jpg
image002.jpg
image003.jpg
Can you help me with a bash script that copies the images in reverse order so that the final result is:
image001.jpg
image002.jpg
image003.jpg
image004.jpg <-- copy of image003.jpg
image005.jpg <-- copy of image002.jpg
image006.jpg <-- copy of image001.jpg
The text in parentheses is not part of the file name.
Why do I need it? I am creating video from a sequence of images and would like the video to play "forwards" and then "backwards" (looping the resulting video).
You can use printf to print a number with leading 0s.
$ printf '%03d\n' 1
001
$ printf '%03d\n' 2
002
$ printf '%03d\n' 3
003
Throwing that into a for loop yields:
MAX=6
for ((i=1; i<=MAX; i++)); do
cp $(printf 'image%03d.jpg' $i) $(printf 'image%03d.jpg' $((MAX-i+1)))
done
I think that I'd use an array for this... that way, you don't have to hard code a value for $MAX.
image=( image*.jpg )
MAX=${#image[*]}
for i in ${image[*]}
do
num=${i:5:3} # grab the digits
compliment=$(printf '%03d' $(echo $MAX-$num | bc))
ln $i copy_of_image$compliment.jpg
done
I used 'bc' for arithmetic because bash interprets leading 0s as an indicator that the number is octal, and the parameter expansion in bash isn't powerful enough to strip them without jumping through hoops. I could have done that in sed, but as long as I was calling something outside of bash, it made just as much sense to do the arithmetic directly.
I suppose that Kuegelman's script could have done something like this:
MAX=(ls image*.jpg | wc -l)
That script has bigger problems though, because it's overwriting half of the images:
cp image001.jpg image006.jpg # wait wait!!! what happened to image006.jpg???
Also, once you get above 007, you run into the octal problem.
How do I create an unmodified hex dump of a binary file in Linux using bash? The od and hexdump commands both insert spaces in the dump and this is not ideal.
Is there a way to simply write a long string with all the hex characters, minus spaces or newlines in the output?
xxd -p file
Or if you want it all on a single line:
xxd -p file | tr -d '\n'
Format strings can make hexdump behave exactly as you want it to (no whitespace at all, byte by byte):
hexdump -ve '1/1 "%.2x"'
1/1 means "each format is applied once and takes one byte", and "%.2x" is the actual format string, like in printf. In this case: 2-character hexadecimal number, leading zeros if shorter.
It seems to depend on the details of the version of od. On OSX, use this:
od -t x1 -An file |tr -d '\n '
(That's print as type hex bytes, with no address. And whitespace deleted afterwards, of course.)
Perl one-liner:
perl -e 'local $/; print unpack "H*", <>' file
The other answers are preferable, but for a pure Bash solution, I've modified the script in my answer here to be able to output a continuous stream of hex characters representing the contents of a file. (Its normal mode is to emulate hexdump -C.)
I think this is the most widely supported version (requiring only POSIX defined tr and od behavior):
cat "$file" | od -v -t x1 -A n | tr -d ' \n'
This uses od to print each byte as hex without address without skipping repeated bytes and tr to delete all spaces and linefeeds in the output. Note that not even the trailing linefeed is emitted here. (The cat is intentional to allow multicore processing where cat can wait for filesystem while od is still processing previously read part. Single core users may want replace that with < "$file" od ... to save starting one additional process.)
tldr;
$ od -t x1 -A n -v <empty.zip | tr -dc '[:xdigit:]' && echo
504b0506000000000000000000000000000000000000
$
Explanation:
Use the od tool to print single hexadecimal bytes (-t x1) --- without address offsets (-A n) and without eliding repeated "groups" (-v) --- from empty.zip, which has been redirected to standard input. Pipe that to tr which deletes (-d) the complement (-c) of the hexadecimal character set ('[:xdigit:]'). You can optionally print a trailing newline (echo) as I've done here to separate the output from the next shell prompt.
References:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/od.html
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tr.html
This code produces a "pure" hex dump string and it runs faster than the all the
other examples given.
It has been tested on 1GB files filled with binary zeros, and all linefeeds.
It is not data content dependent and reads 1MB records instead of lines.
perl -pe 'BEGIN{$/=\1e6} $_=unpack "H*"'
Dozens of timing tests show that for 1GB files, these other methods below are slower.
All tests were run writing output to a file which was then verified by checksum.
Three 1GB input files were tested: all bytes, all binary zeros, and all LFs.
hexdump -ve '1/1 "%.2x"' # ~10x slower
od -v -t x1 -An | tr -d "\n " # ~15x slower
xxd -p | tr -d \\n # ~3x slower
perl -e 'local \$/; print unpack "H*", <>' # ~1.5x slower
- this also slurps the whole file into memory
To reverse the process:
perl -pe 'BEGIN{$/=\1e6} $_=pack "H*",$_'
You can use Python for this purpose:
python -c "print(open('file.bin','rb').read().hex())"
...where file.bin is your filename.
Explaination:
Open file.bin in rb (read binary) mode.
Read contents (returned as bytes object).
Use bytes method .hex(), which returns hex dump without spaces or new lines.
Print output.