I ran a command to split a file of 2TB into files of 50GB each.
...| split -b 50G
Now, I need to join and uncompress them, on the same command. Because I don't have enough free disk space to have them both.
To join: cat part-* > big_file.gz
To uncompress: tar -xvf big_file.gz
How could I run those commands above on the same line?
To run several commands in bash use ; as a separator, like:
cat part-* > big_file.gz;tar -xvf big_file.gz
you can also combine them with some bash loop commands like while, for, etc
Related
I have a file needs to copy unique directory call test
directory structure as below
/contentroot/path/a/x/test
/contentroot/path/a/y/test
/contentroot/path/a/z/test
--------------------------
as above I have more then 250 combination test directory
I have try below command ( by using asterisk) but it's only copy one test directly only and giving issue (cp: omitting directory )
cp myfile.txt /contentroot/path/a/*/test
any Help
Perhaps a for loop?
for FOLDER in /contentroot/path/a/*/test; do
cp myfile.txt $FOLDER
done
You can feed the output of an echo as an input to xargs. xargs will then run the cp command three times, appending the next directory path piped to it from the echo.
The -n 1 option on the xargs command is so it only appends one of those arguments at a time to the cp each time it runs.
echo /contentroot/path/a/x/test /contentroot/path/a/y/test /contentroot/path/a/z/test | xargs -n 1 cp myfile.txt
Warnings! Firstly this will over-write files (if they exist) and secondlt any bash command should be tested and used at the runners risk! ;)
I have a C++ program that can be run in the following format (two cases):
./main 10000 file1.txt
OR
./main 10000 file1.txt file2.txt
where file1.txt and file2.txt are huge text files. I have the file.tar.gz that basically may include:
Just one file (file1.txt)
Two files file1.txt and file2.txt
Is there a way in bash to use pipe to read the files from the gz file directly, for both cases, i.e., even if the gz file contains one or two files? I have checked at Pipe multiple files (gz) into C program but I am not too savvy at bash and thus, I have trouble understanding the answers there.
This isn't going to be particularly simple. Your question is really too broad, as it stands. One approach would be:
Determine whether the archive contains one or two files
Set up named pipes (fifos) for each of the files ("mkfifo" command)
Run commands to output the content of the files in the archive to the appropriate fifo, as a background process
Run the primary command, specifying the fifos as the filename arguments
Giving a full rundown of all of this is, I think, beyond what should be the scope of a Stackoverflow question. For (1), you could probably do something like:
FILECOUNT=`tar -vzf (filename.tar.gz) | wc -l`
This lists the files within the archive (tar -vzf) and counts the number of lines of output from that command (wc -l). It's not foolproof but should work if the filenames are simple names like the ones you suggested (file1.txt, file2.txt).
For (2), make either one or two fifos as appropriate:
mkfifo file1-fifo.txt
if [ $FILECOUNT = 2 ]; then
mkfifo file2-fifo.txt
fi
For (3), use tar with -O to extract file contents from the archive, and redirect it to the fifo(s), as a background process:
tar -O -xf (filename.tar.gz) file1.txt > file1-fifo.txt &
if [ $FILECOUNT = 2 ]; then
tar -O -xf (filename.tar.gz) file2.txt > file2-fifo.txt &
fi
And then (4) is just:
SECONDFILE=""
if [ $FILECOUNT = 2 ]; then
SECONDFILE=file2-fifo.txt
fi
./main 1000 file1-fifo.txt $SECONDFILE
Finally, you should delete the fifo nodes:
rm file1-fifo.txt
rm file2-fifo.txt
Note that this will involve extracting the archive contents twice (in parallel), once for each file. There's no way (that I can think of) of getting around this.
I would like to know how many files are in any given folder/directory/branch of a Perforce depot, but I don't see a way to do this. The p4 fstat command was my first thought, but it doesn't appear to have options to return file counts. Is there a simple way to get a count of files in a folder using either the graphical or command-line client?
While the p4 fstat doesn't offer a way to obtain file counts per se, you can easily parse its output to obtain this information. Note, this works in Windows, but I would imagine it is easily modified for other OSes. Here's how you do it:
p4 fstat -T depotFile //depot/some/folder/... | find /c "... depotFile"
It can also be done with the p4 files command as thusly:
p4 files //depot/some/folder/... | find /c "//"
Use the p4 sizes -s command.
C:\test>p4 sizes -s ...
... 5 files 342 bytes
or if you want just the count:
C:\test>p4 -F %fileCount% sizes -s ...
5
(obviously the file argument can be anything else in place of ... which is the current directory)
Try this command:
p4 files //depot/dir1/dir2/... | wc -l
Explanation:
p4 files //depot/dir1/dir2/... is the command that recursively prints the files under dir2 to screen.
Pipe that through wordcount lines (| wc -l) to count the number of output lines.
This works in Linux/Unix systems, and in Windows if you're using cygwin (or some other Linux-like terminal).
Powershell method:
To count the files in a specific directory:
p4 files //depot/dir1/dir2/... | Measure-Object
Alternatively:
(p4 files //depot/dir1/dir2/...).Count
The following will give you a list of all child directories and their file counts in an object that you can view as a table, sort, export to csv, etc:
p4 dirs //depot/dir1/* | Select-Object -Property #{Name="Path";Expression={$PSItem}},#{Name="FileCounts";Expression={(p4 files $PSItem/...).Count}}
Currently I am in this directory-
/data/real/test
When I do ls -lt at the command prompt. I get like below something-
REALTIME_235000.dat.gz
REALTIME_234800.dat.gz
REALTIME_234600.dat.gz
REALTIME_234400.dat.gz
REALTIME_234200.dat.gz
How can I consolidate the above five dat.gz files into one dat.gz file in Unix without any data loss. I am new to Unix and I am not sure on this. Can anyone help me on this?
Update:-
I am not sure which is the best way whether I should unzip each of the five file then combine into one? Or
combine all those five dat.gz into one dat.gz?
If it's OK to concatenate files content in random order, then following command will do the trick:
zcat REALTIME*.dat.gz | gzip > out.dat.gz
Update
This should solve order problem:
zcat $(ls -t REALTIME*.dat.gz) | gzip > out.dat.gz
What do you want to happen when you gunzip the result? If you want the five files to reappear, then you need to use something other than the gzip (.gz) format. You would need to either use tar (.tar.gz) or zip (.zip).
If you want the result of the gunzip to be the concatenation of the gunzip of the original files, then you can simply cat (not zcat or gzcat) the files together. gunzip will then decompress them to a single file.
cat [files in whatever order you like] > combined.gz
Then:
gunzip combined.gz
will produce an output that is the concatenation of the gunzip of the original files.
The suggestion to decompress them all and then recompress them as one stream is completely unnecessary.
There is an iso file with 5.5 GB in size. I want to split that with tar command each part should be 500M. What is the command for that?
It's not the job of tar to split files. Use split to split the tar file.
You can use split:
tar [your params] |split -b 500m - output_prefix
...Or use zip (or any other compressor):
tar [your params] | zip -s 500m output.zip -