I have some large files that I need to concatenate into one giant file to put through a software package that does not accept stdin. I would rather not duplicate the content of existing files on the hard drive if necessary, and am looking for a shortcut that basically does cat files*.txt silently when opened.
You can use process substitution to make the output of a command appear to be a file.
some_command <(cat files*.txt)
But if the application reads from standard input, you can just pipe it:
cat files*.txt | some_command
Another solution I just discovered, using named pipes...
mkfifo files.star.txt
chmod 666 files.star.txt
cat files*.txt > files.star.txt &
some_command files.star.txt
Related
In a particular directory, I made a file named "fileName" and add contents to it. When I typed cat fileName, it's content are printed on the terminal. Now I used the following command:
cat fileName>fileName
No error was shown. Now when I try to see contents of file using,
cat fileName
nothing was shown in the terminal and file is empty (when I checked it). What is the reason for this?
> i.e. redirection to the same file will create/truncate the file before cat command is invoked as it has a higher precedence. You could avoid the same by using intermediate file and then from intermediate to actual file or you could use tee like:
cat fileName | tee fileName
To clarify on SMA's answer, the file is truncated because redirection is handled by the shell, which opens the file for writing before invoking the command. when you run cat file > file,the shell truncates and opens the file for writing, sets stdout to the file, and then execute ["cat", "file"]. So you will have to use some other command for the task like tee
The answers given here are wrong. You will have a problem with truncating regardless of using the redirect or pipeline, although it may APPEAR to work sometimes, depending on size of file or length of your pipeline. It is a race condition, as the reader may have a chance to read some or all of the file before the writer starts, but the point of the pipeline is to run all these at the same time so they will be starting at the same time and the first thing tee executable will do is open the output file (and truncate it in the process). The only way you will not have a problem in this scenario is if the end of the pipeline would load the entirety of the output into memory and only write it to file on shutdown. It is unlikely to happen and defeats the point of having a pipeline.
Proper solution for making this reliable is to just write to a temp file and then rename the temp file back to original filename:
TMP="$(mktemp fileName.XXXXXXXX)"
cat fileName | grep something | tee "${TMP}"
mv "${TMP}" fileName
Assume that there are three files in a directory of my Ubuntu-Linux. One of them is open in a music player, but two others are not open in any application and are not in use of any Process.
I want to delete those two files. In the other words, I want to detect files that are not used by any process, and then delete them.
I tried this:
rm !(lsof | grep "thePath")
but it failed. There are a problem with lsof here.
Is there any way?
Your syntax is wildly wrong. In the shell, unlike in many other programming languages, putting a command (as if it were a function call) in parentheses does not produce its result.
Instead, you want something like this:
for f in *; do
fuser -s "$f" || rm "$f"
done
I preferred fuser over lsof as it allows you to query an individual file.
I have created this code which will allow user to change the port in a specific file,
#Change Port
IRSSIPORT1=`head -n 1 /etc/ports.txt | tail -n 1`
sudo perl -pi -e "s/^$IRSSIPORT1.*\n$//g" /etc/ports.txt
sudo perl -pi -e "s/web_port = 8081/web_port = $IRSSIPORT1/g" .sickbread/config.ini
echo "sickbread Port: $IRSSIPORT1" | sudo tee -a $HOME/private/SBinfo.txt
What this code do is it takes a number from a file and then put it in the config file where it is required to change and deletes that number from the initial file from where it took it, but it requires read access as well as write access,
I tried everything in my knowledge to get it work without sudo, but i failed to do it.
Any suggestion?
I get this error -
Can't remove /etc/ports.txt: Permission denied, skipping file.
You can't do inplace edit on 666 files inside /etc as -i switch makes new file and deletes old one inside directory.
Since users don't have sufficient permissions to add/delete files from /etc (nor it would be good idea to do so), you have to read all file content at once, change it, and write back to the same file. Using a temporary file is also a workable solution.
While it may seem that the question is more about system administration rather than about programming, it's actually somewhat about perl so it's may be a good place for it here.
Doing chmod 666 /etc/ports.txt grants all users read-write access to this particular file (of course you don't need to put 777 as it's not an executable or script). So anyone will be able to open this file for writing and put any contents in it.
But when you do perl -pi -e ... /etc/ports.txt you don't only write into that file. Instead, perl will want to delete and then recreate this file, as shown here in strace output:
# strace perl -pi -e 's/a/b/' /etc/ports.txt 2>&1 | grep /etc/ports.txt
...
open("/etc/ports.txt", O_RDONLY) = 3
unlink("/etc/ports.txt") = 0
open("/etc/ports.txt", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
To delete the file it will need to have a write access not to the file itself, but to the directory /etc, which of course you cannot give to any user.
So I suppose you just don't need to try using in-place edit as it's always related to removing or renaming files, but instead get the contents of the file, make required changes and then write it back to the same file.
Is there a way to "spoof" the file extension of a file in bash for consumption by another program? I can think of doing some shell scripting and making lots of soft-links, but that isn't very scalable.
Let's imagine I have a program I'm trying to use that requires input files to be of a specific file extension, and it has no method of turning off this check.
You could make a fifo with the requisite extension and cat any other file type into it. So, if your crazy program needs to see files that end in .funky, you can do this:
mkfifo file.funky
cat someotherfile > file.funky &
someprogram file.funky
Create a symbolic link for each file you want to have a particular extension, then pass the name of the symlink to the command.
For example suppose you have files with names of the form *.foo and you need to refer to them with extensions of .bar:
for file in *.foo ; do
ln -s $file _$$_$file.bar
done
I precede each symlink name with _$$_ to avoid the possibility of colliding with an existing file name (you don't want to do ln -s file.foo file.bar if file.bar already exists).
With a little more programming, your script can keep track of which symlinks it created and, if you like, clean them up after executing the command.
This assumes, as you stated in the question, that the command can't be forced to accept a different extension.
You could, without too much difficulty, create a wrapper script that replaces the command in question, creating the symlinks, invoking the command, and cleaning up after itself automatically.
Consider something like:
cat file | command > file
Is this good practice? Could this overwrite the input file as the same time as we are reading it, or is it always read first in memory then piped to second command?
Obviously, I can use temp files as intermediary step, but I'm just wondering..
t=$(mktemp)
cat file | command > ${t} && mv ${t} file
No, it is not ok. All commands in a pipeline execute at the same time, and the shell prepares redirections before executing the commands. So, it is likely that the command will overwrite the file before cat reads it.
You need sponge(1) from moreutils.
You can also use something like this (not recommended, use explicit temp files in production code):
{ rm file && your_command > file; } < file
Not only should you NOT write your output to your input, but also you should avoid looping your output back to your input.
When dealing with big files, I tried
cat *allfastq30 > Sample_All_allfastq30
and it generated error messages:
cat: Sample_All_allfastq30: input file is output file