I'm trying to write a perl one liner for listing the content of a directory like with the "ls -l" Unix command however I only want to output directories and not files
I have managed to do it using a perl script but I want to try and see if it can be reduced to a single line using a pipe like
ls -l | grep "something"
or
ls -l | perl -...
the way I have done it so far is this
#!/usr/bin/perl
open(LS_D, "ls -l |");
while(<LS_D>) {
print if /^d.*/; #print line if line starts with d
}
Also could you tell me why this works with the pipe in "ls -l |" but not with "ls -l"?
Thanks
The long listing of directories can be directly obtained with
ls -ld */
without a need to filter the output.
As for the |, that is how you tell open to open a process and replace its STDIN or STDOUT
For three or more arguments if MODE is |-, the filename is interpreted as a command to which output is to be piped, and if MODE is -|, the filename is interpreted as a command that pipes output to us. In the two-argument (and one-argument) form, one should replace dash (-) with the command. See Using open() for IPC in perlipc for more examples of this.
Without |, so with open(LS_D, "ls -l"), you are trying to open a file, with the name 'ls -l'
The three-argument open, which is recommended with its lexical filehandle, would be
open my $read_fh, '-|', 'ls', '-l';
where the command with arguments is supplied as a list.
Finally, what you have in your Perl script can be done with a one-liner for example as
perl -we'print grep { /^d/ } `ls -l`'
but, as shown above, you can get that directly with ls on the command line.
I should add
Always have use warnings; and use strict; at the top of your programs
Always check calls like open and you'll know about, and see the reason for, the failure
open my $fh ... or die "Can't open ... : $!";
See Error Variables in perlvar for $!
Related
If I execute the following command:
> read someVariable _ < <(echo "54 41")
and:
> echo $someVariable
The result is: 54.
What does < < (with spaces) do?
Why is _ giving the first word from the result in the "echo" command?
The commands above are just example.
Thanks a lot
Process Substitution
As tldp.org explains,
Process substitution feeds the output of a process (or processes) into
the stdin of another process.
So in effect this is similar to piping stdout of one command to the other , e.g. echo foobar barfoo | wc . But notice: in the [bash manpage][3] you will see that it is denoted as <(list). So basically you can redirect output of multiple (!) commands.
Note: technically when you say < < you aren't referring to one thing, but two redirection with single < and process redirection of output from <( . . .).
Now what happens if we do just process substitution?
$ echo <(echo bar)
/dev/fd/63
As you can see, the shell creates temporary file descriptor /dev/fd/63 where the output goes. That means < redirects that file descriptor as input into a command.
So very simple example would be to make process substitution of output from two echo commands into wc:
$ wc < <(echo bar;echo foo)
2 2 8
So here we make shell create a file descriptor for all the output that happens in the parenthesis and redirect that as input to wc .As expected, wc receives that stream from two echo commands, which by itself would output two lines, each having a word, and appropriately we have 2 words, 2 lines, and 6 characters plus two newlines counted.
Side Note: Process substitution may be referred to as a bashism (a command or structure usable in advanced shells like bash, but not specified by POSIX), but it was implemented in ksh before bash's existence as ksh man page. Shells like tcsh and mksh however do not have process substitution. So how could we go around redirecting output of multiple commands into another command without process substitution? Grouping plus piping!
$ (echo foo;echo bar) | wc
2 2 8
Effectively this is the same as above example, However, this is different under the hood from process substitution, since we make stdout of the whole subshell and stdin of wc [linked with the pipe][5]. On the other hand, process substitution makes a command read a temporary file descriptor.
So if we can do grouping with piping, why do we need process substitution? Because sometimes we cannot use piping. Consider the example below - comparing outputs of two commands with diff (which needs two files, and in this case we are giving it two file descriptors)
diff <(ls /bin) <(ls /usr/bin)
I have a complex command I am passing via ssh to a remote server. I am trying to unzip a file and then change its naming structure and extension in a second ssh command. The command I have is:
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print $1$3".log"}'"
Obviously the " around the .log portion of the print statement are failing me. The idea is that I would strip the .out portion from the filename and end up with file20171119.log as an ending result. I am just a bit confused on the syntax or on how to escape that properly so bash interprets the .log appropriately.
The easiest way to deal with this problem is to avoid it. Don't bother trying to escape your script to go on a command line: Pass it on stdin instead.
ssh root#server1 bash -s <<'EOF'
gzip -d /tmp/file.out-20171119.gz
# note that (particularly w/o a cd /tmp) this doesn't do anything at all related to the
# line above; thus, probably buggy as given in the original question.
echo file* | awk -F'[.-]' '{print $1$3".log"}'
EOF
A quoted heredoc -- one with <<'EOF' or <<\EOF instead of <<EOF -- is passed literally, without any shell expansions; thus, $1 or $3 will not be replaced by the calling shell as they would with an unquoted heredoc.
If you don't want to go the avoidance route, you can have the shell do the quoting for you itself. For example:
external_function() {
gzip -d /tmp/file.out-20171119.gz
echo file* | awk -F'[.-]' '{print $1$3".log"}'
}
ssh root#server1 "$(declare -f external_function); external_function"
declare -f prints a definition of a function. Putting that function literally into your SSH command ensures that it's run remotely.
You need to escape the " to prevent them from closing your quoted string early, and you need to escape the $ in the awk script to prevent local parameter expansion.
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
The most probable reason (as you don't show the contents of the root home directory in the server) is that you are uncompressing the file in the /tmp directory, but feeding to awk filenames that should exist in the root home directory.
" allows escaping sequences with \. so the correct way to do is
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
(like you wrote in your question) this means the following command is executed with a shell in the server machine.
gzip -d /tmp/file.out-20171119.gz; echo file* | awk - F'[.-]' '{print $1$3".log"}'
You are executing two commands, the first to gunzip /tmp/file.out-2017119.gz (beware, as it will be gunzipped in /tmp). And the second can be the source for the problem. It is echoing all the files in the local directory (this is, the root user home directory, probably /root in the server) that begin with file in the name (probably none), and feeding that to the next awk command.
As a general rule.... test your command locally, and when it works locally, just escape all special characters that will go unescaped, after being parsed by the first shell.
another way to solve the problem is to use gzip(1) as a filter... so you can decide the name of the output file
ssh root#server1 "gzip -d </tmp/file.out-20171119.gz >file20171119.log"
this way you save an awk(1) execution just to format the output file. Or if you have the date from an environment variable.
DATE=`date +%Y%m%d`
ssh root#server1 "gzip -d </tmp/file.out-${DATE}.gz >file${DATE}.log"
Finally, let me give some advice: Don't use /tmp to uncompress files. /tmp is used by several distributions as a high speed temporary dir. It is normally ram based, too quick, but limited space, so uncompressing a log file there can fill up the memory of the kernel used for the ram based filesystem, which is not a good idea. Also, a log file normally expands a lot and /tmp is a local system general directory, where other users can store files named file<something> and you can clash with those files (in case you do searches with wildcard patterns, like you do in your command) Also, it is common once you know the name of the file to assign it to environment variables and use those variables, so case you need to change the format of the filename, you do it in only one place.
I searched online but I didn't find anything that could answer my question.
I'm using a java tool in Ubuntu Linux, calling it with bash command; this tool has two paths for two different input files:
java -Xmx8G -jar picard.jar FastqToSam \
FASTQ=6484_snippet_1.fastq \ #first read file of pair
FASTQ2=6484_snippet_2.fastq \ #second read file of pair
[...]
What I'd like to do is for example, instead of specify the path of a single FASTQ, specify the path of two different files.
So instead of having cat file1 file2 > File and using File as input of FASTQ, I'd like that this operation would be executed on the fly and create the File on the fly, without saving it on the file system (that would be what happens with the command cat file1 file2 > File).
I hope that I've been clear in explaining my question, in case just ask me and I'll try to explain better.
Most well-written shell commands which accept a file name argument also usually accept a list of file name arguments. Like cat file or cat file1 file2 etc.
If the program you are trying to use doesn't support this, and cannot easily be fixed, perhaps your OS or shell makes /dev/stdin available as a pseudo-file.
cat file1 file2 | java -mumble -crash -burn FASTQ=/dev/stdin
Some shells also have process substitutions, which (typically) look to the calling program like a single file containing whatever the process substitution produces on standard output.
java -mumble -crash -burn FASTQ=<(cat file1 file2) FASTQ2=<(cat file3 file4)
If neither of these work, a simple shell script which uses temporary files and deletes them when it's done is a tried and true solution.
#!/bin/sh
: ${4?Need four file name arguments, will process them pairwise}
t=$(mktemp -d -t fastqtwoness.XXXXXXX) || exit
trap 'rm -rf $t' EXIT HUP INT TERM # remove in case of failure or when done
cat "$1" "$2" >$t/1.fastq
cat "$3" "$4" >$t/2.fastq
exec java -mumble -crash -burn FASTQ=$t/1.fastq FASTQ2=$t/2.fastq
I want to make a simple dmenu command that reads a file of commands and names. Then takes the names and displays them using dmenu then takes dmenu's output and runs the associated command using the file again.
I got to the point where dmenu displays the names, but I don't really know where to go from there. Learning bash is a really daunting task to me and I don't really know where to start with this seemingly simple script/command.
here is the file:
Pushbullet
google-chrome-stable --app=https://www.pushbullet.com
Steam
steam
Chrome
google-chrome-stable
Libre Office
libreoffice
Transmission
transmission-qt
Audio Control Panel
sudo pavucontrol & bluberry
and here is what I have so far for my command:
awk 'NR % 2 != 0' /home/rocco/programlist | dmenu | ??(grep -l "stdout" /home/rocco/programlist....)
It was my thinking that I could somehow pipe into grep or awk with the name of the application then get the line number then add one and pipe that into sh.
Thanks
I have no experience with dmenu but if I understand how it works correctly, this should do what you want. Wrapping a command in $(…) returns the output as a variable, which we can pass on to another command.
#!/bin/bash
plist="/home/rocco/programlist"
# pipe every second line to dmenu
selected=$(awk 'NR % 2 != 0' "$plist" | dmenu)
# search for the selected item, get the command after it
cmd=$(grep -A1 "$selected" "$plist" | tail -n 1)
# run the command
$cmd
Worth mentioning a mistake in your question. dmenu sends to stdout, or standard output, but the next program in line would be reading stdin, or standard input. In any case, grep can't take patterns on standard input, which is why I've saved to a variable instead of trying to pipe it somewhere.
Assuming you have programlist.txt in the working directory you can use:
awk 'NR%2 !=0' programlist.txt |dmenu |awk '{system("grep --no-group-separator -A 1 '"'"'"$0"'"'"' programlist.txt");}' |awk '{if(NR==2){system($0);}}'
Note the quoting of the $0 in the first awk envocation. This is necessary to get names with spaces in them like "Libre Office"
I searched the Internet, but maybe I used the wrong keyword, but I couldn't find the syntax to my very simple problem below:
How do I redirect a file as command line arguments to the Linux command "touch"? I want to create a file with "touch abc.txt", but the filename should come from the filename "file.txt" which contains "abc.txt", not manually typed-in.
[root#machine ~]# touch < file.txt
touch: missing file operand
Try `touch --help' for more information.
[root#machine ~]# cat file.txt
abc.txt
Try
$ touch $(< file.txt)
to expand the content of file.txt and give it as argument to touch
Alternatively, if you have multiple filenames stored in a file, you could use xargs, e.g.,
xargs touch <file.txt
(It would work for just one, but is more flexible than a simple "echo").