I was working my way through a primer on Shell (Bash) Scripting and had the following doubt :
Why does not the following command print the contents of cp's directory : which cp | ls -l
Does not piping by definition mean that we pass the output of one command to another i.e. redirect the output ?
Can someone help me out ? I am a newbie ..
The output of which is being piped to the standard input of ls. However, ls doesn't take anything on standard input. You want it (I presume) to be passed as a parameter. There are a couple of ways of doing that:
which cp | xargs ls -l
or
ls -l `which cp`
or
ls -l $(which cp)
In the first example the xargs command takes the standard output of the previous previous command and makes each line a parameter to the command whose name immediately follows xargs. So, for instance
find / | xargs ls -l
will do an ls -l on each file in the filesystem (there are some issues with this with peculiarly named files but that's beyond the scope of this answer).
The remaining two are broadly equivalent and use the shell to do this, expanding the output from which into the command line for cp.
It would be,
$ ls -l $(which cp)
-rwxr-xr-x 1 root root 130304 Mar 24 2014 /bin/cp
OR
$ which cp | xargs ls -l
-rwxr-xr-x 1 root root 130304 Mar 24 2014 /bin/cp
To pass the output of one command as parameter of another command, you need to use xargs along with the pipe symbol.
From man xargs
xargs - build and execute command lines from standard input.xargs reads items
from the standard input, delimited by blanks (which can be protected
with double or single quotes or a backslash) or newlines, and executes
the command (default is /bin/echo) one or more times with any initial-
arguments followed by items read from standard input. Blank lines on
the standard input are ignored.
Related
In Linux I use InfluxDB which can make a backup of the database for archival purposes. Each backup comprises a series of files with the same prefix "/tank/Backups/var/Influxdb/20191225T235655Z." and different extensions.
I wanted to write a bash script which first deletes the oldest existing backups, then creates a new one (here I paste only the removal):
ls -tp /tank/Backups/var/Influxdb/* | grep -v '/$' | sed -E 's/\..+//' | \
sort -ru | sed 's/$/.*/' | tail -n +4 | xargs -d '\n' -r rm --
However, when I run the script as "sudo", I get
rm: cannot remove '/tank/Backups/var/Influxdb/20191225T235655Z.*': No such file or directory
When I run the quoted script, except the latest part, I get:
/tank/Backups/var/Influxdb/20190930T215357Z.*
/tank/Backups/var/Influxdb/20190930T215352Z.*
which is correct. Also, if I manually write
sudo /tank/Backups/var/Influxdb/20190930T215357Z.*
the command succeeds.
Why is the script reporting an error?
I'm using Ubuntu 18.04 and the folder "/tank" is a ZFS volume.
Better do :
find /tank/Backups/var/Influxdb/* -mtime +5 -delete
to remove files older than 5 days.
Then, you can run the next command
Explaining the Error
This answer is only here to explain the error and give a deeper understanding of what is happening. If you are simply looking for an elegant solution search for other answers.
When I run the quoted script, except the latest part, I get:
/tank/Backups/var/Influxdb/20190930T215357Z.*
/tank/Backups/var/Influxdb/20190930T215352Z.*
which is correct
The listed strings are not what you want. When you pass these paths to rm it sees them just as literal strings, that is, two files whose names end with a literal *. Since you don't have such files you get an error.
When you type rm * manually into your console bash (not rm!) does globbing. bash searches files and replaces the * with the list of found files. Only after that bash executes rm foundFile1 foundFile2 .... rm never sees the *.
Strings inside a pipeline are not processed by bash, but by the commands in the pipeline, in your case rm. rm does not glob.
You could run bash inside your pipeline and let it expand the * you inserted earlier. To this end, replace the last command in your pipeline with xargs -r bash -c 'rm -- $*' --. However, note that your paths are not quoted here. If there are spaces or literal * in your filenames the command will break. This is necessary for globbing as quoted "*" are not expanded by bash.
To quote your files you have to insert the * glob inside the bash command:
ls -tp /tank/Backups/var/Influxdb/* | grep -v '/$' | sed -E 's/\..+//' |
sort -ru | tail -n +4 | xargs -d\\n -L1 -r bash -c 'rm -- "$0."*'
Above command is only a simple fix for your command. It is neither elegant nor very robust. Using tools like find is strongly recommended.
This question already has answers here:
Unix pipe into ls
(3 answers)
Closed 6 years ago.
I am a newbie to linux . I have been learning about cat command when i tried this .
harish#harish-Lenovo-G50-45:~$ cat file
Videos/Arrow/Season*
harish#harish-Lenovo-G50-45:~$ cat file | ls -l
The command displays the content of the current folder instead of the folder mentioned in the file .. But when i did this
harish#harish-Lenovo-G50-45:~$ cat file
Videos/Arrow/Season*
harish#harish-Lenovo-G50-45:~$ ls -l $(cat file)
The contents of the expected folder displays correctly . Why cant i not use the pipeline in this case ?
In the first example you are passing the output of cat file to the input of ls -l. Since ls -l does not take any input, it does not do anything regarding the output of cat file. However in the second example you are using $(cat file) which puts the output of cat file in the place of an argument passed to ls -l, and this time ls -l has the text inside file in the right place for doing something with it. The issue here is noticing the difference between the standard input of a program and the arguments of a program. Standard input is what you have when you call scanf in C, for example; and the arguments are what you get in the argv pointer passed as parameter to the main procedure.
I am trying to write a bash script to automatically change the desktop background by selecting from the images in a given directory.The problem I am facing is that when I try to use the "ls" command to list the images, it also shows its attributes.Is there a way that I could only list the file names and none of its other attributes.
Here, if if TEST_DIR= /home/vivek/Downloads
then
ls $TEST_DIR/*.jpeg -o $TEST_DIR/*.png -o $TEST_DIR/*.jpg
gives output as
-rw-rw-r-- 1 vivek 7221 Sep 5 20:42 /home/vivek/Downloads/viv.jpeg
I just need the output as "/home/vivek/Downloads/viv.jpeg".
EDIT:
The script works just fine after removing the "-o" from the command.
Earlier I was using "-o" to denote an "or" keyword.
But now,
ls $TEST_DIR/*.jpeg $TEST_DIR/*.png $TEST_DIR/*.jpg
gives me exactly the output I needed. Thanks.
You're passing -o to ls (twice, even). Remove that, because it means:
-o like -l, but do not list group information
You can also shorten it to:
ls $TEST_DIR/*.{jpeg,png,jpg}
Your ls is probably aliased to something. See what this prints:
$ alias ls
It should work fine from a script, because aliases are ignored there by default.
Try this, using cut command as follows;
ls $TEST_DIR/*.{jpeg,png,jpg}|cut -d':' -f2|cut -d' ' -f2
It will exclude unwanted text using : and space to find result.
It will give you this output:
/home/vivek/Downloads/viv.jpeg
i have the next code:
files=$(lftp -u mgtwrk35,Unix11! sftp://illin634<<EOF
cd some_dir
ls | tail -1
EOF)
why does the code above work,
but if i add to the ls function -tr like:
ls -tr | tail -1
it doesn't work and gives me the next message:
ls: invalid option -- t
ls: invalid option -- r
Thanks in advance for the answer
The ls in lftp is not the real thing. It's also named "ls" but does not support all the parameters you can find in the standard ls.
find lets you recursively list files.
cls --sort=date lets you sort by modification date.
I don't know how to combine the two. There's a work-around mentioned by Nicolas Noble, which is to use awk to post-process the result of find.
I'm using ls -a command to get the file names in a directory, but the output is in a single line.
Like this:
. .. .bash_history .ssh updater_error_log.txt
I need a built-in alternative to get filenames, each on a new line, like this:
.
..
.bash_history
.ssh
updater_error_log.txt
Use the -1 option (note this is a "one" digit, not a lowercase letter "L"), like this:
ls -1a
First, though, make sure your ls supports -1. GNU coreutils (installed on standard Linux systems) and Solaris do; but if in doubt, use man ls or ls --help or check the documentation. E.g.:
$ man ls
...
-1 list one file per line. Avoid '\n' with -q or -b
Yes, you can easily make ls output one filename per line:
ls -a | cat
Explanation: The command ls senses if the output is to a terminal or to a file or pipe and adjusts accordingly.
So, if you pipe ls -a to python it should work without any special measures.
Ls is designed for human consumption, and you should not parse its output.
In shell scripts, there are a few cases where parsing the output of ls does work is the simplest way of achieving the desired effect. Since ls might mangle non-ASCII and control characters in file names, these cases are a subset of those that do not require obtaining a file name from ls.
In python, there is absolutely no reason to invoke ls. Python has all of ls's functionality built-in. Use os.listdir to list the contents of a directory and os.stat or os to obtain file metadata. Other functions in the os modules are likely to be relevant to your problem as well.
If you're accessing remote files over ssh, a reasonably robust way of listing file names is through sftp:
echo ls -1 | sftp remote-site:dir
This prints one file name per line, and unlike the ls utility, sftp does not mangle nonprintable characters. You will still not be able to reliably list directories where a file name contains a newline, but that's rarely done (remember this as a potential security issue, not a usability issue).
In python (beware that shell metacharacters must be escapes in remote_dir):
command_line = "echo ls -1 | sftp " + remote_site + ":" + remote_dir
remote_files = os.popen(command_line).read().split("\n")
For more complex interactions, look up sftp's batch mode in the documentation.
On some systems (Linux, Mac OS X, perhaps some other unices, but definitely not Windows), a different approach is to mount a remote filesystem through ssh with sshfs, and then work locally.
you can use ls -1
ls -l will also do the work
You can also use ls -w1
This allows to set number of columns.
From manpage of ls:
-w, --width=COLS
set output width to COLS. 0 means no limit
ls | tr "" "\n"
Easy, as long as your filenames don't include newlines:
find . -maxdepth 1
If you're piping this into another command, you should probably prefer to separate your filenames by null bytes, rather than newlines, since null bytes cannot occur in a filename (but newlines may):
find . -maxdepth 1 -print0
Printing that on a terminal will probably display as one line, because null bytes are not normally printed. Some programs may need a specific option to handle null-delimited input, such as sort's -z. Your own script similarly would need to account for this.
-1 switch is the obvious way of doing it but just to mention, another option is using echo and a command substitution within a double quote which retains the white-spaces(here \n):
echo "$(ls)"
Also how ls command behaves is mentioned here:
If standard output is a terminal, the output is in columns (sorted
vertically) and control characters are output as question marks;
otherwise, the output is listed one per line and control characters
are output as-is.
Now you see why redirecting or piping outputs one per line.