How to separate output for each command generated by a bash file? - linux

Let's say we have a bash script like the one bellow:
echo test
ls -alh
pwd
echo test2
So the file can have any number of commands on it each producing or not its own output.
Then the above file is run like this /bin/bash -xe test.sh which will produce the following output:
+ echo test
test
+ ls -alh
total 32
drwx------+ 6 daniels staff 204B Apr 3 23:33 .
drwxr-xr-x+ 64 daniels staff 2.1K Apr 4 01:53 ..
-rw-r--r--# 1 daniels staff 6.0K Apr 3 23:33 .DS_Store
drwxr-xr-x 5 daniels staff 170B Mar 15 17:03 Todo
-rw-r--r--# 1 daniels staff 282B Apr 3 20:39 test.py
-rw-r--r--# 1 daniels staff 97B Apr 4 01:52 test.sh
+ pwd
/Users/daniels/Desktop
+ echo test2
test2
Is there any way to parse the generated output reliable and figure out how to separate the output based on each command?
For the above example we should be able to separate and extract one group with:
+ echo test
test
another one with
+ ls -alh
total 32
drwx------+ 6 daniels staff 204B Apr 3 23:33 .
drwxr-xr-x+ 64 daniels staff 2.1K Apr 4 01:53 ..
-rw-r--r--# 1 daniels staff 6.0K Apr 3 23:33 .DS_Store
drwxr-xr-x 5 daniels staff 170B Mar 15 17:03 Todo
-rw-r--r--# 1 daniels staff 282B Apr 3 20:39 test.py
-rw-r--r--# 1 daniels staff 97B Apr 4 01:52 test.sh
etc.
I was thinking of parsing the output and look if the line starts with + then take that as the start of one command but then you can easily have something like echo + ok which will make this logic fail.
Another option would've been if we can modify the char that is outputted by /bin/bash -x so that instead of + to output something like https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text but looks like + is hardcoded in bash and not configurable.
Any ideas?

+ is not hard-coded, and this is readily described in man bash and under help set for -x:
-x After expanding each simple command, for command,
case command, select command, or arithmetic for
command, display the expanded value of PS4, fol‐
lowed by the command and its expanded arguments or
associated word list.
And here's a further description of PS4, also from man bash:
PS4 The value of this parameter is expanded as with PS1 and
the value is printed before each command bash displays
during an execution trace. The first character of PS4 is
replicated multiple times, as necessary, to indicate mul‐
tiple levels of indirection. The default is ``+ ''.
Here's an example:
$ PS4=$'\nAnd now, a word from '; set -x; date; uptime
And now, a word from date
Mon Apr 3 16:20:35 PDT 2017
And now, a word from uptime
16:20:35 up 65 days, 1:24, 6 users, load average: 1.20, 1.42, 1.37
You can use this to embed special markers or characters as you see fit.

Related

no such file or directory: mount_smbfs {mount point} {local directory}

How can I get mount_smbfs to work on a Mac zsh terminal from within a loop?
The identical commands work when run one by one, but they throw an error when issued from within a loop: no such file or directory: mount_smbfs {mount point} {local directory}.
So, when I source something like this (the export commands actually live in their own script that gets sourced first):
export H_DIR="/mnt/${USER}"
export I_DIR="/mnt/I"
export DUM_HOST="dummy.host.ninja"
declare -A mounts
mounts[${H_DIR}]="//${USER}#${DUM_HOST}/${USER}"
mounts[${I_DIR}]="//${DUM_HOST}/I"
for dir mount_point in ${(kv)mounts}; do
if [ ! -d $dir ]; then
echo "Making $dir and intermediates."
mkdir -p $dir
#else
#command="diskutil unmount $dir"
#echo "command: $command"
#$command
fi
command="mount_smbfs $mount_point $dir"
echo "command: $command"
$command
ls -alt $dir
done
I get something like this:
command: mount_smbfs //me#dummy.host.ninja/me /mnt/me
/Users/me/.zsh_setup/mount_all.zsh:24: no such file or directory: mount_smbfs ///me#dummy.host.ninja/me /mnt/me
total 0
drwxr-xr-x 3 me staff 96 Dec 20 14:30 ..
drwxr-xr-x 2 me staff 64 Dec 20 14:30 .
command: mount_smbfs //dummy.host.ninja/I /mnt/I
/Users/me/.zsh_setup/mount_all.zsh:24: no such file or directory: mount_smbfs ///dummy.host.ninja/I /mnt/I
total 0
drwxr-xr-x 3 me staff 96 Dec 20 14:30 ..
drwxr-xr-x 2 me staff 64 Dec 20 14:30 .
But, I can just paste the same command that was run in the loop, and it works.
me#my_mac ~ % mount_smbfs //me#dummy.host.ninja/me /mnt/me
Password for dummy.host.ninja:
me#my_mac ~ %
me#my_mac ~ % ls -alt /mnt/me
total 1499808
drwxr-xr-x 3 me staff 96 Dec 20 14:30 ..
drwx------ 1 me staff 16384 Dec 20 14:02 .
-rwx------ 1 me staff 10621 Dec 20 14:02 .bash_history
... A bunch more folders that live on the mounted host ...
me#my_mac ~ %
me#my_mac ~ %
me#my_mac ~ % mount_smbfs //dummy.host.ninja/I /mnt/I
me#my_mac ~ %
me#my_mac ~ % ls -alt /mnt/I
total 960
drwx------ 1 me staff 16384 Dec 20 18:33 .snapshot
drwxr-xr-x 7 me staff 224 Dec 20 14:30 ..
drwx------ 1 me staff 16384 Sep 21 11:22 .
... A bunch more folders that live on the mounted host ...
me#my_mac~ %
For context:
me#my_mac ~ % ls -l /
total 9
drwxrwxr-x 21 root admin 672 Dec 17 11:03 Applications
drwxr-xr-x 67 root wheel 2144 Dec 16 10:03 Library
drwxr-xr-x# 9 root wheel 288 Oct 12 23:06 System
drwxr-xr-x 6 root admin 192 Dec 16 09:45 Users
drwxr-xr-x 3 root wheel 96 Dec 20 12:38 Volumes
drwxr-xr-x# 38 root wheel 1216 Oct 12 23:06 bin
drwxr-xr-x 2 root wheel 64 Oct 12 23:06 cores
dr-xr-xr-x 3 root wheel 4507 Dec 20 12:38 dev
lrwxr-xr-x# 1 root wheel 11 Oct 12 23:06 etc -> private/etc
lrwxr-xr-x 1 root wheel 25 Dec 20 12:38 home -> /System/Volumes/Data/home
lrwxr-xr-x 1 root wheel 26 Dec 20 12:38 mnt -> Users/me/mounts
... and so on ...
me#my_mac ~ %
me#my_mac ~ % ls -l /Users/me/mounts
total 32
drwx------ 1 me staff 16384 Sep 21 11:22 I
drwxr-xr-x 3 me staff 96 Dec 20 14:30 me
me#my_mac ~ %
I need to enter my password once for the first mount, but not again because I use ssh keys. If I mount one manually to open the ssh key then run the for loop with other locations, that still won't work, so I don't think that's the issue.
As #GordonDavisson mentioned in the comments, the issue isn't with mount_smbfs but with how the command is parsed when assigned to a string variable. The best solution is to not do this, as seen here: Assigning command to a variable in Zsh. There are some suggestions for how to use a string variable as your actual command call, but I opted for just not doing it.
Making this minimum change works:
command="mount_smbfs $mount_point $dir"
echo "command: $command"
# Replace ...
# $command
# ... with the actual command:
mount_smbfs $mount_point $dir
Assigning the full command to a string was just for the convenience of echoing the command to the console before calling it, for logging and debugging. You can still do this, but it just won't be so tightly associated with the command itself, which isn't a real problem in this context where the command string variable is only used once right before calling the actual command. There are some suggestions in the linked post for how to get around this if you really want to use a string variable as your actual command call.
Also as mentioned in the comments and the linked post, it is perhaps best to wrap the command in a function. I tend to disagree in cases like this where it is a one-line command that is essentially function call itself. I'm not doing anything but calling it, no extra validation or anything. Wrapping it just adds an unnecessary layer of convolution. However, I haven't sought out a full explanation for why it might still be best practice.
That said, I will re-wrap the full script (minus the exports) back up into a set of functions as I originally did before troubleshooting and posting here. That's a good case for a function encapsulating a command or set of commands.

How to use wget with multithread and wildcards?

I want to use wget to download multiple files at once in a script using wildcards, like this:
wget -r -nd --no-parent --no-remove-listing $ftpUrl -l1 -A file1*.txt &
wget -r -nd --no-parent --no-remove-listing $ftpUrl -l1 -A file2*.txt &
wget -r -nd --no-parent --no-remove-listing $ftpUrl -l1 -A file3*.txt &
The problem is that wget downloads .listing file everytime and because there are multiple instances running, sometimes the file is being downloaded when another instance is reading it.
Is there a way to lock .listing file or ask wget to not download it (I can do it manually as the first command)? I don't understand how it reads this .listing file since it's not a plain list of the URLs, but rather something like this:
drwxr-xr-x 3 4015 4015 16384 Dec 14 21:23 .
drwxr-xr-x 4 4015 4015 4096 Dec 14 21:23 ..
-rw-r--r-- 1 4015 4015 327 Feb 15 2022 file1-bla.txt
-rw-r--r-- 1 4015 4015 10716 Feb 15 2022 file2-bla.txt
-rw-r--r-- 1 4015 4015 163 Feb 15 2022 file2-bla.txt
If I try to use -i .listing (or even if I rename .listing to list.txt and use -i list.txt) I get an error saying that the URLs are not valid or something.
RFC 959 stipulates that
LIST (LIST) This command causes a list to be sent from the server to the passive DTP. If the pathname specifies a directory or other
group of files, the server should transfer a list of files in the
specified directory. If the pathname specifies a file then the server
should send current information on the file. A null argument implies
the user's current working or default directory. The data transfer is
over the data connection in type ASCII or type EBCDIC. (The user must
ensure that the TYPE is appropriately ASCII or EBCDIC). Since the
information on a file may vary widely from system to system, this
information may be hard to use automatically in a program, but may be
quite useful to a human user.
Observe that it does not impose formal requirement what exactly will be returned.
drwxr-xr-x 3 4015 4015 16384 Dec 14 21:23 .
drwxr-xr-x 4 4015 4015 4096 Dec 14 21:23 ..
-rw-r--r-- 1 4015 4015 327 Feb 15 2022 file1-bla.txt
-rw-r--r-- 1 4015 4015 10716 Feb 15 2022 file2-bla.txt
-rw-r--r-- 1 4015 4015 163 Feb 15 2022 file2-bla.txt
This looks akin to output of ls -lah command, if you are interested in using that to form URL list acceptable by wget then you should take last column (excluding . and ..) and prefix them with URL of FTP server, if that URL is ftp://ftp.example.com and you got list as above then your list of URLs should look as follows
ftp://ftp.example.com/file1-bla.txt
ftp://ftp.example.com/file2-bla.txt
ftp://ftp.example.com/file3-bla.txt

java -xvf test.jar not working

i tried to extract the test.jar
command is executing successfully but no output.
user#host:home/test->ll
drwxr-xr-x 107 user abc 6040 Apr 4 09:55 ..
drwxr-xr-x 2 user abc 26 Apr 4 10:06 .
-rw-r--r-- 1 user abc 51241 Apr 4 10:06 test.jar
user#host:home/test->jar -xvf test.jar
user#host:home/test->ll
total 262
drwxr-xr-x 107 user abc 6040 Apr 4 09:55 ..
drwxr-xr-x 2 user abc 26 Apr 4 10:06 .
-rw-r--r-- 1 user abc 51241 Apr 4 10:06 test.jar
Kindly help me to resolve this
Actual Requirement:
Need to extract and access a resource in jar file.
According to Oracle Java toutorials:
https://docs.oracle.com/javase/tutorial/deployment/jar/unpack.html
You should unpack it with:
jar xfv test.jar
Without '-' sign.
x option indicates that you want to extract files from the JAR archive.
f options indicates that the JAR file from which files are to be extracted is specified on the command line, rather than through stdin.
v is verbose
try to this command also unzip test.jar and below commands help you view some file without extract all the files.
unzip -q -c test.jar META-INF/MANIFEST.MF

ls to list matches in specified directory

How do I list matched results in a specified directory?
On my Ubuntu server if I list the contents of a directory it correctly lists it. My working directory is /var/crash.
#pwd
/var/crash
# ls -l
-rw-r--r-- 1 bob bob 121876 Aug 8 2015 results.xml
-rw-rw-r-- 1 bob bob 126 Nov 3 2015 start.txt
-rw-rw-r-- 1 bob bob 43 Jul 28 2015 exit.txt
Let's say I want to list all files that contain 'tar'. In this example there should only be one match i.e. start.txt
# ls -l *tar*
-rw-rw-r-- 1 bob bob 126 Nov 3 2015 start.txt
All's good so far. However if I include the directory (/var/crash) it lists all files.
# ls -l *tar* /var/crash
-rw-r--r-- 1 bob bob 121876 Aug 8 2015 results.xml
-rw-rw-r-- 1 bob bob 126 Nov 3 2015 start.txt
-rw-rw-r-- 1 bob bob 43 Jul 28 2015 exit.txt
I'm guessing my syntax is telling ls to list all matches of tar AND everything in /var/crash. What is the correct syntax to list matches in a specified directory?
You need to specify the pattern together with the directory:
ls -l /var/crash/*tar*
Otherwise, with ls -l *tar* /var/crash you are telling ls to act against two parameters: /var/crash and *tar*. In fact, *tar* will be expanded before ls reaches it, so there might be more parameters for ls.

Linux: Finding Newly Added Files

I am trying to obtain a backup of 'newly' added files to a Fedora system. Files can be copied through a Windows Samba share and appear to retain the original created timestamp. However, because it retains this timestamp I am having issues identifying which files were newly added to the system.
Currently, the only way I can think of doing this is to have a master list snapshot of all the files on the system at a specific time. Then when I perform the backup I compare the previous snapshot with a current snapshot. It would detect files that were removed from the system but it seems excessive and I was thinking there must be an easier way to backup newly added files.
Terry
Try using find. Something like this:
find . -ctime -10
That will give you a list of files and directories, starting from within your current directory, that has had its state changed within the last 10 days.
Example:
My Downloads directory looks like this:
kobus#akira:~/Downloads$ ll
total 2025284
drwxr-xr-x 4 kobus kobus 4096 Nov 4 11:25 ./
drwxr-xr-x 41 kobus kobus 4096 Oct 30 09:26 ../
-rw-rw-r-- 1 kobus kobus 8042383 Oct 28 14:08 apache-maven-3.3.3- bin.tar.gz
drwxrwxr-x 2 kobus kobus 4096 Oct 14 09:55 ELKImages/
-rw-rw-r-- 1 kobus kobus 1469054976 Nov 4 11:25 Fedora-Live-Workstation-x86_64-23-10.iso
-rw------- 1 kobus kobus 351004 Sep 21 14:07 GrokConstructor-master.zip
drwxrwxr-x 11 kobus kobus 4096 Jul 11 2014 jboss-eap-6.3/
-rw-rw-r-- 1 kobus kobus 183399393 Oct 19 16:26 jboss-eap-6.3.0-installer.jar
-rw-rw-r-- 1 kobus kobus 158177216 Oct 19 16:26 jboss-eap-6.3.0.zip
-rw-rw-r-- 1 kobus kobus 71680110 Oct 13 13:51 jre-8u60-linux-x64.tar.gz
-rw-r--r-- 1 kobus kobus 4680 Oct 12 12:34 nginx-release-centos-7-0.el7.ngx.noarch.rpm
-rw-r--r-- 1 kobus kobus 3479765 Oct 12 14:22 ngx_openresty-1.9.3.1.tar.gz
-rw------- 1 kobus kobus 16874455 Sep 15 16:49 Oracle_VM_VirtualBox_Extension_Pack-5.0.4-102546.vbox-extpack
-rw-r--r-- 1 kobus kobus 7505310 Oct 6 10:29 sublime_text_3_build_3083_x64.tar.bz2
-rw------- 1 kobus kobus 41467245 Sep 7 10:37 tagspaces-1.12.0-linux64.tar.gz
-rw-rw-r-- 1 kobus kobus 42658300 Nov 4 10:14 tagspaces-2.0.1-linux64.tar.gz
-rw------- 1 kobus kobus 70046668 Sep 15 16:49 VirtualBox-5.0-5.0.4_102546_el7-1.x86_64.rpm
Here's what the find returns:
kobus#akira:~/Downloads$ find . -ctime -10
.
./tagspaces-2.0.1-linux64.tar.gz
./apache-maven-3.3.3-bin.tar.gz
./Fedora-Live-Workstation-x86_64-23-10.iso
kobus#akira:~/Downloads$
Most unices do not have a concept of file creation time. You can't make ls print it because the information is not recorded. If you need creation time, use a version control system: define creation time as the check-in time.
If your unix variant has a creation time, look at its documentation. For example, on Mac OS X (the only example I know of¹), use ls -tU. Windows also stores a creation time, but it's not always exposed to ports of unix utilities, for example Cygwin ls doesn't have an option to show it. The stat utility can show the creation time, called “birth time” in GNU utilities, so under Cygwin you can show files sorted by birth time with stat -c '%W %n' * | sort -k1n.
Note that the ctime (ls -lc) is not the file creation time, it's the inode change time. The inode change time is updated whenever anything about the file changes (contents or metadata) except that the ctime isn't updated when the file is merely read (even if the atime is updated). In particular, the ctime is always more recent than the mtime (file content modification time) unless the mtime has been explicitly set to a date in the future.
"Newly added files, Fedora" : The below examples will show a list with date and time.
Example, all installed packages : $ rpm -qa --last
Example, the latest 100 packages : $ rpm -qa --last | head -100
Example, create a text file : $ rpm -qa --last | head -100 >> last-100-packages.txt

Resources