It is desirable to have PBSPro std output files that are easily understood while making use of job arrays. I have not found a way.
Here are a set of jobs for discussion -
Job id Name User Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
651902.srvname pl_0000 xxxxxxxxx 00:00:00 R large
651903[].srvname dp_0000-0001 xxxxxxxxx 0 H large
651904.srvname bp_0100 xxxxxxxxx 0 H large
651905[].srvname dp_0000-bpx6 xxxxxxxxx 0 H large
Running qsub without changing the output name results in files for each of the subjobs for job arrays. For example, the following files are produced for job 651905[]:
651905[1].srvname.OU
651905[2].srvname.OU
...
651905[x].srvname.OU
Using qsub -o [JOBNAME] which is known when launching, results in a single file for the job array id so that the std output is only available for one of the subjobs.
The desired output file set is:
dp_0000-bpx6[1].OU
dp_0000-bpx6[2].OU
...
dp_0000-bpx6[x].OU
How can this be accomplished? In other words, how can the output file name be set to something more understandable while preserving the array index?
A secondary question is how can I include the sequence number along with the job name? Something like -
dp_0000-bpx6.651905[1].OU
dp_0000-bpx6.651905[2].OU
...
dp_0000-bpx6.651905[x].OU
This won't get you fully there, but close.
qsub -J "0-512:512" -N pl_0000 -o pl_0000.^array_index^ -- /usr/bin/echo "HI"
Produces this output. You can see each array index with their own output.
-rw------- 1 pbsdev pbsdev 3 Apr 29 21:00 pl_0000.0
-rw------- 1 pbsdev pbsdev 3 Apr 29 21:00 pl_0000.512
-rw------- 1 pbsdev pbsdev 0 Apr 29 21:00 pl_0000.e1441.0
-rw------- 1 pbsdev pbsdev 0 Apr 29 21:00 pl_0000.e1441.512
This one will take the jobid and put it in the output filename along with the index.
jobid=$(qsub -h -J "0-512:512" -N pl_0000 -- /usr/bin/echo "HI") && qalter -o pl_0000.${jobid}.^array_index^.OU ${jobid} && qrls ${jobid}; echo ${jobid}
1446[].pdw-s1
Produces this output. You can see each jobid and array index with their own output.
drwxrwxr-x 2 root pbsdev 69632 Apr 29 21:05 .
drwxrwxr-x 6 root pbsdev 20480 Apr 29 20:49 ..
-rw------- 1 pbsdev pbsdev 3 Apr 29 21:05 pl_0000.1446[].pdw-s1.0.OU
-rw------- 1 pbsdev pbsdev 3 Apr 29 21:05 pl_0000.1446[].pdw-s1.512.OU
-rw------- 1 pbsdev pbsdev 0 Apr 29 21:05 pl_0000.e1446.0
-rw------- 1 pbsdev pbsdev 0 Apr 29 21:05 pl_0000.e1446.512
JOBID v v Array Index
pl_0000.1446[].pdw-s1.512.OU
You can add a -e and change the error output in the qalter. Note, the -h is needed to hold the job, so then qalter has time to modify it, then qrls releases it.
Related
Below files are present in current directory.
-rw-r--r-- 1 kazama kazama 0 Feb 16 08:50 london_july_2001_001.jpeg
-rw-r--r-- 1 kazama kazama 0 Feb 16 08:50 london_march_2004_002.png
-rw-r--r-- 1 kazama kazama 0 Feb 16 08:50 newyork_dec_2012_005.jpeg
-rw-r--r-- 1 kazama kazama 0 Feb 16 08:50 paris_sept_2007_003.jpg
I want to filter all images except which starts with "paris" text.
I have tried below command which works as per my expectation.
ls -l !(paris*)
But, I do not understand why below 2 commands do not give me expected output. In both of the cases it shows all the 4 files.
ls -l !(paris)*.*(jpeg|jpg|png)
ls -l !(paris)*
How bash interprets this extended globbing syntax internally ?Is it first trying to match !(paris) then it tries to match (jpeg|jpg|png) ?
!(paris) matches anything but paris, which includes paris_, pari, par, pa, p, and even the empty string. Bash will find something that doesn't match paris and try to match the rest of the pathname against the rest of the pattern. See:
$ echo !(paris)london*
london_july_2001_001.jpeg london_march_2004_002.png
$ echo !(paris)_*
london_july_2001_001.jpeg london_march_2004_002.png newyork_dec_2012_005.jpeg paris_sept_2007_003.jpg
$ echo !(paris)_*_*_*
london_july_2001_001.jpeg london_march_2004_002.png newyork_dec_2012_005.jpeg
I have some 1080x1920 png files and I want to crop them into 1080x1728 with offset-y 65, so I tried and it returns this:
gimp -i -b '(gimp-image-crop "*.png" 1080 1728 0 65)' -b '(gimp-quit 0)'
$ ll
total 1796
drwxrwxr-x 2 cat cat 4096 Jul 25 15:06 ./
drwxrwxr-x 4 cat cat 4096 Jul 25 14:37 ../
-rwxrwxr-x 1 cat cat 278356 Jul 9 10:56 1.png*
-rwxrwxr-x 1 cat cat 278356 Jul 9 10:56 Screenshot_20210709-105437.png*
-rwxrwxr-x 1 cat cat 58088 Jul 9 10:56 Screenshot_20210709-105445.png*
-rwxrwxr-x 1 cat cat 108385 Jul 9 14:54 Screenshot_20210709-145331.png*
-rwxrwxr-x 1 cat cat 130486 Jul 9 22:18 Screenshot_20210709-221631.png*
-rwxrwxr-x 1 cat cat 133602 Jul 9 22:20 Screenshot_20210709-221834.png*
-rwxrwxr-x 1 cat cat 153976 Jul 9 22:22 Screenshot_20210709-222035.png*
-rwxrwxr-x 1 cat cat 149365 Jul 9 22:23 Screenshot_20210709-222039.png*
-rwxrwxr-x 1 cat cat 124369 Jul 11 17:01 Screenshot_20210711-165959.png*
-rwxrwxr-x 1 cat cat 138719 Jul 24 23:06 Screenshot_20210724-230338.png*
-rwxrwxr-x 1 cat cat 88272 Jul 24 23:06 Screenshot_20210724-230346.png*
-rwxrwxr-x 1 cat cat 104779 Jul 24 23:06 Screenshot_20210724-230356.png*
-rwxrwxr-x 1 cat cat 63867 Jul 24 23:05 Screenshot_20210724-230408.png*
$ gimp -i -b '(gimp-image-crop "*.png" 1080 1728 0 65)' -b '(gimp-quit 0)'
batch command experienced an execution error:
Error: Invalid type for argument 1 to gimp-image-crop
$
It always returns Invalid type for argument 1 to gimp-image-crop.
I check in Procedure Browser in GIMP and it tells:
The argument 1 is "THE IMAGE". I also tried "1.png" and ".+png" but it always returns the same error: Error: Invalid type for argument 1 to gimp-image-crop
I don't understand which part I do wrong.
My OS is Linux Mint 20.2 Cinnamon and GIMP version is 2.10.18.
Please help.
The image argument of gimp-image-crop is the handle of a loaded image in Gimp, not an image file. So you have to gimp-file-load the image (this gives you a handle), apply your operations, and then gimp-file-save. Note that saving an image to a "flat" format (JPG, PNG...) is actually saving a layer, so if your image is more than one layer you may have to first create a layer to save using gimp-layer-new-from-visible.
This said, agreed that in most cases it is faster to use ImageMagick.
I have a bunch of MP4 files that look like this:
-rw-rw-r-- 1 116M Apr 19 06:08 lULIqx9Akn4.mp4
These are youtube videos. When I try to do anything with all of them, I get a weird error. Every command I try says that I'm using invalid options (that I am not using). Here are some examples.
$ ls *.mp4
/bin/ls: invalid option -- '7'
Try '/bin/ls --help' for more information.
$ mv *.mp4 videos/
mv: invalid option -- 'L'
Try 'mv --help' for more information.
$ cp *.mp4 videos/.
cp: invalid option -- '7'
Try 'cp --help' for more information.
It doesn't do the same thing with a different extension (*.mp3, *.txt, *.sh).
What's going on? How do I fix this?
I used this as a cheap workaround,
find . -name "*.mp4" -exec mv {} videos/. \;
but I want to understand what's happening, not just get the job done.
One of your filenames starts with a hyphen, e.g,. -7 or -L. Try ls -- *.mp4 or cp -- *.mp4 videos. Also, allow me to suggest UNIX and Linux Stack Exchange for shell questions :) .
Solution:
Either move the files,
mv -- *.mp4 ./videos
or rename the files in situ...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Explanation:
My sense is you have a file with a leading - in the directly... most commands stop you creating such files but if you copy them from another operating system it can occur. Thus, you need to rename any files with a leading - in their filename...
Let me explain with an example...
Let's try to create a file with a leading -:
touch "-7ULIqx9Akn4.mp4"
touch: illegal option -- 7
we can get around this as follows:
>touch -- "-7ULIqx9Akn4.mp4"
> ls -al -- -*.mp4
total 0
-rw-r--r--# 1 n staff 0 Apr 29 13:02 -7ULIqx9Akn4.mp4
ok, now lets set up an example and demonstrate a solution...
> ls -la
total 0
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -75438752.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:49 -85438750.mp4
drwxr-xr-x# 7 n staff 238 Apr 29 12:49 .
drwxr-xr-x# 6 n staff 204 Apr 29 11:18 ..
-rw-r--r--# 1 n staff 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n staff 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-rw-r--# 1 n staff 0 Apr 29 11:19 lULIqx9Akn4.mp4
next:
ls -- -*.mp4
-75438752.mp4 -85438750.mp4
ok, lets now rename these files...
A little explanation here, the following command uses mv to remove the leading character. i.e. Find files with a leading - and remove the leading character {$file:1} from the filename...
for file in -*.mp4; do mv -- "$file" "${file:1}"; done
Result:
> for file in -*.mp4; do mv -- "$file" "${file:1}"; done
> ll
total 0
drwxr-xr-x# 7 n 238 Apr 29 12:52 ./
drwxr-xr-x# 6 n 204 Apr 29 11:18 ../
-rw-r--r--# 1 n 0 Apr 29 12:36 75438750.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 75438752.mp4
-rw-r--r--# 1 n 0 Apr 29 12:33 7ULIqx9Akn4.mp4
-rw-r--r--# 1 n 0 Apr 29 12:49 85438750.mp4
-rw-rw-r--# 1 n 0 Apr 29 11:19 lULIqx9Akn4.mp4
Note
The above does not account for duplicate file names...
Here is my scenario:
well i use csh
1)
$ ls -l /corefiles/ | grep "root"
-rw-r----- 1 root root 0 Sep 22 2014 core.3.4.
-rwxr-x--- 1 root root 92 Sep 22 2014 ss.sh
2)
$ set textInfo=`ls -l /corefiles/ | grep "root"`
$ echo $textInfo
-rw-r----- 1 root root 0 Sep 22 2014 core.3.4. -rwxr-x--- 1 root root 92 Sep 22 2014 ss.sh
But I need echo $textInfo to give output like 1).
How can I achieve this? I do not want to redirect the content into a file.
I need to store console output in a variable but with the same format as present in the console.
I need a variable which has content as below:
$ echo $textInfo
-rw-r----- 1 root root 0 Sep 22 2014 core.3.4.
-rwxr-x--- 1 root root 92 Sep 22 2014 ss.sh
Use echo "$textInfo" instead of echo $textInfo. Otherwise the variable is expanded as part of the command line instead of as a string, so the newlines aren't preserved.
try this:
textInfo=$(ls -l /corefiles/ | grep "root")
then
echo "$textInfo"
I am new in shell script.Will you please suggest how to write backup shell script. I am having following formated data in target directory.
StoreID_date_time.zip
Like:
-rw------- 1 rupesh ldapusers 8267310 Mar 22 12:00 44_22032014_115629.zip
-rw------- 1 rupesh ldapusers 8269938 Mar 22 12:07 44_22032014_120013.zip
-rw------- 1 rupesh ldapusers 8267110 Mar 22 12:14 44_22032014_120704.zip
-rw------- 1 rupesh ldapusers 8254223 Mar 22 14:25 45_22032014_142155.zip
-rw------- 1 rupesh ldapusers 7871060 Mar 22 12:11 48_22032014_120813.zip
-rw------- 1 rupesh ldapusers 8314418 Mar 22 12:22 48_22032014_121038.zip
-rw------- 1 rupesh ldapusers 8254699 Mar 24 12:13 49_22032014_145338.zip
Now I want to backup files with following way:
Backup directory : /backup/date/storeid/zip files of that store
like:
/backup/22032014/44/44_22032014_115629.zip,44_22032014_120013.zip...so on
/backup/22032014/45/45_22032014_142155.zip
/backup/22032014/48/48_22032014_120813.zip,48_22032014_121038.zip
/backup/22032014/49/49_22032014_145338.zip
for next day /backup/23032014/respective_storeIDfolder&files
Please give some hint or code example so I can move foreword.
I have coded in bare minimum steps without doing a real check but verified it. It works fine with some dummy files I created on my box :)
#!/bin/bash
for i in $(find * -type f -iname '*.zip' )
do
echo "Zip file : "$i
store_id=$(echo $i | cut -d "_" -f 1 );
timestamp=$(echo $i | cut -d "_" -f 2 );
echo Store id = ${store_id}
# I am assuming all these directories here will be of teh same pattern name. Else put a numeric check down.
mkdir -p /backup/${timestamp}/${store_id}
cp -f $i /backup/${timestamp}/${store_id}/
done;