I borrowed the SGE system, but it doesn't work when I qsub.
#!/bin/bash
#PBS -l nodes=1:ppn=4
#PBS -N pix2pix
#PBS -o pix2pix.out
#PBS -e pix2pix.err
#PBS -q GPU_HIGH
#PBS -l walltime=72000:00:00
#cd $PBS_0_WORKDIR
cd /public/home/chensu/others/Guicai/pix2pix-tensorflow
source activate tensorflow
python /public/home/chensu/others/Guicai/pix2pix-tensorflow/pix2pix.py --mode train --output_dir /public/home/chensu/others/Guicai/pix2pix-tensorflow/face2face-model --max_epochs 2000 --input_dir /public/home/chensu/others/Guicai/pix2pix-tensorflow/photos/combined/train --which_direction AtoB --batch_size 4
there are many free queues.
qstat -q
server: admin1
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- --- --- -- -----
FAT_HIGH -- -- -- -- 0 0 0 E R
high -- -- -- -- 10 0 10 E R
MIC_HIGH -- -- -- -- 0 0 0 E R
GPU_HIGH -- -- -- -- 0 0 0 E R
low -- -- -- -- 4 0 20 E R
batch -- -- -- -- 0 0 20 E R
middle -- -- -- -- 0 0 10 E R
----- -----
14 0
when I qsub my test.pbs with qsub test.pbs
MGMT1:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
20414.MGMT1 chensu GPU_HIGH pix2pix -- 1 4 -- 72000:00: C --
Also there are no log, so I don't know what happened.
Any suggestions will be appreciated
Related
with the commands
$>squeue -u mnyber004
I can visualize all the submitted jobs on my cluster account (slurm)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
16884 ada CPUeq6 mnyber00 R 1-01:26:17 1 srvcnthpc105
16882 ada CPUeq4 mnyber00 R 1-01:26:20 1 srvcnthpc104
16878 ada CPUeq2 mnyber00 R 1-01:26:31 1 srvcnthpc104
20126 ada CPUeq1 mnyber00 R 22:32:28 1 srvcnthpc103
22004 curie WRI_0015 mnyber00 R 16:11 1 srvcnthpc603
22002 curie WRI_0014 mnyber00 R 16:13 1 srvcnthpc603
22000 curie WRI_0013 mnyber00 R 16:14 1 srvcnthpc603
How to cancel all the jobs running on the partition ada?
In your case, scancel offers the appropriate filters, so you can simply run
scancel -u mnyber004 -p ada
Should it not have been the case, a frequent idiom is to use the more powerful filtering properties of squeue and the --format option to build the proper command and then feed it to sh:
squeue -u mnyber004 -p ada --format "scancel %i" | sh
You can play it safer by first saving to a file and then sourcing the file.
squeue -u mnyber004 -p ada --format "scancel %j" > /tmp/remove.sh
source remove.sh
On the command line, I'm able to get this rolling with no problem:
java weka.Run weka.classifiers.timeseries.WekaForecaster -W
"weka.classifiers.functions.MultilayerPerceptron -L 0.01 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 20 " -t "C:\MyFile.arff" -F DirectionNumeric -L 1 -M 3 -prime 3 -horizon 6 -holdout 100 -G TradeDay -dayofweek -weekend -future
But once I try to put the skip list, I start to get errors saying that it's missing a date that is not in the skip list even though the date is in fact on it:
java weka.Run weka.classifiers.timeseries.WekaForecaster -W "weka.classifiers.functions.MultilayerPerceptron -L 0.01 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 20 " -t "C:\MyFile.arff" -F DirectionNumeric -L 1 -M 3 -prime 3 -horizon 6 -holdout 100 -G TradeDay -dayofweek -weekend -future -skip ""2014-06-07#yyyy-MM-dd, 2014-06-12"
Does anybody knows how to get this working? Weka is low on documentation as far as I know.
Thank's in advance!
Forget it. I got it, the problem was the 's' must be in capital letters:
-Skip
instead of
-skip.
I am working with SVM at Weka
I have some data and I try SVM (I tried different values of C) technique to analize the data. But the output has totally confused me, that is why I hope some help.
This is the output for a polynomial kernel of degree 1:
Scheme:weka.classifiers.meta.CVParameterSelection -P "C 0.001 10.0 5.0" -X 10 -S 1 -W weka.classifiers.functions.SMO -- -C 0.7 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 1.0"
Classifier Options: -C 7.5003 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 1.0"
=== Summary ===
Correctly Classified Instances 83 79.0476 %
Incorrectly Classified Instances 22 20.9524 %
Kappa statistic 0.6555
Mean absolute error 0.0487
Root mean squared error 0.1549
Relative absolute error 91.5633 %
Root relative squared error 100.2828 %
Total Number of Instances 105
This is the output for a polynomial kernel of degree 2:
Scheme:weka.classifiers.meta.CVParameterSelection -P "C 0.001 10.0 5.0" -X 10 -S 1 -W weka.classifiers.functions.SMO -- -C 0.7 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 2.0"
Classifier Options: -C 2.5008 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 2.0"
=== Summary ===
Correctly Classified Instances 87 82.8571 %
Incorrectly Classified Instances 18 17.1429 %
Kappa statistic 0.7236
Mean absolute error 0.0486
Root mean squared error 0.1547
Relative absolute error 91.4748 %
Root relative squared error 100.1845 %
Total Number of Instances 105
This is the output for a gaussian kernel and gamma value of 1.0:
Scheme:weka.classifiers.meta.CVParameterSelection -P "C 0.001 10.0 5.0" -X 10 -S 1 -W weka.classifiers.functions.SMO -- -C 0.7 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.RBFKernel -C 250007 -G 1.0"
Classifier Options: -C 2.5008 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.RBFKernel -C 250007 -G 1.0"
=== Summary ===
Correctly Classified Instances 87 82.8571 %
Incorrectly Classified Instances 18 17.1429 %
Kappa statistic 0.721
Mean absolute error 0.0486
Root mean squared error 0.1547
Relative absolute error 91.4571 %
Root relative squared error 100.1651 %
Total Number of Instances 105
These are my questions:
Why the classifiers at “Classifier Options: -Cxxxxx” have different values of C? I think it is related to the values of C that I am checking with “C 0.001 10.0 5.0”, but I may be wrong and I may be getting wrong conclusions. I think that with “C 0.001 10.0 5.0” it tries the values 0.001, 0.01, 0.1, 1.0 and 10.0. If that is true why the values of C are: 7.5003 and 2.5008??
The percentage of the Correctly Classified Instances is very similar in all of the different runs, I don’t understand why… What does it mean? If it were 12% I would think SVM was not a proper technique (it wasn't linearly separable) but with 80% (no or little variations after changing the parameters) I don’t know what to think…
If I check the data in the “Visualize” of Weka the data doesn’t seem linearly separable but due to the conversion of a bigger number of dimensions when I use SVM I don’t think it is possible to get conclusions from the visualize.
1) Yes you use CVParameterSelection which trys different parameters in your case from 0.001 to 10 in 5 steps. The steps would be 0.001 + k * (10-0.001)/4) because your first step is allready defined as 0.001. If you round the following values (for k= 0/1/2/3/4) you see that they fit.
Step 1) 0.001
Step 2) 2.50075
Step 3) 5.005
Step 4) 7.50025
Step 5) 10.0
2) If you would have 12% accuracy there would be something really strange. Random classification (assuming you have balanced data) would result in 50%. I cannot tell you if 82% is a good result because i don't know about your data.
3) To your question about linar separability. Thats exactly why the SVM is so good. SVM with the right Kernel transforms your data into a higher featurespace to get rid of the non-separability. An RBF kernel can transform your data even into an infinite-dimensional feature space. Thats why a perfect linar separation is allways possible. At this point overfitting can occur. To avoid that and reach a good generalisation you have your complexity parameter C.
Here you can read more in a good stackexchange post about the last point
https://stats.stackexchange.com/questions/80398/how-can-svm-find-an-infinite-feature-space-where-linear-separation-is-always-p
What's a difference between obj-m=md1.o md2.o and obj-m+=md1.o md2.o in a kernel make-file?
You can try to infer it with an example:
obj-m= a.o b.o
obj-m1= 1 2 3
obj-m= c.o d.o
obj-m1+= 4 5 6
ALL :
echo $(obj-m) -- $(obj-m1)
Its output is
c.o d.o -- 1 2 3 4 5 6
So: = assigns (overwriting), and += appends.
Is there a way for ps (or similar tool) to display the pthread's name?
I wrote the following simple program:
// th_name.c
#include <stdio.h>
#include <pthread.h>
void * f1() {
printf("f1 : Starting sleep\n");
sleep(30);
printf("f1 : Done sleep\n");
}
int main() {
pthread_t f1_thread;
pthread_create(&f1_thread, NULL, f1, NULL);
pthread_setname_np(f1_thread, "f1_thread");
printf("Main : Starting sleep\n");
sleep(40);
printf("Main : Done sleep\n");
return 0;
}
Is there a command/utility (like ps) that I can use to display the threads for the above program, along with their name.
$ /tmp/th_name > /dev/null &
[3] 2055
$ ps -eLf | egrep "th_name|UID"
UID PID PPID LWP C NLWP STIME TTY TIME CMD
aal 31088 29342 31088 0 2 10:01 pts/4 00:00:00 /tmp/th_name
aal 31088 29342 31089 0 2 10:01 pts/4 00:00:00 /tmp/th_name
aal 31095 29342 31095 0 1 10:01 pts/4 00:00:00 egrep th_name|UID
I am running my program on Ubuntu 12.10.
With procps-ng (https://gitlab.com/procps-ng/procps) there are output option -L and -T which will print threads names:
$ ps -eL
$ ps -eT
-l long format may be used with them:
$ ps -eLl
$ ps -eTl
but -f option will replace thread name with full command line which is the same for all threads.
note the man page of pthread_setname_np(),which have showed how to get the threads' names:
pthread_setname_np() internally writes to the thread specific comm
file under /proc filesystem: /proc/self/task/[tid]/comm.
pthread_getname_np() retrieves it from the same location.
and
Example
The program below demonstrates the use of pthread_setname_np() and
pthread_getname_np().
The following shell session shows a sample run of the program:
$ ./a.out
Created a thread. Default name is: a.out
The thread name after setting it is THREADFOO.
^Z #Suspend the program
1+ Stopped ./a.out
$ ps H -C a.out -o 'pid tid cmd comm'
PID TID CMD COMMAND
5990 5990 ./a.out a.out
5990 5991 ./a.out THREADFOO
$ cat /proc/5990/task/5990/comm
a.out
$ cat /proc/5990/task/5991/comm
THREADFOO
Show the thread IDs and names of the process with PID 12345:
ps H -o 'tid comm' 12345