Mainframe ICETOOL joins two files - mainframe
I have two VB files.
File1 is 406 bytes long, and the key is in position 401 to 406.
File2 is 565 bytes long, and the key is in position 87 to 92.
I would like the result file to have all the 406 bytes from file1 plus 8 bytes from file2 starts from column 42.
I have the following codes:
//COPY01 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN01 DD DSN=CCSY.CCSYAC.BNS.C2,DISP=SHR
//IN02 DD DSN=CCSY.CCSYAC.UID.XBAS,DISP=SHR
//OUT01 DD DSN=CCSY.CCSYAC.BNS.C2.ID,DISP=(,CATLG,DELETE),
// DCB=(RECFM=FB,LRECL=410,BLKSIZE=41000),
// SPACE=(CYL,(200,100),RLSE)
//TOOLIN DD *
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
/*
//BNSCCNTL DD *
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
/*
I got S013. Anyone would like to help? I'll appreciate.
JESMSGLG:
13.02.33 JOB38364 ---- WEDNESDAY, 10 JAN 2018 ----
13.02.33 JOB38364 IRR010I USERID CCSYAC IS ASSIGNED TO THIS JOB.
13.02.34 JOB38364 ICH70001I CCSYAC LAST ACCESS AT 12:56:19 ON WEDNESDAY, JANUARY 10, 2018
13.02.34 JOB38364 $HASP373 CCSYAC02 STARTED - INIT 68 - CLASS J - SYS SY07
13.02.34 JOB38364 IEF403I CCSYAC02 - STARTED - TIME=13.02.34
13.02.35 JOB38364 IEC141I 013-68,IFG0196L,CCSYAC02,COPY01,OUT01,B275,STRP49, 005
005 CCSY.CCSYAC.BNS.C2.ID
13.02.35 JOB38364 IEA995I SYMPTOM DUMP OUTPUT 006
006 SYSTEM COMPLETION CODE=013 REASON CODE=00000068
006 TIME=13.02.35 SEQ=61720 CPU=0000 ASID=0058
006 PSW AT TIME OF ERROR 075C1000 80E70476 ILC 2 INTC 0D
006 NO ACTIVE MODULE FOUND
006 NAME=UNKNOWN
006 DATA AT PSW 00E70470 - 4100302C 0A0D010D A7E5014B
006 AR/GR 0: 008FE990/00E70780 1: 00000000/A4013000
006 2: 00000000/0000DCC8 3: 00000000/00E70754
006 4: 00000000/008C4388 5: 00000000/008C471C
006 6: 00000000/008C46C4 7: 00000000/008C471C
006 8: 00000000/008C46E4 9: 00000000/008D1C50
006 A: 00000000/80E72A62 B: 00000000/00E73622
006 C: 00000000/80E7387E D: 00000000/7F59FCE8
006 E: 00000000/80E6FCD4 F: 00000000/00000068
006 END OF SYMPTOM DUMP
13.02.47 JOB38364 IEF450I CCSYAC02 COPY01 - ABEND=S013 U0000 REASON=00000068 011
011 TIME=13.02.47
13.02.47 JOB38364 IEF404I CCSYAC02 - ENDED - TIME=13.02.47
13.02.47 JOB38364 $HASP395 CCSYAC02 ENDED - ABEND=S013
------ JES2 JOB STATISTICS ------
10 JAN 2018 JOB EXECUTION DATE
26 CARDS READ
563 SYSOUT PRINT RECORDS
0 SYSOUT PUNCH RECORDS
54 SYSOUT SPOOL KBYTES
0.22 MINUTES EXECUTION TIME
TOOLMSG:
ICE600I 0 DFSORT ICETOOL UTILITY RUN STARTED
ICE650I 0 VISIT http://www.ibm.com/storage/dfsort FOR ICETOOL PAPERS,
EXAMPLES AND MORE
ICE632I 0 SOURCE FOR ICETOOL STATEMENTS: TOOLIN
ICE630I 0 MODE IN EFFECT: STOP
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
DFSMSG:
ICE200I 0 IDENTIFIER FROM CALLING PROGRAM IS 0001
ICE411I 0 THIS IS THE JOINKEYS MAIN TASK FOR JOINING F1 AND F2
ICE416I 0 JOINKEYS IS USING THE F1 SUBTASK FOR IN01 - SEE JNF1JMSG MESSAGES
ICE416I 1 JOINKEYS IS USING THE F2 SUBTASK FOR IN02 - SEE JNF2JMSG MESSAGES
ICE419I 0 JOINED RECORDS: TYPE=F, LENGTH=410
ICE201I A RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E9-I40658 C9-I35397 E5-
I38877 E7-I40658
ICE143I 0 BLOCKSET COPY TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
ICE146I 0 END OF STATEMENTS FROM BNSCCNTL - PARAMETER LIST STATEMENTS FOLLOW
DEBUG NOABEND,ESTAE
OPTION MSGDDN=DFSMSG,LIST,MSGPRT=ALL,RESINV=0,SORTDD=BNSC,SORTOUT=OUT01*
,DYNALLOC
SORT FIELDS=COPY
RECORD TYPE=V
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE089I 0 CCSYAC02.COPY01 . , INPUT LRECL = 410, TYPE = F
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1024000,MINLIM=122800,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=DFSMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR OUT01
ICE751I 2 EF-I35397 DA-I40658
ICE805I 0 JOBNAME: CCSYAC02 , STEPNAME: COPY01
ICE802I 0 BLOCKSET TECHNIQUE IN CONTROL
ICE906I 0 ST=ABOVE,SR=6291456,RC=0
ICE907I 0 ST=ABOVE,SA=6291440,NF=1,LF=6291440,SF=6291440
ICE906I 0 ST=BELOW,SR=1036288,RC=0
ICE907I 0 ST=BELOW,SA=1036272,NF=1,LF=1036272,SF=1036272
ICE889I 0 CT=MAX , SB=8, L=0, D=0000, CCW=1MAM
ICE902I 0 O RP10 I
ICE906I 1 ST=ABOVE,SR=6234096,RC=0
ICE907I 1 ST=ABOVE,SA=6234080,NF=1,LF=6234080,SF=6234080
ICE906I 1 ST=BELOW,SR=94680,RC=0
ICE907I 1 ST=BELOW,SA=49608,NF=1,LF=49608,SF=49608
ICE185A 0 AN S013 ABEND WAS ISSUED BY DFSORT, ANOTHER PROGRAM OR AN EXIT (PHASE C 3)
JNF1JMSG:
ICE201I A RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E4-I40658 C9-I35397 E5-
I38877 E6-I31999 C4-I31999 E7-I40658
ICE417I 0 THIS IS THE JOINKEYS F1 SUBTASK FOR IN01
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
SORT FORMAT=BI,FIELDS=(401,6,A)
RECORD TYPE=F
DEBUG NOABEND,ESTAE
OPTION EQUALS,MSGPRT=ALL,LIST,NOCHECK,RESINV=0,DYNALLOC,SORTDD=JNF1,MSG*
DDN=JNF1JMSG,SORTIN=IN01
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE088I 0 CCSYAC02.COPY01 . , INPUT LRECL = 406, BLKSIZE = 27998, TYPE = VB
ICE093I 0 MAIN STORAGE = (MAX,9251025,9251025)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (9193665,9193665)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=9251025,MAXLIM=1024000,MINLIM=122800,EQUALS=Y,LIST=Y,ERET=RC16 ,MSGDDN=JNF1JMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT==N,DYNALOC=(DISK ,004),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=128
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR IN01
ICE750I 0 DC 589973550 TC 0 CS DSVXX KSZ 10 VSZ 10
ICE752I 0 FSZ=589973550 BC IGN=0 E AVG=203 0 WSP=766274 C DYN=520 53216
ICE751I 1 D8-I35397 D4-I35397 EA-I35933 F1-I35933 E8-I40658
ICE091I 0 OUTPUT LRECL = 406, TYPE = V
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 1433042, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 581815052
ICE253I 0 RECORDS SORTED - PROCESSED: 1433042, EXPECTED: 2906273
ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 406, EXPECTED: 203
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 540 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 563M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT
JNF2JMSG:
ICE201I A RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E4-I40658 C9-I35397 E5-
I38877 E6-I31999 E7-I40658
ICE417I 0 THIS IS THE JOINKEYS F2 SUBTASK FOR IN02
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
SORT FORMAT=BI,FIELDS=(87,6,A)
RECORD TYPE=F
DEBUG NOABEND,ESTAE
OPTION EQUALS,MSGPRT=ALL,LIST,NOCHECK,RESINV=0,DYNALLOC,SORTDD=JNF2,MSG*
DDN=JNF2JMSG,SORTIN=IN02
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE088I 0 CCSYAC02.COPY01 . , INPUT LRECL = 565, BLKSIZE = 27998, TYPE = VB
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1024000,MINLIM=122800,EQUALS=Y,LIST=Y,ERET=RC16 ,MSGDDN=JNF2JMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(DISK ,004),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR IN02
ICE750I 0 DC 1063772 TC 0 CS DSVXX KSZ 10 VSZ 10
ICE752I 0 FSZ=1063772 BC IGN=0 E AVG=282 0 WSP=1381 C DYN=0 0
ICE751I 1 D8-I35397 D4-I35397 D1-I35397 E8-I40658
ICE091I 0 OUTPUT LRECL = 565, TYPE = V
ICE080I 0 IN MAIN STORAGE SORT
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 3603, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 1034061
ICE253I 0 RECORDS SORTED - PROCESSED: 3603, EXPECTED: 3772
ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 287, EXPECTED: 282
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 0 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT
Here is the description for the S013 error you encountered:
The SORTIN, SORTOUT, or OUTFIL BLKSIZE parameter:
Was greater than 32760 for a disk data set, or was greater than the maximum block size
supported by the access method for a tape data set.
Was not an integer multiple of the LRECL parameter for fixed-length records, or was not
at least four bytes longer than the LRECL for variable-length
records.
and if you look at the JCL you provided:
//COPY01 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN01 DD DSN=CCSY.CCSYAC.BNS.C2,DISP=SHR
//IN02 DD DSN=CCSY.CCSYAC.UID.XBAS,DISP=SHR
//OUT01 DD DSN=CCSY.CCSYAC.BNS.C2.ID,DISP=(,CATLG,DELETE),
// DCB=(RECFM=FB,LRECL=410,BLKSIZE=41000),
// SPACE=(CYL,(200,100),RLSE)
//TOOLIN DD *
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
/*
//BNSCCNTL DD *
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
/*
You can see that your block size of 41000 is greater than that 32760 maximum. Try not specifying the block size. If you do not specify the block size in the JCL, the system will calculate an efficient one for you.
Related
Is it possible to get current CPU utilisation from a specific core via /sys in Linux?
I would like to write a shellscript that reads the current CPU utilisation on a per-core basis. Is it possible to read this from the /sys directory in Linux (CentOS 8)? I have found /sys/bus/cpu/drivers/processor/cpu0 which does give me a fair bit of information (like current frequency), but I've yet to figure out how to read CPU utilisation. In other words: Is there a file that gives me current utilisation of a specific CPU core in Linux, specifically CentOS 8?
I believe that you should be able to extract information from /proc/stat - the lines that start with cpu$N, where $N is 0, 1, 2, ...... For example: Strongly suggesting reading articles referenced on other answer. cpu0 101840 1 92875 80508446 4038 0 4562 0 0 0 cpu1 81264 0 68829 80842548 4424 0 2902 0 0 0 Repeated call will show larger values: cpu 183357 1 162020 161382289 8463 0 7470 0 0 0 cpu0 102003 1 93061 80523961 4038 0 4565 0 0 0 cpu1 81354 0 68958 80858328 4424 0 2905 0 0 0 Notice CPU0 5th column (idle count) moving from 80508446 to 80523961 Format of each line in cpuN user-time nice-time system-time idle-time io-wait ireq softirq steal guest guest_nice So a basic solution: while true ; for each cpu read current counters, at least user-time system-time and idle usage = current(user-time + system-time) - prev(user-time+system-time) idle = current(idle) - prev(idle) utilization = usage/(usage+idle) // print or whatever. set prev=current done
SGE faild to submit job, attribute is not a memory value
I failed to submit a job with mem attribute. Since I am novice and after google two days, I look for help here. Any suggestion would be grateful! Following is what had I done: \1. Submit my script: qsub -S /bin/bash -A assembly -pe threads 16 -l mem=2GB -cwd -N "pBcR_correct_asm" -j y -o /dev/null runCorrection.sh Unable to run job: unknown resource "mem". Exiting. \2. Considering that I had replace "h" to "host", which solve my problem according to SGE unknown resource "nodes", I replace "m" to "mem”， and it didn't work. \3. After google, I know "h" is the shortcut that was defined in "/opt/gridengine/util/resources/centry/ hostname", and can be confirmed with "qconf -sc": qconf -sc #name shortcut type relop requestable consumable default urgency #---------------------------------------------------------------------------------------- arch a RESTRING == YES NO NONE 0 calendar c RESTRING == YES NO NONE 0 cpu cpu DOUBLE >= YES NO 0 0 display_win_gui dwg BOOL == YES NO 0 0 h_core h_core MEMORY <= YES NO 0 0 h_cpu h_cpu TIME <= YES NO 0:0:0 0 h_data h_data MEMORY <= YES NO 0 0 h_fsize h_fsize MEMORY <= YES NO 0 0 h_rss h_rss MEMORY <= YES NO 0 0 h_rt h_rt TIME <= YES NO 0:0:0 0 h_stack h_stack MEMORY <= YES NO 0 0 h_vmem h_vmem MEMORY <= YES NO 0 0 hostname h HOST == YES NO NONE 0 load_avg la DOUBLE >= NO NO 0 0 load_long ll DOUBLE >= NO NO 0 0 load_medium lm DOUBLE >= NO NO 0 0 load_short ls DOUBLE >= NO NO 0 0 m_core core INT <= YES NO 0 0 m_socket socket INT <= YES NO 0 0 m_topology topo RESTRING == YES NO NONE 0 m_topology_inuse utopo RESTRING == YES NO NONE 0 mem_free mf MEMORY <= YES NO 0 0 mem_total mt MEMORY <= YES NO 0 0 mem_used mu MEMORY >= YES NO 0 0 \4. I thus replaced "mt" to "mem", but it complained attribute problem. According to above output, it seemed the mem_total is almost same as "hostname" that worked previously. Then, I think jsv may be a problem after going through the SGE guide, but I can't find any scripts contain "Unable to run job: attribute......", which under the director of "/opt/gridengine/util/resources/jsv". I think I have to configure some files, but what these files, and what should I do? qsub -S /bin/bash -A assembly -pe threads 16 -l mt=2GB -cwd -N "pBcR_correct_asm" -j y -o test.out runCorrection.sh Unable to run job: attribute "mem_total" is not a memory value. Exiting.
#Vince!
Thanks for your reply very much.
Finally I solve my problem, by using "h_vmem=2g" ("2GB" would give error), but I don't know where to find how to design the value of the complex (MEMORY).
Following information is un-necessary now.
I had read the website you gave, and configured the attribute of h_vmem and s_vmeme in complex to "consumable", but it didn't work. I guess I have to configure "complex_value" of queue which is "NONE" at the moment. However, I can't open the web http://gridscheduler.sourceforge.net/htmlman/htmlman5/sge_types.html?pathrev=V62u5_TAG that may tell me how to configure. Am I right to configure to configure queue? Do I have to configure host too?
Any suggestions would be grateful!
Fowllowing is what had I done:
\1. Change the attribute of consumable to "YES" for h_vmem and s_vmem:
qconf -sc
#name shortcut type relop requestable consumable default urgency
#----------------------------------------------------------------------------------------
arch a RESTRING == YES NO NONE 0
calendar c RESTRING == YES NO NONE 0
cpu cpu DOUBLE >= YES NO 0 0
display_win_gui dwg BOOL == YES NO 0 0
h_core h_core MEMORY <= YES NO 0 0
h_cpu h_cpu TIME <= YES NO 0:0:0 0
h_data h_data MEMORY <= YES NO 0 0
h_fsize h_fsize MEMORY <= YES NO 0 0
h_rss h_rss MEMORY <= YES NO 0 0
h_rt h_rt TIME <= YES NO 0:0:0 0
h_stack h_stack MEMORY <= YES NO 0 0
h_vmem h_vmem MEMORY <= YES YES 0 0
hostname h HOST == YES NO NONE 0
load_avg la DOUBLE >= NO NO 0 0
load_long ll DOUBLE >= NO NO 0 0
load_medium lm DOUBLE >= NO NO 0 0
load_short ls DOUBLE >= NO NO 0 0
m_core core INT <= YES NO 0 0
m_socket socket INT <= YES NO 0 0
m_topology topo RESTRING == YES NO NONE 0
m_topology_inuse utopo RESTRING == YES NO NONE 0
mem_free mf MEMORY <= YES NO 0 0
mem_total mt MEMORY <= YES NO 0 0
mem_used mu MEMORY >= YES NO 0 0
min_cpu_interval mci TIME <= NO NO 0:0:0 0
np_load_avg nla DOUBLE >= NO NO 0 0
np_load_long nll DOUBLE >= NO NO 0 0
np_load_medium nlm DOUBLE >= NO NO 0 0
np_load_short nls DOUBLE >= NO NO 0 0
num_proc p INT == YES NO 0 0
qname q RESTRING == YES NO NONE 0
rerun re BOOL == NO NO 0 0
s_core s_core MEMORY <= YES NO 0 0
s_cpu s_cpu TIME <= YES NO 0:0:0 0
s_data s_data MEMORY <= YES NO 0 0
s_fsize s_fsize MEMORY <= YES NO 0 0
s_rss s_rss MEMORY <= YES NO 0 0
s_rt s_rt TIME <= YES NO 0:0:0 0
s_stack s_stack MEMORY <= YES NO 0 0
s_vmem s_vmem MEMORY <= YES YES 0 0
seq_no seq INT == NO NO 0 0
slots s INT <= YES YES 1 1000
swap_free sf MEMORY <= YES NO 0 0
swap_rate sr MEMORY >= YES NO 0 0
swap_rsvd srsv MEMORY >= YES NO 0 0
swap_total st MEMORY <= YES NO 0 0
swap_used su MEMORY >= YES NO 0 0
tmpdir tmp RESTRING == NO NO NONE 0
virtual_free vf MEMORY <= YES NO 0 0
virtual_total vt MEMORY <= YES NO 0 0
virtual_used vu MEMORY >= YES NO 0 0
# >#< starts a comment but comments are not saved across edits --------
\2. Submit my job to the queue of smp.q, and it complained the same problem:
qsub -S /bin/bash -A assembly -q smp.q -pe newPe 16 -l h_vmem=2GB -cwd -N "pBcR_correct_asm" -j y -o runCorrection.sh
Unable to run job: attribute "h_vmem" is not a memory value.
Exiting.
\3. The information of smp.q. I think "complex_values" should be changed and "h_vmem" can keep unchanged:
qconf -sq smp.q
qname smp.q
hostlist #smp.q
seq_no 0
load_thresholds np_load_avg=1.75
suspend_thresholds NONE
nsuspend 1
suspend_interval 00:05:00
priority 0
min_cpu_interval 00:05:00
processors UNDEFINED
qtype BATCH INTERACTIVE
ckpt_list NONE
pe_list make newPe
rerun FALSE
slots 160
tmpdir /tmp
shell /bin/csh
prolog NONE
epilog NONE
shell_start_mode posix_compliant
starter_method NONE
suspend_method NONE
resume_method NONE
terminate_method NONE
notify 00:00:60
owner_list NONE
user_lists NONE
xuser_lists NONE
subordinate_list NONE
complex_values NONE
projects NONE
xprojects NONE
calendar NONE
initial_state default
s_rt INFINITY
h_rt INFINITY
s_cpu INFINITY
h_cpu INFINITY
s_fsize INFINITY
h_fsize INFINITY
s_data INFINITY
h_data INFINITY
s_stack INFINITY
h_stack INFINITY
s_core INFINITY
h_core INFINITY
s_rss INFINITY
h_rss INFINITY
s_vmem INFINITY
h_vmem INFINITY
\4. The information of hosts in #smp.q:
qconf -sconf smp03.local
#smp03.local:
mailer /bin/mail
xterm /usr/bin/X11/xterm
execd_spool_dir /opt/gridengine/default/spool
\5. The global information. Have I add h_vmem and s_vmem here?
qconf -sconf
#global:
execd_spool_dir /opt/gridengine/default/spool
mailer /bin/mail
xterm /usr/bin/X11/xterm
load_sensor none
prolog none
epilog none
shell_start_mode posix_compliant
login_shells sh,ksh,csh,tcsh
min_uid 0
min_gid 0
user_lists none
xuser_lists none
projects none
xprojects none
enforce_project false
enforce_user auto
load_report_time 00:00:40
max_unheard 00:05:00
reschedule_unknown 00:00:00
loglevel log_warning
administrator_mail none
set_token_cmd none
pag_cmd none
token_extend_time none
shepherd_cmd none
qmaster_params none
execd_params ENABLE_ADDGRP_KILL=TRUE H_MEMORYLOCKED=infinity
reporting_params accounting=true reporting=true \
flush_time=00:00:15 joblog=true sharelog=00:00:00
finished_jobs 100
gid_range 20000-20100
qlogin_command builtin
qlogin_daemon builtin
rlogin_command builtin
rlogin_daemon builtin
rsh_command builtin
rsh_daemon builtin
max_aj_instances 2000
max_aj_tasks 75000
max_u_jobs 0
max_jobs 0
max_advance_reservations 0
auto_user_oticket 0
auto_user_fshare 0
auto_user_default_project none
auto_user_delete_time 86400
delegated_file_staging false
reprioritize 0
jsv_url none
jsv_allowed_mod ac,h,i,e,o,j,M,N,p,w
What you likely want is h_vmem. At least that is the attribute I always use to specify the memory I want request for a job. See: http://gridscheduler.sourceforge.net/htmlman/htmlman5/queue_conf.html?pathrev=V62u5_TAG Specifically, The resource limit parameters s_vmem and h_vmem are imple- mented by Sun Grid Engine as a job limit. They impose a limit on the amount of combined virtual memory consumed by all the processes in the job. If h_vmem is exceeded by a job running in the queue, it is aborted via a SIGKILL signal (see kill(1)). If s_vmem is exceeded, the job is sent a SIGXCPU signal which can be caught by the job. If you wish to allow a job to be "warned" so it can exit gracefully before it is killed then you should set the s_vmem limit to a lower value than h_vmem. For parallel processes, the limit is applied per slot which means that the limit is mul- tiplied by the number of slots being used by the job before being applied. Also, you may need to set this as consumable using qconf.
write error: No space left on device in embedded linux
all I have a embedded board, run linux OS. and I use yaffs2 as rootfs. I run a program on it, but after some times, it got a error "error No space left on device.". but I checked the flash, there still have a lot free space. I just write some config file. the config file is rarely update. the program will write some log to flash. log size is limited to 2M. I don't know why, and how to solve. Help me please!(my first language is not English,sorry. hope you understand what I say) some debug info: # ./write_test version 1.0 close file :: No space left on device return errno 28 # cat /proc/yaffs YAFFS built:Nov 23 2015 16:57:34 Device 0 "rootfs" start_block........... 0 end_block............. 511 total_bytes_per_chunk. 2048 use_nand_ecc.......... 1 no_tags_ecc........... 1 is_yaffs2............. 1 inband_tags........... 0 empty_lost_n_found.... 0 disable_lazy_load..... 0 refresh_period........ 500 n_caches.............. 10 n_reserved_blocks..... 5 always_check_erased... 0 data_bytes_per_chunk.. 2048 chunk_grp_bits........ 0 chunk_grp_size........ 1 n_erased_blocks....... 366 blocks_in_checkpt..... 0 n_tnodes.............. 749 n_obj................. 477 n_free_chunks......... 23579 n_page_writes......... 6092 n_page_reads.......... 11524 n_erasures............ 96 n_gc_copies........... 5490 all_gcs............... 1136 passive_gc_count...... 1136 oldest_dirty_gc_count. 95 n_gc_blocks........... 96 bg_gcs................ 96 n_retired_writes...... 0 n_retired_blocks...... 0 n_ecc_fixed........... 0 n_ecc_unfixed......... 0 n_tags_ecc_fixed...... 0 n_tags_ecc_unfixed.... 0 cache_hits............ 0 n_deleted_files....... 0 n_unlinked_files...... 289 refresh_count......... 1 n_bg_deletions........ 0 Device 2 "data" start_block........... 0 end_block............. 927 total_bytes_per_chunk. 2048 use_nand_ecc.......... 1 no_tags_ecc........... 1 is_yaffs2............. 1 inband_tags........... 0 empty_lost_n_found.... 0 disable_lazy_load..... 0 refresh_period........ 500 n_caches.............. 10 n_reserved_blocks..... 5 always_check_erased... 0 data_bytes_per_chunk.. 2048 chunk_grp_bits........ 0 chunk_grp_size........ 1 n_erased_blocks....... 10 blocks_in_checkpt..... 0 n_tnodes.............. 4211 n_obj................. 24 n_free_chunks......... 658 n_page_writes......... 430 n_page_reads.......... 467 n_erasures............ 7 n_gc_copies........... 421 all_gcs............... 20 passive_gc_count...... 13 oldest_dirty_gc_count. 3 n_gc_blocks........... 6 bg_gcs................ 4 n_retired_writes...... 0 n_retired_blocks...... 0 n_ecc_fixed........... 0 n_ecc_unfixed......... 0 n_tags_ecc_fixed...... 0 n_tags_ecc_unfixed.... 0 cache_hits............ 0 n_deleted_files....... 0 n_unlinked_files...... 2 refresh_count......... 1 n_bg_deletions........ 0 # log and config file stored in "data". thanks!!
In General this could be your disk space (here Flash), first of all check your flash space with with df -h (or other commands you have.. df is present in BusyBox). But if your flash space (specially on your program partition) is ok, this could be your "inode" (directory) space problem, you could see your inode usage with df -i command. (a good link for this: https://wiki.gentoo.org/wiki/Knowledge_Base:No_space_left_on_device_while_there_is_plenty_of_space_available) If non of these is the problem cause, I think you have to have a deeper look at your code, specially if you deal with disk I/O! Also good to mention that be aware of memory & heap space & free all allocated spaces in you functions.
What dtrace script output means?
I am tracing DTrace probes in my restify.js application (restify it is http server in node.js that provides dtrace support). I am using sample dtrace script from restify documentation:
#!/usr/sbin/dtrace -s
#pragma D option quiet
restify*:::route-start
{
track[arg2] = timestamp;
}
restify*:::handler-start
/track[arg3]/
{
h[arg3, copyinstr(arg2)] = timestamp;
}
restify*:::handler-done
/track[arg3] && h[arg3, copyinstr(arg2)]/
{
#[copyinstr(arg2)] = quantize((timestamp - h[arg3, copyinstr(arg2)]) / 1000000);
h[arg3, copyinstr(arg2)] = 0;
}
restify*:::route-done
/track[arg2]/
{
#[copyinstr(arg1)] = quantize((timestamp - track[arg2]) / 1000000);
track[arg2] = 0;
}
And the output is:
use_restifyRequestLogger
value ------------- Distribution ------------- count
-1 | 0
0 |######################################## 2
1 | 0
use_validate
value ------------- Distribution ------------- count
-1 | 0
0 |######################################## 2
1 | 0
pre
value ------------- Distribution ------------- count
0 | 0
1 |#################### 1
2 |#################### 1
4 | 0
handler
value ------------- Distribution ------------- count
128 | 0
256 |######################################## 2
512 | 0
route_user_read
value ------------- Distribution ------------- count
128 | 0
256 |######################################## 2
512 | 0
I was wondering what is value value field - what does it mean?
Why there is 124/256/512 for example? I guess it means the time/duration but it is in strange format - is it possible to show miliseconds for example?
The output is a histogram. You are getting a histogram because you are using the quantize function in your D script. The DTrace documentation says the following on quantize: A power-of-two frequency distribution of the values of the specified expressions. Increments the value in the highest power-of-two bucket that is less than the specified expression. The 'value' columns is the result of (timestamp - track[arg2]) / 1000000 where timestamp is the current time in nanoseconds. So the value shown is duration in milliseconds. Putting this all together, the route_user_read result graph is telling you that you had 2 requests that took between 128 and 256 milliseconds. This output is useful when you have a lot of requests and want to get a general sense of how your server is performing (you can quickly identify a bi-modal distribution for example). If you just want to see how long each request is taking, try using the printf function instead of quantize.
Decipher garbage collection output
I was running a sample program program using rahul#g3ck0:~/programs/Remodel$ GOGCTRACE=1 go run main.go gc1(1): 0+0+0 ms 0 -> 0 MB 422 -> 346 (422-76) objects 0 handoff gc2(1): 0+0+0 ms 0 -> 0 MB 2791 -> 1664 (2867-1203) objects 0 handoff gc3(1): 0+0+0 ms 1 -> 0 MB 4576 -> 2632 (5779-3147) objects 0 handoff gc4(1): 0+0+0 ms 1 -> 0 MB 3380 -> 2771 (6527-3756) objects 0 handoff gc5(1): 0+0+0 ms 1 -> 0 MB 3511 -> 2915 (7267-4352) objects 0 handoff gc6(1): 0+0+0 ms 1 -> 0 MB 6573 -> 2792 (10925-8133) objects 0 handoff gc7(1): 0+0+0 ms 1 -> 0 MB 4859 -> 3059 (12992-9933) objects 0 handoff gc8(1): 0+0+0 ms 1 -> 0 MB 4554 -> 3358 (14487-11129) objects 0 handoff gc9(1): 0+0+0 ms 1 -> 0 MB 8633 -> 4116 (19762-15646) objects 0 handoff gc10(1): 0+0+0 ms 1 -> 0 MB 9415 -> 4769 (25061-20292) objects 0 handoff gc11(1): 0+0+0 ms 1 -> 0 MB 6636 -> 4685 (26928-22243) objects 0 handoff gc12(1): 0+0+0 ms 1 -> 0 MB 6741 -> 4802 (28984-24182) objects 0 handoff gc13(1): 0+0+0 ms 1 -> 0 MB 9654 -> 5097 (33836-28739) objects 0 handoff gc1(1): 0+0+0 ms 0 -> 0 MB 209 -> 171 (209-38) objects 0 handoff Help me understand the first part i.e. 0 + 0 + 0 => Mark + Sweep + Clean times Does 422 -> 346 means that there has been memory cleanup from 422MB to 346 MB? If yes, then how come the memory is been reduced when there was nothing to be cleaned up?
In Go 1.5, the format of this output has changed considerably. For the full documentation, head over to http://godoc.org/runtime and search for "gctrace:" gctrace: setting gctrace=1 causes the garbage collector to emit a single line to standard error at each collection, summarizing the amount of memory collected and the length of the pause. Setting gctrace=2 emits the same summary but also repeats each collection. The format of this line is subject to change. Currently, it is: gc # ##s #%: #+...+# ms clock, #+...+# ms cpu, #->#-># MB, # MB goal, # P where the fields are as follows: gc # the GC number, incremented at each GC ##s time in seconds since program start #% percentage of time spent in GC since program start #+...+# wall-clock/CPU times for the phases of the GC #->#-># MB heap size at GC start, at GC end, and live heap # MB goal goal heap size # P number of processors used The phases are stop-the-world (STW) sweep termination, scan, synchronize Ps, mark, and STW mark termination. The CPU times for mark are broken down in to assist time (GC performed in line with allocation), background GC time, and idle GC time. If the line ends with "(forced)", this GC was forced by a runtime.GC() call and all phases are STW.
The output is generated from this line: http://golang.org/src/pkg/runtime/mgc0.c?#L2147 So the different parts are: 0+0+0 ms : mark, sweep and clean duration in ms 1 -> 0 MB : heap before and after in MB 209 - 171 : objects before and after (209-38) objects : number of allocs and frees handoff (and in Go 1.2 steal and yields) are internals of the algorithm.