Different behavior when printing a bytes.Buffer in Go - string

When I execute this:
buf := new(bytes.Buffer)
buf.WriteString("Hello world")
fmt.Println(buf)
it prints Hello World.
But if I execute this:
var buf bytes.Buffer
buf.WriteString("Hello world")
fmt.Println(buf)
it prints: {[72 101 108 108 111 32 119 111 114 108 100] 0 [72 101 108 108 111 32 119 111 114 108 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0}
I understand that this is the content of the structure byte.Buffer but why it is printed in a different format?

Because a value of type *bytes.Buffer has a String() method (the method set of *bytes.Buffer contains the String() method), and a value of type bytes.Buffer does not.
And the fmt package checks if the value being printed has a String() string method, and if so, it is called to produce the string representation of the value.
Quoting from package doc of fmt:
Except when printed using the verbs %T and %p, special formatting considerations apply for operands that implement certain interfaces. In order of application:
If the operand is a reflect.Value, the operand is replaced by the concrete value that it holds, and printing continues with the next rule.
If an operand implements the Formatter interface, it will be invoked. Formatter provides fine control of formatting.
If the %v verb is used with the # flag (%#v) and the operand implements the GoStringer interface, that will be invoked.
If the format (which is implicitly %v for Println etc.) is valid for a string (%s %q %v %x %X), the following two rules apply:
If an operand implements the error interface, the Error method will be invoked to convert the object to a string, which will then be formatted as required by the verb (if any).
If an operand implements method String() string, that method will be invoked to convert the object to a string, which will then be formatted as required by the verb (if any).
The Buffer.String() method returns its content as a string, that's what you see printed when you pass a pointer of type *bytes.Buffer. And when you pass a non-pointer value of type bytes.Buffer, it is simply printed like a normal struct value, for which the default format is:
{field0 field1 ...}
See related / similar questions:
The difference between t and *t
Why not use %v to print int and string
Why does Error() have priority over String()

Related

Mainframe ICETOOL joins two files

I have two VB files.
File1 is 406 bytes long, and the key is in position 401 to 406.
File2 is 565 bytes long, and the key is in position 87 to 92.
I would like the result file to have all the 406 bytes from file1 plus 8 bytes from file2 starts from column 42.
I have the following codes:
//COPY01 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN01 DD DSN=CCSY.CCSYAC.BNS.C2,DISP=SHR
//IN02 DD DSN=CCSY.CCSYAC.UID.XBAS,DISP=SHR
//OUT01 DD DSN=CCSY.CCSYAC.BNS.C2.ID,DISP=(,CATLG,DELETE),
// DCB=(RECFM=FB,LRECL=410,BLKSIZE=41000),
// SPACE=(CYL,(200,100),RLSE)
//TOOLIN DD *
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
/*
//BNSCCNTL DD *
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
/*
I got S013. Anyone would like to help? I'll appreciate.
JESMSGLG:
13.02.33 JOB38364 ---- WEDNESDAY, 10 JAN 2018 ----
13.02.33 JOB38364 IRR010I USERID CCSYAC IS ASSIGNED TO THIS JOB.
13.02.34 JOB38364 ICH70001I CCSYAC LAST ACCESS AT 12:56:19 ON WEDNESDAY, JANUARY 10, 2018
13.02.34 JOB38364 $HASP373 CCSYAC02 STARTED - INIT 68 - CLASS J - SYS SY07
13.02.34 JOB38364 IEF403I CCSYAC02 - STARTED - TIME=13.02.34
13.02.35 JOB38364 IEC141I 013-68,IFG0196L,CCSYAC02,COPY01,OUT01,B275,STRP49, 005
005 CCSY.CCSYAC.BNS.C2.ID
13.02.35 JOB38364 IEA995I SYMPTOM DUMP OUTPUT 006
006 SYSTEM COMPLETION CODE=013 REASON CODE=00000068
006 TIME=13.02.35 SEQ=61720 CPU=0000 ASID=0058
006 PSW AT TIME OF ERROR 075C1000 80E70476 ILC 2 INTC 0D
006 NO ACTIVE MODULE FOUND
006 NAME=UNKNOWN
006 DATA AT PSW 00E70470 - 4100302C 0A0D010D A7E5014B
006 AR/GR 0: 008FE990/00E70780 1: 00000000/A4013000
006 2: 00000000/0000DCC8 3: 00000000/00E70754
006 4: 00000000/008C4388 5: 00000000/008C471C
006 6: 00000000/008C46C4 7: 00000000/008C471C
006 8: 00000000/008C46E4 9: 00000000/008D1C50
006 A: 00000000/80E72A62 B: 00000000/00E73622
006 C: 00000000/80E7387E D: 00000000/7F59FCE8
006 E: 00000000/80E6FCD4 F: 00000000/00000068
006 END OF SYMPTOM DUMP
13.02.47 JOB38364 IEF450I CCSYAC02 COPY01 - ABEND=S013 U0000 REASON=00000068 011
011 TIME=13.02.47
13.02.47 JOB38364 IEF404I CCSYAC02 - ENDED - TIME=13.02.47
13.02.47 JOB38364 $HASP395 CCSYAC02 ENDED - ABEND=S013
------ JES2 JOB STATISTICS ------
10 JAN 2018 JOB EXECUTION DATE
26 CARDS READ
563 SYSOUT PRINT RECORDS
0 SYSOUT PUNCH RECORDS
54 SYSOUT SPOOL KBYTES
0.22 MINUTES EXECUTION TIME
TOOLMSG:
ICE600I 0 DFSORT ICETOOL UTILITY RUN STARTED
ICE650I 0 VISIT http://www.ibm.com/storage/dfsort FOR ICETOOL PAPERS,
EXAMPLES AND MORE
ICE632I 0 SOURCE FOR ICETOOL STATEMENTS: TOOLIN
ICE630I 0 MODE IN EFFECT: STOP
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
DFSMSG:
ICE200I 0 IDENTIFIER FROM CALLING PROGRAM IS 0001
ICE411I 0 THIS IS THE JOINKEYS MAIN TASK FOR JOINING F1 AND F2
ICE416I 0 JOINKEYS IS USING THE F1 SUBTASK FOR IN01 - SEE JNF1JMSG MESSAGES
ICE416I 1 JOINKEYS IS USING THE F2 SUBTASK FOR IN02 - SEE JNF2JMSG MESSAGES
ICE419I 0 JOINED RECORDS: TYPE=F, LENGTH=410
ICE201I A RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E9-I40658 C9-I35397 E5-
I38877 E7-I40658
ICE143I 0 BLOCKSET COPY TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
ICE146I 0 END OF STATEMENTS FROM BNSCCNTL - PARAMETER LIST STATEMENTS FOLLOW
DEBUG NOABEND,ESTAE
OPTION MSGDDN=DFSMSG,LIST,MSGPRT=ALL,RESINV=0,SORTDD=BNSC,SORTOUT=OUT01*
,DYNALLOC
SORT FIELDS=COPY
RECORD TYPE=V
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE089I 0 CCSYAC02.COPY01 . , INPUT LRECL = 410, TYPE = F
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1024000,MINLIM=122800,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=DFSMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR OUT01
ICE751I 2 EF-I35397 DA-I40658
ICE805I 0 JOBNAME: CCSYAC02 , STEPNAME: COPY01
ICE802I 0 BLOCKSET TECHNIQUE IN CONTROL
ICE906I 0 ST=ABOVE,SR=6291456,RC=0
ICE907I 0 ST=ABOVE,SA=6291440,NF=1,LF=6291440,SF=6291440
ICE906I 0 ST=BELOW,SR=1036288,RC=0
ICE907I 0 ST=BELOW,SA=1036272,NF=1,LF=1036272,SF=1036272
ICE889I 0 CT=MAX , SB=8, L=0, D=0000, CCW=1MAM
ICE902I 0 O RP10 I
ICE906I 1 ST=ABOVE,SR=6234096,RC=0
ICE907I 1 ST=ABOVE,SA=6234080,NF=1,LF=6234080,SF=6234080
ICE906I 1 ST=BELOW,SR=94680,RC=0
ICE907I 1 ST=BELOW,SA=49608,NF=1,LF=49608,SF=49608
ICE185A 0 AN S013 ABEND WAS ISSUED BY DFSORT, ANOTHER PROGRAM OR AN EXIT (PHASE C 3)
JNF1JMSG:
ICE201I A RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E4-I40658 C9-I35397 E5-
I38877 E6-I31999 C4-I31999 E7-I40658
ICE417I 0 THIS IS THE JOINKEYS F1 SUBTASK FOR IN01
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
SORT FORMAT=BI,FIELDS=(401,6,A)
RECORD TYPE=F
DEBUG NOABEND,ESTAE
OPTION EQUALS,MSGPRT=ALL,LIST,NOCHECK,RESINV=0,DYNALLOC,SORTDD=JNF1,MSG*
DDN=JNF1JMSG,SORTIN=IN01
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE088I 0 CCSYAC02.COPY01 . , INPUT LRECL = 406, BLKSIZE = 27998, TYPE = VB
ICE093I 0 MAIN STORAGE = (MAX,9251025,9251025)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (9193665,9193665)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=9251025,MAXLIM=1024000,MINLIM=122800,EQUALS=Y,LIST=Y,ERET=RC16 ,MSGDDN=JNF1JMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT==N,DYNALOC=(DISK ,004),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=128
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR IN01
ICE750I 0 DC 589973550 TC 0 CS DSVXX KSZ 10 VSZ 10
ICE752I 0 FSZ=589973550 BC IGN=0 E AVG=203 0 WSP=766274 C DYN=520 53216
ICE751I 1 D8-I35397 D4-I35397 EA-I35933 F1-I35933 E8-I40658
ICE091I 0 OUTPUT LRECL = 406, TYPE = V
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 1433042, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 581815052
ICE253I 0 RECORDS SORTED - PROCESSED: 1433042, EXPECTED: 2906273
ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 406, EXPECTED: 203
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 540 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 563M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT
JNF2JMSG:
ICE201I A RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE751I 0 C5-I40658 C6-I35397 C7-I35397 C8-I40658 E4-I40658 C9-I35397 E5-
I38877 E6-I31999 E7-I40658
ICE417I 0 THIS IS THE JOINKEYS F2 SUBTASK FOR IN02
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5650-ZOS, Z/OS DFSORT V2R2 - 13:02 ON WED JAN 10, 2018 -
SORT FORMAT=BI,FIELDS=(87,6,A)
RECORD TYPE=F
DEBUG NOABEND,ESTAE
OPTION EQUALS,MSGPRT=ALL,LIST,NOCHECK,RESINV=0,DYNALLOC,SORTDD=JNF2,MSG*
DDN=JNF2JMSG,SORTIN=IN02
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE088I 0 CCSYAC02.COPY01 . , INPUT LRECL = 565, BLKSIZE = 27998, TYPE = VB
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1024000,MINLIM=122800,EQUALS=Y,LIST=Y,ERET=RC16 ,MSGDDN=JNF2JMSG
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(DISK ,004),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y,TUNE=STOR,EXPMAX=MAX ,EXPOLD=50% ,EXPRES=10%
ICE084I 0 EXCP ACCESS METHOD USED FOR IN02
ICE750I 0 DC 1063772 TC 0 CS DSVXX KSZ 10 VSZ 10
ICE752I 0 FSZ=1063772 BC IGN=0 E AVG=282 0 WSP=1381 C DYN=0 0
ICE751I 1 D8-I35397 D4-I35397 D1-I35397 E8-I40658
ICE091I 0 OUTPUT LRECL = 565, TYPE = V
ICE080I 0 IN MAIN STORAGE SORT
ICE055I 0 INSERT 0, DELETE 0
ICE054I 0 RECORDS - IN: 3603, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 1034061
ICE253I 0 RECORDS SORTED - PROCESSED: 3603, EXPECTED: 3772
ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 287, EXPECTED: 282
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 0 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT
Here is the description for the S013 error you encountered:
The SORTIN, SORTOUT, or OUTFIL BLKSIZE parameter:
Was greater than 32760 for a disk data set, or was greater than the maximum block size
supported by the access method for a tape data set.
Was not an integer multiple of the LRECL parameter for fixed-length records, or was not
at least four bytes longer than the LRECL for variable-length
records.
and if you look at the JCL you provided:
//COPY01 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN01 DD DSN=CCSY.CCSYAC.BNS.C2,DISP=SHR
//IN02 DD DSN=CCSY.CCSYAC.UID.XBAS,DISP=SHR
//OUT01 DD DSN=CCSY.CCSYAC.BNS.C2.ID,DISP=(,CATLG,DELETE),
// DCB=(RECFM=FB,LRECL=410,BLKSIZE=41000),
// SPACE=(CYL,(200,100),RLSE)
//TOOLIN DD *
COPY JKFROM TO(OUT01) VSAMTYPE(V) USING(BNSC)
/*
//BNSCCNTL DD *
JOINKEYS F1=IN01,FIELDS=(401,6,A)
JOINKEYS F2=IN02,FIELDS=(87,6,A)
REFORMAT FIELDS=(F1:5,402,F2:42,8)
/*
You can see that your block size of 41000 is greater than that 32760 maximum. Try not specifying the block size. If you do not specify the block size in the JCL, the system will calculate an efficient one for you.

SGE faild to submit job, attribute is not a memory value

I failed to submit a job with mem attribute. Since I am novice and after google two days, I look for help here. Any suggestion would be grateful!
Following is what had I done:
\1. Submit my script:
qsub -S /bin/bash -A assembly -pe threads 16 -l mem=2GB -cwd -N "pBcR_correct_asm" -j y -o /dev/null runCorrection.sh
Unable to run job: unknown resource "mem".
Exiting.
\2. Considering that I had replace "h" to "host", which solve my problem according to SGE unknown resource "nodes", I replace "m" to "mem”, and it didn't work.
\3. After google, I know "h" is the shortcut that was defined in "/opt/gridengine/util/resources/centry/
hostname", and can be confirmed with "qconf -sc":
qconf -sc
#name shortcut type relop requestable consumable default urgency
#----------------------------------------------------------------------------------------
arch a RESTRING == YES NO NONE 0
calendar c RESTRING == YES NO NONE 0
cpu cpu DOUBLE >= YES NO 0 0
display_win_gui dwg BOOL == YES NO 0 0
h_core h_core MEMORY <= YES NO 0 0
h_cpu h_cpu TIME <= YES NO 0:0:0 0
h_data h_data MEMORY <= YES NO 0 0
h_fsize h_fsize MEMORY <= YES NO 0 0
h_rss h_rss MEMORY <= YES NO 0 0
h_rt h_rt TIME <= YES NO 0:0:0 0
h_stack h_stack MEMORY <= YES NO 0 0
h_vmem h_vmem MEMORY <= YES NO 0 0
hostname h HOST == YES NO NONE 0
load_avg la DOUBLE >= NO NO 0 0
load_long ll DOUBLE >= NO NO 0 0
load_medium lm DOUBLE >= NO NO 0 0
load_short ls DOUBLE >= NO NO 0 0
m_core core INT <= YES NO 0 0
m_socket socket INT <= YES NO 0 0
m_topology topo RESTRING == YES NO NONE 0
m_topology_inuse utopo RESTRING == YES NO NONE 0
mem_free mf MEMORY <= YES NO 0 0
mem_total mt MEMORY <= YES NO 0 0
mem_used mu MEMORY >= YES NO 0 0
\4. I thus replaced "mt" to "mem", but it complained attribute problem. According to above output, it seemed the mem_total is almost same as "hostname" that worked previously. Then, I think jsv may be a problem after going through the SGE guide, but I can't find any scripts contain "Unable to run job: attribute......", which under the director of "/opt/gridengine/util/resources/jsv". I think I have to configure some files, but what these files, and what should I do?
qsub -S /bin/bash -A assembly -pe threads 16 -l mt=2GB -cwd -N "pBcR_correct_asm" -j y -o test.out runCorrection.sh
Unable to run job: attribute "mem_total" is not a memory value.
Exiting.
#Vince!
Thanks for your reply very much.
Finally I solve my problem, by using "h_vmem=2g" ("2GB" would give error), but I don't know where to find how to design the value of the complex (MEMORY).
Following information is un-necessary now.
I had read the website you gave, and configured the attribute of h_vmem and s_vmeme in complex to "consumable", but it didn't work. I guess I have to configure "complex_value" of queue which is "NONE" at the moment. However, I can't open the web http://gridscheduler.sourceforge.net/htmlman/htmlman5/sge_types.html?pathrev=V62u5_TAG that may tell me how to configure. Am I right to configure to configure queue? Do I have to configure host too?
Any suggestions would be grateful!
Fowllowing is what had I done:
\1. Change the attribute of consumable to "YES" for h_vmem and s_vmem:
qconf -sc
#name shortcut type relop requestable consumable default urgency
#----------------------------------------------------------------------------------------
arch a RESTRING == YES NO NONE 0
calendar c RESTRING == YES NO NONE 0
cpu cpu DOUBLE >= YES NO 0 0
display_win_gui dwg BOOL == YES NO 0 0
h_core h_core MEMORY <= YES NO 0 0
h_cpu h_cpu TIME <= YES NO 0:0:0 0
h_data h_data MEMORY <= YES NO 0 0
h_fsize h_fsize MEMORY <= YES NO 0 0
h_rss h_rss MEMORY <= YES NO 0 0
h_rt h_rt TIME <= YES NO 0:0:0 0
h_stack h_stack MEMORY <= YES NO 0 0
h_vmem h_vmem MEMORY <= YES YES 0 0
hostname h HOST == YES NO NONE 0
load_avg la DOUBLE >= NO NO 0 0
load_long ll DOUBLE >= NO NO 0 0
load_medium lm DOUBLE >= NO NO 0 0
load_short ls DOUBLE >= NO NO 0 0
m_core core INT <= YES NO 0 0
m_socket socket INT <= YES NO 0 0
m_topology topo RESTRING == YES NO NONE 0
m_topology_inuse utopo RESTRING == YES NO NONE 0
mem_free mf MEMORY <= YES NO 0 0
mem_total mt MEMORY <= YES NO 0 0
mem_used mu MEMORY >= YES NO 0 0
min_cpu_interval mci TIME <= NO NO 0:0:0 0
np_load_avg nla DOUBLE >= NO NO 0 0
np_load_long nll DOUBLE >= NO NO 0 0
np_load_medium nlm DOUBLE >= NO NO 0 0
np_load_short nls DOUBLE >= NO NO 0 0
num_proc p INT == YES NO 0 0
qname q RESTRING == YES NO NONE 0
rerun re BOOL == NO NO 0 0
s_core s_core MEMORY <= YES NO 0 0
s_cpu s_cpu TIME <= YES NO 0:0:0 0
s_data s_data MEMORY <= YES NO 0 0
s_fsize s_fsize MEMORY <= YES NO 0 0
s_rss s_rss MEMORY <= YES NO 0 0
s_rt s_rt TIME <= YES NO 0:0:0 0
s_stack s_stack MEMORY <= YES NO 0 0
s_vmem s_vmem MEMORY <= YES YES 0 0
seq_no seq INT == NO NO 0 0
slots s INT <= YES YES 1 1000
swap_free sf MEMORY <= YES NO 0 0
swap_rate sr MEMORY >= YES NO 0 0
swap_rsvd srsv MEMORY >= YES NO 0 0
swap_total st MEMORY <= YES NO 0 0
swap_used su MEMORY >= YES NO 0 0
tmpdir tmp RESTRING == NO NO NONE 0
virtual_free vf MEMORY <= YES NO 0 0
virtual_total vt MEMORY <= YES NO 0 0
virtual_used vu MEMORY >= YES NO 0 0
# >#< starts a comment but comments are not saved across edits --------
\2. Submit my job to the queue of smp.q, and it complained the same problem:
qsub -S /bin/bash -A assembly -q smp.q -pe newPe 16 -l h_vmem=2GB -cwd -N "pBcR_correct_asm" -j y -o runCorrection.sh
Unable to run job: attribute "h_vmem" is not a memory value.
Exiting.
\3. The information of smp.q. I think "complex_values" should be changed and "h_vmem" can keep unchanged:
qconf -sq smp.q
qname smp.q
hostlist #smp.q
seq_no 0
load_thresholds np_load_avg=1.75
suspend_thresholds NONE
nsuspend 1
suspend_interval 00:05:00
priority 0
min_cpu_interval 00:05:00
processors UNDEFINED
qtype BATCH INTERACTIVE
ckpt_list NONE
pe_list make newPe
rerun FALSE
slots 160
tmpdir /tmp
shell /bin/csh
prolog NONE
epilog NONE
shell_start_mode posix_compliant
starter_method NONE
suspend_method NONE
resume_method NONE
terminate_method NONE
notify 00:00:60
owner_list NONE
user_lists NONE
xuser_lists NONE
subordinate_list NONE
complex_values NONE
projects NONE
xprojects NONE
calendar NONE
initial_state default
s_rt INFINITY
h_rt INFINITY
s_cpu INFINITY
h_cpu INFINITY
s_fsize INFINITY
h_fsize INFINITY
s_data INFINITY
h_data INFINITY
s_stack INFINITY
h_stack INFINITY
s_core INFINITY
h_core INFINITY
s_rss INFINITY
h_rss INFINITY
s_vmem INFINITY
h_vmem INFINITY
\4. The information of hosts in #smp.q:
qconf -sconf smp03.local
#smp03.local:
mailer /bin/mail
xterm /usr/bin/X11/xterm
execd_spool_dir /opt/gridengine/default/spool
\5. The global information. Have I add h_vmem and s_vmem here?
qconf -sconf
#global:
execd_spool_dir /opt/gridengine/default/spool
mailer /bin/mail
xterm /usr/bin/X11/xterm
load_sensor none
prolog none
epilog none
shell_start_mode posix_compliant
login_shells sh,ksh,csh,tcsh
min_uid 0
min_gid 0
user_lists none
xuser_lists none
projects none
xprojects none
enforce_project false
enforce_user auto
load_report_time 00:00:40
max_unheard 00:05:00
reschedule_unknown 00:00:00
loglevel log_warning
administrator_mail none
set_token_cmd none
pag_cmd none
token_extend_time none
shepherd_cmd none
qmaster_params none
execd_params ENABLE_ADDGRP_KILL=TRUE H_MEMORYLOCKED=infinity
reporting_params accounting=true reporting=true \
flush_time=00:00:15 joblog=true sharelog=00:00:00
finished_jobs 100
gid_range 20000-20100
qlogin_command builtin
qlogin_daemon builtin
rlogin_command builtin
rlogin_daemon builtin
rsh_command builtin
rsh_daemon builtin
max_aj_instances 2000
max_aj_tasks 75000
max_u_jobs 0
max_jobs 0
max_advance_reservations 0
auto_user_oticket 0
auto_user_fshare 0
auto_user_default_project none
auto_user_delete_time 86400
delegated_file_staging false
reprioritize 0
jsv_url none
jsv_allowed_mod ac,h,i,e,o,j,M,N,p,w
What you likely want is h_vmem. At least that is the attribute I always use to specify the memory I want request for a job.
See:
http://gridscheduler.sourceforge.net/htmlman/htmlman5/queue_conf.html?pathrev=V62u5_TAG
Specifically,
The resource limit parameters s_vmem and h_vmem are imple-
mented by Sun Grid Engine as a job limit. They impose a
limit on the amount of combined virtual memory consumed by
all the processes in the job. If h_vmem is exceeded by a job
running in the queue, it is aborted via a SIGKILL signal
(see kill(1)). If s_vmem is exceeded, the job is sent a
SIGXCPU signal which can be caught by the job. If you wish
to allow a job to be "warned" so it can exit gracefully
before it is killed then you should set the s_vmem limit to
a lower value than h_vmem. For parallel processes, the
limit is applied per slot which means that the limit is mul-
tiplied by the number of slots being used by the job before
being applied.
Also, you may need to set this as consumable using qconf.

write error: No space left on device in embedded linux

all
I have a embedded board, run linux OS. and I use yaffs2 as rootfs.
I run a program on it, but after some times, it got a error "error No space left on device.". but I checked the flash, there still have a lot free space.
I just write some config file. the config file is rarely update. the program will write some log to flash. log size is limited to 2M.
I don't know why, and how to solve.
Help me please!(my first language is not English,sorry. hope you understand what I say)
some debug info:
# ./write_test
version 1.0
close file :: No space left on device
return errno 28
# cat /proc/yaffs
YAFFS built:Nov 23 2015 16:57:34
Device 0 "rootfs"
start_block........... 0
end_block............. 511
total_bytes_per_chunk. 2048
use_nand_ecc.......... 1
no_tags_ecc........... 1
is_yaffs2............. 1
inband_tags........... 0
empty_lost_n_found.... 0
disable_lazy_load..... 0
refresh_period........ 500
n_caches.............. 10
n_reserved_blocks..... 5
always_check_erased... 0
data_bytes_per_chunk.. 2048
chunk_grp_bits........ 0
chunk_grp_size........ 1
n_erased_blocks....... 366
blocks_in_checkpt..... 0
n_tnodes.............. 749
n_obj................. 477
n_free_chunks......... 23579
n_page_writes......... 6092
n_page_reads.......... 11524
n_erasures............ 96
n_gc_copies........... 5490
all_gcs............... 1136
passive_gc_count...... 1136
oldest_dirty_gc_count. 95
n_gc_blocks........... 96
bg_gcs................ 96
n_retired_writes...... 0
n_retired_blocks...... 0
n_ecc_fixed........... 0
n_ecc_unfixed......... 0
n_tags_ecc_fixed...... 0
n_tags_ecc_unfixed.... 0
cache_hits............ 0
n_deleted_files....... 0
n_unlinked_files...... 289
refresh_count......... 1
n_bg_deletions........ 0
Device 2 "data"
start_block........... 0
end_block............. 927
total_bytes_per_chunk. 2048
use_nand_ecc.......... 1
no_tags_ecc........... 1
is_yaffs2............. 1
inband_tags........... 0
empty_lost_n_found.... 0
disable_lazy_load..... 0
refresh_period........ 500
n_caches.............. 10
n_reserved_blocks..... 5
always_check_erased... 0
data_bytes_per_chunk.. 2048
chunk_grp_bits........ 0
chunk_grp_size........ 1
n_erased_blocks....... 10
blocks_in_checkpt..... 0
n_tnodes.............. 4211
n_obj................. 24
n_free_chunks......... 658
n_page_writes......... 430
n_page_reads.......... 467
n_erasures............ 7
n_gc_copies........... 421
all_gcs............... 20
passive_gc_count...... 13
oldest_dirty_gc_count. 3
n_gc_blocks........... 6
bg_gcs................ 4
n_retired_writes...... 0
n_retired_blocks...... 0
n_ecc_fixed........... 0
n_ecc_unfixed......... 0
n_tags_ecc_fixed...... 0
n_tags_ecc_unfixed.... 0
cache_hits............ 0
n_deleted_files....... 0
n_unlinked_files...... 2
refresh_count......... 1
n_bg_deletions........ 0
#
log and config file stored in "data".
thanks!!
In General this could be your disk space (here Flash), first of all check your flash space with with df -h (or other commands you have.. df is present in BusyBox). But if your flash space (specially on your program partition) is ok, this could be your "inode" (directory) space problem, you could see your inode usage with df -i command. (a good link for this: https://wiki.gentoo.org/wiki/Knowledge_Base:No_space_left_on_device_while_there_is_plenty_of_space_available)
If non of these is the problem cause, I think you have to have a deeper look at your code, specially if you deal with disk I/O!
Also good to mention that be aware of memory & heap space & free all allocated spaces in you functions.

MPI_FILE_READ_AT ignores file (don't read the file)

I am new at MPI and parallel computing.I wrote simple MPI program in Fortran. But the problem is the MPI_FILE_READ_AT ignores and does not read the file.
My code is:
program O72
use mpi
!implicit none
INTEGER :: NINTS, DATATYPE, STATUS(MPI_STATUS_SIZE), mpierr, FH, i
integer :: taskid, no_tasks, FILESIZE, FILENO_2
INTEGER(KIND=MPI_OFFSET_KIND) :: OFFSET
INTEGER :: BUF(10)
character :: filename='ab.dat'
FILESIZE=10
BUF(:)=0
call MPI_INIT(mpierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD,no_tasks,mpierr)
call MPI_COMM_RANK(MPI_COMM_WORLD,taskid,mpierr)
NINTS = FILESIZE/no_tasks !*INTSIZE)
OFFSET= taskid*NINTS !*INTSIZE
call MPI_File_open(MPI_COMM_WORLD, "ab.dat", MPI_MODE_RDONLY, &
MPI_INFO_NULL, FH, mpierr)
call MPI_FILE_READ_AT(FH, OFFSET, BUF, NINTS, MPI_INT, STATUS, mpierr)
write(6,*) taskid, no_tasks
call MPI_BARRIER(MPI_COMM_WORLD, mpierr)
write(6,*) 'taskid=',taskid,'NINTS=', NINTS
write(6,*) 'taskid_',taskid,'OFFSET=',OFFSET
write(6,*) BUF(:)
!call MPI_FILE_WRITE(FILENO_2, OFFSET, BUF, NINTS, DATATYPE, STATUS, mpierr)
call MPI_FILE_CLOSE(FH, mpierr)
!call MPI_FILE_CLOSE(FILENO_2)
call MPI_FINALIZE(mpierr)
end
Then, the input file is an ASCII file with 2 columns in which 1st column is always 1 and 2nd column is always 2.
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
And the output comes like this with 2 processor used:
0 2
1 2
taskid= 1 NINTS= 5
taskid_ 1 OFFSET= 5
taskid= 0 NINTS= 5
taskid_ 0 OFFSET= 0
171051313 171051313 171051313 171051313 171051313 0 0 0 0 0
822751753 822751753 822751753 822751753 822751753 0 0 0 0 0
I don't understand where the problem is and why random values are assigned to BUF(:)
Could you please help me?
I guess the simple answer is that MPI-IO only performs binary IOs, while what you try to achieve here is formatted IOs.
So starting from here, you see that you have a few problems in your code:
You file is formatted. Since as I said in preamble, MPI-IO only performs binary IO, you'll have to, either switch to a binary input file, or to read the file as a set of strings which you'll have to make sense of internally. Let's assume for the rest of this post that your input file is now binary.
The offset to use in MPI_File_read_at() is expected in bytes. What you pass is a number of elements. You therefore need to multiply this offset by the size of what you want to read, aka the size of an integer.
Finally, you print the full of buf which is an array of 10 integer whereas you only read nints of them. You probably should only print buf(1:nints).
With that, your code should work.

What dtrace script output means?

I am tracing DTrace probes in my restify.js application (restify it is http server in node.js that provides dtrace support). I am using sample dtrace script from restify documentation:
#!/usr/sbin/dtrace -s
#pragma D option quiet
restify*:::route-start
{
track[arg2] = timestamp;
}
restify*:::handler-start
/track[arg3]/
{
h[arg3, copyinstr(arg2)] = timestamp;
}
restify*:::handler-done
/track[arg3] && h[arg3, copyinstr(arg2)]/
{
#[copyinstr(arg2)] = quantize((timestamp - h[arg3, copyinstr(arg2)]) / 1000000);
h[arg3, copyinstr(arg2)] = 0;
}
restify*:::route-done
/track[arg2]/
{
#[copyinstr(arg1)] = quantize((timestamp - track[arg2]) / 1000000);
track[arg2] = 0;
}
And the output is:
use_restifyRequestLogger
value ------------- Distribution ------------- count
-1 | 0
0 |######################################## 2
1 | 0
use_validate
value ------------- Distribution ------------- count
-1 | 0
0 |######################################## 2
1 | 0
pre
value ------------- Distribution ------------- count
0 | 0
1 |#################### 1
2 |#################### 1
4 | 0
handler
value ------------- Distribution ------------- count
128 | 0
256 |######################################## 2
512 | 0
route_user_read
value ------------- Distribution ------------- count
128 | 0
256 |######################################## 2
512 | 0
I was wondering what is value value field - what does it mean?
Why there is 124/256/512 for example? I guess it means the time/duration but it is in strange format - is it possible to show miliseconds for example?
The output is a histogram. You are getting a histogram because you are using the quantize function in your D script. The DTrace documentation says the following on quantize:
A power-of-two frequency distribution of the values of the specified expressions. Increments the value in the highest power-of-two bucket that is less than the specified expression.
The 'value' columns is the result of (timestamp - track[arg2]) / 1000000 where timestamp is the current time in nanoseconds. So the value shown is duration in milliseconds.
Putting this all together, the route_user_read result graph is telling you that you had 2 requests that took between 128 and 256 milliseconds.
This output is useful when you have a lot of requests and want to get a general sense of how your server is performing (you can quickly identify a bi-modal distribution for example). If you just want to see how long each request is taking, try using the printf function instead of quantize.

Resources