How can I solve "WER046A SORT CAPACITY EXCEEDED" in SYNCSORT JCL - mainframe

Question

I am trying to sort a data set and writing into a new data set, Every time i am executing the job, its abending (ABEND=U0016).
message form SYSOUT.
WER276B SYSDIAG= 24646562, 29667262, 29667262, 27500165
WER164B 307,288K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
WER164B 0 BYTES RESERVE REQUESTED, 307,272K BYTES USED
WER036B G=5174,B=1,BIAS=99
WER162B 75 PREALLOCATED SORTWORK TRACKS, 3,750,000 DYNAMICALLY
ALLOCATED,
WER162B 26,721,480 ACQUIRED IN 2,230 SECONDARY EXTENTS, 0 RELEASED,
TOTAL 30,471,555 TRACKS USED
WER046A SORT CAPACITY EXCEEDED
WER493I ZIIP PROCESSOR USED
WER211B SYNCSMF CALLED BY SYNCSORT; RC=0000
WER449I SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
WER066A APROX RCD CNT 30430982
Here is my code:
//STEP50 EXEC SORTD
//SORTIN DD DSN=FILEXYZ(0),
// DISP=SHR,DCB=BUFNO=192
//SORTOUT DD DSN=FILE2XXY,
// DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),
// DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),
// UNIT=(TAPE,2)
//SYSOUT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)
OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),
OVERLAY=(70:C' encoding="UTF-8"'))
OPTION DYNALLOC=(SYSDA,255)
/*
Here is code from SORTD from the jcl:
4 //STEP40 EXEC SORTD
5 XXSORTD PROC CYLS=1,SYSFICH='*',
XX DMPCLAS='1,DEST=ABNORMAL'
XX*
XX* REMOVED SEP PARAMETER 89/07/20
XX*
6 XXSORTD EXEC PGM=SORT
7 //SYSOUT DD SYSOUT=*
X/SYSOUT DD SYSOUT=&SYSFICH
IEFC653I SUBSTITUTION JCL - SYSOUT=*
8 XXSORTWK01 DD DSN=&WORK1,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK
IEFC653I SUBSTITUTION JCL - DSN=&WORK1,SPACE=(CYL,
(1)),UNIT=SORTWORK
9 XXSORTWK02 DD DSN=&WORK2,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK
IEFC653I SUBSTITUTION JCL - DSN=&WORK2,SPACE=(CYL,
(1)),UNIT=SORTWORK
10 XXSORTWK03 DD DSN=&WORK3,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK
IEFC653I SUBSTITUTION JCL - DSN=&WORK3,SPACE=(CYL,
(1)),UNIT=SORTWORK
11 XXSORTWK04 DD DSN=&WORK4,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK
IEFC653I SUBSTITUTION JCL - DSN=&WORK4,SPACE=(CYL,
(1)),UNIT=SORTWORK
12 XXSORTWK05 DD DSN=&WORK5,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK
IEFC653I SUBSTITUTION JCL - DSN=&WORK5,SPACE=(CYL,
(1)),UNIT=SORTWORK
13 XXSYSABEND DD SYSOUT=&DMPCLAS
IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL
14 XXSYSUDUMP DD SYSOUT=&DMPCLAS
IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL
15 //SORTIN DD DSN=FILEXYZ(0),
// DISP=SHR,DCB=BUFNO=192
16 //SORTOUT DD DSN=FILE2XXY,
// DISP=(NEW,PASS,DELETE),RETPD=365,VOL=(,,,99),
// DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),
// UNIT=TAPE
17 //SYSPRINT DD SYSOUT=*
18 //SYSIN DD *
Please suggest me some hints to over come this issue.
Thank you.

Answer 1

I'll go out-on-a-limb here and assume that the files in question here are the same as previous question with same file attributes by same OP at:
How can I reduce CPU in SORT operation
With 80,000,000 records at an LRECL of 30050, your input file is approximately 2.2TB.
A generally accepted rule-of-thumb for sort work space is 1.3x the input file size. So, you’ll need to target ~2.85TB (I.e ~3.7 million CYLS or ~55 million tracks) of sort work space.
With SyncSort’s maximum of 255 SORTWK DDs, you’ll need to acquire ~14,500 CYLs across each of the supported 255 DDs to accommodate your space requirements. This is a bit extreme, and will likely present a number of challenges, but there are a few options that could allow you to achieve this.
Disclaimer - I have not personally used SyncSort myself, but after perusing the docs, these concepts look to be consistent with other sort products that I do have experience with.
Rely on DYNALLOC
You have specified DYNALLOC=(SYSDA,255) and this is generally the recommended approach. However, there are a couple of issues with your current setup:
By using the SORTD PROC, you are robbing yourself of 5 DDs that could have been more appropriately utilized. The hardcoded SORTWK DDs in the PROC with SPACE=(CYL,(1)) limit the total work space you can acquire because DYNALLOC now only has 250 DDs to work with. As #SaggingRufus suggests, calling SYNCSORT directly would avoid the 1-CYL allocations of the PROC DDs and allow DYNALLOC to work with all 255 DDs.
SyncSort is not projecting the necessary work space accurately. Based on the output that you provided, SyncSort initially acquired 75 tracks (1 CYL for each of 5 hardcoded DDs in the SORTD proc… in other words, 5 DDs at 1 CYL * 15 Tracks-per-CYL = 75) explicitly, then dynamically acquired an additional 3,750,000 tracks (1,000 CYLs (or 15,000 tracks) * 250 DDs). It then acquired an additional 26.7 million tracks (1.78MM CYLs) across 2,230 secondary extents, which indicates an average of 799 CYLs (or 11,983 tracks) per secondary extent and an average of 8.75 secondary extents per volume/DD. All of this results in a total of ~1.6TB (~2MM Cyls) being allocated for sort work space, which is way below the estimated work space requirements for a 2.2TB input file. Even if we apply the DYNALLOC attributes to the hardcoded SORTWK DDs from the PROC, that only adds about 32GB to the total work space and is still woefully inadequate. So, as #phunsoft suggests, adding the SIZE=Ennnnnnnn parameter may be necessary for SyncSort to project work space appropriately.
Additional note - The fact that you're getting an average of <10 secondary extents per DD/volume indicates that you're exhausting the available free space on volumes in the work space pool for your environment. You should coordinate with your storage admins to ensure that enough disk space is available for your work space.
So, bypassing the PROC to provide access to all 255 DDs and providing an estimated record count could allow SyncSort to more reliably calculate the necessary work space, but this amount of data in a single sort is likely an outlier and default calculations may not be sufficient. Also, the available free space in your work space pool may be a limiting factor.
Hardcode SORTWK DDs
If you hardcode the SORTWK DDs in your JCL, you bypass the DDs in the PROC and eliminate some of the uncertainty of the DYNALLOC calculations. This is actually the approach that I, personally, would take first in this scenario… hardcode each SORTWK DD with the best possible options to acquire ~14,500 CYLS on each DD. So:
Hardcode each of 255 DDs (SORTWK01 - SORTWKFF) in job
Include SPACE=(CYL,(4000,1000)) on each DD (even though SyncSort will supposedly calculate secondary allocations itself)
Include DSNTYPE=LARGE on each DD to allow greater than 4G per volume (each SORTWK is restricted to 1 volume)
The aggregate of these DDs should provide sufficient work space for your sort requirements.
Each of the above options assume that sufficient vols/extents/overall-space are available in the work space pool for your shop. I’d recommend coordinating with your storage management team to ensure availability of resources
Use MAXSORT
SyncSort provides the MAXSORT capability to accommodate extremely large data sets. It sorts sections of an input file into intermediate files, and re-uses SORTWK space for subsequent sections, until the intermediate files for each section can be combined into a single output file. There are particular requirements for using MAXSORT. As referenced earlier, I don’t have access to SyncSort, so I can’t test this capability myself. You'll have to research the SyncSort doc yourself for the details of this capability.

Answer 2

Since your sort input is read from tape, sort cannot reliably calculate the sort work space needed. Try adding the SIZE=Ennnnnnn parameter to the SORT statement. nnnnnnn is the number of records to be read. E tells sort this is an estimate.

Answer 3

try calling SYNCSORT directly instead of calling this proc. This PROC only has 5 workfiles set up. It is usually best to let SYNCSORT allocate as may workfiles as needed rather than telling it how many it can use.
I believe it is the workfiles that have run out of space. Try this:
//SORT10 EXEC PGM=SYNCSORT,
// PARM=('INCORE=OFF,DYNALLOC=(SYSDA,255)',EQUALS)
//SORTIN DD DSN=FILEXYZ(0),
// DISP=SHR,DCB=BUFNO=192
//SORTOUT DD DSN=FILE2XXY,
// DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),
// DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),
// UNIT=(TAPE,2)
//SYSOUT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SORTMSG DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)
OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),
OVERLAY=(70:C' encoding="UTF-8"'))
OPTION DYNALLOC=(SYSDA,255)
/*