I have a code below to write the properties of the gdg to a PS file.
//STEP10 EXEC PGM=IDCAMS
//DD1 DD DSN='GDGBASE(0)',DISP=SHR
//SYSPRINT DD DSN=GDG.VERSION.PS,
// DISP=(NEW,CATLG,DELETE),
// UNIT=WORK,AVGREC=K,
// SPACE=(108,(5,5),RLSE),
// DCB=(BUFNO=10,RECFM=FB,LRECL=108)
//SYSIN DD *
PRINT INFILE(DD1) COUNT(0) CHAR
/*
But I just need the GDG's latest version name to be written in to the PS file.
I don't think you can do this using pure JCL. As a suggestion to get you started here's a short REXX and some JCL - it's not perfect, but you can tune it to fit your specific needs.
Part 1: a short REXX-script to display the DSN of an allocated file:
/* REXX find DSN for DD */
arg mydd
x= LISTDSI(mydd file)
say sysdsname
Part 2: JCL to invoke it using batch-TSO:
//FINDDSN EXEC PGM=IKJEFT01,PARM='DSNFIND MYFILE'
//MYFILE DD DSN=MY.GDG.FILE(+0),DISP=SHR
//SYSEXEC DD DSN=PDS.CONTAINING.REXX,DISP=SHR
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD DUMMY
where DSNFIND is the membername of the given REXX-procedure.
Output in SYSTSPRT is:
MY.GDG.FILE.G0338V00
READY
END
I haven't found a quick way to suppress the READY and END - but instead of SAY you could e.g. use EXECIO to write to an other file - READY and END would still go to SYSTSPRT.
Related
So how do I pass binary data using stdin to a executable command that I want to run using subprocess.run()?
The documentation is pretty vague about using the stdin for passing the data to external executable. I'm working on linux machine using python3 and I want to invoke dd of=/somefile.data bs=32 (which takes the input from stdin if I understand the man page correctly) and I have the binary data in bytearray that I want to pass to the command through stdin so that I do not have to write it to a temporary file and invoke the dd using that file as a input.
My requirement is simply to pass data that I have in bytearray to the dd command to be written to disk. What is the correct way to achieve this using the subprocess.run() and stdin?
Edit: I meant something like the following:
ba = bytearray(b"some bytes here")
#Run the dd command and pass the data from ba variable to its stdin
You can pass the output of one command to another by calling Popen directly.
file_cmd1 = <your dd command>
file_cmd2 = <command you want to pass dd output to>
proc1 = Popen(sh_split(file_cmd1), stdout=subprocess.PIPE)
proc2 = Popen(file_cmd2, [shell=True], stdin=proc1.stdout, stdout=subprocess.PIPE)
proc1.stdout.close()
This, as far as I know, will work just fine on a byte output from command 1.
In your case what you most like want to do is the following when you just want to pass data to the stdin of the process:
out = bytearray(b"Some data here")
p = subprocess.Popen(sh_split("dd of=/../../somefile.data bs=32"), stdin=subprocess.PIPE)
out = p.communicate(input=b''.join(out))[0]
print(out.decode())#Prints the output from the dd
Specifically for stdin to subprocess.run() as asked by the OP, use input as follows:
#!/usr/bin/python3
import subprocess
data = bytes("Hello, world!", "ascii")
p = subprocess.run(
"cat -", # The - means 'cat from stdin'
input=data,
# stdin=... <-- don't use this
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.decode("ascii"))
print(p.returncode)
# Hello, world!
# 0
Would like to know how to include in a SORT step a file that has the SORTIN content, meaning that the file has been created dynamically and its content is something like this
//SORTIN DD DSN=DES.LQHISE03.ENTRADA,DISP=SHR
Step 1: Please have the following 3 datasets.
Dataset 1: Job card and the EXEC statement.
//JOBNAME JOB ('ACCOUNT INFORMATION'),
// JOBCARD,CLASS=2,MSGCLASS=H,NOTIFY=&SYSUID
//STEP1 EXEC PGM=SORT
Dataset 2: SORTIN content - This should be the dynamically created file.
//SORTIN DD DSN=DES.LQHISE03.ENTRADA,DISP=SHR
Dataset 3: Rest of the content
//SORTOUT DD DSN=OUTPUT.DATASET.NAME,
// DISP=(,CATLG,DELETE),
// SPACE=(CYL,(100,0),RLSE),
// DCB=(LRECL=80,RECFM=FB,BLKSIZE=0)
//SYSOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=COPY
/*
//IDCAMS EXEC PGM=IDCAMS,COND=(0,LT,STEP1)
//SYSPRINT DD SYSOUT=*
//DD1 DD DISP=SHR,DSN=YOUR.PDS
DELETE 'YOUR.PDS(MEMBER)' FILE(DD1)
/*
Step 2: Concatenate the datasets.
//STEP2 EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSUT1 DD DSN=Dataset 1,DISP=SHR
// DD DSN=Dataset 2,DISP=SHR
// DD DSN=Dataset 3,DISP=SHR
//SYSUT2 DD DSN=Sort Step,
// DISP=(,CATLG,DELETE),
// SPACE=(CYL,(100,0),RLSE),
// DCB=(LRECL=80,RECFM=FB,BLKSIZE=0)
//SYSIN DD DUMMY
Step 3: Have IEBGENER to write the Sort Step (output from STEP 2) as a member in a PDS.
//STEP3 EXEC PGM=IEBGENER,COND=(0,LT,STEP2)
//SYSUT1 DD DSN=*.STEP2.SYSUT2,DISP=SHR
//SYSUT2 DD DSN=YOUR.PDS(MEMBER),DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
You will now have the complete JCL in a member inside your PDS (Note: The member inside the PDS will be deleted after the completion of the SORT job). You just need to submit the member. You can make use of the IKJEFT01 utility to run a TSO function, LISTDS. This function displays data set attributes. If a member exists, RC 0 is issued & you may add code to SUBMIT the member, else you may issue some return code (to signify non availability of the member). More details here
I am trying to do the following:
1) Search members of a PDS for a string.
2) Exclude the members with the above mentioned string and get a list of members.
I want to do this using a JCL or any other mainframe tool or utility.
Any help is appreciated.
I think you're looking for the ISPF Search-For utility, which can also be executed in batch.
You can also use the SRCHFOR command at an ISPF member listing screen, then sort by the prompt column which indicates which members had matches.
There are probably much better ways of doing this but this should work for you.
Copy the following JCL and replace:
your-job-car-info with whatever your job card needs
pds-to-search with the name of the PDS you want to search
temp-pds-name with a valid PDS name that won't duplicate anything you already have
string-to-search-for with the string you want to search PDS members for
The list of PDS members that do not contain string-to-search-for will be listed under DD LISTING,
which you can redirect wherever appropriate.
The first job step (SEARCH) invokes the ISPF SEARCHFOR utility, to identify all the PDS members that contain the string.
The second step (LSTPDS) invokes the TSO LISTDS utility to list all members of the PDS. The third step (REXXPGM) writes a little
REXX procedure to a temporary PDS for use in the next step. The last step (LSTMBR) runs the REXX procedure to build the LISTING dataset
and deletes the temporary files built along the way.
If one were smart enough, I think ICETOOL could probably replace the final 2 steps (but what the heck... Actually I did find a better process, see next JCL after this one)
//LOOKFOR JOB your-job-card-info
//*====================================================================
//SEARCH EXEC PGM=ISRSUPC,PARM=(SRCHCMP,'ANYC')
//NEWDD DD DSN=pds-to-search,DISP=SHR
//OUTDD DD DSN=&&SRCH,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(133,(5,5)),
// DCB=(LRECL=133,BLKSIZE=133,RECFM=FB)
//SYSIN DD *
SRCHFOR 'string-to-search-for'
/*
//*====================================================================
//LSTPDS EXEC PGM=IKJEFT01
//SYSTSPRT DD DSN=&&MBRS,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(80,(5,5)),
// DCB=(LRECL=80,BLKSIZE=80,RECFM=FB)
//SYSTSIN DD *
LISTDS 'pds-to-search' MEMBERS
/*
//REXXPDS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
ALLOCATE DSN('temp-pds-name') NEW KEEP -
DSORG(PO) RECFM(F,B) LRECL(80) -
AVGREC(U) SPACE(80,100) DIR(5) DSNTYPE(PDS)
/*
//*====================================================================
//REXXPGM EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
REPRO INFILE(REXXPROG) OUTDATASET('temp-pds-name(MBRLIST)')
/*
//REXXPROG DD *
'EXECIO * DISKR SRCH(FINIS STEM X.'
FND = ''
DO I = 1 TO X.0
IF WORD(X.I, 3) = 'STRING(S)' THEN FND = FND WORD(X.I, 1)
END
DROP X.
'EXECIO * DISKR MBRS(FINIS STEM X.'
MBRS = 0
DO I = 1 TO X.0 - 2
IF MBRS & WORDPOS(WORD(X.I, 1), FND) = 0 THEN
QUEUE WORD(X.I,1)
ELSE
IF X.I = '--MEMBERS--' THEN MBRS = 1
END
'EXECIO * DISKW LISTING(FINIS'
RETURN
/*
//*====================================================================
//LISTMBR EXEC PGM=IKJEFT01,PARM=('%MBRLIST')
//SYSEXEC DD DSN=NXB123.JUNK,DISP=(OLD,DELETE,DELETE)
//SRCH DD DSN=&&SRCH,DISP=(OLD,DELETE,DELETE)
//MBRS DD DSN=&&MBRS,DISP=(OLD,DELETE,DELETE)
//LISTING DD SYSOUT=A
//SYSTSPRT DD SYSOUT=A
//SYSTSIN DD *
/*
//
And this an even better answer... Just took a while to get there
//MYJOB JOB your-job-card-info
//SEARCH EXEC PGM=ISRSUPC,PARM=(SRCHCMP,'LNFMTO')
//NEWDD DD DSN=pds-to-search,DISP=SHR
//OUTDD DD DSN=&&SRCH,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(133,(5,5)),
// DCB=(LRECL=133,BLKSIZE=133,RECFM=FB)
//SYSIN DD *
SRCHFOR 'string-to-search-for'
/*
//STEP02 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTMSG DD SYSOUT=*
//SORTIN DD DSN=&&SRCH,DISP=(OLD,DELETE,DELETE)
//SORTOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=COPY -
INCLUDE COND=(30,6,CH,EQ,C' 0 ') -
OUTREC FIELDS=(1,8)
/*
The key is the LNFMTO paramter fed to ISRSUPC. This parameter tells ISRSUPC to list only those members where
the search string was not found. Exactly what was asked for. The next step just
formats the results from the previous step. The list of members that do not contain the search
string are listed in SYSOUT which can be redirected where needed.
Another approach: "punch" the whole PDS, in unsorted order, to a "flat" file; use JOINKEYS with data prepared using SS, for Sub-String, field type.
//PUNCHIT EXEC PGM=IEBPTPCH
//SYSUT1 DD DISP=SHR,DSN=your.pds
//SYSUT2 DD DSN=output.dataset,RECFM=FB,LRECL=81,otherstuffyouneed
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
PUNCH TYPORG=PO
/*
After that, a JOINKEYS step using SORT:
//FINDMISS EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTOUT DD SYSOUT=*
//SYSIN DD *
OPTION COPY
JOINKEYS F1=INA,FIELDS=(1,8,A)
JOINKEYS F2=INB,FIELDS=(82,8,A)
JOIN UNPAIRED,F1,ONLY
/*
//JNF1CNTL DD *
INCLUDE COND=(2,13,CH,EQ,C'MEMBER NAME')
INREC BUILD=(15,8)
/*
//JNF2CNTL DD *
INCLUDE COND=(2,13,CH,EQ,C'MEMBER NAME',
OR,2,72,SS,EQ,C'yoursearchvalue')
INREC IFTHEN=(WHEN=GROUP,BEGIN=(2,13,CH,EQ,C'MEMBER NAME'),
PUSH=(82:15,8)),
IFTHEN=(WHEN=(2,13,CH,EQ,C'MEMBER NAME'),
OVERLAY=(82:8X))
/*
//INA DD DISP=SHR,DSN=YOUR.PUNCHED.FILE
//INB DD DISP=SHR,DSN=YOUR.PUNCHED.FILE (yes, exactly the same one)
Then processes the punched file, reading it twice in the same step, and using JOINKEYS to "match" one extract from the file to another extract from the file.
On INA via JNF1CNTL it extracts all the member names.
On INB via JNF2CNTL it extracts all the member names, plus all data-lines which match the required search. IFTHEN=(WHEN=GROUP is then used to mark each data line that has matched with the member-name they are part of, and the member-name itself gets given a blank member-name.
The JOINKEYS then match on the member-name keys. All those from INA via JNF1CNTL which do not match are those members which do not contain the searched-for string. Both files require sorting, which happens by default for each JOINKEYS statement.
This will work with DFSORT. With SyncSort it will depend whether your installation supports JNFnCNTL files or not (you have to try it, it is not documented). If you do not have JNFnCNTL support, it would require two earlier, separate, steps to extract the data.
If I look at /proc/6945/stat then I get a series of numbers, one of which is the number of CPU-centiseconds for which the process has been running.
But I'm running these processes on heavily-loaded boxes, and what I'm interested in is the clock-time when the job will finish, for which I want to know the clock-time that it started.
The timestamps on files in /proc/6945 look to be in the right sort of range but I can't find a particular file which consistently has the right clock-time on it.
As always I can't modify the process.
Timestamps of the directories in /proc are useless.
I was advised to look at 'man proc'; this says that /proc/$PID/stat field 21 records the start-time of the process in kernel jiffies since boot ... so:
open A,"< /proc/stat"; while (<A>) { if (/^btime ([0-9]*)/) { $btime = $1 } }
to obtain the boot time, then
my #statl = split " ",`cat /proc/$i/stat`;
$starttime_jiffies = $statl[21];
$starttime_ut = $btime + $starttime_jiffies / $jiffies_per_second;
$cputime = time-$starttime_ut
but I set $jiffies_per_second to 100 because I don't know how to ask the kernel for its value from perl.
I have a project on github that does this in perl. You can find it here:
https://github.com/cormander/psj
The code you're wanting is in lib/Proc/PID.pm, and here is the snippit (with comments removed):
use POSIX qw(ceil sysconf _SC_CLK_TCK);
sub _start_time {
my $pid = shift->pid;
my $tickspersec = sysconf(_SC_CLK_TCK);
my ($secs_since_boot) = split /\./, file_read("/proc/uptime");
$secs_since_boot *= $tickspersec;
my $start_time = (split / /, file_read("/proc/$pid/stat"))[21];
return ceil(time() - (($secs_since_boot - $start_time) / $tickspersec));
}
Beware the non-standard code function file_read in here, but that should be pretty straight forward.
Use the creation timestamp of the /proc/6945 directory (or whatever PID), rather than looking at the files it contains. For example:
ls -ld /proc/6945
Bash command to get the start date of some process:
date -d #$(cat /proc/PID/stat | awk "{printf \"%.0f\", $(grep btime /proc/stat | cut -d ' ' -f 2)+\$22/$(getconf CLK_TCK);}")
I am now doing some tests of my application again corrupted files. But I found it is hard to find test files.
So I'm wondering whether there are some existing tools, which can write random/garbage bytes into a file of some format.
Basically, I need this tool to:
It writes random garbage bytes into the file.
It does not need to know the format of the file, just writing random bytes are OK for me.
It is best to write at random positions of the target file.
Batch processing is also a bonus.
Thanks.
The /dev/urandom pseudo-device, along with dd, can do this for you:
dd if=/dev/urandom of=newfile bs=1M count=10
This will create a file newfile of size 10M.
The /dev/random device will often block if there is not sufficient randomness built up, urandom will not block. If you're using the randomness for crypto-grade stuff, you can steer clear of urandom. For anything else, it should be sufficient and most likely faster.
If you want to corrupt just bits of your file (not the whole file), you can simply use the C-style random functions. Just use rnd() to figure out an offset and length n, then use it n times to grab random bytes to overwrite your file with.
The following Perl script shows how this can be done (without having to worry about compiling C code):
use strict;
use warnings;
sub corrupt ($$$$) {
# Get parameters, names should be self-explanatory.
my $filespec = shift;
my $mincount = shift;
my $maxcount = shift;
my $charset = shift;
# Work out position and size of corruption.
my #fstat = stat ($filespec);
my $size = $fstat[7];
my $count = $mincount + int (rand ($maxcount + 1 - $mincount));
my $pos = 0;
if ($count >= $size) {
$count = $size;
} else {
$pos = int (rand ($size - $count));
}
# Output for debugging purposes.
my $last = $pos + $count - 1;
print "'$filespec', $size bytes, corrupting $pos through $last\n";
# Open file, seek to position, corrupt and close.
open (my $fh, "+<$filespec") || die "Can't open $filespec: $!";
seek ($fh, $pos, 0);
while ($count-- > 0) {
my $newval = substr ($charset, int (rand (length ($charset) + 1)), 1);
print $fh $newval;
}
close ($fh);
}
# Test harness.
system ("echo =========="); #DEBUG
system ("cp base-testfile testfile"); #DEBUG
system ("cat testfile"); #DEBUG
system ("echo =========="); #DEBUG
corrupt ("testfile", 8, 16, "ABCDEFGHIJKLMNOPQRSTUVWXYZ ");
system ("echo =========="); #DEBUG
system ("cat testfile"); #DEBUG
system ("echo =========="); #DEBUG
It consists of the corrupt function that you call with a file name, minimum and maximum corruption size and a character set to draw the corruption from. The bit at the bottom is just unit testing code. Below is some sample output where you can see that a section of the file has been corrupted:
==========
this is a file with nothing in it except for lowercase
letters (and spaces and punctuation and newlines).
that will make it easy to detect corruptions from the
test program since the character range there is from
uppercase a through z.
i have to make it big enough so that the random stuff
will work nicely, which is why i am waffling on a bit.
==========
'testfile', 344 bytes, corrupting 122 through 135
==========
this is a file with nothing in it except for lowercase
letters (and spaces and punctuation and newlines).
that will make iFHCGZF VJ GZDYct corruptions from the
test program since the character range there is from
uppercase a through z.
i have to make it big enough so that the random stuff
will work nicely, which is why i am waffling on a bit.
==========
It's tested at a basic level but you may find there are edge error cases which need to be taken care of. Do with it what you will.
Just for completeness, here's another way to do it:
shred -s 10 - > my-file
Writes 10 random bytes to stdout and redirects it to a file. shred is usually used for destroying (safely overwriting) data, but it can be used to create new random files too.
So if you have already have a file that you want to fill with random data, use this:
shred my-existing-file
You could read from /dev/random:
# generate a 50MB file named `random.stuff` filled with random stuff ...
dd if=/dev/random of=random.stuff bs=1000000 count=50
You can specify the size also in a human readable way:
# generate just 2MB ...
dd if=/dev/random of=random.stuff bs=1M count=2
You can also use cat and head. Both are usually installed.
# write 1024 random bytes to my-file-to-override
cat /dev/urandom | head -c 1024 > my-file-to-override