So how do I pass binary data using stdin to a executable command that I want to run using subprocess.run()?
The documentation is pretty vague about using the stdin for passing the data to external executable. I'm working on linux machine using python3 and I want to invoke dd of=/somefile.data bs=32 (which takes the input from stdin if I understand the man page correctly) and I have the binary data in bytearray that I want to pass to the command through stdin so that I do not have to write it to a temporary file and invoke the dd using that file as a input.
My requirement is simply to pass data that I have in bytearray to the dd command to be written to disk. What is the correct way to achieve this using the subprocess.run() and stdin?
Edit: I meant something like the following:
ba = bytearray(b"some bytes here")
#Run the dd command and pass the data from ba variable to its stdin
You can pass the output of one command to another by calling Popen directly.
file_cmd1 = <your dd command>
file_cmd2 = <command you want to pass dd output to>
proc1 = Popen(sh_split(file_cmd1), stdout=subprocess.PIPE)
proc2 = Popen(file_cmd2, [shell=True], stdin=proc1.stdout, stdout=subprocess.PIPE)
proc1.stdout.close()
This, as far as I know, will work just fine on a byte output from command 1.
In your case what you most like want to do is the following when you just want to pass data to the stdin of the process:
out = bytearray(b"Some data here")
p = subprocess.Popen(sh_split("dd of=/../../somefile.data bs=32"), stdin=subprocess.PIPE)
out = p.communicate(input=b''.join(out))[0]
print(out.decode())#Prints the output from the dd
Specifically for stdin to subprocess.run() as asked by the OP, use input as follows:
#!/usr/bin/python3
import subprocess
data = bytes("Hello, world!", "ascii")
p = subprocess.run(
"cat -", # The - means 'cat from stdin'
input=data,
# stdin=... <-- don't use this
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.decode("ascii"))
print(p.returncode)
# Hello, world!
# 0
I have a code below to write the properties of the gdg to a PS file.
//STEP10 EXEC PGM=IDCAMS
//DD1 DD DSN='GDGBASE(0)',DISP=SHR
//SYSPRINT DD DSN=GDG.VERSION.PS,
// DISP=(NEW,CATLG,DELETE),
// UNIT=WORK,AVGREC=K,
// SPACE=(108,(5,5),RLSE),
// DCB=(BUFNO=10,RECFM=FB,LRECL=108)
//SYSIN DD *
PRINT INFILE(DD1) COUNT(0) CHAR
/*
But I just need the GDG's latest version name to be written in to the PS file.
I don't think you can do this using pure JCL. As a suggestion to get you started here's a short REXX and some JCL - it's not perfect, but you can tune it to fit your specific needs.
Part 1: a short REXX-script to display the DSN of an allocated file:
/* REXX find DSN for DD */
arg mydd
x= LISTDSI(mydd file)
say sysdsname
Part 2: JCL to invoke it using batch-TSO:
//FINDDSN EXEC PGM=IKJEFT01,PARM='DSNFIND MYFILE'
//MYFILE DD DSN=MY.GDG.FILE(+0),DISP=SHR
//SYSEXEC DD DSN=PDS.CONTAINING.REXX,DISP=SHR
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD DUMMY
where DSNFIND is the membername of the given REXX-procedure.
Output in SYSTSPRT is:
MY.GDG.FILE.G0338V00
READY
END
I haven't found a quick way to suppress the READY and END - but instead of SAY you could e.g. use EXECIO to write to an other file - READY and END would still go to SYSTSPRT.
I am trying to do the following:
1) Search members of a PDS for a string.
2) Exclude the members with the above mentioned string and get a list of members.
I want to do this using a JCL or any other mainframe tool or utility.
Any help is appreciated.
I think you're looking for the ISPF Search-For utility, which can also be executed in batch.
You can also use the SRCHFOR command at an ISPF member listing screen, then sort by the prompt column which indicates which members had matches.
There are probably much better ways of doing this but this should work for you.
Copy the following JCL and replace:
your-job-car-info with whatever your job card needs
pds-to-search with the name of the PDS you want to search
temp-pds-name with a valid PDS name that won't duplicate anything you already have
string-to-search-for with the string you want to search PDS members for
The list of PDS members that do not contain string-to-search-for will be listed under DD LISTING,
which you can redirect wherever appropriate.
The first job step (SEARCH) invokes the ISPF SEARCHFOR utility, to identify all the PDS members that contain the string.
The second step (LSTPDS) invokes the TSO LISTDS utility to list all members of the PDS. The third step (REXXPGM) writes a little
REXX procedure to a temporary PDS for use in the next step. The last step (LSTMBR) runs the REXX procedure to build the LISTING dataset
and deletes the temporary files built along the way.
If one were smart enough, I think ICETOOL could probably replace the final 2 steps (but what the heck... Actually I did find a better process, see next JCL after this one)
//LOOKFOR JOB your-job-card-info
//*====================================================================
//SEARCH EXEC PGM=ISRSUPC,PARM=(SRCHCMP,'ANYC')
//NEWDD DD DSN=pds-to-search,DISP=SHR
//OUTDD DD DSN=&&SRCH,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(133,(5,5)),
// DCB=(LRECL=133,BLKSIZE=133,RECFM=FB)
//SYSIN DD *
SRCHFOR 'string-to-search-for'
/*
//*====================================================================
//LSTPDS EXEC PGM=IKJEFT01
//SYSTSPRT DD DSN=&&MBRS,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(80,(5,5)),
// DCB=(LRECL=80,BLKSIZE=80,RECFM=FB)
//SYSTSIN DD *
LISTDS 'pds-to-search' MEMBERS
/*
//REXXPDS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
ALLOCATE DSN('temp-pds-name') NEW KEEP -
DSORG(PO) RECFM(F,B) LRECL(80) -
AVGREC(U) SPACE(80,100) DIR(5) DSNTYPE(PDS)
/*
//*====================================================================
//REXXPGM EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
REPRO INFILE(REXXPROG) OUTDATASET('temp-pds-name(MBRLIST)')
/*
//REXXPROG DD *
'EXECIO * DISKR SRCH(FINIS STEM X.'
FND = ''
DO I = 1 TO X.0
IF WORD(X.I, 3) = 'STRING(S)' THEN FND = FND WORD(X.I, 1)
END
DROP X.
'EXECIO * DISKR MBRS(FINIS STEM X.'
MBRS = 0
DO I = 1 TO X.0 - 2
IF MBRS & WORDPOS(WORD(X.I, 1), FND) = 0 THEN
QUEUE WORD(X.I,1)
ELSE
IF X.I = '--MEMBERS--' THEN MBRS = 1
END
'EXECIO * DISKW LISTING(FINIS'
RETURN
/*
//*====================================================================
//LISTMBR EXEC PGM=IKJEFT01,PARM=('%MBRLIST')
//SYSEXEC DD DSN=NXB123.JUNK,DISP=(OLD,DELETE,DELETE)
//SRCH DD DSN=&&SRCH,DISP=(OLD,DELETE,DELETE)
//MBRS DD DSN=&&MBRS,DISP=(OLD,DELETE,DELETE)
//LISTING DD SYSOUT=A
//SYSTSPRT DD SYSOUT=A
//SYSTSIN DD *
/*
//
And this an even better answer... Just took a while to get there
//MYJOB JOB your-job-card-info
//SEARCH EXEC PGM=ISRSUPC,PARM=(SRCHCMP,'LNFMTO')
//NEWDD DD DSN=pds-to-search,DISP=SHR
//OUTDD DD DSN=&&SRCH,DISP=(NEW,PASS,DELETE),
// AVGREC=K,
// SPACE=(133,(5,5)),
// DCB=(LRECL=133,BLKSIZE=133,RECFM=FB)
//SYSIN DD *
SRCHFOR 'string-to-search-for'
/*
//STEP02 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTMSG DD SYSOUT=*
//SORTIN DD DSN=&&SRCH,DISP=(OLD,DELETE,DELETE)
//SORTOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=COPY -
INCLUDE COND=(30,6,CH,EQ,C' 0 ') -
OUTREC FIELDS=(1,8)
/*
The key is the LNFMTO paramter fed to ISRSUPC. This parameter tells ISRSUPC to list only those members where
the search string was not found. Exactly what was asked for. The next step just
formats the results from the previous step. The list of members that do not contain the search
string are listed in SYSOUT which can be redirected where needed.
Another approach: "punch" the whole PDS, in unsorted order, to a "flat" file; use JOINKEYS with data prepared using SS, for Sub-String, field type.
//PUNCHIT EXEC PGM=IEBPTPCH
//SYSUT1 DD DISP=SHR,DSN=your.pds
//SYSUT2 DD DSN=output.dataset,RECFM=FB,LRECL=81,otherstuffyouneed
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
PUNCH TYPORG=PO
/*
After that, a JOINKEYS step using SORT:
//FINDMISS EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTOUT DD SYSOUT=*
//SYSIN DD *
OPTION COPY
JOINKEYS F1=INA,FIELDS=(1,8,A)
JOINKEYS F2=INB,FIELDS=(82,8,A)
JOIN UNPAIRED,F1,ONLY
/*
//JNF1CNTL DD *
INCLUDE COND=(2,13,CH,EQ,C'MEMBER NAME')
INREC BUILD=(15,8)
/*
//JNF2CNTL DD *
INCLUDE COND=(2,13,CH,EQ,C'MEMBER NAME',
OR,2,72,SS,EQ,C'yoursearchvalue')
INREC IFTHEN=(WHEN=GROUP,BEGIN=(2,13,CH,EQ,C'MEMBER NAME'),
PUSH=(82:15,8)),
IFTHEN=(WHEN=(2,13,CH,EQ,C'MEMBER NAME'),
OVERLAY=(82:8X))
/*
//INA DD DISP=SHR,DSN=YOUR.PUNCHED.FILE
//INB DD DISP=SHR,DSN=YOUR.PUNCHED.FILE (yes, exactly the same one)
Then processes the punched file, reading it twice in the same step, and using JOINKEYS to "match" one extract from the file to another extract from the file.
On INA via JNF1CNTL it extracts all the member names.
On INB via JNF2CNTL it extracts all the member names, plus all data-lines which match the required search. IFTHEN=(WHEN=GROUP is then used to mark each data line that has matched with the member-name they are part of, and the member-name itself gets given a blank member-name.
The JOINKEYS then match on the member-name keys. All those from INA via JNF1CNTL which do not match are those members which do not contain the searched-for string. Both files require sorting, which happens by default for each JOINKEYS statement.
This will work with DFSORT. With SyncSort it will depend whether your installation supports JNFnCNTL files or not (you have to try it, it is not documented). If you do not have JNFnCNTL support, it would require two earlier, separate, steps to extract the data.
I can do that in php with the following code:
$dt1 = '2011-11-11 11:11:11';
$t1 = strtotime($dt1);
$dt2 = date('Y-m-d H:00:00');
$t2 = strtotime($dt2);
$tDiff = $t2 - $t1;
$hDiff = round($tDiff/3600);
$hDiff will give me the result in hours.
How do I implement the above in bash shell?
You could use date command to achieve this. man date will provide you with more details. A bash script could be something on these lines (seems to work fine on Ubuntu 10.04 bash 4.1.5):
#!/bin/bash
# Date 1
dt1="2011-11-11 11:11:11"
# Compute the seconds since epoch for date 1
t1=$(date --date="$dt1" +%s)
# Date 2 : Current date
dt2=$(date +%Y-%m-%d\ %H:%M:%S)
# Compute the seconds since epoch for date 2
t2=$(date --date="$dt2" +%s)
# Compute the difference in dates in seconds
let "tDiff=$t2-$t1"
# Compute the approximate hour difference
let "hDiff=$tDiff/3600"
echo "Approx hour diff b/w $dt1 & $dt2 = $hDiff"
Hope this helps!
I am now doing some tests of my application again corrupted files. But I found it is hard to find test files.
So I'm wondering whether there are some existing tools, which can write random/garbage bytes into a file of some format.
Basically, I need this tool to:
It writes random garbage bytes into the file.
It does not need to know the format of the file, just writing random bytes are OK for me.
It is best to write at random positions of the target file.
Batch processing is also a bonus.
Thanks.
The /dev/urandom pseudo-device, along with dd, can do this for you:
dd if=/dev/urandom of=newfile bs=1M count=10
This will create a file newfile of size 10M.
The /dev/random device will often block if there is not sufficient randomness built up, urandom will not block. If you're using the randomness for crypto-grade stuff, you can steer clear of urandom. For anything else, it should be sufficient and most likely faster.
If you want to corrupt just bits of your file (not the whole file), you can simply use the C-style random functions. Just use rnd() to figure out an offset and length n, then use it n times to grab random bytes to overwrite your file with.
The following Perl script shows how this can be done (without having to worry about compiling C code):
use strict;
use warnings;
sub corrupt ($$$$) {
# Get parameters, names should be self-explanatory.
my $filespec = shift;
my $mincount = shift;
my $maxcount = shift;
my $charset = shift;
# Work out position and size of corruption.
my #fstat = stat ($filespec);
my $size = $fstat[7];
my $count = $mincount + int (rand ($maxcount + 1 - $mincount));
my $pos = 0;
if ($count >= $size) {
$count = $size;
} else {
$pos = int (rand ($size - $count));
}
# Output for debugging purposes.
my $last = $pos + $count - 1;
print "'$filespec', $size bytes, corrupting $pos through $last\n";
# Open file, seek to position, corrupt and close.
open (my $fh, "+<$filespec") || die "Can't open $filespec: $!";
seek ($fh, $pos, 0);
while ($count-- > 0) {
my $newval = substr ($charset, int (rand (length ($charset) + 1)), 1);
print $fh $newval;
}
close ($fh);
}
# Test harness.
system ("echo =========="); #DEBUG
system ("cp base-testfile testfile"); #DEBUG
system ("cat testfile"); #DEBUG
system ("echo =========="); #DEBUG
corrupt ("testfile", 8, 16, "ABCDEFGHIJKLMNOPQRSTUVWXYZ ");
system ("echo =========="); #DEBUG
system ("cat testfile"); #DEBUG
system ("echo =========="); #DEBUG
It consists of the corrupt function that you call with a file name, minimum and maximum corruption size and a character set to draw the corruption from. The bit at the bottom is just unit testing code. Below is some sample output where you can see that a section of the file has been corrupted:
==========
this is a file with nothing in it except for lowercase
letters (and spaces and punctuation and newlines).
that will make it easy to detect corruptions from the
test program since the character range there is from
uppercase a through z.
i have to make it big enough so that the random stuff
will work nicely, which is why i am waffling on a bit.
==========
'testfile', 344 bytes, corrupting 122 through 135
==========
this is a file with nothing in it except for lowercase
letters (and spaces and punctuation and newlines).
that will make iFHCGZF VJ GZDYct corruptions from the
test program since the character range there is from
uppercase a through z.
i have to make it big enough so that the random stuff
will work nicely, which is why i am waffling on a bit.
==========
It's tested at a basic level but you may find there are edge error cases which need to be taken care of. Do with it what you will.
Just for completeness, here's another way to do it:
shred -s 10 - > my-file
Writes 10 random bytes to stdout and redirects it to a file. shred is usually used for destroying (safely overwriting) data, but it can be used to create new random files too.
So if you have already have a file that you want to fill with random data, use this:
shred my-existing-file
You could read from /dev/random:
# generate a 50MB file named `random.stuff` filled with random stuff ...
dd if=/dev/random of=random.stuff bs=1000000 count=50
You can specify the size also in a human readable way:
# generate just 2MB ...
dd if=/dev/random of=random.stuff bs=1M count=2
You can also use cat and head. Both are usually installed.
# write 1024 random bytes to my-file-to-override
cat /dev/urandom | head -c 1024 > my-file-to-override