I have a command that have the structure :
xrdcp "root://server/file?authz=ENVELOPE&Param1=Val1" local_file_path
The problem is that ENVELOPE in text that should be unquoted in command line
and it contains a lot of new-lines
I cannot use repr as it will replace new-line with \n
Moreover subprocess seems to automatically use repr on the items from the list arguments
In bash this command is usually run with
xrdcp "root://server/file?authz=$(<ENVELOPE)&Param1=Val1" local_file
So, is there a way to run a command while keeping the new lines in the arguments?
Thank you!
Later Edit:
my actual code is :
envelope = server['envelope']
complete_url = "\"" + server['url'] + "?" + "authz=" + "{}".format(server['envelope']) + xrdcp_args + "\""
xrd_copy_list = []
xrd_copy_list.extend(xrdcp_cmd_list)
xrd_copy_list.append(complete_url)
xrd_copy_list.append(dst_final_path_str)
xrd_job = subprocess.Popen(xrd_copy_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = xrd_job.communicate()
print(stdout)
print(stderr)
Related
Good evening everyone, this is the first time I post here.
I have many codes I need to run in CMD but I'm not good at coding with CMD, so I want to use python to modify the code that I have to repeat often
for example
temp1 = ["C:/dir1", "C:/dir2", "C:/dir3"]
temp2 = ["1", "2", "3"]
for i, p in zip(temp1, temp2):
os.system('7z -a "{}" C:/target/output"{}" -m5'.format(i, p))
With my code, I also need to see the Shell prompt, because some of the codes I do show a very long terminal data output that I want to monitor actively, I prefer to see it in the python terminal as an output, or print or something similar.
Thanks in advance, it is my first post hope it's clear.
os.system() implicitly uses cmd.exe on Windows (by passing the specified command line to cmd /c) and implicitly passes the command's output through to stdout.
Therefore, your command should work in principle, though I recommend double-quoting the target path in full ("C:/target/output{}"), not just the substituted part (C:/target/output"{}")[1]:
import os
temp1 = ["C:/dir1", "C:/dir2", "C:/dir3"]
temp2 = ["1", "2", "3"]
for i, p in zip(temp1, temp2):
os.system('7z -a "{}" "C:/target/output{}" -m5'.format(i, p))
If you're running this code from a GUI script (a script invoked via pythonw.exe / pyw.exe), the console window created for each os.system() call automatically closes when the command finishes, which doesn't allow you to inspect the output after the fact. To address this, you have two options:
Append & pause to the command, which waits for keypress before closing the console window:
import os
temp1 = ["C:/dir1", "C:/dir2", "C:/dir3"]
temp2 = ["1", "2", "3"]
for i, p in zip(temp1, temp2):
os.system('7z -a "{}" "C:/target/output{}" -m5 & pause'.format(i, p))
Invoke your command via cmd /k, which starts an interactive cmd.exe session that stays open after the command finishes; note that each such session then blocks further execution of your Python script until you manually close the console window in which the persistent cmd.exe session runs.
import os
temp1 = ["C:/dir1", "C:/dir2", "C:/dir3"]
temp2 = ["1", "2", "3"]
for i, p in zip(temp1, temp2):
os.system('cmd /k " 7z -a "{}" "C:/target/output{}" -m5 "'.format(i, p))
Note the unusual quoting (an outer "..." string in which embedded " aren't escaped), which is what cmd.exe supports, however.
[1] Typically, this shouldn't make a difference, but you never know how programs parse their command line on Windows, and double-quoting an argument in full is more likely to work.
I am trying to call a python script from another using subprocess.Popen(). It works perfectly when I debug the program line-by-line, however when I run the program normally I get the following error:
C:\Python38\python.exe: can't open file '"<filepath>\thumbs.py"': [Errno 22] Invalid argument
I'm stumped as to what the issue is as it works without fault when the program is being debugged line-by-line so not sure what changes exactly when the program is run normally.
I am trying to pass a set of arguments into the subprocesses, and am then parsing these using argparse in the child process.
This is how I am calling the process:
cmd = ['python', "\"" + pythonPath + "thumbs.py\"", '-d', "\"" + outputdb + "\"", '-c', "\"" + path + cache + "\""]
subprocess.Popen(cmd).wait()
And here is how I am parsing the arguments in the child process:
if __name__ == '__main__':
global session
args = None
parser = argparse.ArgumentParser()
parser.add_argument("-d", "--database", action="store", dest="dbname", help="Database to output results to", required=True)
parser.add_argument("-c", "--cache", action="store", dest="cachefile", help="Cache file to scrape.", required=True)
while args is None:
args = parser.parse_args()
Can anyone spot what I'm missing?
I've managed to fix this issue by creating my command and arguments as a string first and then using shlex.Split() to split it into an arguments list.
cmdStr = "python \"" + pythonPath + "thumbs.py\" -d \"" + outputdb + "\" -c \"" + path + cache + "\""
cmd = shlex.split(cmdStr)
subprocess.Popen(cmd).wait()
Further info here: https://docs.python.org/3.4/library/subprocess.html#popen-constructor
When in Powershell I can run the following command
.\exiftool.exe '-GPSLatitude*=56.9359839838' '-GPSLongitude*=-4.4651045874' DSC01008.JPG '-overwrite_original_in_place'
This works just fine, and the placement of single quotes around those params are required. I went through various iterations of different placements and the above is the only way I could get it to work.
My issue is -> That I'm trying to replace those values with programmatic values. As in something like the following.
$lat = 56.9359839838
$lon = -4.4651045874
$fileName = 'DSC01008.JPG'
.\exiftool.exe -GPSLatitude*=$lat -GPSLongitude*=$lon $fileName '-overwrite_original_in_place'
I've gone through numerous attempts with single/backtick/double quotes, trying to concatenate and anything else I can think of - but that magic format hasn't appeared!
Any thoughts on what I am doing wrong?
As an example I thought this was really work, yet didn't. But it matches up with the command that does when it's hard coded.
EDIT/Update -
The * are required inside the command, otherwise it doesn't work. They are used to get around not passing in reference locators.
If you were to run these commands in PS, then these are the errors that you get.
cmd ->
.\exiftool.exe "-GPSLatitude*=$lat" "-GPSLongtitude*=$lon" $FileName
error ->
No Error, but the file does not get any GPS tags. Nothing actually modified.
cmd ->
$combLat = "`'-GPSLatitude*=$lat`'" # + "
$combLon = "`'-GPSLongitude*=$lon`'" # + "'"
$combined = "$combLat $combLon $fileName"
# $combined will output the following: '-GPSLatitude*=-3.4651045874' '-GPSLongitude*=55.9359839838' 'E:\CameraAddGPS\TestFolder\DSC01010.JPG'
.\exiftool.exe $combined
error ->
.\exiftool.exe : Wildcards don't work in the directory specification
At E:\CameraAddGPS\TestFolder\demo.ps1:25 char:1
+ .\exiftool.exe $combined
+ ~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Wildcards don't...y specification:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
No matching files
Update 2 -
This will work, but I do not want to update the image twice.
$combLat = '-GPSLatitude*=' + $lat
$combLon = '-GPSLongitude*=' + $lon
.\exiftool.exe $combLat $fileName '-overwrite_original_in_place'
.\exiftool.exe $combLon $fileName '-overwrite_original_in_place'
The problem is that the latitude and longtitude argumets are "attached" directly to the -GPSLatitute* and -GPSLongtitude* parameter names after the = sign. (The - initial character prevents PowerShell from invoking variable expansion - see GitHub issue #14587.)
One way to work around this is to wrap the -GPSLatitude* and -GPSLongtitude* parameters in " quotes; e.g.
.\exiftool.exe "-GPSLatitude*=$lat" "-GPSLongtitude*=$lon" $FileName -overwrite_original_in_place
Another way is to prefix the - character with a backquote (`); e.g.:
.\exiftool.exe `-GPSLatitude*=$lat `-GPSLongitude*=$lon $fileName -overwrite_original_in_place
I prefer the first variation as it seems more readable to me.
Aside #1: There's no need for the single ' quotes around that last parameter. They don't hurt, but they don't do anything, either.
Aside #2: You can see the actual command line that PowerShell is running if you prefix your command with the getargs command available here:
https://github.com/Bill-Stewart/getargs/releases
The getargs utility outputs the actual command line without any parsing or interpretation and will show the actual command that PowerShell will run.
When I run sqlplus in Python using subprocess I get no output when there are SQL errors, or update or insert statements returning number of rows updated or inserted. When I run select statements with no errors I do get the output.
Here is my code:
This creates a string with newlines that are then written to process.stdin.write()
def write_sql_string(process, args):
sql_commands = ''
sql_commands += "WHENEVER SQLERROR EXIT SQL.SQLCODE;\n"
sql_line = '#' + args.sql_file_name
if crs_debug:
print('DEBUG: ' + 'sys.argv', ' '.join(sys.argv))
if len(args.sql_args) > 0:
len_argv = len(args.sql_args)
sql_commands += "SET VERIFY OFF\n"
for i in range(0, len_argv):
sql_line += ' "' + args.sql_args[i] + '"'
sql_commands += sql_line + "\n"
# if prod_env:
sql_commands += "exit;\n"
if crs_debug:
print('DEBUG: ' + 'sql_line: ' + sql_line)
process.stdin.write(sql_commands)
This code executes the SQL commands
def execute_sql_file(username, dbpass, args):
db_conn_str = username + '/' + dbpass + '#' + args.dbname
# '-S', - Silent
sqlplus_cmd = ['sqlplus', '-S', '-L', db_conn_str]
if crs_debug:
print('DEBUG: ' + ' '.join(sqlplus_cmd))
process = subprocess.Popen(sqlplus_cmd,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
write_sql_string(process, args)
stdout, stderr = process.communicate()
# Get return code of sql query
stdout_lines = stdout.split("\n")
print('STDOUT')
for line in stdout_lines:
line = line.rstrip()
print(line)
stderr_lines = stderr.split("\n")
print('STDERR')
for line in stderr_lines:
line = line.rstrip()
print(line)
sqlplus_rc = process.poll()
# Check if sqlplus returned an error
if sqlplus_rc != 0:
print("FAILURE in " + script_name + " in connecting to Oracle exit code: " + str(sqlplus_rc))
print(stderr_data)
sys.exit(sqlplus_rc)
When I run I run my code for a SQL file that requires parameters, if I run it with missing parameters I get no output. If I run it with parameters I get the correct output.
Here is an example SQL file sel_dual.sql:
SELECT 'THIS IS TEXT &1 &2' FROM dual;
As an example command line:
run_sql_file.py dbname sql_file [arg1]...[argn]
If I run the script with
run_sql_file.py dbname sel_dual.py
I get no output, even though it should ask for a parameter and give other error output.
If I run the script with
run_sql_file.py dbname sel_dual.py Seth F
I get the proper output:
'THISISTEXTSETHF'
----------------------------------------------------------------------------
THIS IS TEXT Seth F
The args referred to is the result of processing args with the argparse module:
parser = argparse.ArgumentParser(description='Run a SQL file with ptional arguments using SQLPlus')
parser.add_argument('dbname', help='db (environment) name')
parser.add_argument('sql_file_name', help='sql file')
parser.add_argument('sql_args', nargs='*', help='arguments for sql file')
args = parser.parse_args()
Does anybody know what could be causing this? I've omitted the rest of the script since it basically gets command arguments and validates that the SQL file exists.
I am running sqlplus version Release 12.1.0.2.0 Production. I am running Python version 3.7.6. I am running on Linux (not sure what version). The kernel release is 4.1.12-124.28.5.el7uek.x86_64.
I am using spawn to execute latexmk with child_process.spawn command, args, options where command = 'latexmk' and the args array is
["-interaction=nonstopmode", "-f", "-cd", "-pdf", "-synctex=1", "-file-line-error", "-shell-escape", "-xelatex", "-outdir="/Users/user/Documents/path with space/.auxdir"", ""/Users/user/Documents/path with space/main.tex""]
The options just set some environment variables. I get the following from stderr:
stderr: mkdir ": Permission denied at /usr/texbin/latexmk line 1889.
latexmk seems to be trying to search for a path that contains the ". The quotes are necessary in case of whitespace in the path. What can I do to solve this?
EDIT:
To be clear, I need to populate outdir and the filePath like so:
args.push("-outdir=\"#{outdir}\"")
args.push("\"#{filePath}\"")
Problem is that "\'#{}\'" gives me the literal strings.
For a given program, I needed to have two cases in which I would use it. One when it would return a buffer and the other when it returned a stream. From the nodejs documentation, exec is used for the former while spawn is used for the latter. I wanted to share the same arguments except for maybe and extra arg_i thrown in there for spawn. The issue I faced was finding a solution that handled whitespace. The following resolves the issue:
args= ["arg1", "arg2", "...", "arg_i", "...", "arg_n", "path /with /space"]
For exec:
escapedArgs = (item.replace(' ', '\\ ') for item in args)
command = "program #{escapedArgs.join(' ')}"
proc = child_process.exec command, options, (error, stdout, stderr)
For spawn, args does not need to be modified:
proc = child_process.spawn command, args, options