I have no idea if my Python is on the right track here, but with this example, I need to combine both r and f-strings whilst escaping a handful of characters in a particularly ugly command.
import subprocess
cust = input("Cust: ")
subprocess.run(r"""pd service:list -j | grep -i {cust} | awk '/name/ {print $0} /description/ {print $0} /summary/ {print $0 "\n"}' >> services_detailed.txt""".format(cust=cust), shell=True)
I have tried a few different methods, but this is the closest I have come to getting one to work.
A single r-string works absolutely fine (if I don't need to take user input), but when I need to take user input it becomes a problem.
Python syntax looks fine in VSCode but when running it spits out:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'print $0'
This doesn't really send me anywhere and I am not sure if I should be continuing with finding a way to get this string to work or going back to the drawing board.
I figured I would ask the question in case anyone else falls down the 'rf-string' rabbit hole.
Related
I was writing a python code using subprocess module and I got stuck in this situation where I need to use pipes to pass a result of a commnad to another to obtain specific data I need.
However, this also can be achieved through pure Python code.
Ex)
from subprocess import Popen
cmd_result = Popen('ls -l ./ | awk -F " " \'{if ($5 > 10000) print $0}\'' | grep $USER', shell=True).communicate().split('\n')
Or
cmd_result = Popen('ls -l ./', shell=True).communicate().split('\n')
result_lst = []
for result in cmd_result:
result_items = result.split()
if int(result_item[4]) > 10000 and result_item[2] == "user_name":
result_lst.append(result)
And I am wondering which method is better than the other in efficiency-wise.
I found that the one with pure python code is slower than the one with pipelines, but not sure if that means using pipes is more efficient.
Thank you in advance.
The absolutely best solution to this is to avoid using a subprocess at all.
import os
myuid = os.getuid()
for file in os.scandir("."):
st = os.stat(file)
if st.st_size > 10000 and st.st_uid == myuid:
print(file)
In general, if you want to run and capture the output of a command, the simplest by far is subprocess.check_output; but really, don't parse ls output, and, of course, try to avoid superfluous subprocesses like useless greps if efficiency is important.
files = subprocess.check_output(
"""ls -l . | awk -v me="$USER" '$5 > 10000 && $2 == me { print $9 }'""",
text=True, shell=True)
This has several other problems; $4 could contain spaces (it does, on my system) and $9 could contain just the beginning of the file name if it contains spaces.
If you need to run a process which could produce a lot of output concurrently and fetch its output as it arrives, not when the process has finished, the Stack Overflow subprocess tag info page has a couple of links to questions about how to do that; I am guessing it is not worth the effort for this simple task you are asking about, though it could be useful for more complex ones.
Need help with this, appreciate if someone can try to suggest a fix.
$ echo "print('This works fine')"|python3
This works fine.
But:
$ echo "input('This is NOT working! ')"|python3
This is the output received:
This is NOT working! Traceback (most recent call last):
File "<stdin>", line 1, in <module>
EOFError: EOF when reading a line
It seems you just want to execute python commands inline from bash. For this task you can use this:
python3 -c "input('This also works)"
EDIT:
If you want to use the pipe, you can just take the input from bash and pipe it to python, like this:
#!/bin/bash
read -p "Enter your name : " name
echo "x='$name';print('My name is', x)"|python3
I'm porting some code to Python 3 and stumbled upon an invocation of numpy.fromfile(src) which works just fine when src is a handle of a real file but fails when src is sys.stdin. I boiled down the issue to the following two commands that will let you see how the problem only exists in Python 3 but not in Python 2:
$ echo 1 2 3 | python -c 'import numpy,sys; print(numpy.fromfile(sys.stdin, dtype=int, sep=" "))'
[1 2 3]
$ echo 1 2 3 | python3 -c 'import numpy,sys; print(numpy.fromfile(sys.stdin, dtype=int, sep=" "))'
Traceback (most recent call last):
File "<string>", line 1, in <module>
OSError: obtaining file position failed
When I look at /usr/lib/python2.7/dist-packages/numpy/core/records.py then it seems that even in the Python2 version of numpy it says for fromfile:
The file object must support random access (i.e. it must have tell and seek methods).
So I guess the fact that the above works with Python2 was never meant to be supported. Still the code I want to port to Python3 relied on this functionality.
I tried reading sys.stdin into an io.BytesIO but this will error out with io.UnsupportedOperation: fileno.
In my hopes that others also stumbled across this problem, I wanted to ask for an easy drop-in replacement of numpy.fromfile() which allows reading from stdin even on Python3. Certainly other projects already had to implement a solution to this? I wasn't able to find one.
You could try reading bytes from stdin like this:
data = sys.stdin.buffer.read()
And then using numpy.frombuffer to read it. (You should also provide the datatype)
arr = np.frombuffer(data, dtype=np.float)
If you are reading text data, you can do this:
data = sys.stdin.read()
arr = numpy.fromstring(data, sep=" ")
When you don't provide sep it acts in some legacy mode.
In linux, I am trying to select a variable from a specific column and row of CSV file and then use this variable as the end of a file location hierarchy. When I type the following into a bash terminal window, it seems to work by outputting the variable in correct row and column on screen.
awk -F "," 'FNR == 2 {print $8}' /sdata/images/projects/ASD_SSD/1/ruths_data/ruth/imaging\ study/imaging\ study\ working/delete2.csv
However, I am trying to go do the following substitution within a script, this fails to work...
r=2
c=8
s=awk -F "," 'FNR == $r {print $c}' /sdata/images/projects/ASD_SSD/1/ruths_data/ruth/imaging\ study/imaging\ study\ working/delete2.csv
I then try to use the s output as the end of a hierarchy file location. For example, /home/ork/js/s*
I keep getting the following error, so this looks like it's not creating the s variable and then not inserting it into the actual file location.
omitting directory `/home/ork/js/'
I have spent a few hours trying to figure out what is preventing this from working and am a new user (so I am sure it is something simple, sorry).
I hope I was clear enough, please let me know if this requires further clarification.
This is a common question here. The single quotes are protecting the variables from the shell, so they never get expanded. Also command substitution is needed when assigning to variable s. One way to do it would be:
s=$(awk -F, 'FNR==r{print c}' r="$r" c="$c" file)
I have a 2GB text file on my linux box that I'm trying to import into my database.
The problem I'm having is that the script that is processing this rdf file is choking on one line:
mismatched tag at line 25462599, column 2, byte 1455502679:
<link r:resource="http://www.epuron.de/"/>
<link r:resource="http://www.oekoworld.com/"/>
</Topic>
=^
I want to replace the </Topic> with </Line>. I can't do a search/replace on all lines but I do have the line number so I'm hoping theres some easy way to just replace that one line with the new text.
Any ideas/suggestions?
sed -i yourfile.xml -e '25462599s!</Topic>!</Line>!'
sed -i '25462599 s|</Topic>|</Line>|' nameoffile.txt
The tool for editing text files in Unix, is called ed (as opposed to sed, which as the name implies is a stream editor).
ed was once intended as an interactive editor, but it can also easily scripted. The way ed works, is that all commands take an address parameter. The way to address a specific line is just the line number, and the way to change the addressed line(s) is the s command, which takes the same regexp that sed would. So, to change the 42nd line, you would write something like 42s/old/new/.
Here's the entire command:
FILENAME=/path/to/whereever
LINENUMBER=25462599
ed -- "${FILENAME}" <<-HERE
${LINENUMBER}s!</Topic>!</Line>!
w
q
HERE
The advantage of this is that ed is standardized, while the -i flag to sed is a proprietary GNU extension that is not available on a lot of systems.
Use "head" to get the first 25462598 lines and use "tail" to get the remaining lines (starting at 25462601). Though... for a 2GB file this will likely take a while.
Also are you sure the problem is just with that line and not somewhere previous (ie. the error looks like an XML parse error which might mean the actual problem is someplace else).
My shell script:
#!/bin/bash
awk -v line=$1 -v new_content="$2" '{
if (NR == line) {
print new_content;
} else {
print $0;
}
}' $3
Arguments:
first: line number you want change
second: text you want instead original line contents
third: file name
This script prints output to stdout then you need to redirect. Example:
./script.sh 5 "New fifth line text!" file.txt
You can improve it, for example, by taking care that all your arguments has expected values.