I need to modify a config file using python. The file has a format similar to
property_one = 0
property_two = 5
i.e there aren't any section names.
Python's configparser module doesn't support sectionless files, but I can use it to load them easily anyway using the trick from here: https://stackoverflow.com/a/26859985/11637934
parser = ConfigParser()
with open("foo.conf") as lines:
lines = chain(("[top]",), lines) # This line does the trick.
parser.read_file(lines)
The problem is, I can't find a clean way to write the parser back to a file without the section header. The best solution I have at the moment is to write the parser to a StringIO buffer, skip the first line and then write it to a file:
with open('foo.conf', 'w') as config_file, io.StringIO() as buffer:
parser.write(buffer)
buffer.seek(0)
buffer.readline()
shutil.copyfileobj(buffer, config_file)
It works but it's a little ugly and involves creating a second copy of the file in memory. Is there a better or more concise way of achieving this?
Stumbled on a less ugly way of doing this:
text = '\n'.join(['='.join(item) for item in parser.items('top')])
with open('foo.conf', 'w') as config_file:
config_file.write(text)
Related
A sensor provides a stream of frames containing object coordinates, which are stored in ProtoBuf format in a gzipped file. I would like to read this file in Julia.
Using protoc, I have generated the Protobuf files for both Python and Julia, coordinate_push.py and coordinate_push.jl
My Python code is as follows:
frameList = []
with gzip.open(filePath) as f:
data = f.read()
next_pos, pos = 0, 0
while pos < len(data):
msg = coordinate_push.CoordinatesFrame()
next_pos, pos = _DecodeVarint32(data, pos)
msg.ParseFromString(data[pos:pos + next_pos])
frameList.append(msg)
pos += next_pos
I'd like to rewrite the above in Julia, and don't know where to start. Part of the problem is that I haven't fully understood the Python script (IO is not my strong point).
I understand that I need:
to open the gzip file, presumably using using GZip; file = GZip.open(file_path, "r")
to read in the data, along the lines of using ProtoBuf; data = readproto(iob, CoordinatesFrame())
What I don't understand is:
how to define iob, and especially how to link it to file (in the Julia Protobuf manual, we had iob = PipeBuffer(), but here it's a gzip-file that we'd like to read)
how to replicate the while-loop in Julia, and in particular the mysterious _DecodeVarint32 (I'm on Windows, if it's related to that.)
whether the file coordinate_push.jl has to be in the same directory as my main file, and if not, how I can properly import it (it is currently in a proto subfolder, and in Python I'd import it using from src.proto import coordinate_push)
Insight on any of the three points would be highly appreciated.
You should open an issue on the Gzip GitHub repo and ask this first part of your question there (I am not a Gzip expert unfortunately).
On the second point, I suggest looking here: https://github.com/JuliaIO/FileIO.jl/blob/master/README.md for lots of examples of FileIO loops which seems exactly what you need to replicate that Python loop. For the second part of this question, you best bet for that function is to try and hunt down the definition on GitHub or in the docs somewhere.
For the 3rd questions, coordinate_push.jl does not need to be in the same folder as your "main file" (I am not sure what you mean by this so perhaps it would help to add context on the structure of your files). To import that file all you need to do is add include("path/to/coordinate_push.jl") at the top of the file you want to call/run the code from. It's worth noting that the path can either be the absolute path or the relative project path (in some cases).
im a noobie in python. I want to get some data from csv with pandas and after write a new csv file with extra data in format
"type";"currency";"amount";"comment"
"type1";"currency1";"amount1";"comment1"
etc
import pandas as pd
import csv
req=pd.read_csv('/Users/user/web/python/Bookcopy.csv')
type="type"
comment = "2week"
i=0
while i<3:
Currency = req['Currency'].values[i]
ReqAmount = req['Request'].values[i]
r = round(ReqAmount,-1)
i+=1
data =[type,Currency,r,comment]
#print(data)
csv_file = open('data2.csv', 'w')
with csv_file:
writer = csv.writer(csv_file)
writer.writerow(data)
print("DONE")
writer.writerows(data)
_csv.Error: iterable expected, not numpy.float64
I have multiple things to criticize here. I hope it doesn't come across mean, but rather educational. While your code would work regardless of those points, it is good coding style to follow them.
Variables should not start with a capital letter. Currency and ReqAmount should be currency and reqAmount.
Don't use keywords for variable names. type is a python keyword.
Make sure your formatting doesn't get destroyed when posting here. Especially for python, which relies on tab formatting. Read here for more information: https://stackoverflow.com/editing-help#code
That said, let me try to go through your code and give you tips and tricks:
Don't run code in python directly. Always use the main() method. It's just better coding style.
When looping, don't use the i=0; while i<3; i+=1 construct, rather use for i in range(3). While it works, it is not very pythonic and a lot harder to read.
Never assume anything that cannot be guaranteed in your code. In this case, you assume that the csv-file has at least 3 lines, otherwise your program would crash. Instead, read the number of lines from the csv file, with len(req).
data =[type,Currency,r,comment] keeps overwriting your data variable. You could either append to data and then write everything to the output file at the end, or directly write to the output file in every iteration.
Don't use open to create a variable (except absolutely necessary). Instead, use open in a with statement. This ensures that the file will get closed properly. I have seen that you do use the with statement, nonetheless you usually use it like with open(...) as variable_name:.
CSV files usually start with the column names. Therefore you should write the column names before you write data.
I won't fix that, because it will change the appearance of the program completely, but normally, don't mix different libraries. If you use pandas for csv reading, also use it for writing. If you use the csv library for writing, also use it for reading. While it isn't wrong to mix them, it is bad style and creates more dependencies than would be necessary.
I don't really understand what your code is supposed to do, so I just guess and hope it goes in the right direction.
When fixing all those points, you might have something like that:
import pandas as pd
import csv
def main():
req=pd.read_csv('/Users/user/web/python/Bookcopy.csv')
transferType = "type"
comment = "2week"
with open('data2.csv', 'w') as csv_file:
writer = csv.writer(csv_file)
writer.writerow(["type","currency","amount","comment"])
for i in range(len(req)):
currency = req['Currency'].values[i]
reqAmount = req['Request'].values[i]
r = round(reqAmount,-1)
data = [transferType,currency,r,comment]
#print(data)
writer.writerow(data)
print("DONE")
# Whenever you run a program, __name__ will be set to '__main__' in the initial
# script. This makes it easier later when you work with multiple code files.
if __name__ == '__main__':
main()
Is there a way that I can use mx.recordio.MXRecordIO to read from a bytes object rather than a file object?
For example I'm currently doing:
import mxnet as mx
results_file = 'results.rec'
with open(results_file, 'wb') as f:
f.write(results)
recordio = mx.recordio.MXRecordIO(results_file, 'r')
temp = recordio.read()
But if possible I'd rather not have to write to file as an intermediate step. I've tried using BytesIO, but can't seem to get it to work.
Currently they is no way of achieving this sorry. This is non-trivial because the RecordIO reading/parsing is done in C++ and you can't simply forward the stream to the C++ API.
I have a CSV file with two columns in it, the one of the left being an old string, and the one directly to right being the new one. I have a heap of .xml files that contain the old strings, which I need to replace/update with the new ones.
The script is supposed to open each .xml file one at a time and replace all of the old strings in the CSV file with the new ones. I have tried to use a replace function to replace instances of the old string, called 'column[0]' with the new string, called 'column[1]'. However I must be missing something as this seems to do nothing. If I the first variable in the replace function to an actual string with quotation marks, the replace function works. However if both the terms in the replace function are variables, it doesn't.
Does anyone know what I am doing wrong?
import os
import csv
with open('csv.csv') as csv:
lines = csv.readline()
column = lines.split(',')
fileNames=[f for f in os.listdir('.') if f.endswith('.xml')]
for f in fileNames:
x=open(f).read()
x=x.replace(column[0],column[1])
print(x)
Example of CSV file:
oldstring1,newstring1
oldstring2,newstring2
Example of .xml file:
Word words words oldstring1 words words words oldstring2
What I want in the new .xml files:
Word words words newstring1 words words words newstring2
The problem over here is you are treating the csv file as normal text file not looping over the all the lines in the csv file.
You need to read file using csv reader
Following code will work for your task
import os
import csv
with open('csv.csv') as csvfile:
reader = csv.reader(csvfile)
fileNames=[f for f in os.listdir('.') if f.endswith('.xml')]
for f in fileNames:
x=open(f).read()
for row in reader:
x=x.replace(row[0],row[1])
print(x)
It looks like this is better done using sed. However.
If we want to use Python, it seems to me that what you want to do is best achieved
reading all the obsolete - replacements pairs and store them in a list of lists,
have a loop over the .xml files, as specified on the command line, using the handy fileinput module, specifying that we want to operate in line and that we want to keep around the backup files,
for every line in each of the .xml s operate all the replacements,
put back the modified line in the original file (using simply a print, thanks to fileinput's magic) (end='' because we don't want to strip each line to preserve eventual white space).
import fileinput
import sys
old_new = [line.strip().split(',') for line in open('csv.csv')]
for line in fileinput.input(sys.argv[1:], inplace=True, backup='.bak'):
for old, new in old_new:
line = line.replace(old, new)
print(line, end='')
If you save the code in replace.py, you will execute it like this
$ python3 replace.py *.xml subdir/*.xml another_one/a_single.xml
stackoverflow.
I've been trying to get the following code to create a .txt file, write some string on it and then print some message if said string was in the file. This is merely a study for a more complex project, but even given it's simplicity, it's still not working.
Code:
import io
file = open("C:\\Users\\...\\txt.txt", "w+") #"..." is the rest of the file destination
file.write('wololo')
if "wololo" in file.read():
print ("ok")
This function always skips the if as if there was no "wololo" inside the file, even though I've checked it all times and it was properly in there.
I'm not exactly sure what could be the problem, and I've spend a great deal of time searching everywhere for a solution, all to no avail. What could be wrong in this simple code?
Oh, and if I was to search for a string in a much bigger .txt file, would it still be wise to use file.read()?
Thanks!
When you write to your file, the cursor is moved to the end of your file. If you want to read the data aferwards, you'll have to move the cursor to the beginning of the file, such as:
file = open("txt.txt", "w+")
file.write('wololo')
file.seek(0)
if "wololo" in file.read():
print ("ok")
file.close() # Remember to close the file
If the file is big, you should consider to iterate over the file line by line instead. This would avoid that the entire file is stored in memory. Also consider using a context manager (the with keyword), so that you don't have to explicitly close the file yourself.
with open('bigdata.txt', 'rb') as ifile: # Use rb mode in Windows for reading
for line in ifile:
if 'wololo' in line:
print('OK')
else:
print('String not in file')