The new Path package from the pathlib library, which has been added from Python 3.4, seems a powerful replacement of approaches such as os.path.join(), but I've some trouble working with it.
I have a path that can be anything from
folder_foo/file.csv
to
long/path/to/folder_foo/file.csv
I read the .csv file in folder_foo with pandas, modify it and want to save it to
folder_bar/file.csv
or
long/path/to/folder_bar/file.csv
Essentially I want to rename folder_foo to folder_bar in the Path object.
EDIT: example path code
csv_path = Path("long/path/to/folder_foo/file.csv")
Attempts
1
csv_path.parents[0] = csv_path.parents[0] + "_clean")
Which leads to the error TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str', which means you cannot use + to combine a PosixPath with a str as described in TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str'.
2
To solve this I tried the following:
csv_path.parents[0] = Path(str(csv_path.parents[0]) + "_clean")
Which however results in the error : TypeError: '_PathParents' object does not support item assignment.
Since PosixPath is not a list, this error is understandable.
3
Maybe .parts is a better approach, but
csv_path.parts[-2] = csv_path.parts[-2][:-3] + "bar"
results in: TypeError: 'tuple' object does not support item assignment.
Question
How can I easily rename the file's parent folder?
Would rather split this up for readability:
bar_folder = csv_path.parent.parent / 'folder_bar'
csv_path2 = bar_folder / csv_path.name
Having the destination folder as a variable also enables you to create the folder using for example:
bar_folder.mkdir(exist_ok=True)
You could also write a little function to replace the part of the path you want to change. Here's a runnable example:
from pathlib import Path
path1 = Path("a/b/c.txt")
path2 = Path("b/c.txt")
def rename_dir(path, src, dst):
# convert to list so that we can change elements
parts = list(path.parts)
# replace part that matches src with dst
parts[parts.index(src)] = dst
return Path(*parts)
rename_dir(path1, 'b', 'q')
#> PosixPath('a/q/c.txt')
rename_dir(path2, 'b', 'q')
#> PosixPath('q/c.txt')
Created at 2021-03-06 10:44:00 PST by reprexlite v0.4.2
EDIT: Found a cleaner solution without str()
csv_path2 = csv_path.parents[1] / (csv_path.parts[-2][:-3] + "bar") / csv_path.parts[-1]
# result
PosixPath('long/path/to/folder_bar/file.csv')
Path.parents gets the whole path to the folder minus the file. Path.parents[1] goes 2 levels up (long/path/to/), which is still a Path object. Then we get the last folder name with csv_path.parts[-2], which is a string. We apply [:-3] to get all string characters except "foo". This means we have "folder_". Then with + "bar" we get "folder_bar", which is added to our Path object. Finally we re-add the file name to our Path object with / csv_path.parts[-1].
Hack like solution
csv_path = Path(str(csv_path.parents[0])[:-3] + 'bar/' + csv_path.parts[-1])
It seems to me a bit unintuitive, however. There should be a more clean solution?
Related
Ultimately, I want to loop through every pdf in specified directory ('C:\Users\dude\pdfs_for_parsing') and print the metadata for each pdf. The issue is that when I try to loop through the "directory" I'm receiving the error "FileNotFoundError: [Errno 2] No such file or directory:". I understand this error is occurring because I now have double slashes in my filepaths for some reason.
Example Code
import PyPDF2
import os
path_of_the_directory = r'C:\Users\dude\pdfs_for_parsing'
directory = []
ext = ('.pdf')
def isolate_pdfs():
for files in os.listdir(path_of_the_directory):
if files.endswith(ext):
x = os.path.abspath(files)
directory.append(x)
for pdf in directory:
reader = PyPDF2.PdfReader(pdf)
information = reader.metadata
print(information)
isolate_pdfs()
If I print the file paths one at a time, I see that the files have single '/' like I'm expecting:
for pdf in directory:
print(pdf)
The '//' seems to get added when I try to open each of the PDFs 'PDFFile = open(pdf,'rb')'
Your issue has nothing to do with //, it's here:
os.path.abspath(files)
Say you have C:\Users....\x.pdf, you list that directory, so the files will contain x.pdf. You then take the absolute path of x.pdf, which the abspath supposes to be in the current directory. You should replace it with:
x = os.path.join(path_of_the_directory, files)
Other notes:
PDFFile and PDF shouldn't be in uppercase. Prefer pdf_file and pdf_reader. The latter also avoids the confusion with the for pdf in...
Try to use a debugger rather than print statements. This is how I found your bug. It can be in your IDE or in command line with python -i You can step through your code, test a few variations, fiddle with the variables...
Why is ext = ('.pdf') with braces ? It doesn't do anything but leads to think that it might be a tuple (but isn't).
As an exercise the first for can be written as: directory = [os.path.join(path_of_the_directory, x) for x in os.listdir(path_of_the_directory) if x.endswith(ext)]
Python 3.6, OS Windows 7
I am trying to read a .txt using pd.read_csv() using relative filepath. So, from pd.read_csv() API checked out that the filepath argument can be any valid string path.
So, in order to define the relative path I use pathlib module. I have defined the relative path as:
df_rel_path = pathlib.Path.cwd() / ("folder1") / ("folder2") / ("file.txt")
a = str(df_rel_path)
Finally, I just want to use it to feed pd.read_csv() as:
df = pd.read_csv(a, engine = "python", sep = "\s+")
However, I am just getting an error stating "No such file or directory: ..." showing double backslashes on the folder path.
I have tried to manually write the path on pd.read_csv() using a raw string, that is, using r"relative/path". However, I am still getting the same result, double backslashes. Is there something I am overlooking?
You can get what you want by using os module
df_rel_path = os.path.abspath(os.path.join(os.getcwd(), "folder1", "folder2"))
This way the os module will deal with the joining the path parts with the proper separator. You can omit os.path.abspath if you read a file that's within the same directory but I wrote it for the sake of completeness.
For more info, refer to this SO question: Find current directory and file's directory
You need a filename to call pd.read_csv. In the example 'a' is a only the path and does not point to a specific file. You could do something like this:
df_rel_path = pathlib.Path.cwd() / ("folder1") / ("folder2")
a = str(df_rel_path)
df = pd.read_csv(a+'/' +'filename.txt')
With the filename your code works for me (on Windows 10):
df_rel_path = pathlib.Path.cwd() / ("folder1") / ("folder2")/ ("file.txt")
a = str(df_rel_path)
df = pd.read_csv(a)
I am new to Python coding. I am able to create the output XML file. I want to use a variable which holds a string value and pass it to 'predicate' of 'find()'. Is this achievable? How to make this work?
I am using LXML package with Python 3.6. Below is my code. Area of problem is commented at the end of the code.
import lxml.etree as ET
# Create root element
root = ET.Element("Base", attrib={'Name': 'My Base Node'})
# Create first child element
FirstElement = ET.SubElement(root, "FirstNode", attrib={'Name': 'My First Node', 'Comment':'Hello'})
# Create second child element
SecondElement = ET.SubElement(FirstElement, "SecondNode", attrib={'Name': 'My Second Node', 'Comment': 'World'})
# Create XML file
XML_data_as_string = ET.tostring(root, encoding='utf8')
with open("TestFile.xml", "wb") as f:
f.write(XML_data_as_string)
# Variable to substitute in second portion of predicate
NewValue = "My Second Node"
# #### AREA OF PROBLEM ###
# Question. How to pass variable 'NewValue' in the predicate?
# Gives "SyntaxError: invalid predicate"
x = root.find("./FirstNode/SecondNode[#Name={subs}]".format(subs=NewValue))
# I commented above line and reexecuted the code with this below line
# enabled. It gave "ValueError: empty namespace prefix must be passed as None,
# not the empty string"
x = root.find("./FirstNode/SecondNode[#Name=%s]", NewValue)
As Daniel Haley said - you're missing a single quotes in #Name={subs}.
The following line works for me:
x = root.find("./FirstNode/SecondNode[#Name='{subs}']".format(subs=NewValue))
Since you use Python 3.6, you can utilize f-strings:
x = root.find(f"./FirstNode/SecondNode[#Name='{NewValue}']")
The "proper" way to solve this would be to use XPath variables, which are not supported by find() (and consequently, aren't supported by xml.etree from the standard library either) but are supported by xpath().
NewValue = "AJL's Second Node" # Uh oh, that apostrophe is going to break something!!
x_list = root.xpath("./FirstNode/SecondNode[#Name=$subs]", subs=NewValue)
x = x_list[0]
This avoids any sort of issue you might otherwise run into with quoting and escaping.
The main caveat of this method is namespace support, since it doesn't use the bracket syntax of find.
x = root.find("./{foobar.xsd}FirstNode")
# Brackets are doubled to avoid conflicting with `.format()`
x = root.find("./{{foobar.xsd}}FirstNode/SecondNode[#Name='{subs}']".format(subs=NewValue))
Instead, you must specify those in a separate dict:
ns_list = {'hello':'foobar.xsd'}
x_list = root.xpath("./hello:FirstNode/SecondNode[#Name=$subs]", namespaces=ns_list , subs=NewValue)
x = x_list[0]
I want to read some CSVs from a given directory and I want to name the resulted dataframes similarly to the name of the csv.
So I wrote the code bellow but I am aware that it is not the rigth syntax.
Beside I have the error :
TypeError: 'str' object does not support item assignment
My code :
import os
for element in os.listdir('.'):
element[:-4] = read_csv(element)
Thank you for your help
you can do that by add tempering the global scope as follows:
import os
for i in os.listdir('.'):
globals()[i] = pd.read_csv(i)
But, that's very ugly and, as #JonClements pointed out, won't work if the filename doesn't follow the python variable naming rules. As a reminder, variable naming rules are :
Variables names must start with a letter or an underscore, such as:
_underscore
underscore_
The remainder of your variable name may consist of letters, numbers and underscores.
password1
n00b
un_der_scores
check this link for more explanation.
The best way is to create a dictionary:
import os
d = {}
for i in os.listdir('.'):
d[i] = pd.read_csv(i)
Then you can access any dataframe you want as follows d['file1.csv']
My application offers the ability to the user to export its results. My application exports text files with name Exp_Text_1, Exp_Text_2 etc. I want it so that if a file with the same file name pre-exists in Desktop then to start counting from this number upwards. For example if a file with name Exp_Text_3 is already in Desktop, then I want the file to be created to have the name Exp_Text_4.
This is my code:
if len(str(self.Output_Box.get("1.0", "end"))) == 1:
self.User_Line_Text.set("Nothing to export!")
else:
import os.path
self.txt_file_num = self.txt_file_num + 1
file_name = os.path.join(os.path.expanduser("~"), "Desktop", "Exp_Txt" + "_" + str(self.txt_file_num) + ".txt")
file = open(file_name, "a")
file.write(self.Output_Box.get("1.0", "end"))
file.close()
self.User_Line_Text.set("A text file has been exported to Desktop!")
you likely want os.path.exists:
>>> import os
>>> help(os.path.exists)
Help on function exists in module genericpath:
exists(path)
Test whether a path exists. Returns False for broken symbolic links
a very basic example would be create a file name with a formatting mark to insert the number for multiple checks:
import os
name_to_format = os.path.join(os.path.expanduser("~"), "Desktop", "Exp_Txt_{}.txt")
#the "{}" is a formatting mark so we can do file_name.format(num)
num = 1
while os.path.exists(name_to_format.format(num)):
num+=1
new_file_name = name_to_format.format(num)
this would check each filename starting with Exp_Txt_1.txt then Exp_Txt_2.txt etc. until it finds one that does not exist.
However the format mark may cause a problem if curly brackets {} are part of the rest of the path, so it may be preferable to do something like this:
import os
def get_file_name(num):
return os.path.join(os.path.expanduser("~"), "Desktop", "Exp_Txt_" + str(num) + ".txt")
num = 1
while os.path.exists(get_file_name(num)):
num+=1
new_file_name = get_file_name(num)
EDIT: answer to why don't we need get_file_name function in first example?
First off if you are unfamiliar with str.format you may want to look at Python doc - common string operations and/or this simple example:
text = "Hello {}, my name is {}."
x = text.format("Kotropoulos","Tadhg")
print(x)
print(text)
The path string is figured out with this line:
name_to_format = os.path.join(os.path.expanduser("~"), "Desktop", "Exp_Txt_{}.txt")
But it has {} in the place of the desired number. (since we don't know what the number should be at this point) so if the path was for example:
name_to_format = "/Users/Tadhg/Desktop/Exp_Txt_{}.txt"
then we can insert a number with:
print(name_to_format.format(1))
print(name_to_format.format(2))
and this does not change name_to_format since str objects are Immutable so the .format returns a new string without modifying name_to_format. However we would run into a problem if out path was something like these:
name_to_format = "/Users/Bob{Cat}/Desktop/Exp_Txt_{}.txt"
#or
name_to_format = "/Users/Bobcat{}/Desktop/Exp_Txt_{}.txt"
#or
name_to_format = "/Users/Smiley{:/Desktop/Exp_Txt_{}.txt"
Since the formatting mark we want to use is no longer the only curly brackets and we can get a variety of errors:
KeyError: 'Cat'
IndexError: tuple index out of range
ValueError: unmatched '{' in format spec
So you only want to rely on str.format when you know it is safe to use. Hope this helps, have fun coding!