I have multiple folders in a directory and each folder has multiple files. I have a code which checks for a specific file in each folder and does some data preprocessing and analysis if the specific file is present.
A snippet of it is given below.
import pandas as pd
import json
import os
rootdir = os.path.abspath(os.getcwd())
df_list = []
for subdir, dirs, files in os.walk(rootdir):
for file in files:
if file.startswith("StudyParticipants") and file.endswith(".csv"):
temp = pd.read_csv(os.path.join(subdir, file))
.....
.....
'some analysis'
Merged_df.to_excel(path + '\Processed Data Files\Study_Participants_Merged.xlsx')
Now, I want to automate this process. I want this script to be executed whenever a new folder is added. This is my first in exploring automation process and I ham stuck on this for quite a while without major progress.
I am using windows system and Jupyter notebook to create these dataframes and perform analysis.
Any help is greatly appreciated.
Thanks.
I've wrote a script which you should only run once and it will work.
Please note:
1.) This solution does not take into account which folder was created. If this information is required I can rewrite the answer.
2.) This solution assumes folders won't be deleted from the main folder. If this isn't the case, I can rewrite the answer as well.
import time
import os
def DoSomething():
pass
if __name__ == '__main__':
# go to folder of interest
os.chdir('/home/somefolders/.../A1')
# get current number of folders inside it
N = len(os.listdir())
while True:
time.sleep(5) # sleep for 5 secs
if N != len(os.listdir()):
print('New folder added! Doing something useful...')
DoSomething()
N = len(os.listdir()) # update N
take a look at watchdog.
http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/
you could also code a very simple watchdog service on your own.
list all files in the directory you want to observe
wait a time span you define, say every few seconds
make again a list of the filesystem
compare the two lists, take the difference of them
the resulting list from this difference are your filesystem changes
Best greetings
I have been using this whole script flawlessly on my PC. I attempted to put it on my coworkers PC, but this particular part doesn't seem to work. I am using a tkinter interface to take data from psql and fill a premade fillable PDF using pypdftk, then either saving it using asksaveasfilename and opening it with subprocess.Popen or not saving it and opening it as a temp file using subprocess.run. On my PC both work great. On coworkers PC, neither work.
On my coworkers PC, the save option opens the save dialog with all the correct info as far as I can tell, and lets me go through the process of saving a file as it normally would, but then the file just doesn't save and never actually appears. If I open as a temp file, it throws the exception.
import tkinter as tk
from tkinter import *
from tkinter.ttk import *
import tkinter.messagebox
import pypdftk
from tkinter.filedialog import asksaveasfilename
import os.path
import os
import subprocess
from pathlib import Path
def file_handler(form_path, data, fname):
try:
tl2 = tk.Toplevel()
tl2.wm_title('File Handler')
w = 340
h = 55
ws = tl2.winfo_screenwidth()
hs = tl2.winfo_screenheight()
x = (ws/2) - (w/2)
y = (hs/2) - (h/2)
tl2.geometry('%dx%d+%d+%d' % (w, h, x, y))
def save_and_open():
savefile = asksaveasfilename(defaultextension=".pdf", initialdir="C:\\Desktop", filetypes=[('pdf file', '*.pdf')], initialfile=fname)
generated_pdf = pypdftk.fill_form(form_path, data, savefile)
subprocess.Popen(generated_pdf,shell=True)
def open_without_save():
try:
generated_pdf = pypdftk.fill_form(form_path, data)
os.rename(generated_pdf, generated_pdf+".pdf")
generated_pdf = generated_pdf+".pdf"
subprocess.run(generated_pdf,shell=True)
except:
tk.messagebox.showerror("Unable to open", "An error has occurred. Please try again.")
else:
tl2.destroy()
finally:
if os.path.exists(generated_pdf):
os.remove(generated_pdf)
print("Successfully removed temp file.")
save = tk.Button(tl2,text='Save and Open', width=20, command=save_and_open)
nosave = tk.Button(tl2,text='Open without saving', width=20,command=open_without_save)
save.grid(row=0, columnspan=2, sticky='NESW', padx=5, pady=10, ipadx=5, ipady=5)
nosave.grid(row=0, column=2, columnspan=2, sticky='NESW', padx=5, pady=10, ipadx=5, ipady=5)
tl2.mainloop()
except:
tk.messagebox.showerror("Unable to open", "An error has occurred. Please try again.")
As far as I can tell, everything works until you get into the save_and_open and open_without_save functions. I left in all the libraries I believe are relevant.
I should also mention, I am quite a novice at python. So if any of this is ugly coding, feel free to shame me for it.
update:
I now believe the problem to be here in the pypdftk.py file:
if os.getenv('PDFTK_PATH'):
PDFTK_PATH = os.getenv('PDFTK_PATH')
else:
PDFTK_PATH = '/usr/bin/pdftk'
if not os.path.isfile(PDFTK_PATH):
PDFTK_PATH = 'pdftk'
My error states pdftk is not a known command. My guess is that there is no environment variable, then it looks to the /usr/bin and cannot find the pdftk file, so it's just making "pdftk" a string? I don't know much about /usr/bin, but is there a way to check that?
What's going on in lines 19-21
if os.getenv('PDFTK_PATH'): is checking to see if the environment variable PDFTK_PATH even exists on your machine. If so, the constant PDFTK_PATH is set to the value provided by the PDFTK_PATH environment key/variable.
Otherwise, it sets PDFTK_PATH to /usr/bin/pdftk. Two things are happening here... First, it provides a path to the binary, i.e., /usr/bin. Second, it provides the name of the binary, i.e., pdftk. In other words, it sets PDFTK_PATH to path + executable filename.
(NOTE: The directory usr/bin is where most executables are stored on machines running Unix-like operating systems, e.g., Mac, Ubuntu, etc. This is alluded to in the repo, see here.)
To err on the side of caution, if not os.path.isfile(PDFTK_PATH): checks to see if the pdftk binary can indeed be found in the /usr/bin/ folder. If not, it sets PDFTK_PATH to pdftk, i.e., it sets the path to the pdftk binary to the very directory in which pypdftk.py is located.
Farther down, the try block runs a test call on whatever the value of PDFTK_PATH was ultimately set to. See lines 46-49. If the binary, pdftk.exe, is not where PDFTK_PATH says it is, then you get the error that you got.
Concerning the r-string
As for casting the string literal to an r-string, that actually did nothing. The r prefix simply redefines the \ as just that, a backslash, rather than allowing the \ to continue to function as cue to an escape sequence. See here and here. You'll notice that neither /usr/bin/pdftk nor pdftk, i.e., where you prepended the r, contain any backslashes.
What I think happened...
After you took the advice of acw1668, and installed the PDF Toolkit (pdftk); and reinstalled the pypdftk package, you had probably fixed the problem. I don't know, but maybe you had not restarted your IDE/editor during these steps? (Side note: Sometimes you need to restart your IDE/editor after changes have been made to your machine's environment variables/settings.)
The short answer: If you're on a windows machine, the install of the pdftk toolkit added PDFTK_PATH to your environment; or, if you're on a Unix-based machine, the install placed the binary in the /usr/bin directory.
Regardless, I assure you the r had nothing to do with it. Now that you know it's working, let's prove it... take out the r, you'll see that it is still working.
I was able to fix this problem by going into the pypdftk.py file and changing the paths to raw strings like such:
if os.getenv('PDFTK_PATH'):
PDFTK_PATH = os.getenv('PDFTK_PATH')
else:
PDFTK_PATH = r'/usr/bin/pdftk'
if not os.path.isfile(PDFTK_PATH):
PDFTK_PATH = r'pdftk'
Well, I've got the need of automate a process in my job(actually I'm an intern), and I just wondered if I could use Python for such process. I'm still processing my ideas of how to do those stuffs, and now I'm currently trying to understand how to download a file from a web URL using python3. I've found a guide on another website, but there's no active help there. I was told to use the module requests to download the actual file, and the module re to get the real file name.
The code was working fine, but then I tried to add some features like GUI, and it just stopped working. I took off the GUI code, and it didn't work again. Now I have no idea of what to do to get the code working, pls someone helo me, thanks :)
code:
import os
import re
# i have no idea of how this function works, but it gets the real file name
def getFilename(cd):
if not cd:
print("check 1")
return None
fname = re.findall('filename=(.+)', cd)
if len(fname) == 0:
print("check 2")
return None
return fname[0]
def download(url):
# get request
response = requests.get(url)
# get the real file name, cut off the quota and take the second element of the list(actual file name)
filename = getFilename(response.headers.get('content-disposition'))
print(filename)
# open in binary mode and write to file
#open(filename, "wb").write(response.content)
download("https://pixabay.com/get/57e9d14b4957a414f6da8c7dda353678153fd9e75b50704b_1280.png?attachment=")
os.system("pause")```
I know the title is really vague but anyways, I have a script for downloading the subtitles of a series or movie once the torrent is done downloading. The Input needs to be the filepath of the downloaded file. Conveniently uTorrent has a support for running the script once a torrent finishes downloading and has the filepath as one of its "parameters". I tried running the script with
C:\python\subtitles.py %D
where %D is the supported utorrent parameter for the filepath. This did not work as the script loaded then prompted for user input.Any help on how to automate this would be helpful.
from datetime import timedelta
from babelfish import Language
from subliminal import download_best_subtitles, region, save_subtitles, scan_videos
import os
# configure the cache
region.configure('dogpile.cache.dbm', arguments={'filename': 'cachefile.dbm'})
path = str(input("enter filepath:"))
# scan for videos newer than 1 week and their existing subtitles in a folder
videos = scan_videos(path, age=timedelta(days=7))
print("scan success")
# download best subtitles
subtitles = download_best_subtitles(videos, {Language('eng')})
print("downloads done")
# save them to disk, next to the video
for v in videos:
save_subtitles(v, subtitles[v])
That's because you're trying to get the "parameters" from stdin while your bittorrent client is passing the path as a command-line argument.
Replace path = str(input("enter filepath:")) with
import sys
path = sys.argv[1]
and it'll work.
I have created a program that calculates size of files. I have created another program that sends mail to user. Name of programs is filesize.py and mailsend.py respectively. The mailsend.py is working perfectly, when I run it, I can see mail in my gmail inbox. But I want to run this mailsend.py from code of filesize.py.
I have done this in filesize.py: ( I am sure what I am trying to do is foolish, but I am new to programming let alone python )
import mailsend.py
cmd="python mailsend.py"
os.system(cmd)
And this is not working. Any suggestions?
Replace import mailsend.py to import os.
More recommended way is...
mailsend.py
def main():
# Put old `mailsend.py` content here.
if __name__ == '__main__':
main()
filesize.py
import mailsend
mailsend.main()