I am new Python3 so for give me for asking such question but I couldn't find answer on Google, I have Python scanning a file directory and I need to add in an open and close bracket and a new var into the Python result, so that I can insert it into the database.
Mysql requires inserts to be wrapped in brackets in the val =[('test.mp4',newip) ] This works as I get 1 was inserted when I run the hard coded script.
So what I am trying to archive is to modify the result of the scan and add the following
open/close brackets and the new newip into the result of the scan like the following example
Scan result
['test.mp4', 'test_2.mp4', 'test_3.mp4', test_4.mp4']
Insert new result (modified)
[('test',newip), ('test_2.mp4',newip), ('test_3.mp4',newip), ('test_4.mp4',newip)]
When hard coded its works
root#ubuntu:~# python3 testscan.py
['test.mp4', 'test_2.mp4', 'test_3.mp4', test_4.mp4']
1 was inserted.
Please can anyone advise how to achieve this, below is the full code
import os, mysql.connector, re, uuid
files = [f.name for f in (os.scandir('/var/www/html/media/usb1')) if f.name.endswith('.mp4')]
print(files)
newip = (':'.join(re.findall('..', '%012x' % uuid.getnode())))
mydb = mysql.connector.connect(
host="127.0.0.1",
user="user",
password="password",
database="database"
)
mycursor = mydb.cursor()
sql = "INSERT IGNORE INTO files (file,ip) VALUES (%s,%s)"
val =[('test.mp4',newip)]
mycursor.executemany(sql, val)
mydb.commit()
print(mycursor.rowcount, "was inserted.")
So if you want to add the newip to the scan you can use a list comprehension:
files = ['test.mp4', 'test_2.mp4', 'test_3.mp4', 'test_4.mp4']
sql_values = [(file, newip) for file in files]
the result looks like this:
[('test.mp4', newip), ('test2.mp4', newip), ('test3.mp4', newip), ('test4.mp4', newip)]
Related
So I'm trying to move my csv files from the source folder to the dest folder after performing an action on each file using nested for loops
Below are the nested for loops.
What's happening now is that the top file gets copied into the table in the database, but it doesn't get moved to destination folder after its contents are inserted into the sql table, and then the loop breaks after first run and prints the error in try block.
If I remove the shutil statement, all rows from each csv file successfully copies into database.
Essentially I want that to happen, but I also want to move each file, after I've copied all the data into the table, to the dest folder.
This script will be triggered on a power automate action that will run once a file is added to the folder. So I don't want to add/duplicate the rows in my database from the same file.
I'm also adding variables below this code so you can get an idea of what the function is doing as well.
Thanks for any help you can provide on this, and let me know if more clarification is needed.
My attempt:
for file in dir_list:
source = r"C:\Users\username\source\{}".format(file)
df = pd.read_csv(path2)
df = df.dropna()
rows= df_to_row_tuples(df)
for row in rows:
cursor.execute(sql, row)
conn.commit()
shutil.move(source, destination)
Variables:
def df_to_row_tuples(df):
df = df.fillna('')
rows = [tuple(cell) for cell in df.values]
return rows
conn = sqlite3.connect(r'C:\Users\some.db')
cursor = conn.cursor()
sql = "INSERT INTO tblrandomtble VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
path = r'C:\Users\username\source'
dir_list = os.listdir(path)
source=""
destination= r"C:\Users\username\destination"
df = pd.DataFrame()
rows = tuple()
If the file already exists, the move function will overwrite it, provided you pass the whole path...including the file name
So add the file name to the destination arg of the shutil.move function...
I am looking for a way to read a bunch of files from S3, but there is a potential for a path to not exist. I would just like to ignore the fact that the path does not exist, and process all information possible. Example I want to read in files:
files_to_read = []
for id in ids_to_process:
for date in dates_to_process:
files_to_read.append('s3://bucket/date=' + date + '/id=' + id + '/*.parquet')
sqlContext.read.parquet(*files_to_read)
The issue is that some id's may not have started until a certain date, an while I can figure that out, it's not very easy to do it programmatically. What would the easiest way be to either a) ignore a file if the path does not exist. b) check if a path exists.
I have tried sqlContext.sql("spark.sql.files.ignoreMissingFiles=true"), which does not seem to work. Would there be any similar option that I am missing?
Here, missing file really means the deleted file under directory after you construct the DataFrame.
It is recommended to judge whether the target file exists in python in advance instead of handing it over to spark.
You could try something like this, maybe looking to catch the specific exception that is being thrown when a file does not exist (I believe in Scala it's an AnalysisException):
df = None
for path in paths_to_read:
try:
temp_df = sqlContext \
.read \
.parquet(path)
if df is None:
df = temp_df
else:
df = df.union(temp_df)
except:
# Ignoring this path
# print("Path " + path + " cannot be read. Ignoring.")
pass
Haven't seen something in pyspark that can do that. I also faced this and this is what I did:
Have a list of S3 addresses that you want to read.
addrs = ["s3a://abc", "s3a://xyz", ... ]
Test the links beforehand, and remove them if not accessible
for add in addrs:
try:
spark.read.format("parquet").load(add)
except:
print(add)
addrs.remove(add)
Read the updated list using spark
sdf_a = spark\
.read\
.format("parquet")\
.load(addrs)
Currently working on a program in Python that has to take data from a text file and input it into appropriate place in SQLite. I have created my database and the columns, now I am stuck on how I process the text data in and read it into the sqlite database.
Here are a couple lines from text file.
Kernel version: Windows 10 Enterprise, Multiprocessor Free
Product type: Professional
Product version: 6.3
Service pack: 0
Here is what I have so far,
import sqlite3
conn = sqlite3.connect('systeminfo.db')
c = conn.cursor()
def create_table():
c.execute("""CREATE TABLE IF NOT EXISTS system_information (
Machine_Name text,
Kernel_version text,
Product_type text,
product_version text,
Registered_organization text,
registered_owner text,
system_root text,
processors text,
physical_memory text
)""")
create_table1()
This creates my database and my table just how I want it, now I am stuck on taking the for example Kernel version from text file and putting the "Windows 10 Enterprise" into the database under the Kernel_Version Column.
UPDATE:
After using #zedfoxus tips, I was able to successfully get data, here is what I have, now how can I do the next lines more efficient? I am using elif, getting errors,
def insert_data(psinfo):
with open(psinfo) as f:
file_data = f.readlines()
for item in file_data:
if 'Kernel version' in item:
info = item.strip().split(':')
val = info[1].strip().split(',')
elif 'Product type' in item:
info = item.strip().split(':')
val = info[1].strip().split(',')
c.execute(
'INSERT INTO system_information (Kernel_version,Product_type ) values(?,?)',
(val[1].strip(),)
)
conn.commit()
Let's say you have a file called kernel.txt that contains
Kernel version: Windows 10 Enterprise, Multiprocessor Free
Product type: Professional
Product version: 6.3
Service pack: 0
Your python code would just have to read that text file and insert data into SQLite like so:
import sqlite3
conn = sqlite3.connect('systeminfo.db')
c = conn.cursor()
def create_table():
# same thing you had...just removing it for brevity
def insert_data(filename):
# read all the lines of the file
with open(filename) as f:
file_data = f.readlines()
# if Kernel version exists in the line, split the line by :
# take the 2nd item from the split and split it again by ,
# take the first item and pass it to the insert query
# don't forget to commit changes
for item in file_data:
if 'Kernel version' in item:
info = item.strip().split(':')
val = info[1].strip().split(',')
c.execute(
'insert into system_information (Kernel_version) values(?)',
(val[0].strip(),)
)
conn.commit()
create_table()
insert_data('kernel.txt')
You will have to change this code if you have multiple files containing such information, or if you have a single file containing multiple blocks of similar information. This code will get you started, though.
Update
I have separated the data parsing into its own function that I can call multiple times. Note how I have created 3 variables to store additional information like product type and version. The insert execution is happening outside of the loop. We are, basically, collecting all information we need and then inserting in one shot.
import sqlite3
conn = sqlite3.connect('systeminfo.db')
c = conn.cursor()
def create_table():
# same thing you had...just removing it for brevity
pass
def get_value(item):
info = item.strip().split(':')
val = info[1].strip().split(',')
return val[0].strip()
def insert_data(filename):
# read all the lines of the file
with open(filename) as f:
file_data = f.readlines()
# if Kernel version exists in the line, split the line by :
# take the 2nd item from the split and split it again by ,
# take the first item and pass it to the insert query
# don't forget to commit changes
kernel_version = ''
product_type = ''
product_version = ''
for item in file_data:
if 'Kernel version' in item:
kernel_version = get_value(item)
elif 'Product type' in item:
product_type = get_value(item)
elif 'Product version' in item:
product_version = get_value(item)
c.execute(
'''insert into system_information
(Kernel_version, Product_type, Product_version)
values(?, ?, ?)''',
(kernel_version, product_type, product_version,)
)
conn.commit()
create_table()
insert_data('kernel.txt')
First off i must say i am VERY new to programming (less then a week experience in total). I set out to write a program that generates a series of documents of an .odt template. I want to use a template with a specific keyword lets say "X1234X" and so on. This will then be replaced by values generated from the program. Each document is a little different and the values are entered and calculated via a prompt (dates and other things)
I wrote most of the code so far but i am stuck since 2 days on that problem. I used the ezodf module to generate a new document (with different filenames) from a template but i am stuck on how to edit the content.
I googled hard but came up empty hope someone here could help. I tried reading the documentations but i must be honest...its a bit tough to understand. I am not familiar with the "slang"
Thanks
PS: a ezodf method would be great, but any other ways will do too. The program doesnt have to be pretty it just has to work (so i can work less ^_^)
Well i figured it out. nd finished the program. I used a ezodf to create the file, then zipfile to extract and edit the content.xml and then repacked the whole thing via a nice >def thingy< from here. I tried to mess with etree...but i couldnt figure it out...
from ezodf import newdoc
import os
import zipfile
import tempfile
for s in temp2:
input2 = s
input2 = str(s)
input1 = cname[0]
file1 = '.odt'
namef = input2 + input1 + file1
odt = newdoc(doctype='odt', filename=namef, template='template.odt')
odt.save()
a = zipfile.ZipFile('template.odt')
content = a.read('content.xml')
content = str(content.decode(encoding='utf8'))
content = str.replace(content,"XXDATEXX", input2)
content = str.replace(content, 'XXNAMEXX', input1)
def updateZip(zipname, filename, data):
# generate a temp file
tmpfd, tmpname = tempfile.mkstemp(dir=os.path.dirname(zipname))
os.close(tmpfd)
# create a temp copy of the archive without filename
with zipfile.ZipFile(zipname, 'r') as zin:
with zipfile.ZipFile(tmpname, 'w') as zout:
zout.comment = zin.comment # preserve the comment
for item in zin.infolist():
if item.filename != filename:
zout.writestr(item, zin.read(item.filename))
# replace with the temp archive
os.remove(zipname)
os.rename(tmpname, zipname)
# now add filename with its new data
with zipfile.ZipFile(zipname, mode='a', compression=zipfile.ZIP_DEFLATED) as zf:
zf.writestr(filename, data)
updateZip(namef, 'content.xml', content)
I have this code that simply creates a list from user input. I want to load this into sqlite Db instead of list shown but am not conversant with Sqlite. please help
HERE IS THE CODE
listQ = []
while True:
read = input("Type in a line. ").lower().split()
for item in read:
listQ.append( input("Type in a line. ") )
for line in listQ:
import sqlite3
conn = sqlite3.connect('/C/project/new/sqlite_file.db')
c = conn.cursor()
for item in listQ:
c.execute('insert into tablename values (?,?,?)', item)
#print(line)