Unknown column added in user input form - python-3.x

I have a simple data entry form that writes the inputs to a csv file. Everything seems to be working ok, except that there are extra columns being added to the file in the process somewhere, seems to be during the user input phase. Here is the code:
import pandas as pd
#adds all spreadsheets into one list
Batteries= ["MAT0001.csv","MAT0002.csv", "MAT0003.csv", "MAT0004.csv",
"MAT0005.csv", "MAT0006.csv", "MAT0007.csv", "MAT0008.csv"]
#User selects battery to log
choice = (int(input("Which battery? (1-8):")))
def choosebattery(c):
done = False
while not done:
if(c in range(1,9)):
return Batteries[c]
done = True
else:
print('Sorry, selection must be between 1-8')
cfile = choosebattery(choice)
cbat = pd.read_csv(cfile)
#Collect Cycle input
print ("Enter Current Cycle")
response = None
while response not in {"Y", "N", "y", "n"}:
response = input("Please enter Y or N: ")
cy = response
#Charger input
print ("Enter Current Charger")
response = None
while response not in {"SC-G", "QS", "Bosca", "off", "other"}:
response = input("Please enter one: 'SC-G', 'QS', 'Bosca', 'off', 'other'")
if response == "other":
explain = input("Please explain")
ch = response + ":" + explain
else:
ch = response
#Location
print ("Enter Current Location")
response = None
while response not in {"Rack 1", "Rack 2", "Rack 3", "Rack 4", "EV001", "EV002", "EV003", "EV004", "Floor", "other"}:
response = input("Please enter one: 'Rack 1 - 4', 'EV001 - 004', 'Floor' or 'other'")
if response == "other":
explain = input("Please explain")
lo = response + ":" + explain
else:
lo = response
#Voltage
done = False
while not done:
choice = (float(input("Enter Current Voltage:")))
modchoice = choice * 10
if(modchoice in range(500,700)):
vo = choice
done = True
else:
print('Sorry, selection must be between 50 and 70')
#add inputs to current battery dataframe
log = pd.DataFrame([[cy,ch,lo,vo]],columns=["Cycle", "Charger", "Location", "Voltage"])
clog = pd.concat([cbat,log], axis=0)
clog.to_csv(cfile, index = False)
pd.read_csv(cfile)
And I receive:
Out[18]:
Charger Cycle Location Unnamed: 0 Voltage
0 off n Floor NaN 50.0
Where is the "Unnamed" column coming from?

There's an 'unnamed' column coming from your csv. The reason most likely is that the lines in your input csv files end with a comma (i.e. your separator), so pandas interprets that as an additional (nameless) column. If that's the case, check whether your lines end with your separator. For example, if your files are separated by commas:
Column1,Column2,Column3,
val_11, val12, val12,
...
Into:
Column1,Column2,Column3
val_11, val12, val12
...
Alternatively, try specifying the index column explicitly as in this answer. I believe some of the confusion stems from pandas concat reordering your columns .

Related

Using ctrader-fix to download historical data from cTrader

I am using the python package ctrader-fix (https://pypi.org/project/ctrader-fix/) to download historical price data from ctrader's API (https://help.ctrader.com/fix/).
The code does not make clear to me at least where exactly I declare the symbol (e.g. 'NatGas') through its SymbolID code number (in the case of 'NatGas' the SymbolID code number is 10055) for which I request historical data but also it does not make clear where I specify the timeframe I am interested on (e.g. 'H' for hourly data) and the number of records I want to retrieve.
section of ctrader where the FIX SymbolID number of 'NatGas' is provided
The code that is provided is the following (I have filled the values except the username).
config = {
'Host': '',
'Port': 5201,
'SSL': False,
'Username': '****************',
'Password': '3672075',
'BeginString': 'FIX.4.4',
'SenderCompID': 'demo.pepperstoneuk.3672025',
'SenderSubID': 'QUOTE',
'TargetCompID': 'cServer',
'TargetSubID': 'QUOTE',
'HeartBeat': '30'
}
client = Client(config["Host"], config["Port"], ssl = config["SSL"])
def send(request):
diferred = client.send(request)
diferred.addCallback(lambda _: print("\nSent: ", request.getMessage(client.getMessageSequenceNumber()).replace("", "|")))
def onMessageReceived(client, responseMessage): # Callback for receiving all messages
print("\nReceived: ", responseMessage.getMessage().replace("", "|"))
# We get the message type field value
messageType = responseMessage.getFieldValue(35)
# we send a security list request after we received logon message response
if messageType == "A":
securityListRequest = SecurityListRequest(config)
securityListRequest.SecurityReqID = "A"
securityListRequest.SecurityListRequestType = 0
send(securityListRequest)
# After receiving the security list we send a market order request by using the security list first symbol
elif messageType == "y":
# We use getFieldValue to get all symbol IDs, it will return a list in this case
# because the symbol ID field is repetitive
symboldIds = responseMessage.getFieldValue(55)
if config["TargetSubID"] == "TRADE":
newOrderSingle = NewOrderSingle(config)
newOrderSingle.ClOrdID = "B"
newOrderSingle.Symbol = symboldIds[1]
newOrderSingle.Side = 1
newOrderSingle.OrderQty = 1000
newOrderSingle.OrdType = 1
newOrderSingle.Designation = "From Jupyter"
send(newOrderSingle)
else:
marketDataRequest = MarketDataRequest(config)
marketDataRequest.MDReqID = "a"
marketDataRequest.SubscriptionRequestType = 1
marketDataRequest.MarketDepth = 1
marketDataRequest.NoMDEntryTypes = 1
marketDataRequest.MDEntryType = 0
marketDataRequest.NoRelatedSym = 1
marketDataRequest.Symbol = symboldIds[1]
send(marketDataRequest)
# after receiving the new order request response we stop the reactor
# And we will be disconnected from API
elif messageType == "8" or messageType == "j":
print("We are done, stopping the reactor")
reactor.stop()
def disconnected(client, reason): # Callback for client disconnection
print("\nDisconnected, reason: ", reason)
def connected(client): # Callback for client connection
print("Connected")
logonRequest = LogonRequest(config)
send(logonRequest)
# Setting client callbacks
client.setConnectedCallback(connected)
client.setDisconnectedCallback(disconnected)
client.setMessageReceivedCallback(onMessageReceived)
# Starting the client service
client.startService()
# Run Twisted reactor, we imported it earlier
reactor.run()
Can you explain the code to me and provide instructions on how to get for example hourly data for NatGas (1,000 observations)?`

How to handle blank line,junk line and \n while converting an input file to csv file

Below is the sample data in input file. I need to process this file and turn it into a csv file. With some help, I was able to convert it to csv file. However not fully converted to csv since I am not able to handle \n, junk line(2nd line) and blank line(4th line). Also, i need help to filter transaction_type i.e., avoid "rewrite" transaction_type
{"transaction_type": "new", "policynum": 4994949}
44uu094u4
{"transaction_type": "renewal", "policynum": 3848848,"reason": "Impressed with \n the Service"}
{"transaction_type": "cancel", "policynum": 49494949, "cancel_table":[{"cancel_cd": "AU"}, {"cancel_cd": "AA"}]}
{"transaction_type": "rewrite", "policynum": 5634549}
Below is the code
import ast
import csv
with open('test_policy', 'r') as in_f, open('test_policy.csv', 'w') as out_f:
data = in_f.readlines()
writer = csv.DictWriter(
out_f,
fieldnames=[
'transaction_type', 'policynum', 'cancel_cd','reason'],lineterminator='\n',
extrasaction='ignore')
writer.writeheader()
for row in data:
dict_row = ast.literal_eval(row)
if 'cancel_table' in dict_row:
cancel_table = dict_row['cancel_table']
cancel_cd= []
for cancel_row in cancel_table:
cancel_cd.append(cancel_row['cancel_cd'])
dict_row['cancel_cd'] = ','.join(cancel_cd)
writer.writerow(dict_row)
Below is my output not considering the junk line,blank line and transaction type "rewrite".
transaction_type,policynum,cancel_cd,reason
new,4994949,,
renewal,3848848,,"Impressed with
the Service"
cancel,49494949,"AU,AA",
Expected output
transaction_type,policynum,cancel_cd,reason
new,4994949,,
renewal,3848848,,"Impressed with the Service"
cancel,49494949,"AU,AA",
Hmm I try to fix them but I do not know how CSV file work, but my small knoll age will suggest you to run this code before to convert the file.
txt = {"transaction_type": "renewal",
"policynum": 3848848,
"reason": "Impressed with \n the Service"}
newTxt = {}
for i,j in txt.items():
# local var (temporar)
lastX = ""
correctJ = ""
# check if in J is ascii white space "\n" and get it out
if "\n" in f"b'{j}'":
j = j.replace("\n", "")
# for grammar purpose check if
# J have at least one space
if " " in str(j):
# if yes check it closer (one by one)
for x in ([j[y:y+1] for y in range(0, len(j), 1)]):
# if 2 spaces are consecutive pass the last one
if x == " " and lastX == " ":
pass
# if not update correctJ with new values
else:
correctJ += x
# remember what was the last value checked
lastX = x
# at the end make J to be the correctJ (just in case J has not grammar errors)
j = correctJ
# add the corrections to a new dictionary
newTxt[i]=j
# show the resoult
print(f"txt = {txt}\nnewTxt = {newTxt}")
Termina:
txt = {'transaction_type': 'renewal', 'policynum': 3848848, 'reason': 'Impressed with \n the Service'}
newTxt = {'transaction_type': 'renewal', 'policynum': 3848848, 'reason': 'Impressed with the Service'}
Process finished with exit code 0

How to swap string and save

I am having problems saving the file i modifyed basicly i need to replace in original file string called DTC_5814_removing and switch_data and save it as a seperate file how would i do that, so here is what program basicly does, it opens eeprom file, then searches for a string between two strings and groups it, then counts the data and by that given data searches for other string that is between two strings and modyfies data,basicly the code works i have a question how is the best way to save that as a seperate file, filesave function currently has no functin
here is the code:
import re
#checking the structures counting
file = open ("eeprom", "rb") .read().hex()
filesave = open("eepromMOD", "wb")
DTC_data = re.search("ffff30(.*)100077", file)
DTC_data_final = print (DTC_data.group(1))
#finds string between two strings in 2nd line of eeprom file
switch_data = re.search("010607(.*)313132", file)
switch_data_final = print (switch_data.group(1))
#finds string betwenn two strings in 3rd line of eeprom file
DTC_data_lenght = (len(DTC_data.group(1)))
#lenght of the whole DTC_data group
DTC_312D = re.search("ffff30(.*)312d", file)
DTC_3036 = re.search("ffff30(.*)3036", file)
DTC_5814 = re.search("ffff30(.*)5814", file)
#searching for DTC 312D
DTC_312D_lenght = (len(DTC_312D.group(1))+4)
DTC_312D_lenght_start =(len(DTC_312D.group(1)))
DTC_3036_lenght = (len(DTC_3036.group(1))+4)
DTC_3036_lenght_start =(len(DTC_3036.group(1)))
DTC_5814_lenght = (len(DTC_5814.group(1))+4)
DTC_5814_lenght_start =(len(DTC_5814.group(1)))
#confirming the lenght of the DTC table
if DTC_312D_lenght <= DTC_data_lenght and DTC_312D_lenght%4==0 :
#If dtc lenght shorter than whole table and devidable by 4
print("Starting DTC removal")
#Printing for good count
switch_data_lenght = (len(switch_data.group(1)))
#Counting switch data table
DTC_312D_removing = switch_data.group(1)[:DTC_312D_lenght_start] + "0000" + switch_data.group(1)[DTC_312D_lenght:]
#Read from data group (data[:define start] + "mod to wish value" + data[define end]
print(DTC_312D_removing)
else:
print("DTC non existant or incorrect")
if DTC_3036_lenght <= DTC_data_lenght and DTC_3036_lenght%4==0 :
#If dtc lenght shorter than whole table and devidable by 4
print("Starting DTC removal")
#Printing for good count
switch_data_lenght = (len(switch_data.group(1)))
#Counting switch data table
DTC_3036_removing = DTC_312D_removing[:DTC_3036_lenght_start] + "0000" + switch_data.group(1)[DTC_3036_lenght:]
#Read from data group (data[:define start] + "mod to wish value" + data[define end]
print(DTC_3036_removing)
else:
print("DTC non existant or incorrect")
if DTC_5814_lenght <= DTC_data_lenght and DTC_5814_lenght%4==0 :
#If dtc lenght shorter than whole table and devidable by 4
print("Starting DTC removal")
#Printing for good count
switch_data_lenght = (len(switch_data.group(1)))
#Counting switch data table
DTC_5814_removing = DTC_3036_removing[:DTC_5814_lenght_start] + "0000" + switch_data.group(1)[DTC_5814_lenght:]
#Read from data group (data[:define start] + "mod to wish value" + data[define end]
print(DTC_5814_removing)
else:
print("DTC non existant or incorrect")
Solved with
File_W = file.replace(switch_data.group(1), DTC_5814_removing)
File_WH = binascii.unhexlify(File_W)
filesave.write(File_WH)
filesave.close()

Syntax Error I Can't Understand [Python 3]

Here's my code:
cont = "y"
cont1 = "0"
cont2 = "yes"
log = 0
data = open("songs.txt", "r")
songs = data.readlines()
songs.sort()
while cont1 == "0":
cont = input("Do you want to make a playlist? [Y or N]") #Need to change to change cont1 if a playlist is not wanted to be created
if cont == "y":
new = "playlist" + str(log) + ".txt"
print(new, "Please remember the number. Thanks")
log = log + 1
else:
break
while cont == "y":
print ("1. ", songs[0],"2. ", songs[1],"3. ", songs[2],"4. ", songs[3],"5. ", songs[4],"6. ", songs[5],"7. ", songs[6],"8. ", songs[7],"9. ", songs[8],"10. ", songs[9],"11. ", songs[10],"12. ", songs[11],"13. ", songs[12],"14. ", songs[13],"15. ", songs[14],"16. ", songs[15],"17. ", songs[16],"18. ", songs[17],"19. ", songs[18],"20. ", songs[19])
choice = int(input("Please enter the first choice of song you would like in your playlist [Type the allocated number please]"))
choice1 = choice - 1
with open("playlist" + str(log) + ".txt", "a") as playlist:
playlist.write(str(songs[choice1]))
cont = input("Do you want to add another song? [Y or N]")
if cont == "n":
break
view = input("Which playlist would you like to view? Please enter the number
you were given before.")
with open("playlist" + str(view) + ".txt", "a") as see:
print(see)
The function of my code is a playlist generator for the user using the songs I already have. That all works up to now, but when trying to allow the user to view the playlist of their choice in this section of the code:
view = input("Which playlist would you like to view? Please enter the number
you were given before.")
with open("playlist" + str(view) + ".txt", "a") as see:
print(see)
I got an error message saying:
<_io.TextIOWrapper name='playlist2.txt' mode='a' encoding='UTF-8'>
Does this make sense to anyone?
Thanks!
By the way, the songs.txt contains a list of 20 songs in it. Cheers!
I think there might be a couple things here. playlist2.txt is not opened in a readable mode, so open needs r instead of a. see is a file object, so print needs see.read() to print the file contents.

Python pyodbc fetchmany() how to select out put to update query

I have code to fetchmany() that will output eg:10 records
And i have added iterating value for each 0 1 2 3 4 5 for print statement , now i want user input 0 or 1 and it should select column. For those input so i can update sql record for those column
cur.execute("select events.SERIALNUM, emp.LASTNAME, emp.SSNO,
events.EVENT_TIME_UTC from AccessControl.dbo.emp,
AccessControl.dbo.events where emp.id = events.empid and emp.SSNO=?
order by EVENT_TIME_UTC desc ", empid)
rows = cur.fetchmany(att_date)
n = 0
for row in rows :
event_date = row.EVENT_TIME_UTC
utc = event_date.replace(tzinfo=from_zone)
utc_to_local = utc.astimezone(to_zone)
local_time = utc_to_local.strftime('%H:%M:%S')
att_date = utc_to_local.strftime('%d:%m:%y')
print (n, row.SERIALNUM, row.LASTNAME, row.SSNO, att_date, local_time)
n = n + 1
seri_al = input("Copy And Past the serial number u want to modifiy: ")
this will output following Data
0 1500448188 FIRST NAME 03249 2017-07-19 17:01:17
1 1500448187 FIRST NAME 03249 2017-07-19 17:01:15
Eg:
seri_al = input("Copy And Past the serial number u want to modifiy: ")
instead of copying and pasting '1500448188' these numbers I want the user to only enter '0' and map that one and update sql query as for where clause serial number.
It appears that you already know how to use input to prompt for the user's choice. The only piece you are missing is to add items to a dictionary as you loop through the rows. Here is a slightly abstracted example:
rows = [('1500448188',),('1500448187',)] # test data
selections = dict()
n = 0
for row in rows:
selections[n] = row[0]
print(n, repr(row[0]))
n += 1
select = input("Enter the index (0, 1, ...) you want to select: ")
selected_key = selections[int(select)]
print("You selected " + repr(selected_key))
which prints
0 '1500448188'
1 '1500448187'
Enter the index (0, 1, ...) you want to select: 1
You selected '1500448187'

Resources