How to parse a list in python3 - python-3.x

So I get this list back from interactive brokers. (API 9.73 using this repo)
ib = IB()
ib.connect('127.0.0.1', 7497, clientId=2)
data = ib.positions()
print((data))
print(type(data))
The data comes back as , but here is the response.
`[Position(account='DUC00074', contract=Contract(conId=43645865, symbol='IBKR', secType='STK', exchange='NASDAQ', currency='USD', localSymbol='IBKR', tradingClass='NMS'), position=2800.0, avgCost=39.4058383), Position(account='DUC00074', contract=Contract(conId=72063691, symbol='BRK B', secType='STK',exchange='NYSE', currency='USD', localSymbol='BRK B', tradingClass='BRK B'), position=250.0, avgCost=163.4365424)]`
I have got this far:
for d in data:
for i in d:
print(i)
But I have no idea as to how I would parse and then dump into a DB, anything after Position(... So to be really clear, I don't know how I would parse, like I would say in php / json.

Okay, Im new to python, not new to programing. So the response from Interactive Brokers threw me off. I'm so used to JSON response. Regardless, what it comes down to is this is a list of objects, (the example above). That might be simple, but missed it. Once I figured it out it became a little easier.
Here is the final code, hopefully this will help someone else down the line.
for obj in data:
#the line below just bascially dumps the object
print("obj={} ".format(obj))
#looking for account number
#"contract" is an object inside an object, so I had to extract that.
if(getattr(obj, 'account') == '123456'):
print("x={} ".format(getattr(obj, 'account')))
account = (getattr(obj, 'account'))
contract = getattr(obj, 'contract')
contractId = getattr(contract, 'conId')
symbol = getattr(contract, 'symbol')
secType = getattr(contract, 'secType')
exdate = getattr(contract, 'lastTradeDateOrContractMonth')
strike = getattr(contract, 'strike')
opt_type = getattr(contract, 'right')
localSymbol = getattr(contract, 'localSymbol')
position = getattr(obj, 'position')
avgCost = getattr(obj, 'avgCost')
sql = "insert into IB_opt (account, contractId, symbol, secType, exdate, opt_type, localSymbol, position, avgCost,strike) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)";
args = (account, contractId, symbol, secType, exdate, opt_type, localSymbol, position, avgCost,strike)
cursor.execute(sql, args)
Of all the sites I looked at this one was pretty helpful.

Related

slider.value values not getting updated using ColumnDataSource(Dataframe).data

I have been working on COVID19 analysis for a dashboard and am using a JSON data source. I have converted the json to dataframe. I am working on plotting bar chart for "Days to reach deaths" over a "States" x-axis (categorical values). I am trying to use a function to update the slider.value. Upon running the bokeh serve with --log-level=DEBUG, I am getting a following error:
Can someone provide me with any direction or help with what might be causing the issue as I am new to Python and any help is appreciated? Or if there's any other alternative.
Please find the code below:
cases_summary = requests.get('https://api.rootnet.in/covid19-in/stats/history')
json_data = cases_summary.json()
#Data Cleaning
cases_summary=pd.json_normalize(json_data['data'], record_path='regional', meta='day')
cases_summary['loc']=np.where(cases_summary['loc']=='Nagaland#', 'Nagaland', cases_summary['loc'])
cases_summary['loc']=np.where(cases_summary['loc']=='Madhya Pradesh#', 'Madhya Pradesh', cases_summary['loc'])
cases_summary['loc']=np.where(cases_summary['loc']=='Jharkhand#', 'Jharkhand', cases_summary['loc'])
#Calculate cumulative days since 1st case for each state
cases_summary['day_count']=(cases_summary['day'].groupby(cases_summary['loc']).cumcount())+1
#Initial plot for default slider value=35
days_reach_death_count=cases_summary.loc[(cases_summary['deaths']>=35)].groupby(cases_summary['loc']).head(1).reset_index()
slider = Slider(start=10, end=max(cases_summary['deaths']), value=35, step=10, title="Total Deaths")
source = ColumnDataSource(data=dict(days_reach_death_count[['loc','day_count', 'deaths']]))
q = figure(x_range=days_reach_death_count['loc'], plot_width=1200, plot_height=600, sizing_mode="scale_both")
q.title.align = 'center'
q.title.text_font_size = '17px'
q.xaxis.axis_label = 'State'
q.yaxis.axis_label = 'Days since 1st Case'
q.xaxis.major_label_orientation = math.pi/2
q.vbar('loc', top='day_count', width=0.9, source=source)
deaths = slider.value
q.title.text = 'Days to reach %d Deaths' % deaths
hover = HoverTool(line_policy='next')
hover.tooltips = [('State', '#loc'),
('Days since 1st Case', '#day_count'), # #$name gives the value corresponding to the legend
('Deaths', '#deaths')
]
q.add_tools(hover)
def update(attr, old, new):
days_death_count = cases_summary.loc[(cases_summary['deaths'] >= slider.value)].groupby(cases_summary['loc']).head(1).reindex()
source.data = [ColumnDataSource().from_df(days_death_count)]
slider.on_change('value', update)
layout = row(q, slider)
tab = Panel(child=layout, title="New Confirmed Cases since Day 1")
tabs= Tabs(tabs=[tab])
curdoc().add_root(tabs)
Your code has 2 issues
(critical) source.data must be a dictionary, but you're assigning it an array
(minor) from_df is a class method, you don't have to construct an object of it
Try using source.data = ColumnDataSource.from_df(days_death_count) instead.

Twitter API: How to make query keep running?

Novice programmer here seeking help.
I already set up my code to my requirements to use the Twitter's premium API.
SEARCH_TERM = '#AAPL OR #FB OR #KO OR #ABT OR #PEPCO'
PRODUCT = 'fullarchive'
LABEL = 'my_label'
r = api.request('tweets/search/%s/:%s' % (PRODUCT, LABEL),
{'query':SEARCH_TERM, 'fromDate':201501010000, 'toDate':201812310000})
However, when I run it I obtain the maximum number of tweets per search which is 500.
My question is should I add to the query maxResults = 500? And how do I use the next parameter to keep the code running until all the tweets that correspond to my query are obtained?
To up the results from the default of 100 to 500, yes, add maxResults to the query like this:
r = api.request('tweets/search/%s/:%s' % (PRODUCT, LABEL),
{
'query':SEARCH_TERM,
'fromDate':201501010000, 'toDate':201812310000,
'maxResults':500
})
You can make successive queries to get more results by using the next parameter. But, even easier, you can let TwitterAPI do this for you by using the TwitterPager class. Here is an example:
from TwitterAPI import TwitterAPI, TwitterPager
SEARCH_TERM = '#AAPL OR #FB OR #KO OR #ABT OR #PEPCO'
PRODUCT = 'fullarchive'
LABEL = 'my_label'
api = TwitterAPI(<consumer key>,
<consumer secret>,
<access token key>,
<access token secret>)
pager = TwitterPager(api, 'tweets/search/%s/:%s' % (PRODUCT, LABEL),
{
'query':SEARCH_TERM,
'fromDate':201501010000, 'toDate':201812310000
})
for item in pager.get_iterator():
print(item['text'] if 'text' in item else item)
This example will keep making successive requests with the next parameter until no tweets can be downloaded.
Use the count variable in a raw_query, for example:
results = api.GetSearch(
raw_query="q=twitter%20&result_type=recent&since=2014-07-19&count=100")

Pygal bar chart says “No data”

I am trying to create a bar graph in pygal that uses the api for hacker news and charts the most active news based on comments. I posted my code below, but I cannot figure out why my graph keep saying "No data"??? Any suggestions? Thanks!
import requests
import pygal
from pygal.style import LightColorizedStyle as LCS, LightenStyle as LS
from operator import itemgetter
# Make an API call, and store the response.
url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
r = requests.get(url)
print("Status code:", r.status_code)
# Process information about each submission.
submission_ids = r.json()
submission_dicts = []
for submission_id in submission_ids[:30]:
# Make a separate API call for each submission.
url = ('https://hacker-news.firebaseio.com/v0/item/' +
str(submission_id) + '.json')
submission_r = requests.get(url)
print(submission_r.status_code)
response_dict = submission_r.json()
submission_dict = {
'comments': int(response_dict.get('descendants', 0)),
'title': response_dict['title'],
'link': 'http://news.ycombinator.com/item?id=' + str(submission_id),
}
submission_dicts.append(submission_dict)
# Visualization
my_style = LS('#336699', base_style=LCS)
my_config = pygal.Config()
my_config.show_legend = False
my_config.title_font_size = 24
my_config.label_font_size = 14
my_config.major_label_font_size = 18
my_config.show_y_guides = False
my_config.width = 1000
chart = pygal.Bar(my_config, style=my_style)
chart.title = 'Most Active News on Hacker News'
chart.add('', submission_dicts)
chart.render_to_file('hn_submissons_repos.svg')
The values in the array passed to the add function need to be either numbers or dicts that contain the key value (or a mixture of the two). The simplest solution would be to change the keys used when creating submission_dict:
submission_dict = {
'value': int(response_dict.get('descendants', 0)),
'label': response_dict['title'],
'xlink': 'http://news.ycombinator.com/item?id=' + str(submission_id),
}
Notice that link has become xlink, this is one of the optional parameters that are defined in the Value Configuration section of the pygal docs.

Python - Error querying Solarwinds N-Central via SOAP

I'm using python 3 to write a script that generates a customer report for Solarwinds N-Central. The script uses SOAP to query N-Central and I'm using zeep for this project. While not new to python I am new to SOAP.
When calling the CustomerList fuction I'm getting the TypeError: __init__() got an unexpected keyword argument 'listSOs'
import zeep
wsdl = 'http://' + <server url> + '/dms/services/ServerEI?wsdl'
client = zeep.CachingClient(wsdl=wsdl)
config = {'listSOs': 'true'}
customers = client.service.CustomerList(Username=nc_user, Password=nc_pass, Settings=config)
Per the perameters below 'listSOs' is not only a valid keyword, its the only one accepted.
CustomerList
public com.nable.nobj.ei.Customer[] CustomerList(String username, String password, com.nable.nobj.ei.T_KeyPair[] settings) throws RemoteException
Parameters:
username - MSP N-central username
password - Corresponding MSP N-central password
settings - A list of non default settings stored in a T_KeyPair[]. Below is a list of the acceptable Keys and Values. If not used leave null
(Key) listSOs - (Value) "true" or "false". If true only SOs with be shown, if false only customers and sites will be shown. Default value is false.
I've also tried passing the dictionary as part of a list:
config = []
key = {'listSOs': 'true'}
config += key
TypeError: Any element received object of type 'str', expected lxml.etree._Element or builtins.dict or zeep.objects.T_KeyPair
Omitting the Settings value entirely:
customers = client.service.CustomerList(Username=nc_user, Password=nc_pass)
zeep.exceptions.ValidationError: Missing element Settings (CustomerList.Settings)
And trying zeep's SkipValue:
customers = client.service.CustomerList(Username=nc_user, Password=nc_pass, Settings=zeep.xsd.SkipValue)
zeep.exceptions.Fault: java.lang.NullPointerException
I'm probably missing something simple but I've been banging my head against the wall off and on this for awhile I'm hoping someone can point me in the right direction.
Here's my source code from my getAssets.py script. I did it in Python2.7, easily upgradeable though. Hope it helps someone else, N-central's API documentation is really bad lol.
#pip2.7 install zeep
import zeep, sys, csv, copy
from zeep import helpers
api_username = 'your_ncentral_api_user'
api_password='your_ncentral_api_user_pw'
wsdl = 'https://(yourdomain|tenant)/dms2/services2/ServerEI2?wsdl'
client = zeep.CachingClient(wsdl=wsdl)
response = client.service.deviceList(
username=api_username,
password=api_password,
settings=
{
'key': 'customerId',
'value': 1
}
)
# If you can't tell yet, I code sloppy
devices_list = []
device_dict = {}
dev_inc = 0
max_dict_keys = 0
final_keys = []
for device in response:
# Iterate through all device nodes
for device_properties in device.items:
# Iterate through each device's properties and add it to a dict (keyed array)
device_dict[device_properties.first]=device_properties.second
# Dig further into device properties
device_properties = client.service.devicePropertyList(
username=api_username,
password=api_password,
deviceIDs=device_dict['device.deviceid'],
reverseOrder=False
)
prop_ind = 0 # This is a hacky thing I did to make my CSV writing work
for device_node in device_properties:
for prop_tree in device_node.properties:
for key, value in helpers.serialize_object(prop_tree).items():
prop_ind+=1
device_dict["prop" + str(prop_ind) + "_" + str(key)]=str(value)
# Append the dict to a list (array), giving us a multi dimensional array, you need to do deep copy, as .copy will act like a pointer
devices_list.append(copy.deepcopy(device_dict))
# check to see the amount of keys in the last item
if len(devices_list[-1].keys()) > max_dict_keys:
max_dict_keys = len(devices_list[-1].keys())
final_keys = devices_list[-1].keys()
print "Gathered all the datas of N-central devices count: ",len(devices_list)
# Write the data out to a CSV
with open('output.csv', 'w') as csvfile:
fieldnames = final_keys
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
for csv_line in devices_list:
writer.writerow(csv_line)

Assign Class attributes from list elements

I'm not sure if the title accurately describes what I'm trying to do. I have a Python3.x script that I wrote that will issue flood warning to my facebook page when the river near my home has reached it's lowest flood stage. Right now the script works, however it only reports data from one measuring station. I would like to be able to process the data from all of the stations in my county (total of 5), so I was thinking that maybe a class method may do the trick but I'm not sure how to implement it. I've been teaching myself Python since January and feel pretty comfortable with the language for the most part, and while I have a good idea of how to build a class object I'm not sure how my flow chart should look. Here is the code now:
#!/usr/bin/env python3
'''
Facebook Flood Warning Alert System - this script will post a notification to
to Facebook whenever the Sabine River # Hawkins reaches flood stage (22.3')
'''
import requests
import facebook
from lxml import html
graph = facebook.GraphAPI(access_token='My_Access_Token')
river_url = 'http://water.weather.gov/ahps2/river.php?wfo=SHV&wfoid=18715&riverid=203413&pt%5B%5D=147710&allpoints=143204%2C147710%2C141425%2C144668%2C141750%2C141658%2C141942%2C143491%2C144810%2C143165%2C145368&data%5B%5D=obs'
ref_url = 'http://water.weather.gov/ahps2/river.php?wfo=SHV&wfoid=18715&riverid=203413&pt%5B%5D=147710&allpoints=143204%2C147710%2C141425%2C144668%2C141750%2C141658%2C141942%2C143491%2C144810%2C143165%2C145368&data%5B%5D=all'
def checkflood():
r = requests.get(river_url)
tree = html.fromstring(r.content)
stage = ''.join(tree.xpath('//div[#class="stage_stage_flow"]//text()'))
warn = ''.join(tree.xpath('//div[#class="current_warns_statmnts_ads"]/text()'))
stage_l = stage.split()
level = float(stage_l[2])
#check if we're at flood level
if level < 22.5:
pass
elif level == 37:
major_diff = level - 23.0
major_r = ('The Sabine River near Hawkins, Tx has reached [Major Flood Stage]: #', stage_l[2], 'Ft. ', str(round(major_diff, 2)), ' Ft. \n Please click the link for more information.\n\n Current Warnings and Alerts:\n ', warn)
major_p = ''.join(major_r)
graph.put_object(parent_object='me', connection_name='feed', message = major_p, link = ref_url)
<--snip-->
checkflood()
Each station has different 5 different catagories for flood stage: Action, Flood, Moderate, Major, each different depths per station. So for Sabine river in Hawkins it will be Action - 22', Flood - 24', Moderate - 28', Major - 32'. For the other statinos those depths are different. So I know that I'll have to start out with something like:
class River:
def __init__(self, id, stage):
self.id = id #station ID
self.stage = stage #river level'
#staticmethod
def check_flood(stage):
if stage < 22.5:
pass
elif stage.....
but from there I'm not sure what to do. Where should it be added in(to?) the code, should I write a class to handle the Facebook postings as well, is this even something that needs a class method to handle, is there any way to clean this up for efficiency? I'm not looking for anyone to write this up for me, but some tips and pointers would sure be helpful. Thanks everyone!
EDIT Here is what I figured out and is working:
class River:
name = ""
stage = ""
action = ""
flood = ""
mod = ""
major = ""
warn = ""
def checkflood(self):
if float(self.stage) < float(self.action):
pass
elif float(self.stage) >= float(self.major):
<--snip-->
mineola = River()
mineola.name = stations[0]
mineola.stage = stages[0]
mineola.action = "13.5"
mineola.flood = "14.0"
mineola.mod = "18.0"
mineola.major = "21.0"
mineola.alert = warn[0]
hawkins = River()
hawkins.name = stations[1]
hawkins.stage = stages[1]
hawkins.action = "22.5"
hawkins.flood = "23.0"
hawkins.mod = "32.0"
hawkins.major = "37.0"
hawkins.alert = warn[1]
<--snip-->
So from here I'm tring to stick all the individual river blocks into one block. What I have tried so far is this:
class River:
... name = ""
... stage = ""
... def testcheck(self):
... return self.name, self.stage
...
>>> for n in range(num_river):
... stations[n] = River()
... stations[n].name = stations[n]
... stations[n].stage = stages[n]
...
>>> for n in range(num_river):
... stations[n].testcheck()
...
<__main__.River object at 0x7fbea469bc50> 4.13
<__main__.River object at 0x7fbea46b4748> 20.76
<__main__.River object at 0x7fbea46b4320> 22.13
<__main__.River object at 0x7fbea46b4898> 16.08
So this doesn't give me the printed results that I was expecting. How can I return the string instead of the object? Will I be able to define the Class variables in this manner or will I have to list them out individually? Thanks again!
After reading many, many, many articles and tutorials on class objects I was able to come up with a solution for creating the objects using list elements.
class River():
def __init__(self, river, stage, flood, action):
self.river = river
self.stage = stage
self.action = action
self.flood = flood
self.action = action
def alerts(self):
if float(self.stage < self.flood):
#alert = "The %s is below Flood Stage (%sFt) # %s Ft. \n" % (self.river, self.flood, self.stage)
pass
elif float(self.stage > self.flood):
alert = "The %s has reached Flood Stage(%sFt) # %sFt. Warnings: %s \n" % (self.river, self.flood, self.stage, self.action)
return alert
'''this is the function that I was trying to create
to build the class objects automagically'''
def riverlist():
river_list = []
for n in range(len(rivers)):
station = River(river[n], stages[n], floods[n], warns[n])
river_list.append(station)
return river_list
if __name__ == '__main__':
for x in riverlist():
print(x.alerts())

Resources