I have the following query in Excel using BQL:
=BQL("MEMBERS('INEMCBI LX Equity',type=holdings)";"name";"cols=2;rows=223")
This shows a table of securities of a specific mutual fund. Like this:
enter image description here
I want to get the same information using python in Dataframe structure but I dont know any way to get it.
If you run BQNT<GO> on the terminal, you'll get access to Python BQL examples, but the examples won't work outside of BQNT. Since you mention using =BQL, I'm assuming you have access to a terminal.
Example in BQNT:
import bql
bq = bql.Service()
fund = bq.univ.members("INEMCBI LX Equity", type="HOLDINGS")
name = bq.data.name()
req = bql.Request(fund, {"Fund Holdings": name})
res = bq.execute(req)
data = res[0].df()
data
Run BQNT on your Bloomberg terminal to ensure the BQL environment is installed.
Follow the steps exactly as followed.
Open file explorer
Navigate to C:\blp\bqnt\environments\bqnt-3\Lib\site-packages and copy these folders:
bqapi
bqbreg
bql
bqlmetadata
bqrequest
bqutil
ciso8601
Paste them to your python installation folder %userprofile%\Anaconda3\envs{mypythonenv}\lib\site-packages\
Then you can test this code in your code editor. I use Vscode.
import seaborn as sb
import pandas as pd
import matplotlib.pyplot as plt
import bql
bq = bql.Service()
query = """get(px_last)for('AAPL US EQUITY')with(dates=range(-1y,0d),fill='prev')"""
data = bql.combined_df(bq.execute(query)).reset_index()
fig = plt.figure(figsize=(12,8))
sb.lineplot(data=data, x='DATE',y='px_last')
plt.show()
Related
I can't get the below code to navigate through the disclaimer page of the website, I think the issue is how I try and collect the cookie.
I want to try and use requests rather than selenium.
import requests
import pandas as pd
from pandas import read_html
# open the page with the disclaimer just to get the cookies
disclaimer = "https://umm.gassco.no/disclaimer"
disclaimerdummy = requests.get(disclaimer)
# open the actual page and use the cookies from the fake page opened before
actualpage = "https://umm.gassco.no/disclaimer/acceptDisclaimer"
actualpage2 = requests.get(actualpage, cookies=disclaimerdummy.cookies)
# store the content of the actual page in text format
actualpagetext = (actualpage2.text)
# identify relevant data sources by looking at the 'msgTable' class in the webpage code
# This is where the tables with the realtime data can be found
gasscoflow = read_html(actualpagetext, attrs={"class": "msgTable"})
# create the dataframes for the two relevant tables
Table0 = pd.DataFrame(gasscoflow[0])
Table1 = pd.DataFrame(gasscoflow[1])
Table2 = pd.DataFrame(gasscoflow[2])
Table3 = pd.DataFrame(gasscoflow[3])
Table4 = pd.DataFrame(gasscoflow[4])
After Seeing the website first of all it has only 2 tables and you could use session to use cookies across request instead of storing in a variable follow the below code to get all your expected data it is printing only last 2 rows as I have used tail command, you can modify and get your desired data from those tables.
import requests
import pandas as pd
from pandas import read_html
s=requests.session()
s1=s.get("https://umm.gassco.no")
s2=s.get("https://umm.gassco.no/disclaimer/acceptDisclaimer?")
data = read_html(s2.text, attrs={"class": "msgTable"})
t0 = pd.DataFrame(data[0])
t1 = pd.DataFrame(data[1])
print(t0.tail(2))
print(t1.tail(2))
Output:
Let me know if you have any questions :)
I have found this python code online (twitter_map_clustered.py) which (I think) help create a map using the geodata of different tweets.:
from argparse import ArgumentParser
import folium
from folium.plugins import MarkerCluster
import json
def get_parser():
parser = ArgumentParser()
parser.add_argument('--geojson')
parser.add_argument('--map')
return parser
def make_map(geojson_file, map_file):
tweet_map = folium.Map(Location=[50, 5], max_zoom=20)
marker_cluster = MarkerCluster().add_to(tweet_map)
geodata= json.load(open(geojson_file))
for tweet in geodata['features']:
tweet['geometry']['coordinates'].reverse()
marker = folium.Marker(tweet['geometry']['coordinates'], popup=tweet['properties']['text'])
marker.add_to(marker_cluster)
#Save to HTML map file
tweet_map.save(map_file)
if __name__ == '__main__':
parser = get_parser()
args = parser.parse_args()
make_map(args.geojson, args.map)
I managed to extract the geo information of different tweets and save it into a geo_data.json file. However, I have trouble understanding the code, specially the function def get_parser().
It seems that we need to add argument when running the file in the command prompt. The argument should be geo_data.json. However, it is also asking for a map ? parser.add_argument('--map')
Why is it the case? In the code, aren't we creating the map here?
#Save to HTML map file
tweet_map.save(map_file)
Can you please help me. How would you run the python script ? Is there anything important I am missing ?
As explained by argparse documentation, it simply asks for the name of the geojson file and a name that your code will use to save the map.
Therefore, you will run:
python twitter_map_clustered.py --geojson geo_data.json --map mymap.html
and you will get a map named mymap.html.
When trying to run a ScriptRunConfig, using :
src = ScriptRunConfig(source_directory=project_folder,
script='train.py',
arguments=['--input-data-dir', ds.as_mount(),
'--reg', '0.99'],
run_config=run_config)
run = experiment.submit(config=src)
It doesn't work and breaks with this when I submit the job :
... lots of things... and then
TypeError: Object of type 'DataReference' is not JSON serializable
However if I run it with the Estimator, it works. One of the differences is the fact that with a ScriptRunConfig we're using a list for parameters and the other is a dictionary.
Thanks for any pointers!
Being able to use DataReference in ScriptRunConfig is a bit more involved than doing just ds.as_mount(). You will need to convert it into a string in arguments and then update the RunConfiguration's data_references section with the DataReferenceConfiguration created from ds. Please see here for an example notebook on how to do that.
If you are just reading from the input location and not doing any writes to it, please check out Dataset. It allows you to do exactly what you are doing without doing anything extra. Here is an example notebook that shows this in action.
Below is a short version of the notebook
from azureml.core import Dataset
# more imports and code
ds = Datastore(workspace, 'mydatastore')
dataset = Dataset.File.from_files(path=(ds, 'path/to/input-data/within-datastore'))
src = ScriptRunConfig(source_directory=project_folder,
script='train.py',
arguments=['--input-data-dir', dataset.as_named_input('input').as_mount(),
'--reg', '0.99'],
run_config=run_config)
run = experiment.submit(config=src)
you can see this link how-to-migrate-from-estimators-to-scriptrunconfig in official documents.
The core code of using DataReference in ScriptRunConfig is
# if you want to pass a DataReference object, such as the below:
datastore = ws.get_default_datastore()
data_ref = datastore.path('./foo').as_mount()
src = ScriptRunConfig(source_directory='.',
script='train.py',
arguments=['--data-folder', str(data_ref)], # cast the DataReference object to str
compute_target=compute_target,
environment=pytorch_env)
src.run_config.data_references = {data_ref.data_reference_name: data_ref.to_config()} # set a dict of the DataReference(s) you want to the `data_references` attribute of the ScriptRunConfig's underlying RunConfiguration object.
Below is the code I am working on. I am trying to scrape the historical data table that is located at nasdaq.com. I have it referencing a list of stocks compiled in an excel worksheet (can provide if needed,however not the issue). I am having a hard time identifying the table as well as being able to call the correct tag. I tried to follow an online tutorial with no avail.
When I run the code, in the python shell I just get shown empty brackets every few seconds.
<code>
import webbrowser, selenium, os, pyautogui, select, time, openpyxl,
bs4, requests, subprocess
from selenium import webdriver
from selenium.webdriver.support.ui import Select
wb = openpyxl.load_workbook('StockList.xlsx')
sheet = wb.get_sheet_by_name('Merged')
for row in range (2,211):
def clean_history(sel, domains):
temp = sel.get_location()
for domain in domains:
sel.open(domain)
sel.delete_all_visible_cookies()
sel.open(temp)
url = sheet['B' + str(row)].value
root_url = 'http://www.nasdaq.com'
index_url = root_url + '/symbol/' + url + '/historical'
def get_data():
response = requests.get(index_url)
soup = bs4.BeautifulSoup(response.text)
return [div.attrs.get('td') for tr in
soup.select('div.quotes_content_left_pn1AJAX')]
print(get_data())
</code>
Edit:
I only just taught myself python, hence the poor reiterations. I followed a book I picked up for some of this. And the actual interaction of the webpage I followed this link.
http://blog.miguelgrinberg.com/post/easy-web-scraping-with-python
I believe I am mislabeling something in the line of code with the Return line. I want to be able to take this data and store it as a text file or csv, or anything easily readable for further manipulation.
I am using Python3.4.2 and pythonOCC-0.16.0-win32-py34.exe to draw components. Every components are rendered properly with one defined color but that is not look like a real world component.
Above image is my Python implementation which generate 3D image from STEP file with one color.
Below image is rendered one of my windows software and there I have used Step file. I want to render component same as look like in below image so its look like a real world component.
Is there any way to get correct colored output in Python by read STEP file? I have searched a lot but didn't get a way to implement it. Please help me to go in forward direction.
from future import print_function
import sys
#from OCC.STEPCAFControl import STEPCAFControl_Reader
from OCC.STEPControl import STEPControl_Reader
from OCC.IFSelect import IFSelect_RetDone, IFSelect_ItemsByEntity
from OCC.Display.SimpleGui import init_display
from OCC.Display.WebGl import threejs_renderer
step_reader = STEPControl_Reader()
status = step_reader.ReadFile('./models/test.STEP')
if status == IFSelect_RetDone: # check status
failsonly = False
step_reader.PrintCheckLoad(failsonly, IFSelect_ItemsByEntity)
step_reader.PrintCheckTransfer(failsonly, IFSelect_ItemsByEntity)
ok = step_reader.TransferRoot(1)
_nbs = step_reader.NbShapes()
aResShape = step_reader.Shape(1)
else:
print("Error: can't read file.")
sys.exit(0)
#display, start_display, add_menu, add_function_to_menu = init_display()
#display.DisplayShape(aResShape, update=True)
#start_display()
my_renderer = threejs_renderer.ThreejsRenderer(background_color="#123345")
my_renderer.DisplayShape(aResShape)
The above code is used for read STEP file using Python.