I have a IIS log file with lines in the following format :
61.245.163.59 - [16/May/2013:23:55:09 +0530] "GET /ehrm/Recruitment/Images/divider.gif HTTP/1.1" 404 1245
"http://www.example.com/ehrm/Recruitment/MyApplication.aspx?PRF_ID=000005&digest=6LL4BTSuW9YnE5R4T8k27Q" "Mozilla/5.0 (Windows NT
6.1; rv:20.0) Gecko/20100101 Firefox/20.0" GET /ehrm/Recruitment/Images/divider.gif - HTTP/1.1 www.example.com
I want get some columns from this and build a dataframe. In the following method it just build a dataframe with one column. I want to have each of the splitting columns to be one column of the dataframe? And the other thing is the length of log file lines are not unique, so how to improve the accuracy of taking values by splitting like this?
log_list = []
for line in f:
ip = (line.split(' ')[0])
time = (line.split(' ')[2])
method = (line.split(' ')[4])
status = (line.split(' ')[7])
bytes = (line.split(' ')[8])
referrer = (line.split(' ')[9])
agent = (line.split(' ')[10])
data = ip + ' ' + time + ' ' + method + ' ' + status + ' ' + bytes + ' ' + referrer + ' ' + agent
log_list.append(data)
df = pandas.DataFrame(log_list)
The following code should accomplish what you're trying to do:
from pandas import read_csv
log_file = 'filename.log'
df = read_csv(log_file, sep=r'\s+', usecols=[0, 2, 4, 7, 8, 9, 10])
read_csv documentation.
Related
Objective is to acquire stock price data for each stock ticker, then assign a relevant variable its current price. ie. var 'sandp' links to ticker_symbol 'GSPC' which equals the stocks closing price. This bit works. However, I wish to return each variable value which I can then use within another function, but how do I access that variable and its value?
Here is my code:
def live_indices():
"""Acquire stock value from Yahoo Finance using stock ticker as key. Then assign the relevant variable to the respective value.
ie. variable 'sandp' equates to the value gathered from 'GSPC' stock ticker.
"""
import requests
import bs4
ticker_symbol_1 = ['GSPC', 'DJI', 'IXIC', 'FTSE', 'NSEI', 'FCHI', 'N225', 'GDAXI']
ticker_symbol_2 = ['IMOEX.ME', '000001.SS'] # Assigned to seperate list as url for webscraping is different
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'}
all_indices_values = []
for i in range(len(ticker_symbol_1)):
url = 'https://uk.finance.yahoo.com/quote/%5E' + ticker_symbol_1[i] + '?p=%5E' + ticker_symbol_1[i]
tmp_res = requests.get(url, headers=headers)
tmp_res.raise_for_status()
soup = bs4.BeautifulSoup(tmp_res.text, 'html.parser')
indices_price_value = soup.select(
'#quote-header-info > div.My\(6px\).Pos\(r\).smartphone_Mt\(6px\).W\(100\%\) > div.D\(ib\).Va\(m\).Maw\('
'65\%\).Ov\(h\) > div > fin-streamer.Fw\(b\).Fz\(36px\).Mb\(-4px\).D\(ib\)')[
0].text
all_indices_values.append(indices_price_value)
for i in range(len(ticker_symbol_2)):
url = 'https://uk.finance.yahoo.com/quote/' + ticker_symbol_2[i] + '?p=' + ticker_symbol_2[i]
tmp_res = requests.get(url, headers=headers)
tmp_res.raise_for_status()
soup = bs4.BeautifulSoup(tmp_res.text, 'html.parser')
indices_price_value = soup.select(
'#quote-header-info > div.My\(6px\).Pos\(r\).smartphone_Mt\(6px\).W\(100\%\) > div.D\(ib\).Va\(m\).Maw\('
'65\%\).Ov\(h\) > div > fin-streamer.Fw\(b\).Fz\(36px\).Mb\(-4px\).D\(ib\)')[
0].text
all_indices_values.append(indices_price_value)
sandp, dow, nasdaq, ftse100, nifty50, cac40, nikkei, dax, moex, shanghai = [all_indices_values[i] for i in range(10)] # 10 stock tickers in total
return sandp, dow, nasdaq, ftse100, nifty50, cac40, nikkei, dax, moex, shanghai
I want the next function to simply be given the variable name returned from the first function to print out the stock value. I have tried the below to no avail-
def display_value(stock_name):
print(stock_name)
display_value(live_indices(sandp))
The obvious error here is that 'sandp' is not defined.
Additionally, the bs4 code runs fairly slowly, would it be best to use Threads() or is there another way to speed things up?
This looks a bit complicated in my opinion, anyway focus on your question. So you are not returning a variable, you are returning a tuple of values.
def live_indices():
all_indices_values = [1,2,3,4,5,6,7,8,9,10,11,12,13]
sandp, dow, nasdaq, ftse100, nifty50, cac40, nikkei, dax, moex, shanghai = [all_indices_values[i] for i in range(10)] # 10 stock tickers in total
return sandp, dow, nasdaq, ftse100, nifty50, cac40, nikkei, dax, moex, shanghai
live_indices() #-> (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
What you want to make more likely to have is this assignment and it do not need the list comprehension nor the range() simply slice your list:
def live_indices():
all_indices_values = [1,2,3,4,5,6,7,8,9,10,11,12,13]
return all_indices_values
def display_value(x):
print(x)
sandp, dow, nasdaq, ftse100, nifty50, cac40, nikkei, dax, moex, shanghai = live_indices()[:10]
display_value(sandp)#-> 1
You may wanna work with more structured data, so return a dict:
Example
import requests
from bs4 import BeautifulSoup
def live_indices():
all_indices_values = {}
symbols = ['GSPC', 'DJI', 'IXIC', 'FTSE', 'NSEI', 'FCHI', 'N225', 'GDAXI', 'IMOEX.ME', '000001.SS']
for ticker in symbols:
url = f'https://uk.finance.yahoo.com/lookup/all?s={ticker}'
tmp_res = requests.get(url, headers=headers)
tmp_res.raise_for_status()
soup = bs4.BeautifulSoup(tmp_res.text, 'html.parser')
indices_price_value = soup.select('#Main tbody>tr td')[2].text
all_indices_values[ticker] = indices_price_value
return all_indices_values
def display_value(live_indices):
for ticker in live_indices.items():
print(ticker)
display_value(live_indices())
Output
('GSPC', '3,873.33')
('DJI', '30,822.42')
('IXIC', '11,448.40')
('FTSE', '7,236.68')
('NSEI', '17,530.85')
('FCHI', '6,077.30')
('N225', '27,567.65')
('GDAXI', '12,741.26')
('IMOEX.ME', '2,222.51')
('000001.SS', '3,126.40')
Good day, everyone.
I'm trying to get the table on each page from the links appended to 'player_page.'
I want the stats per game for each player in that season, and the table I want is listed on the players' individual page. Each link appended is correct, but I'm having trouble capturing the correct info when running my loops.
Any idea what I'm doing wrong here?
Any help is appreciated.
from bs4 import BeautifulSoup
import requests
import pandas as pd
from numpy import sin
url = 'https://www.pro-football-reference.com'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36'
}
year = 2018
r = requests.get(url + '/years/' + str(year) + '/fantasy.htm')
soup = BeautifulSoup(r.content, 'lxml')
player_list = soup.find_all('td', attrs= {'class': 'left', 'data-stat': 'player'})
player_page = []
for player in player_list:
for link in player.find_all('a', href= True):
#names = str(link['href'])strip('')
link = str(link['href'].strip('.htm'))
player_page.append(url + link + '/gamelog' + '/' + str(year))
for page in player_page:
dfs = pd.read_html(page)
yearly_stats = []
for df in dfs:
yearly_stats.append(df)
final_stats = pd.concat(yearly_stats)
final_stats.to_excel('Fantasy2018.xlsx')
This works. The table columns change according to the player's position, I believe. Not everyone has tackle information, for example.
import pandas as pd
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.pro-football-reference.com'
year = 2018
r = requests.get(url + '/years/' + str(year) + '/fantasy.htm')
soup = BeautifulSoup(r.content, 'lxml')
player_list = soup.find_all('td', attrs= {'class': 'left', 'data-stat': 'player'})
dfs = []
for player in player_list:
for link in player.find_all('a', href= True):
name = link.getText()
link = str(link['href'].strip('.htm'))
try:
df = pd.read_html(url + link + '/gamelog' + '/' + str(year))[0]
for i, columns_old in enumerate(df.columns.levels):
columns_new = np.where(columns_old.str.contains('Unnamed'), '' , columns_old)
df.rename(columns=dict(zip(columns_old, columns_new)), level=i, inplace=True)
df.columns = df.columns.map('|'.join).str.strip('|')
df['Date'] = pd.to_datetime(df['Date'], errors='coerce')
df = df.dropna(subset=['Date'])
df.insert(0,'Name',name)
df.insert(1,'Moment','Regular Season')
dfs.append(df)
except:
pass
try:
df1 = pd.read_html(url + link + '/gamelog' + '/' + str(year))[1]
for i, columns_old in enumerate(df1.columns.levels):
columns_new = np.where(columns_old.str.contains('Unnamed'), '' , columns_old)
df1.rename(columns=dict(zip(columns_old, columns_new)), level=i, inplace=True)
df1.columns = df1.columns.map('|'.join).str.strip('|')
df1['Date'] = pd.to_datetime(df1['Date'], errors='coerce')
df1 = df1.dropna(subset=['Date'])
df1.insert(0,'Name',name)
df1.insert(1,'Moment','Playoffs')
dfs.append(df1)
except:
pass
dfall = pd.concat(dfs)
dfall.to_excel('Fantasy2018.xlsx')
I am trying to automate stock price data extraction from https://www.nseindia.com/. Data is stored as a zip file and url for the zip file file varies by date. If on a certain date stock market is closed eg - weekends and holidays, there would be no file/url.
I want to identify invalid links (links that dont exist) and skip to next link.
This is a valid link -
path = 'https://archives.nseindia.com/content/historical/EQUITIES/2021/MAY/cm05MAY2021bhav.csv.zip'
This is an invalid link - (as 1st May is a weekend and stock market is closed for the day)
path2 = 'https://archives.nseindia.com/content/historical/EQUITIES/2021/MAY/cm01MAY2021bhav.csv.zip'
This is what I do to extract the data
from urllib.request import urlopen
from io import BytesIO
from zipfile import ZipFile
import pandas as pd
import datetime
start_date = datetime.date(2021, 5, 3)
end_date = datetime.date(2021, 5, 7)
delta = datetime.timedelta(days=1)
final = pd.DataFrame()
while start_date <= end_date:
print(start_date)
day = start_date.strftime('%d')
month = start_date.strftime('%b').upper()
year = start_date.strftime('%Y')
start_date += delta
path = 'https://archives.nseindia.com/content/historical/EQUITIES/' + year + '/' + month + '/cm' + day + month + year + 'bhav.csv.zip'
file = 'cm' + day + month + year + 'bhav.csv'
try:
with urlopen(path) as f:
with BytesIO(f.read()) as b, ZipFile(b) as myzipfile:
foofile = myzipfile.open(file)
df = pd.read_csv(foofile)
final.append(df)
except:
print(file + 'not there')
If the path is invalid, python is stuck and I have to restart Python. I am not able to error handle or identify invalid link while looping over multiple dates.
What I have tried so far to differentiate between valid and invalid links -
# Attempt 1
import os
os.path.exists(path)
os.path.isfile(path)
os.path.isdir(path)
os.path.islink(path)
# output is False for both Path and Path2
# Attempt 2
import validators
validators.url(path)
# output is True for both Path and Path2
# Attempt 3
import requests
site_ping = requests.get(path)
site_ping.status_code < 400
# Output for Path is True, but Python crashes/gets stuck when I run requests.get(path2) and I have to restart everytime.
Thanks for your help in advance.
As suggested by SuperStormer - adding a timeout to the request solved the issue
try:
with urlopen(zipFileURL, timeout = 5) as f:
with BytesIO(f.read()) as b, ZipFile(b) as myzipfile:
foofile = myzipfile.open(file)
df = pd.read_csv(foofile)
final.append(df)
except:
print(file + 'not there')
I'm trying to loop pages from this link and extract the interesting part.
Please see the contents in the red circle in the image below.
Here's what I've tried:
url = 'http://so.eastmoney.com/Ann/s?keyword=购买物业&pageindex={}'
for page in range(10):
r = requests.get(url.format(page))
soup = BeautifulSoup(r.content, "html.parser")
print(soup)
xpath for each element (might be helpful for those that don't read Chinese):
/html/body/div[3]/div/div[2]/div[2]/div[3]/h3/span --> 【润华物业】
/html/body/div[3]/div/div[2]/div[2]/div[3]/h3/a --> 润华物业:关于公司购买理财产品的公告
/html/body/div[3]/div/div[2]/div[2]/div[3]/p/label --> 2017-04-24
/html/body/div[3]/div/div[2]/div[2]/div[3]/p/span --> 公告编号：2017-019 证券代码：836007 证券简称：润华物业 主办券商：国联证券
/html/body/div[3]/div/div[2]/div[2]/div[3]/a --> http://data.eastmoney.com/notices/detail/836007/AN201704250530124271,JWU2JWI2JWE2JWU1JThkJThlJWU3JTg5JWE5JWU0JWI4JTlh.html
I need to save the output to an Excel file. How could I do that in Python? Many thanks.
BeautifulSoup won't see this stuff, as it's rendered dynamically by JS, but there's an API endpoint you can query to get what you're after.
Here's how:
import requests
import pandas as pd
def clean_up(text: str) -> str:
return text.replace('</em>', '').replace(':<em>', '').replace('<em>', '')
def get_data(page_number: int) -> dict:
url = f"http://searchapi.eastmoney.com/business/Web/GetSearchList?type=401&pageindex={page_number}&pagesize=10&keyword=购买物业&name=normal"
headers = {
"Referer": f"http://so.eastmoney.com/Ann/s?keyword=%E8%B4%AD%E4%B9%B0%E7%89%A9%E4%B8%9A&pageindex={page_number}",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0",
}
return requests.get(url, headers=headers).json()
def parse_response(response: dict) -> list:
for item in response["Data"]:
title = clean_up(item['NoticeTitle'])
date = item['NoticeDate']
url = item['Url']
notice_content = clean_up(" ".join(item['NoticeContent'].split()))
company_name = item['SecurityFullName']
print(f"{company_name} - {title} - {date}")
yield [title, url, date, company_name, notice_content]
def save_results(parsed_response: list):
df = pd.DataFrame(
parsed_response,
columns=['title', 'url', 'date', 'company_name', 'content'],
)
df.to_excel("test_output.xlsx", index=False)
if __name__ == "__main__":
output = []
for page in range(1, 11):
for parsed_row in parse_response(get_data(page)):
output.append(parsed_row)
save_results(output)
This outputs:
栖霞物业购买资产的公告 - 2019-09-03 16:00:00 - 871792
索克物业购买资产的公告 - 2020-08-17 00:00:00 - 832816
中都物业购买股权的公告 - 2019-12-09 16:00:00 - 872955
开元物业:开元物业购买银行理财产品的公告 - 2015-05-21 16:00:00 - 831971
开元物业:开元物业购买银行理财产品的公告 - 2015-04-12 16:00:00 - 831971
盛全物业:拟购买房产的公告 - 2017-10-30 16:00:00 - 834070
润华物业购买资产暨关联交易公告 - 2016-08-23 16:00:00 - 836007
润华物业购买资产暨关联交易公告 - 2017-08-14 16:00:00 - 836007
萃华珠宝:关于拟购买物业并签署购买意向协议的公告 - 2017-07-10 16:00:00 - 002731
赛意信息:关于购买办公物业的公告 - 2020-12-02 00:00:00 - 300687
And saves this to a .csv file that can be easily handled by excel.
PS. I don't know Chinese (?) so you'd have to look into the response contents and pick more stuff out.
Updated answer based on #baduker's solution, but not working out for loop pages.
import requests
import pandas as pd
for page in range(10):
url = "http://searchapi.eastmoney.com/business/Web/GetSearchList?type=401&pageindex={}&pagesize=10&keyword=购买物业&name=normal"
headers = {
"Referer": "http://so.eastmoney.com/Ann/s?keyword=%E8%B4%AD%E4%B9%B0%E7%89%A9%E4%B8%9A&pageindex={}",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0",
}
response = requests.get(url, headers=headers).json()
output_data = []
for item in response["Data"]:
# print(item)
# print('*' * 40)
title = item['NoticeTitle'].replace('</em>', '').replace(':<em>', '').replace('<em>', '')
url = item['Url']
date = item['NoticeDate'].split(' ')[0]
company_name = item['SecurityFullName']
content = item['NoticeContent'].replace('</em>', '').replace(':<em>', '').replace('<em>', '')
# url_code = item['Url'].split('/')[5]
output_data.append([title, url, date, company_name, content])
names = ['title', 'url', 'date', 'company_name', 'content']
df = pd.DataFrame(output_data, columns = names)
df.to_excel('test.xlsx', index = False)
I am working on webscraping,I am taking names from text file by line by line and searching it on google and scraping address from that results. I want to add that result infront of respective names. this is my text file a.txt:
0.5BN FINHEALTH PRIVATE LIMITED
01 SYNERGY CO.
1 BY 0 SOLUTIONS
and this is my code:
import requests
from bs4 import BeautifulSoup
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
out_fl = open('a.txt','r')
for line in out_fl:
query = line
query = query.replace(' ', '+')
print(line)
URL = f"https://google.com/search?q={query}"
print(URL)
headers = {"user-agent": USER_AGENT}
resp = requests.get(URL, headers=headers)
if resp.status_code == 200:
soup = BeautifulSoup(resp.content, "html.parser")
results = []
newline = '\n'
for g in soup.find_all('span', class_="i4J0ge"):
x = f'{line}:{g.text}{newline}'
results.append(x)
print(results)
with open("results.txt","a") as result:
result.write(str(results))
I am getting result like this but its not formatted properly please help me out.
my expected result is like:
0.5BN FINHEALTH PRIVATE LIMITED : Address: 2nd Floor, BHIVE Forum, GNS Towers #18, Dairy
Circle Road, Adugodi, Koramangala, Bengaluru, Karnataka 560029Hours: Closed ⋅ Opens 9:30AM
MonSaturdayClosedSundayClosedMonday9:30am–7:30pmTuesday9:30am–7:30pmWednesday9:30am–
7:30pmThursday9:30am–7:30pmFriday9:30am–7:30pmSuggest an editUnable to add this file.
Please check that it is a valid photo
01 SYNERGY CO. : 01 SYNERGY CO.\n:Located in: Punjab Agricultural UniversityAddress: 3rd
Floor Kartar Bhawan, Ferozpur Rd, Ludhiana, Punjab 141001Hours: Closes soon ⋅ 5PM ⋅ Opens
9:30AM MonSaturday10am–5pmSundayClosedMonday9:30am–7:30pmTuesday9:30am–
7:30pmWednesday9:30am–7:30pmThursday9:30am–7:30pmFriday9:30am–7:30pmSuggest an editUnable
to add this file. Please check that it is a valid photo.Phone: 098159 18807
Or into excel. Thank you
You can assign your results to pandas data frame and than you can write it into excel or csv
Import pandas as pd
df=pd.DataFrame(columns=["",""]. # Assign column name as required
df = [results]
df.to_excel('filename.xlsx', sheet_name='sheet name', index = False)