I am trying to use python to access pull and plot data from a server (ERRDAP if you're interested).
Here is the code that I am trying to use:
url = 'https://scoos.org/erddap/tabledap/autoss.csv?query'
values = {'time>=' : '2016-07-10T00:00:00Z',
'time<' : '2017-02-10T00:00:00Z',
'temperature' : 'temperature',
'time' :'time',
'temperature_flagPrimary'}
data = urllib.parse.urlencode(values)
data.encode('utf-8')
resp = urllib.request.urlopen(req)
The error message that I get is: "POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str." I think this either has something to do with me trying to request a csv file or improperly using urllib. Any help would be greatly appreciated!
Related
Sending Client request to get data from API.
request =client.get_income('ET',incomeType='PNL',startTime=1611287550000)
The API returns the following data:
[{"symbol":"ET","incomeType":"R","income":"-2.4","time":1611287909000,"info":"50538270"},{"symbol":"ET","incomeType":"R","income":"1.68","time":1611287909000,"info":"50538271"}]
It's a dictionary inside the list. When I try to access the items through a for loop or any of the following methods, It returns the OBJECT.
Methods I tried:
for item in request:
print(item['income'])
returns this : <model.income.Income object at 0x000000684EFB4580>
print(request[0]['income']
ERROR: TypeError: 'Income' object is not subscriptable
None of them works.
I have fixed it myself.
request =client.get_income('ET',incomeType='PNL',startTime=1611287550000)
for x in range(len(request)):
print(request[x].__dict__['income'])
request = [{"symbol":"ET","incomeType":"R","income":"-2.4","time":1611287909000,"info":"50538270"},{"symbol":"ET","incomeType":"R","income":"1.68","time":1611287909000,"info":"50538271"}]
for item in request:
print(item['income'])
it's working fine on my pc, I'm using python 3.9
-2.4
1.68
The data structure you have is a JSON. Use Pandas to parse it and access it elements:
import pandas as pd
request = [{"symbol":"ET","incomeType":"R","income":"-2.4","time":1611287909000,"info":"50538270"},
{"symbol":"ET","incomeType":"R","income":"1.68","time":1611287909000,"info":"50538271"}]
df = pd.DataFrame(request)
print(df)
print(f'Incomes are: {df["income"].tolist()}')
Output:
i want to fetch data from cheat.sh using the requests lib and the discord.py lib....but since discord only allows 2000 characters at length to send at a time, i want to fetch only a certain number of words/digits/newline like 1800. how can i do so?
a small bit of code example showing my idea
import requests
url = "https://cheat.sh/python/string+annotations" #this gets the docs of string annotation in python
response = requests.get(url)
data = response.text # This gives approximately 2403 words...but i want to get only 1809 words
print(data)
import requests
url = "https://cheat.sh/python/string+annotations" #this gets the docs of string
annotation in python
response = requests.get(url)
data = response.text[:1800]
print(data)
This will be the correct code
I am writing a python file server and I want to be able to recv all the data on the buffer. So I made a send function that sends the data along with it's file size but when I send the data there is an attribute error.
I have tried looping around the recv function but couldn't get it to work so I decided to send the size of the data being sent and let the server/client see how much data they have received and if it's less than the amount they are told then they should keep receiving data.
def getsizeof(data, tl=0):
for i in data:
data = sys.getsizeof(i)
tl += data
return tl
def send(conn, request):
length = getsizeof(request)
length_size = sys.getsizeof(length)
byte_size = length_size + length
###########################################################
request.insert(0, byte_size) # THIS IS WHERE I GET 'AttributeError: 'bytes' has no attribute 'insert'
###########################################################
conn.sendall(pickle.dumps(request)) # Encoding & sending data
print(total_length)
I am getting an attribute error but when that is fixed i think that it should work and recv all data in the buffer and send correctly with the byte size of the sent data.
I guess this is kind of a stupid question, but I have tried to search for the answere on google and here with no luck.
So I am connection to a API using a Get request with the http.client package.
The get request is sent to a adress with a ziped file. Sending the request goes ok, and I get a returned object. What I want to do next is to just save the returned file to a folder.
I have tried looking in the http.client documentation withount finding anything about it.
This is an example of how the code looks like. I have had to remove some of it for privacy reasons
conn = http.client.HTTPConnection("Adress to the server")
headers = {
'cache-control': "no-cache",
}
conn.request("GET", path_to_file + "?authTicket=" + ticket,
headers=headers)
res = conn.getresponse()
return res
The code runs ok, but my problem is what to do next. If the returned data was just a normal json file I would just do something like:
data = res.read()
data = data.decode("uft-8")
And then be able to work with it inside the script. But I have no idea how to handel the ziped file that is returned. I would idealy like to unzip it before saving, but as a start it is ok to just save it to file.
Hope someone have a simple solution to this.
I did look a bit more into it, and when using the getheaders() on the returned object I get this:
[('Date', 'Wed, 19 Apr 2017 13:48:41 GMT'),
('Content-Type', 'application/octet-stream; charset=UTF-8'),
('Content-Disposition', 'attachment;filename=filename.csv.gz'),
('Content-Length', '2117014'),
('Server', 'Jetty(9.2.17.v20160517)')]
im not sure if that is relevant, but I guess I now would need a way to download the attachement.
You can extract a zip file easily in Python:
import requests
from zipfile import ZipFile
import io
my_request = requests.get(URL)
ficheiro = ZipFile(io.BytesIO(my_request.content))
ficheiro.extract('file_name')
You can get more info about the zipfile module and the zipfile.ZipFile class here:
https://docs.python.org/3/library/zipfile.html
I have to make use of POST method using urllib.request in Python and have written the following code for POST method.
values = {"abcd":"efgh"}
headers = {"Content-Type": "application/json", "Authorization": "Basic"+str(authKey)}
req = urllib.request.Request(url,values,headers=headers,method='POST')
response = urllib.request.urlopen(req)
print(response.read())
I am able to make use of 'GET' and 'DELETE' but not 'POST'.Could anyone help me out in solving this?
Thanks
If you really have to use urllib.request in POST, you have to:
Encode your data using urllib.parse.urlencode()(if sending a form)
Convert encoded data to bytes
Specify Content-Type header (application/octet-stream for raw binary data, application/x-www-form-urlencoded for forms , multipart/form-data for forms containing files and application/json for JSON)
If you do all of this, your code should be like:
req=urllib.request.Request(url,
urllib.parse.urlencode(data).encode(),
headers={"Content-Type":"application/x-www-form-urlencoded"}
)
urlopen=urllib.request.urlopen(req)
response=urlopen.read()
(for forms)
or
req=urllib.request.Request(url,
json.dumps(data).encode(),
headers={"Content-Type":"application/json"}
)
urlopen=urllib.request.urlopen(req)
response=urlopen.read()
(for JSON).
Sending files is a bit more complicated.
From urllib.request's official documentation:
For an HTTP POST request method, data should be a buffer in the
standard application/x-www-form-urlencoded format. The
urllib.parse.urlencode() function takes a mapping or sequence of
2-tuples and returns an ASCII string in this format. It should be
encoded to bytes before being used as the data parameter.
Read more:
Python - make a POST request using Python 3 urllib
RFC 7578 - Returning Values from Forms: multipart/form-data
You can use the requests module for this.
import requests
...
url="https://example.com/"
print url
data = {'id':"1", 'value': 1}
r = requests.post(url, data=data)
print(r.text)
print(r.status_code, r.reason)
You can send calls without installing any additional packages.
Call this function with your input data and url. function will return the response.
from urllib import request
import json
def make_request(input_data, url):
# dict to Json, then convert to string and then to bytes
input_data = str(json.dumps(input_data)).encode('utf-8')
# Post Method is invoked if data != None
req = request.Request(url, data=input_data)
return request.urlopen(req).read().decode('utf-8')