I'm trying to load odata into
This is my code:
import pandas as pd
import pyodata
import requests
SERVICE_URL = 'http://services.odata.org/V2/Northwind/Northwind.svc/'
HTTP_LIB = requests.Session()
northwind = pyodata.Client(SERVICE_URL, HTTP_LIB)
df = pd.DataFrame(northwind)
I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\KrestenSkovstedBuch\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pandas\core\frame.py", line 730, in __init__
raise ValueError("DataFrame constructor not properly called!")
ValueError: DataFrame constructor not properly called!
I think I need something in between my pyodata object and my DataFrame.
What would that be?
It needs to be some your own code :)
You are passing northwind, which is Pyodata.Client class to DataFrame constructor, which, well, expects something else. You must query your odata service using pyodata and then use the returned data structure (JSON probably) to initialize the Pandas DataFrame correctly, this is a more generic and already answered problem.
Related
I have a function like this:
class Name(BaseModel):
name_id: str
first_name: str
last_name: str
def get_all_names() -> List[Name]:
names = []
try:
# this API returns a list of NAME objects
names = requests.get("some-url")
# I want to validate, that each NAME object comforms to the model "Name" above
# this is what I do currently
validate_name_objects = [Name(**each_name_object) for each_name_object in names]
except Exception as e:
# if any of the NAME object fails the validation check above, then it will automatically
# be caught in this exception block and logged
logger.info(f"log this error (could be requests error, could be validation error: {e}")
return names
FastAPI does this validation check automatically, it somehow takes in type hint of response from function signature which in this case will be List[Name] and then automatically raise exception if the response does not conform to it.
I have these kind of checks in a lot of places in my code with different custom Pydantic models.
So I am looking for a mechanism of this sort where:
# Here SOME_FUNCTION takes in 2 arguments: A custom model to compare(which could be in any custom form
# made from Pydantic models like List[some-model] or Dict[str, some-model], etc)
# and a payload to validate that model against
validate_name_objects = SOME_FUNCTION(List[Name], names)
How to achieve this?
The closest content I found to my problem is provided by Pydantic here - https://pydantic-docs.helpmanual.io/usage/validation_decorator/ but this is only for validating input arguments of a given function and does not accept custom models dynamically.
Update:
After answer by #MatsLindh, here are more flexible ways by which we can use the solution he gave:
(but remember, as pointed out, Pydantic is a parsing library -> not a Validation library)
We can use it with just native python data types
from typing import Dict
from pydantic import parse_obj_as
items = parse_obj_as(Dict[int, str], {"asdasd": "23232"})
print(items)
This will as expected give Validation error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydantic/tools.py", line 35, in pydantic.tools.parse_obj_as
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[Dict]
__root__ -> __key__
value is not a valid integer (type=type_error.integer)
We can also use the same function, for custom data models:
from typing import List, Dict
from pydantic import BaseModel, parse_obj_as
class Item(BaseModel):
id: int
name: str
items = parse_obj_as(Dict[int, Item], {1: "asdfasdasf"})
print(items)
This will as expected give Validation error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydantic/tools.py", line 35, in pydantic.tools.parse_obj_as
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[Dict]
__root__ -> 1
value is not a valid dict (type=type_error.dict)
or let's try much more complicated custom types
from typing import List, Dict, Tuple, Optional
from pydantic import BaseModel, parse_obj_as
class Item(BaseModel):
id: int
name: str
items = parse_obj_as(Dict[Tuple[str, Optional[float], Optional[float]], Item], {(1, "123fdsfds", None): {'id': 1, 'name': 'My Item'}})
print(items)
This gives a validation error too as intended:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydantic/tools.py", line 35, in pydantic.tools.parse_obj_as
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[Dict]
__root__ -> __key__ -> 1
value is not a valid float (type=type_error.float)
You can use parse_obj_as to convert a list of dictionaries to a list of given Pydantic models, effectively doing the same as FastAPI would do when returning the response.
from pydantic import parse_obj_as
...
name_objects = parse_obj_as(List[Name], names)
However, it's important to consider that Pydantic is a parser library, not a validation library - so it will do conversions if your models allow for them.
I've been trying to translate python 2.7 code to python 3. I believe everything above checkpoint 1 should be correct. But I'm getting an error I associated with the second half. I can always download the file I need straight from the link, but I'd like to know what's breaking here.
import urllib
from urllib.request import urlopen
import tarfile
import os
path = 'https://www.cs.cmu.edu/~./enron/enron_mail_20150507.tar.gz'
url = urlopen(path)
#checkpoint 1
os.chdir('..')
tfile = tarfile.open(url, "r:gz")
tfile.extractall(".")
Error:
Traceback (most recent call last):
File "startup.py", line 43, in <module>
tfile = tarfile.open(url, "r:gz")
File "/anaconda3/lib/python3.6/tarfile.py", line 1589, in open
return func(name, filemode, fileobj, **kwargs)
File "/anaconda3/lib/python3.6/tarfile.py", line 1636, in gzopen
fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
File "/anaconda3/lib/python3.6/gzip.py", line 163, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
TypeError: expected str, bytes or os.PathLike object, not HTTPResponse
When confronted with an error like this, closely look at the traceback, and read the documentation for the functions and objects involved.
urllib.request.urlopen returns a HTTPResponse object.
If you look at the error message, you see that tarfile.open expects a str, bytes or os.PathLike object for the parameter name.
However, tarfile.open supports using a file object as a third argument fileobj, and HTTPResponse implements the io.BufferedIOBase interface. The classes in io are basically the file objects that the open function returns.
So you should be able to do this:
tfile = tarfile.open(None, "r:gz", files)
or
tarfile.open(fileobj=url, mode="r:gz")
The latter could be considered more Pythonic ("explicit is better than implicit").
os.chdir('..')
tfile = tarfile.open("enron_mail_20150507.tar.gz", "r:gz")
Instead of doing the above two steps, can you just mention the fully qualified file name as parameter to tarfile.open? just to rule out the possibility that the path is incorrect
>>> from mock import *
>>> from urllib import request
>>> mock = Mock(spec=request.Request)
>>> mock.has_data()
<mock.Mock object at 0x...>
>>> mock.has_data.assret_called_with()
From : https://docs.python.org/3/library/unittest.mock.html#autospeccing
1) How can the statement "mock.has_data()" work as it throws error on my system ?
I do not find has_data() mentod in the request library.
2) Assuming even if we have "has_data()", I doubt how the below call works and doesnt throw attirbute error ?
Eg: mock.has_data.assret_called_with()
In my case with below example it fails but above mentioned example in python documentation doesn't report any error
from mock import *
from urllib import request
mock = Mock(spec=request.Request)
mock.has_header()
mock.has_header.assret_called_with()
Traceback (most recent call last):
File ".\test2.py", line 5, in <module>
mock.has_header.assret_called_with()
File "C:\Users\hai\AppData\Local\Programs\Python\Python35-32\lib\site-packages\mock\mock.py", line 703, in __getattr__
raise AttributeError(name)
AttributeError: assret_called_with
Please clarify, Is something wrong with my understanding or wrong with the documentaion ?
The example in the documentation is outdated (it is a bug. You can help fixing it). It worked as described in Python 3.3. mock.has_data() should raise AttributeError in Python 3.4+:
Changed in version 3.4: The request methods add_data, has_data, get_data, get_type, get_host, get_selector, get_origin_req_host and is_unverifiable that were deprecated since 3.3 have been removed.
The documentation should use a method that exists on request.Request object (such as mock.has_header()). The point is to illustrate the difference between mock.nonexisting (raises AttributeError if request.Request has no such attribute) and mock.some_existing_attribute.some_nonexisting_attribute (the latter doesn't raise AttributeError even if mock.some_existing_attribute has no some_nonexisting_attribute attribute):
>>> from unittest.mock import Mock
>>> from urllib import request
>>> mock = Mock(spec=request.Request)
>>> mock.has_header() # there is request.Request.has_header
<Mock name='mock.has_header()' id='174952746'>
>>> mock.nonexisting # there is no request.Request.nonexisting
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/...python3.6/unittest/mock.py", line 582, in __getattr__
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'nonexisting'
Mock object has no attribute 'nonexisting'
>>> mock.has_header.nonexisting # no exception but there is no has_header.nonexisting
<Mock name='mock.has_header.nonexisting' id='249631353'>
In other words, spec is not recursive.
import cv2
import numpy as np
import os
k=[]
file1=open("TextData.txt",'w')
fn=input("Enter filename : ")
img=cv2.imread(fn,cv2.IMREAD_GRAYSCALE)
l=len(img)
w=len(img[0])
print(str(l)+"\n"+str(w))
for i in range(len(img)):
for j in range(len(img[0])):
k.append(img[i,j])
for a in range(len[k]):
file1.write(str(k[a])+"\n")
file1.close()
Basically, I'm running into the error :
Traceback (most recent call last):
File "imagereads.py", line 17, in <module>
for a in range(len[k]):
TypeError: 'builtin_function_or_method' object is not subscriptable
I'm trying to write a program that will store each image data in a file and access that later on when needed. Can anyone help me in this ? I'm doing this so that I can directly use file1.readLines() to read each data later on.
At first I tried appending each element to k, converting to a string and storing it directly. But I'm having problems getting back the data from the file into a list. Any help on this matter too will be appreciated.
I am trying to write a program that will download mp3's off of a website then join them together but whenever I try to download the files I get this error:
Traceback (most recent call last):
File "/home/tesla/PycharmProjects/OldSpice/Voicemail.py", line 214, in <module> main()
File "/home/tesla/PycharmProjects/OldSpice/Voicemail.py", line 209, in main getMp3s()
File "/home/tesla/PycharmProjects/OldSpice/Voicemail.py", line 134, in getMp3s
raw_mp3.add = urllib.urlretrieve("http://www-scf.usc.edu/~chiso/oldspice/m-b1-hello.mp3")
AttributeError: 'module' object has no attribute 'urlretrieve'
The line that is causing this problem is
raw_mp3.add = urllib.urlretrieve("http://www-scf.usc.edu/~chiso/oldspice/m-b1-hello.mp3")
As you're using Python 3, there is no urllib module anymore. It has been split into several modules.
This would be equivalent to urlretrieve:
import urllib.request
data = urllib.request.urlretrieve("http://...")
urlretrieve behaves exactly the same way as it did in Python 2.x, so it'll work just fine.
Basically:
urlretrieve saves the file to a temporary file and returns a tuple (filename, headers)
urlopen returns a Request object whose read method returns a bytestring containing the file contents
A Python 2+3 compatible solution is:
import sys
if sys.version_info[0] >= 3:
from urllib.request import urlretrieve
else:
# Not Python 3 - today, it is most likely to be Python 2
# But note that this might need an update when Python 4
# might be around one day
from urllib import urlretrieve
# Get file from URL like this:
urlretrieve("http://www-scf.usc.edu/~chiso/oldspice/m-b1-hello.mp3")
Suppose you have following lines of code
MyUrl = "www.google.com" #Your url goes here
urllib.urlretrieve(MyUrl)
If you are receiving following error message
AttributeError: module 'urllib' has no attribute 'urlretrieve'
Then you should try following code to fix the issue:
import urllib.request
MyUrl = "www.google.com" #Your url goes here
urllib.request.urlretrieve(MyUrl)