Python Zeep WSDL Unexpected Elements Traceback - python-3.x

I can't handle this below.
XMLParseError: Unexpected element 'metadata', expected 'id' error.
I also try strict=False settings but that time zeep returned as none.
from zeep import Client
import zeep
wsdl = 'https://api.n11.com/ws/CategoryService.wsdl'
cl = Client(wsdl=wsdl)
request_data = {"auth":{'appKey' : ***,
'appSecret' : **},'categoryId':int(1003221),"pagingData":{'pageSize':1,'currentPage':0}}
print(cl.service.GetCategoryAttributes(**request_data))
Thanks.

I faced the same problem with the same API. My solution is changing zeep's settings.
You should use xsd_ignore_sequence_order=True.
I hope it's not too late.
settings = Settings(strict=False, xml_huge_tree=True, xsd_ignore_sequence_order=True)
read more:
https://docs.python-zeep.org/en/master/settings.html#module-zeep.settings

Solution:
firstly, to check logs:
import logging.config
logging.config.dictConfig({
'version': 1,
'formatters': {
'verbose': {
'format': '%(name)s: %(message)s'
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
'zeep.transports': {
'level': 'DEBUG',
'propagate': True,
'handlers': ['console'],
},
}
})
Second:
from zeep import Client
from zeep.plugins import HistoryPlugin
wsdl = 'https://api.n11.com/ws/CategoryService.wsdl'
from zeep import Client, Settings
settings = Settings(strict=False, xml_huge_tree=True)
history = HistoryPlugin()
from zeep.transports import Transport
transport = Transport(timeout=10)
key={'appKey' : '**',
'appSecret' : '**',
}
cat= {'categoryId':1003221}
dat= {'pageSize':1,'currentPage':1}
client = Client(wsdl=wsdl, transport=transport, plugins=[history], settings=settings)
with client.settings(raw_response=False,strict=False, xml_huge_tree=True):
response = client.service.GetCategoryAttributes(key,1003221,pagingData=dat)
#assert response.status_code == 200
#assert response.content
#node = client.create_message(client.service, 'GetCategoryAttributes',key,1003221,pagingData=dat)
from lxml import etree
try:
for hist in [history.last_sent, history.last_received]:
print(etree.tostring(hist["envelope"], encoding="unicode", pretty_print=True))
except (IndexError, TypeError):
# catch cases where it fails before being put on the wire
pass
lastly, you can parse with bs4 for xml content:
from bs4 import BeautifulSoup
soup = BeautifulSoup(ass, 'lxml')
soup.find('category').find('metadata')

Related

Python - Request and Response logging using HTTPS Connection - Swagger Codegen - OpenAPI

I have a Python FASTAPI application which I run through Uvicorn.
I have a dependent service and I generate a python client using the swaggen-codegen provided by the openapi.json of the dependent service.
My logger dictconfig is:
def get_logging_config():
return {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'()': ColoredFormatter
},
},
'handlers': {
'console': {
'level': logging.getLevelName(logging.DEBUG),
'formatter': 'standard',
'class': 'logging.StreamHandler',
'stream': 'ext://sys.stdout'
}
},
'loggers': {
'': { # root logger
'handlers': ['console'],
'level': logging.getLevelName(get_logging_level()),
'propagate': False
},
'uvicorn': { # uvicorn logger
'handlers': ['console'],
'level': logging.getLevelName(logging.WARN),
'propagate': False
},
'urllib3': { # urllib3 logger
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'urllib3.connectionpool': { # urllib3 logger for HTTP calls
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'requests.packages.urllib3': {
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': True
}
}
}
My logging module init.py contains:
import logging
import logging.config
from .http_client_logger import httpclient_logging_patch
from .logging_config import get_logging_config
httpclient_logging_patch()
logging.config.dictConfig(get_logging_config())
My http_client_logger.py file looks like:
import logging
import http.client
httpclient_logger = logging.getLogger("http.client")
def httpclient_logging_patch(level=logging.DEBUG):
"""Enable HTTPConnection debug logging to the logging framework"""
def httpclient_log(*args):
httpclient_logger.log(level, "test-" + "".join(args))
# mask the print() built-in in the http.client module to use
# logging instead
http.client.print = httpclient_log
# enable debugging
http.client.HTTPConnection.debuglevel = 1
http.client.HTTPSConnection.debuglevel = 1
The logs which I could see are
2023-01-14 20:18:02.384 - DEBUG - https://test-dependent-service-url.com:443 "POST /test-endpoint HTTP/1.1" 200 None (urllib3.connectionpool:_make_request:456)
2023-01-14 20:18:02.385 - DEBUG - response body: b'{"data":[],"id":"93d0f81a-3d4e-4ae6-9c47-07adde47400a","timestamp":"2023-01-14T19:18:02.489899239Z"}' (dependent_service.client.swagger_client.rest:request:219)
I would like to log the request sent (POST, GET, PUT, DELETE) to the dependent service and response received from the dependent service which contains the body of the request and response. Note that I am using an HTTPS connection. But in the logs, the request body is not being logged. Also, in none of the cases, the headers are getting logged.
Please, anyone could help here?
I have tried the following:
Log all requests from the python-requests module
I have also tried to put the debuglevel to 2 but nothing helped
http.client.HTTPConnection.debuglevel = 2
http.client.HTTPSConnection.debuglevel = 2
https://bhoey.com/blog/better-debug-logging-for-the-python-requests-library/ It talks about the file handler but it same as previous and I am using StreamHandler.
Although it works for GET but not for POST.

Type Error and got an unexpected keyword argument: When passing dict to Python Zeep's Client.Service call

I have the following XML which I am sending to the target web application using SOAPUI and it works.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsu="http://www.docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<soapenv:Header/>
<soapenv:Body>
<RequestMessage xmlns="http://www.ercot.com/schema/2007-06/nodal/ews/message">
<Header>
<Verb>create</Verb>
<Noun>OutageSet</Noun>
<ReplayDetection>
<Nonce>72465</Nonce>
<Created>2022-12-27T00:40:00-07:00</Created>
</ReplayDetection>
<Revision>004</Revision>
<Source>TAEPTC</Source>
<!--Optional:-->
<UserID>API_OutplanOSITCC</UserID>
</Header>
<Payload>
<OutageSet xmlns="http://www.ercot.com/schema/2007-06/nodal/ews">
<Outage>
<OutageInfo>
<outageType>PL</outageType>
<Requestor>
<name>API_OutplanOSITCC</name>
<userFullName>APIOutplanOSITCC</userFullName>
</Requestor>
<Disclaimer>tempdisclaimer</Disclaimer>
<disclaimerAck>true</disclaimerAck>
</OutageInfo>
<TransmissionOutage>
<operatingCompany>TAEPTC</operatingCompany>
<equipmentName>1589</equipmentName>
<equipmentIdentifier>_{072D6FCA-D121-49E7-AC9D-CDF5D4DB3D70}</equipmentIdentifier>
<transmissionType>DSC</transmissionType>
<fromStation>AIRLINE</fromStation>
<outageState>O</outageState>
<emergencyRestorationTime>1</emergencyRestorationTime>
<natureOfWork>OE</natureOfWork>
</TransmissionOutage>
<Schedule>
<plannedStart>2023-01-16T10:00:00</plannedStart>
<plannedEnd>2023-01-17T10:00:00</plannedEnd>
<earliestStart>2023-01-16T10:00:00</earliestStart>
<latestEnd>2023-01-17T12:00:00</latestEnd>
</Schedule>
</Outage>
</OutageSet>
</Payload>
</RequestMessage>
</soapenv:Body>
</soapenv:Envelope>
I am trying to generate the XML using python's zeep module and for that I have the modifeid xml and converting that to dict using xmltodict module.
Below is the XMLtoDIct code sinppet -
import xmltodict
import pprint
import json
my_xml = """
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsu="http://www.docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<soapenv:Header/>
<soapenv:Body>
<RequestMessage xmlns="http://www.ercot.com/schema/2007-06/nodal/ews/message">
<Header>
<Verb>create</Verb>
<Noun>OutageSet</Noun>
<ReplayDetection>
<Nonce>72465</Nonce>
<Created>2022-12-27T00:40:00-07:00</Created>
</ReplayDetection>
<Revision>004</Revision>
<Source>TAEPTC</Source>
<!--Optional:-->
<UserID>API_OutplanOSITCC</UserID>
</Header>
<Payload>
<OutageSet xmlns="http://www.ercot.com/schema/2007-06/nodal/ews">
<Outage>
<OutageInfo>
<outageType>PL</outageType>
<Requestor>
<name>API_OutplanOSITCC</name>
<userFullName>APIOutplanOSITCC</userFullName>
</Requestor>
<Disclaimer>tempdisclaimer</Disclaimer>
<disclaimerAck>true</disclaimerAck>
</OutageInfo>
<TransmissionOutage>
<operatingCompany>TAEPTC</operatingCompany>
<equipmentName>1589</equipmentName>
<equipmentIdentifier>_{072D6FCA-D121-49E7-AC9D-CDF5D4DB3D70}</equipmentIdentifier>
<transmissionType>DSC</transmissionType>
<fromStation>AIRLINE</fromStation>
<outageState>O</outageState>
<emergencyRestorationTime>1</emergencyRestorationTime>
<natureOfWork>OE</natureOfWork>
</TransmissionOutage>
<Schedule>
<plannedStart>2023-01-16T10:00:00</plannedStart>
<plannedEnd>2023-01-17T10:00:00</plannedEnd>
<earliestStart>2023-01-16T10:00:00</earliestStart>
<latestEnd>2023-01-17T12:00:00</latestEnd>
</Schedule>
</Outage>
</OutageSet>
</Payload>
</RequestMessage>
</soapenv:Body>
</soapenv:Envelope>
"""
my_dict = xmltodict.parse(my_xml, process_namespaces= True)
#pp = pprint.PrettyPrinter(indent=2)
print(my_dict)
Below is the dict after conversion -
{'http://schemas.xmlsoap.org/soap/envelope/:Envelope': {'http://schemas.xmlsoap.org/soap/envelope/:Header': None, 'http://schemas.xmlsoap.org/soap/envelope/:Body': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:RequestMessage': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Header': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Verb': 'create', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Noun': 'OutageSet', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:ReplayDetection': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Nonce': '72465', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Created': '2022-12-27T00:40:00-07:00'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Revision': '004', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Source': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:UserID': 'API_OutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Payload': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageSet': {'http://www.ercot.com/schema/2007-06/nodal/ews:Outage': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageInfo': {'http://www.ercot.com/schema/2007-06/nodal/ews:outageType': 'PL', 'http://www.ercot.com/schema/2007-06/nodal/ews:Requestor': {'http://www.ercot.com/schema/2007-06/nodal/ews:name': 'API_OutplanOSITCC', 'http://www.ercot.com/schema/2007-06/nodal/ews:userFullName': 'APIOutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Disclaimer': 'tempdisclaimer', 'http://www.ercot.com/schema/2007-06/nodal/ews:disclaimerAck': 'true'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:TransmissionOutage': {'http://www.ercot.com/schema/2007-06/nodal/ews:operatingCompany': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentName': '1589', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentIdentifier': '_{072D6FCA-D121-49E7-AC9D-CDF5D4DB3D70}', 'http://www.ercot.com/schema/2007-06/nodal/ews:transmissionType': 'DSC', 'http://www.ercot.com/schema/2007-06/nodal/ews:fromStation': 'AIRLINE', 'http://www.ercot.com/schema/2007-06/nodal/ews:outageState': 'O', 'http://www.ercot.com/schema/2007-06/nodal/ews:emergencyRestorationTime': '1', 'http://www.ercot.com/schema/2007-06/nodal/ews:natureOfWork': 'OE'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Schedule': {'http://www.ercot.com/schema/2007-06/nodal/ews:plannedStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:plannedEnd': '2023-01-17T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:earliestStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:latestEnd': '2023-01-17T12:00:00'}}}}}}}}
however, when I am passing this dict to the request data like below -
request_data = {'http://www.ercot.com/schema/2007-06/nodal/ews/message:RequestMessage': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Header': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Verb': 'create', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Noun': 'OutageSet', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:ReplayDetection': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Nonce': '72465', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Created': '2022-12-27T00:40:00-07:00'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Revision': '004', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Source': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:UserID': 'API_OutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Payload': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageSet': {'http://www.ercot.com/schema/2007-06/nodal/ews:Outage': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageInfo': {'http://www.ercot.com/schema/2007-06/nodal/ews:outageType': 'PL', 'http://www.ercot.com/schema/2007-06/nodal/ews:Requestor': {'http://www.ercot.com/schema/2007-06/nodal/ews:name': 'API_OutplanOSITCC', 'http://www.ercot.com/schema/2007-06/nodal/ews:userFullName': 'APIOutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Disclaimer': 'tempdisclaimer', 'http://www.ercot.com/schema/2007-06/nodal/ews:disclaimerAck': 'true'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:TransmissionOutage': {'http://www.ercot.com/schema/2007-06/nodal/ews:operatingCompany': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentName': '1589', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentIdentifier': '_{072D6FCA-D121-49E7-AC9D-CDF5D4DB3D70}', 'http://www.ercot.com/schema/2007-06/nodal/ews:transmissionType': 'DSC', 'http://www.ercot.com/schema/2007-06/nodal/ews:fromStation': 'AIRLINE', 'http://www.ercot.com/schema/2007-06/nodal/ews:outageState': 'O', 'http://www.ercot.com/schema/2007-06/nodal/ews:emergencyRestorationTime': '1', 'http://www.ercot.com/schema/2007-06/nodal/ews:natureOfWork': 'OE'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Schedule': {'http://www.ercot.com/schema/2007-06/nodal/ews:plannedStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:plannedEnd': '2023-01-17T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:earliestStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:latestEnd': '2023-01-17T12:00:00'}}}}}}
print("Call URL")
print(client.service.MarketTransactions(**request_data))
I am getting the below error -
Creating client.
zeep.transports: Loading remote data from: http://www.w3.org/2001/xml.xsd
Making request.
Call URL
Traceback (most recent call last):
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\ERCOT_Test.py", line 110, in <module>
print(client.service.MarketTransactions(**request_data))
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\proxy.py", line 46, in __call__
return self._proxy._binding.send(
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\wsdl\bindings\soap.py", line 123, in send
envelope, http_headers = self._create(
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\wsdl\bindings\soap.py", line 73, in _create
serialized = operation_obj.create(*args, **kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\wsdl\definitions.py", line 222, in create
return self.input.serialize(*args, **kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\wsdl\messages\soap.py", line 73, in serialize
body_value = self.body(*args, **kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\xsd\elements\element.py", line 58, in __call__
instance = self.type(*args, **kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\xsd\types\complex.py", line 63, in __call__
return self._value_class(*args, **kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\xsd\valueobjects.py", line 106, in __init__
items = _process_signature(self._xsd_type, args, kwargs)
File "C:\Users\SGUPTA\PycharmProjects\ERCOT\venv\lib\site-packages\zeep\xsd\valueobjects.py", line 241, in _process_signature
raise TypeError(
TypeError: {http://www.ercot.com/schema/2007-06/nodal/ews/message}RequestMessage() got an unexpected keyword argument 'http://www.ercot.com/schema/2007-06/nodal/ews/message:RequestMessage'. Signature: `Header: {http://www.ercot.com/schema/2007-06/nodal/ews/message}HeaderType, Request: {http://www.ercot.com/schema/2007-06/nodal/ews/message}RequestType, Payload: {http://www.ercot.com/schema/2007-06/nodal/ews/message}PayloadType`
Process finished with exit code 1
is this because of the namespaces in both RequestMessage and Outageset nodes?
Here is my complete code -
from zeep.wsse.signature import BinarySignature
from zeep.wsse import utils
from datetime import datetime, timedelta
import contextlib
import os
import requests
from requests_pkcs12 import Pkcs12Adapter
from zeep.transports import Transport
from zeep import Client, Settings
from pathlib import Path
from tempfile import NamedTemporaryFile
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption
from cryptography.hazmat.primitives.serialization.pkcs12 import load_key_and_certificates
import random
import logging.config
# USE THE MOST VERBOSE LOGGING LEVEL
logging.config.dictConfig({
'version': 1,
'formatters': {
'verbose': {
'format': '%(name)s: %(message)s'
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
'zeep.transports': {
'level': 'DEBUG',
'propagate': True,
'handlers': ['console'],
},
}
})
class BinarySignatureTimestamp(BinarySignature):
def apply(self, envelope, headers):
security = utils.get_security_header(envelope)
created = datetime.utcnow()
expired = created + timedelta(seconds=1 * 60)
timestamp = utils.WSU('Timestamp')
timestamp.append(utils.WSU('Created', created.replace(microsecond=0).isoformat() + 'Z'))
timestamp.append(utils.WSU('Expires', expired.replace(microsecond=0).isoformat() + 'Z'))
security.append(timestamp)
super().apply(envelope, headers)
return envelope, headers
def verify(self, envelope):
return envelope
#contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
''' Decrypts the .pfx file to be used with requests. '''
pfx = Path(pfx_path).read_bytes()
private_key, main_cert, add_certs = load_key_and_certificates(pfx, pfx_password.encode('utf-8'), None)
with NamedTemporaryFile(suffix='.pem', delete=False) as t_pem:
with open(t_pem.name, 'wb') as pem_file:
pem_file.write(private_key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()))
pem_file.write(main_cert.public_bytes(Encoding.PEM))
for ca in add_certs:
pem_file.write(ca.public_bytes(Encoding.PEM))
yield t_pem.name
def generate_nonce(length=15):
"""Generate pseudorandom number."""
return ''.join([str(random.randint(0, 9)) for i in range(length)])
# CERTIFICATES PATHS
api_p12_key = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\API Outplan OSI TCC MOTE.p12')
api_certificate = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\OSITCC.crt')
api_pfx_key = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\API Outplan OSI TCC MOTE.pfx')
# SETUP
wsdl_file = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\Nodal.wsdl')
api_base_url = "https://testmisapi.ercot.com"
session = requests.Session()
session.mount(api_base_url,
Pkcs12Adapter(pkcs12_filename=api_p12_key, pkcs12_password='HIDDEN'))
session.verify = None
transport = Transport(session=session)
settings = Settings(forbid_entities=False)
# CREATE CLIENT
print("Creating client.")
with pfx_to_pem(pfx_path=api_pfx_key, pfx_password='AEP') as pem_fle:
client = Client(wsdl=wsdl_file, settings=settings, transport=transport,
wsse=BinarySignatureTimestamp(pem_fle, api_certificate, "AEP"))
print("Making request.")
request_data = {'http://www.ercot.com/schema/2007-06/nodal/ews/message:RequestMessage': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Header': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Verb': 'create', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Noun': 'OutageSet', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:ReplayDetection': {'http://www.ercot.com/schema/2007-06/nodal/ews/message:Nonce': '72465', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Created': '2022-12-27T00:40:00-07:00'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Revision': '004', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Source': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews/message:UserID': 'API_OutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews/message:Payload': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageSet': {'http://www.ercot.com/schema/2007-06/nodal/ews:Outage': {'http://www.ercot.com/schema/2007-06/nodal/ews:OutageInfo': {'http://www.ercot.com/schema/2007-06/nodal/ews:outageType': 'PL', 'http://www.ercot.com/schema/2007-06/nodal/ews:Requestor': {'http://www.ercot.com/schema/2007-06/nodal/ews:name': 'API_OutplanOSITCC', 'http://www.ercot.com/schema/2007-06/nodal/ews:userFullName': 'APIOutplanOSITCC'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Disclaimer': 'tempdisclaimer', 'http://www.ercot.com/schema/2007-06/nodal/ews:disclaimerAck': 'true'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:TransmissionOutage': {'http://www.ercot.com/schema/2007-06/nodal/ews:operatingCompany': 'TAEPTC', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentName': '1589', 'http://www.ercot.com/schema/2007-06/nodal/ews:equipmentIdentifier': '_{072D6FCA-D121-49E7-AC9D-CDF5D4DB3D70}', 'http://www.ercot.com/schema/2007-06/nodal/ews:transmissionType': 'DSC', 'http://www.ercot.com/schema/2007-06/nodal/ews:fromStation': 'AIRLINE', 'http://www.ercot.com/schema/2007-06/nodal/ews:outageState': 'O', 'http://www.ercot.com/schema/2007-06/nodal/ews:emergencyRestorationTime': '1', 'http://www.ercot.com/schema/2007-06/nodal/ews:natureOfWork': 'OE'}, 'http://www.ercot.com/schema/2007-06/nodal/ews:Schedule': {'http://www.ercot.com/schema/2007-06/nodal/ews:plannedStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:plannedEnd': '2023-01-17T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:earliestStart': '2023-01-16T10:00:00', 'http://www.ercot.com/schema/2007-06/nodal/ews:latestEnd': '2023-01-17T12:00:00'}}}}}}
print("Call URL")
print(client.service.MarketTransactions(**request_data))
The same code is working when I am calling another webservice just by sending the header(does not have a namespace) in the request data like below -
from zeep.wsse.signature import BinarySignature
from zeep.wsse import utils
from datetime import datetime, timedelta
import contextlib
import os
import requests
from requests_pkcs12 import Pkcs12Adapter
from zeep.transports import Transport
from zeep import Client, Settings
from pathlib import Path
from tempfile import NamedTemporaryFile
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption
from cryptography.hazmat.primitives.serialization.pkcs12 import load_key_and_certificates
import random
import logging.config
# USE THE MOST VERBOSE LOGGING LEVEL
logging.config.dictConfig({
'version': 1,
'formatters': {
'verbose': {
'format': '%(name)s: %(message)s'
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
'zeep.transports': {
'level': 'DEBUG',
'propagate': True,
'handlers': ['console'],
},
}
})
class BinarySignatureTimestamp(BinarySignature):
def apply(self, envelope, headers):
security = utils.get_security_header(envelope)
created = datetime.utcnow()
expired = created + timedelta(seconds=1 * 60)
timestamp = utils.WSU('Timestamp')
timestamp.append(utils.WSU('Created', created.replace(microsecond=0).isoformat()+'Z'))
timestamp.append(utils.WSU('Expires', expired.replace(microsecond=0).isoformat()+'Z'))
security.append(timestamp)
super().apply(envelope, headers)
return envelope, headers
def verify(self, envelope):
return envelope
#contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
''' Decrypts the .pfx file to be used with requests. '''
pfx = Path(pfx_path).read_bytes()
private_key, main_cert, add_certs = load_key_and_certificates(pfx, pfx_password.encode('utf-8'), None)
with NamedTemporaryFile(suffix='.pem', delete=False) as t_pem:
with open(t_pem.name, 'wb') as pem_file:
pem_file.write(private_key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()))
pem_file.write(main_cert.public_bytes(Encoding.PEM))
for ca in add_certs:
pem_file.write(ca.public_bytes(Encoding.PEM))
yield t_pem.name
def generate_nonce(length=15):
"""Generate pseudorandom number."""
return ''.join([str(random.randint(0, 9)) for i in range(length)])
# CERTIFICATES PATHS
api_p12_key = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\API Outplan OSI TCC MOTE.p12')
api_certificate = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\OSITCC.crt')
api_pfx_key = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\API Outplan OSI TCC MOTE.pfx')
# SETUP
wsdl_file = os.path.join('C:\\Users\\SGUPTA\\ALL\\Projects\\AEPE2\\Nodal.wsdl')
api_base_url = "https://testmisapi.ercot.com"
session = requests.Session()
session.mount(api_base_url,
Pkcs12Adapter(pkcs12_filename=api_p12_key, pkcs12_password='AEP'))
session.verify = None
transport = Transport(session=session)
settings = Settings(forbid_entities=False)
# CREATE CLIENT
print("Creating client.")
with pfx_to_pem(pfx_path=api_pfx_key, pfx_password='hidden') as pem_fle:
client = Client(wsdl=wsdl_file, settings=settings, transport=transport,
wsse=BinarySignatureTimestamp(pem_fle, api_certificate, "AEP"))
print("Making request.")
request_data = {
"Header": {
"Verb": "get",
"Noun": "SystemStatus",
"ReplayDetection": {
"Nonce": generate_nonce(),
"Created": "2022-11-10T15:34:00-06:00"},
"Revision": "1",
"Source": "API Outplan OSI TCC",
"UserID": "API_OutplanOSITCC",
}
}
print("Call URL")
print(client.service.MarketInfo(**request_data))
Could anyone please advise how to fix this and what needs to be changed?
Thanks
Sugata

Python logging with dictConfig and using GMT / UTC zone time

I'm trying to configure some logging across multiple modules within a package. These modules may be executed in different containers, possibly in different regions, so I wanted to explicitly use GMT / UTC time for logging.
When I'm reading about the Formatter class in logging, it indicates you can specify converter to use either local-time or GMT. I'd like to utilize this feature, in conjunction with dictConfig (or possibly fileConfig) to specify the configurations for the different modules, but the documentation is sparse with respect to this feature. Everything specified in the config is working, except for the timezone. The log always uses local time. I can include the '%z' formatting specification in datefmt to specify offset from GMT, but that breaks the .%(msecs)03d formatting.
Below is my code using a defined dictionary and dictConfig. Has anyone had any success specifying timezone in the config? is this possible?
import json
import logging
from logging.config import dictConfig
import time
DEFAULT_CONFIG = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(levelname)s : %(name)s : %(asctime)s.%(msecs)03d : %(message)s',
'datefmt': '%Y-%m-%d %H.%M.%S',
'converter': time.gmtime # also fails with 'time.gmtime', 'gmtime'
}
},
'handlers' : {
'default': {
'level': 'NOTSET',
'class': 'logging.StreamHandler',
'formatter': 'standard',
}
},
'loggers': {
'TEST': { # logging from this module should be logged in DEBUG level
'handlers' : ['default'],
'level': 'INFO',
'propagate': False,
},
},
'root': {
'level': 'INFO',
'handlers': ['default']
},
'incremental': False
}
if __name__ == '__main__':
dictConfig(DEFAULT_CONFIG)
logger = logging.getLogger('TEST')
logger.debug('debug message') # this should not be displayed if level==INFO
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
Output:
INFO : TEST : 2022-07-07 17.20.13.434 : info message
WARNING : TEST : 2022-07-07 17.20.13.435 : warning message
ERROR : TEST : 2022-07-07 17.20.13.435 : error message

AttributeError: module 'google.cloud.dialogflow' has no attribute 'types'

I am building a telegram-bot and using Dialogflow in it, I am getting the following error :
2021-11-19 23:26:46,936 - __main__ - ERROR - Update '{'message': {'entities': [],
'delete_chat_photo': False, 'text': 'hi', 'date': 1637344606, 'new_chat_members': [],
'channel_chat_created': False, 'message_id': 93, 'photo': [], 'group_chat_created':
False, 'supergroup_chat_created': False, 'new_chat_photo': [], 'caption_entities': [],
'chat': {'id': 902424541, 'type': 'private', 'first_name': 'Akriti'},
'from': {'first_name': 'Akriti', 'id': 902424541, 'is_bot': False, 'language_code': 'en'}
}, 'update_id': 624230278}' caused error 'module 'google.cloud.dialogflow' has no
attribute 'types''
It appears there is some issue with the Dialogflow attribute "types", but I don't know what I am doing wrong.
Here is the code where I am using it:
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "client.json"
from google.cloud import dialogflow
dialogflow_session_client = dialogflow.SessionsClient()
PROJECT_ID = "get-informed-ufnl"
from gnewsclient import gnewsclient
client = gnewsclient.NewsClient()
def detect_intent_from_text(text, session_id, language_code='en'):
session = dialogflow_session_client.session_path(PROJECT_ID, session_id)
text_input = dialogflow.types.TextInput(text=text, language_code=language_code)
query_input = dialogflow.types.QueryInput(text=text_input)
response = dialogflow_session_client.detect_intent(session=session, query_input=query_input)
return response.query_result
def get_reply(query, chat_id):
response = detect_intent_from_text(query, chat_id)
if response.intent.display_name == 'get_news':
return "get_news", dict(response.parameters)
else:
return "small_talk", response.fulfillment_text
def fetch_news(parameters):
client.language = parameters.get('language')
client.location = parameters.get('geo-country')
client.topic = parameters.get('topic')
return client.get_news()[:5]
topics_keyboard = [
['Top Stories', 'World', 'Nation'],
['Business', 'Technology', 'Entertainment'],
['Sports', 'Science', 'Health']
]
I figured it out, the problem lies in the import statement. The correct module name should be:
import google.cloud.dialogflow_v2 as dialogflow
I recommend to deactivate your current error handler or use one similar to this example such that you can see the full traceback of the exception :)
Disclaimer: I'm currently the maintainer of python-telegram-bot

Google Cloud logging, Python3.8 standard environment, group request related logs by trace id

I stucked with problem during Google Cloud Logging setup for Python3.8 in Google App Engine Standard environment.
I'm using FastAPI with unicorn. My code logging configuration:
import logging.config
import sys
from google.cloud import logging as google_logging
from app.settings import ENV, _settings
if _settings.ENV == ENV.LOCAL:
MAIN_LOGGER = 'console'
LOGGER_CONF_DICT = {
'class': 'logging.StreamHandler',
'formatter': 'verbose',
'stream': sys.stdout,
'level': _settings.LOG_LEVEL.upper(),
}
else:
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'appengine.googleapis.com%2Frequest_log'
# I've tried other names: stdout, %2FA instead of / symbol, appengine.googleapis.com/stdout
# the same result or no logs at all
}
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '%(log_color)s%(asctime)s [%(levelname)s] [%(name)s] %(message)s (%(filename)s:%(lineno)d)',
'()': 'colorlog.ColoredFormatter',
'log_colors': {
'DEBUG': 'cyan',
'INFO': 'green',
'WARNING': 'yellow',
'ERROR': 'red',
'CRITICAL': 'bold_red',
},
}
},
'handlers': {
MAIN_LOGGER: {**LOGGER_CONF_DICT},
'blackhole': {'level': 'DEBUG', 'class': 'logging.NullHandler'},
},
'loggers': {
'fastapi': {'level': 'INFO', 'handlers': [MAIN_LOGGER]},
'uvicorn.error': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn.access': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'google.cloud.logging.handlers.transports.background_thread': {'level': 'DEBUG', 'handlers': ['blackhole'],
'propagate': False},
'': {
'level': _settings.LOG_LEVEL.upper(),
'handlers': [MAIN_LOGGER],
'propagate': True,
},
}
}
logging.config.dictConfig(LOGGING)
And my logging handler code:
import os
from typing import Any, Dict, Optional
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource
from starlette.requests import Request
from starlette_context import context
from app.settings import _settings
class GCLHandler(CloudLoggingHandler):
def emit(self, record):
message = super(GCLHandler, self).format(record)
request: Optional[Request] = None
trace: Optional[str] = None
span_id: Optional[str] = None
user_id: Optional[int] = None
resource = Resource(
type='gae_app',
labels={
'module_id': os.environ['GAE_SERVICE'],
'project_id': _settings.PROJECT_NAME,
'version_id': os.environ['GAE_VERSION'],
'zone': 'us16' # tried without zone - the same result
}
)
labels: Dict[str, Any] = {}
if context.exists(): # I'm sure that it works
request = context.get('request') # I'm sure that it works
user_id = context.get('user_id') # I'm sure that it works
if user_id is not None:
labels['user_id'] = user_id
if request:
if request.headers.get('X-Cloud-Trace-Context'):
cloud_trace = request.headers.get('X-Cloud-Trace-Context').split('/')
if len(cloud_trace) > 1:
span_id = cloud_trace[1].split(';')[0]
trace = f'projects/{_settings.PROJECT_NAME}/traces/{cloud_trace[0]}'
labels['logging.googleapis.com/trace'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
labels['appengine.googleapis.com/trace_id'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
self.transport.send(
record,
message,
resource=resource,
labels=labels,
trace=trace,
span_id=span_id
)
I've got some strange results in logs viewer that my log has the same trace as request log, but they're not grouped
Any ideas?
There are 2 types of logs in App Engine :
Request log: A log of the requests that are sent to your app. App Engine automatically creates entries in the request log.
App log: log entries that you write to a supported framework or file as described on this page.
The both logs are send to the Cloud Logging Agent automatically by App Engine Standard.
On first request, app logs and request logs are not correlated and that's why there are not shown in a group , this is a known issue stated in App Engine Official Documentation. However in the second request, you can see that the logs are shown in a group.
A feature request in Public Issue Tracker has already been created for this behavior where you will get all the updates regarding the fix.
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'app'
}
Change name to app is helped

Resources