i searched the internet (and stackoverflow :D) to find an answer for the following question - and found none that i understood.
background:
we want to use a python script to connect our companies CMDB with our AWX/Ansible infrastructure.
the CMDB has a REST API which supports a (halfway) proper export.
i'm currently stuck with the implementation of the correct API call.
i can call the API itself and authenticate, but i can't call the proper filter to get the results i need.
the filter is realized by having the following string within the URL (more in the attached code example)
Label LIKE "host*"
it seems that python has a problem with the *.
error message:
InvalidURL(f"URL can't contain control characters. {url!r} "
I found some bug reports that there is an issue within some python versions, but i'm way to new to properly understand if this affects me here :D
used python version 3.7.4
PS: let's see if i can get the markup right :D
i switched the called URL to determine where exactly the problem occurs.
it only occurs when i use the SQL like filter part.
this part is essential since i just want our "hosts" to be returned and not the whole CMDB itself.
#import the required classes and such
from http.client import HTTPConnection
import json
#create a HTTP connection client
client = HTTPConnection("cmdb.example.company")
#basic auth and some header details
headers = {'Content-Type': 'application/json',
'Authorization' : 'Basic my-auth-token'}
#working API call
client.request('GET', '/cmdb/rest/hosts?attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
#broken API call returns - InvalidURL(f"URL can't contain control characters. {url!r} "
client.request('GET', '/cmdb/rest/hosts?filter=Label LIKE "host*"&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
#check and convert the response into a readable (JSON) format
response = client.getresponse()
data = response.read()
#debugging print - show that the returned data is bytes?!
print(data)
#convert the returned data into json
my_json = data.decode('utf8').replace("'", '"')
data = json.loads(my_json)
#only return the data part from the JSON and ignore the meta-overhead
text = json.dumps(data["data"], sort_keys=True, indent=4)
print(text)
so, i want to know how to properly call the API with the described filter and resolve the displayed error.
can you give me an example i can try or pin-point a beginners mistake i made?
am i affected by the mentioned python bug regarding the URL call with * in it?
thanks for helping me out :)
soooo i found my beginners mistake myself:
i used the URL from my browser - and my browser automaticly encodes the special characters within the URL.
i found the following piece of code within Python3 URL encoding guide and modified the string to fit my needs :)
import urllib.parse
query = ' "host*"'
urllib.parse.quote(query)
'%20%22host%2A%22'
Result: '%20%22host%2A%22'
%20 = " "
%22 = " " "
%2A = "*"
so the final code looks somewhat like this:
#broken API call returns - InvalidURL(f"URL can't contain control characters. {url!r} "
client.request('GET', '/cmdb/rest/hosts?filter=Label LIKE "host*"&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
filter=Label LIKE "host*"
#fixed API call
client.request('GET', '/cmdb/rest/hosts?filter=Label%20LIKE%20%22host%2A%22&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
filter=Label%20LIKE%20%22host%2A%22
Related
Hello I'm currently sitting on a problem, thats completely baffling me. I am trying to create a REST API for our companies robot and want the user to be able to post an already created mission to it via a GUI with this function:
def post_mission(host, headers):
data = json.dumps({'mission_id': item_selected.variable})
data = data.replace('"',"'")
url= "mission_queue"
post_mission = requests.post(host + url, json = data, headers = headers)
this gives me a #400 Bad Request error...
however when I replace the data = json.dumps({'mission_id': item_selected.variable}) line with data = {'mission_id': '68754b18-bb1f-11e8-954d-94c691173c1e'} (aka don't take the mission_id I sourced via a textbox where the User is able to search for his desired mission but manually insert it) everything works fine and whats even more bizarre if I use print(host + url, data, headers) for both Code versions it spits out the EXACT same text
I hope this is understandable and someone else has an idea where I'm wrong
I guess you don't have to replace double quote with single quote.
I try to use Python3.7.4 to backup pictures in a blog site, e.g.
http://s2.sinaimg.cn/mw690/001H6t4Fzy7zgC0WLXb01&690
If I input the above address in Firefox address bar, the file is shown correctly.
If I use following code to download picture, server always redirects to a default picture:
from requests import get # just to try different methods
from urllib.request import urlopen
from urllib.parse import urlsplit, urlunsplit, quote
# hard-coded address is randomly selected for debug purpose.
origPict = 'http://s2.sinaimg.cn/mw690/001H6t4Fzy7zgC0WLXb01&690'
p = urlsplit (origPict)
newP = quote (p.path)
origPict = urlunsplit ([p.scheme, p.netloc, newP, p.query, p.fragment])
try:
#url_file = urlopen(origPict)
#u = url_file.geturl ()
url_file = get (origPict)
u = url_file.url
if u != origPict:
raise Exception ('Failed to get picture ' + origPict)
...
Any clue why requests.get or urllib.urlopen don't like '&' in url?
Updates: Thanks for Artur's comments, I realize the question is not on request itself, but on site protection mechanism: js or cookies or something else in webpage feedback something to server to allow it to judge if request comes from scraper. So now the question turns to how to scrape image from web page which is more complex than simply download image from url.
It's not about & symbol, but about redirection. Try adding parameter allow_redirects=False to get, it should be okay
I have written this small PoC for discord webhooks and i am getting error that Can not send empty string. I tried to google but couldn't find a documentation or an answer
here is my code
import requests
discord_webhook_url = 'https://discordapp.com/api/webhooks/xxxxxxxxxxxxxxxxxx/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
data = {'status': 'success'}
headers = {'Content-Type': 'application/json'}
res = requests.post(discord_webhook_url, data=data, headers=headers)
print(res.content)
I'm late, but I came across this issue recently, and seeing as it has not been answered yet, I thought I document my solution to the problem.
For the most part, it is largely due to the structure of the payload being wrong.
https://birdie0.github.io/discord-webhooks-guide/discord_webhook.html provides an example of a working structure. https://discordapp.com/developers/docs/resources/channel#create-message is the official documentation.
I was also able to get a minimum test case working using: {"content": "Test"}.
If it still fails after that with the same error, the likely causes are:
If using curl, check to make sure there are no accidental escape / backslashes \
If using embeds with fields, ensure there are no empty values
When in doubt, ensure all values are populated, and not "". Through trial-and-error / the process of cancellation, you can figure out exactly what key-value pair is causing an issue, so I suggest playing with the webhook via curl before turning it into a full program.
Let's say I want to get a zip file (and then extract it) from a specific URL.
I want to be able to use a wildcard character in the URL like this:
https://www.urlgoeshere.domain/+*+-one.zip
instead of this:
https://www.urlgoeshere.domain/two-one.zip
Here's an example of the code I'm using (URL is contrived):
import requests, zipfile, io
year='2016'
monthnum='01'
month='Jan'
zip_file_url='https://www.urlgoeshere.domain/two-one.zip'
r = requests.get(zip_file_url, stream=True)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()
Thanks in advance!
HTTP does not work that way. You must use the exact URL in order to request a page from the server.
I'm not sure if this helps you, but Flask has a feature that works similarly to what you require. Here's a working example:
#app.route('/categories/<int:category_id>')
def categoryDisplay(category_id):
''' Display a category's items
'''
# Get category and it's items via queries on session
category =session.query(Category).filter_by(id=category_id).one()
items = session.query(Item).filter_by(category_id=category.id)
# Display items using Flask HTML Templates
return render_template('category.html', category=category, items=items,
editItem=editItem, deleteItem=deleteItem, logged_in = check_logged_in())
the route decorator tells the web server to call that method when a url like */categories/(1/2/3/4/232...) is accessed. I'm not sure but I think you could do the same with the name of your zip as a String. See here (project.py) for more details.
I'm writing a Python 3 (3.5) script that will act as a simple command line interface to a user's stikked install. The API is pretty straight forward, and its documentation is available.
My post function is:
def submit_paste(paste):
global settings
data = {
'title': settings['title'],
'name': settings['author'],
'text': paste,
'lang': settings['language'],
'private': settings['private']
}
data = bytes(urllib.parse.urlencode(data).encode())
print(data)
handler = urllib.request.urlopen(settings['url'], data)
print(handler.read().decode('utf-8'))
When I run the script, I get the printed output of data, and the message returned from the API. The data encoding looks correct to me, and outputs:
b'private=0&text=hello%2C+world%0A&lang=text&title=Untitled&name=jacob'
As you can see, that contains the text= attribute, which is the only one actually required for the API call to successfully work. I've been able to successfully post to the API using curl as shown in that link.
The actual error produced by the API is:
Error: Missing paste text
Is the text attribute somehow being encoded incorrectly?
Turns out the problem wasn't with the post function, but with the URL. My virtual host automatically forwards http traffic to https. Apparently, Apache drops the post variables when it forwards.