I have the following GreaseMonkey Script:
GM_xmlhttpRequest({
method: 'GET',
url: "http://www.testurl.com",
headers: {
'User-agent': 'Mozilla/4.0 (compatible) Greasemonkey/0.3',
},
onload: function(responseDetails) {
var tagString = responseDetails.responseText;
var range = document.createRange();
range.selectNode(document.body);
var documentFragment = range.createContextualFragment(tagString);
How do I now extract stuff from documentFragment? documentFragment.getElementById(''), document.body etc all returns undefined.
I suspect this is due to the createContextualFragment method returning a XPCNativeWrapper object, but how do I work around this to access the underlying DOM?
Thanks
Not sure if this is answered by Load and parse remote url with greasemonkey
Depending what you want to do, might be easier to include jquery..
I suspect this is due to the
createContextualFragment method
returning a XPCNativeWrapper object,
but how do I work around this to
access the underlying DOM?
This is done using wrappedJSObject:
var documentFragment = range.createContextualFragment(tagString).wrappedJSObject;
Related
I managed to subscribe to a Youtube Feed through https://pubsubhubbub.appspot.com/subscribe on a web browser, I've been trying to refresh my subscription to PubSubHubbub through Fetch on NodeJS, the code I used is below
const details = {
'hub.mode': 'subscribe',
'hub.topic': 'https://www.youtube.com/xml/feeds/videos.xml?channel_id=${process.env.ID}',
'hub.callback': process.env.callback,
'hub.secret': process.env.secret
};
const endpoint = 'http://pubsubhubbub.appspot.com/subscribe'
var formBody = [];
for (const property in details) {
const encodedKey = encodeURIComponent(property);
const encodedValue = encodeURIComponent(details[property]);
formBody.push(encodedKey + "=" + encodedValue);
}
formBody = formBody.join("&");
const result = await fetch(endpoint, { method: "POST", body: formBody, 'Content-Type': 'application/x-www-form-urlencoded'})
console.log(result.status, await result.text())
Expected:
Status 204
Actual:
Status 400, content: "Invalid value for hub.mode:"
I expected at least for the content to tell me what the invalid value was, however it ended up being blank, it appears to me that it did not manage to read the POSTed content at all. I'm looking for ways to improve my code so that I don't encounter this problem.
I realised that I was supposed to specify in the content type header that charset=UTF-8. I'm not particularly well-versed in sending requests as of yet so this is definitely an eye-opener for me. After adding it in the server responds with 204 as expected and so I'll close my question.
A workout that I tried while trying to fix this was to import the exec() function from child_process module to run cURL and post the request instead. However I would recommend to stick to one language to make things more readable and using exec() runs commands directly from the shell which may have undesirable effects.
I want to click the test product on this page: https://biscuit-bird.myshopify.com/collections/all
I am trying to go fully request based and not use a headless browser like puppeteer or zombie as I have already done that.
Please let me know how to do this by sending a request.
I assume you're using this lib? https://github.com/request/request
If so, inside a callback function that provides body of the page, you can use some lib to parse HTML content, find your link to a test product with CSS selector, and then open the URL stored in href.
This snipped worked for me:
const request = require('request');
var HTMLParser = require('node-html-parser');
request({
url:'https://biscuit-bird.myshopify.com/collections/all',
headers: {
'User-Agent':'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1'
}
}, function (error, response, body) {
// parse method returns a root of the generated DOM
const root = HTMLParser.parse(body)
// similarly to the browser's DOM, you can lookup DOM elements by their selector using method querySelectorAll
const links = root.querySelectorAll("a.grid-link")
const href = links[0].getAttribute('href');
// #TODO: send another request to the URL defined in `href`
});
I tried to use winjs.xhr to POST some data to a URL with no success. I got it working by essentially doing the same thing with XMLHttpRequest. This just doesn't feel right, as winjs.xhr, I thought, wraps XMLHttpRequest anyway. Can anyone explain how I do this in winjs.xhr?
Not working winjs.xhr code
Passing everything in as a URL encoded string
var url = "http://localhost/paramecho.php";
var targetUri = "http://localhost/paramecho.php";
var formParams = "username=foo&password=bar" //prefixing with a '?' makes no difference
//ends up with same response passing an object(below) or string (above)
//var formObj = {username: "foo", password: "bar"}
WinJS.xhr({
type: "post",
url: targetUri,
data: formParams
}).then(function (xhr) {
console.log(xhr.responseText);
});
I end up with my receiving PHP file getting none of the parameters, as though I'd sent no data in the first place.
I tried a few things but the code above is the simplest example. If I was to pass an object into the data parameter it behaves the same way (commented out). I've used a FormData object as well as a plain JSON object.
I changed my app manifest to have the correct network capabilities - and the working example below was done in the same app, so I'm confident it's not capability-related.
Working version using XMLHttpRequest
var username = "foo";
var password = "bar";
var request = new XMLHttpRequest();
try {
request.open("POST", "http://localhost/paramecho.php", false);
request.setRequestHeader('Content-type', "application/x-www-form-urlencoded");
request.send("username=" + encodeURIComponent(username) + "&password=" + encodeURIComponent(password));
console.log(request.responseText);
} catch (e) {
console.log("networkError " + e.message + " " + e.description);
}
And this successfully calls my PHP server-side function, with the parameters I expected.
So, the question is...how do I achieve what I have working in XMLHttpRequest with winjs.xhr? It feels like winjs.xhr the way this is supposed to work (I'm a newbie at Windows 8 app development so I'm happy to be corrected)
You're completely right WiredPrairie. Passing in the header is all that is needed - I think I'd assumed that was the default for a post.
Working version:
WinJS.xhr({
type: "post",
url: targetUri,
data: formParams,
headers: {"Content-type": "application/x-www-form-urlencoded"}
}).then(function (xhr) {
console.log(xhr.responseText);
});
I am trying to get data from the Bing search API, and since the existing libraries seem to be based on old discontinued APIs I though I'd try myself using the request library, which appears to be the most common library for this.
My code looks like
var SKEY = "myKey...." ,
ServiceRootURL = 'https://api.datamarket.azure.com/Bing/Search/v1/Composite';
function getBingData(query, top, skip, cb) {
var params = {
Sources: "'web'",
Query: "'"+query+"'",
'$format': "JSON",
'$top': top, '$skip': skip
},
req = request.get(ServiceRootURL).auth(SKEY, SKEY, false).qs(params);
request(req, cb)
}
getBingData("bookline.hu", 50, 0, someCallbackWhichParsesTheBody)
Bing returns some JSON and I can work with it sometimes but if the response body contains a large amount of non ASCII characters JSON.parse complains that the string is malformed. I tried switching to an ATOM content type, but there was no difference, the xml was invalid. Inspecting the response body as available in the request() callback actually shows bad code.
So I tried the same request with some python code, and that appears to work fine all the time. For reference:
r = requests.get(
'https://api.datamarket.azure.com/Bing/Search/v1/Composite?Sources=%27web%27&Query=%27sexy%20cosplay%20girls%27&$format=json',
auth=HTTPBasicAuth(SKEY,SKEY))
stuffWithResponse(r.json())
I am unable to reproduce the problem with smaller responses (e.g. limiting the number of results) and unable to identify a single result which causes the issue (by stepping up the offset).
My impression is that the response gets read in chunks, transcoded somehow and reassembled back in a bad way, which means the json/atom data becomes invalid if some multibyte character gets split, which happens on larger responses but not small ones.
Being new to node, I am not sure if there is something I should be doing (setting the encoding somewhere? Bing returns UTF-8, so this doesn't seem needed).
Anyone has any idea of what is going on?
FWIW, I'm on OSX 10.8, node is v0.8.20 installed via macports, request is v2.14.0 installed via npm.
i'm not sure about the request library but the default nodejs one works well for me. It also seems a lot easier to read than your library and does indeed come back in chunks.
http://nodejs.org/api/http.html#http_http_request_options_callback
or for https (like your req) http://nodejs.org/api/https.html#https_https_request_options_callback (the same really though)
For the options a little tip: use url parse
var url = require('url');
var params = '{}'
var dataURL = url.parse(ServiceRootURL);
var post_options = {
hostname: dataURL.hostname,
port: dataURL.port || 80,
path: dataURL.path,
method: 'GET',
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': params.length
}
};
obviously params needs to be the data you want to send
I think your request authentication is incorrect. Authentication has to be provided before request.get.
See the documentation for request HTTP authentication. qs is an object that has to be passed to request options just like url and auth.
Also you are using same req for second request. You should know that request.get returns a stream for the GET of url given. Your next request using req will go wrong.
If you only need HTTPBasicAuth, this should also work
//remove req = request.get and subsequent request
request.get('http://some.server.com/', {
'auth': {
'user': 'username',
'pass': 'password',
'sendImmediately': false
}
},function (error, response, body) {
});
The callback argument gets 3 arguments. The first is an error when applicable (usually from the http.Client option not the http.ClientRequest object). The second is an http.ClientResponse object. The third is the response body String or Buffer.
The second object is the response stream. To use it you must use events 'data', 'end', 'error' and 'close'.
Be sure to use the arguments correctly.
You have to pass the option {json:true} to enable json parsing of the response
In a browser, if I send a GET request, the request will send the cookie in the meanwhile. Now I want to simulate a GET request from Node, then how to write the code?
Using the marvelous request library cookies are enabled by default. You can send your own like so (taken from the Github page):
var j = request.jar()
var cookie = request.cookie('your_cookie_here')
j.add(cookie)
request({url: 'http://www.google.com', jar: j}, function () {
request('http://images.google.com')
})
If you want to do it with the native http:request() method, you need to set the appropriate Set-Cookie headers (see an HTTP reference for what they should look like) in the headers member of the options argument; there are no specific methods in the native code for dealing with cookies. Refer to the source code in Mikeal's request library and or the cookieParser code in connect if you need concrete examples.
But Femi is almost certainly right: dealing with cookies is full of rather nitpicky details and you're almost always going to be better off using code that's already been written and, more importantly, tested. If you try to reinvent this particular wheel, you're likely to come up with code that seems to work most of the time, but occasionally and unpredicatably fails mysteriously.
var jar = request.jar();
const jwtSecret = fs.readFileSync(`${__dirname}/.ssh/id_rsa`, 'utf8');
const token = jwt.sign(jwtPayload, jwtSecret, settings);
jar.setCookie(`any-name=${token}`, 'http://localhost:12345/');
const options = {
method: 'GET',
url: 'http://localhost:12345',
jar,
json: true
};
request(options, handleResponse);