In some website for which I have access, there are some input fields. In the sixth field I need to enter some input string from a list of 10000 strings, then a new page appears, for which I would just need to count the number of lines. Finally I would like to get a table with two columns like input string and number of resulting lines. Since I have to manually enter the info for all the different 10000 strings, I wonder therefore what is the best approach to enter a string into a generic formular field and get the resulting text. I heard about curl but I am not sure whether this is the easiest one.
P.S.
Example of interactive way: I type some string o words into google search and then I get a new page with the search results. Previously I have introduced my google username and password, so the results will be probably filtered according to my profile.
Example of non-interactive way: A script somehow introduces my user information, search query and saves to some text file the search results. Imagine the same idea but for a more complicated website like this.
What you want to do is to send a HTTP POST with specific data. This can be done with any proper HTTP client code, and one such is libcurl (or the pycurl binding or even using the curl command line tool). On the response from the post, you probably get a redirect and then the results, or you need to do a separate request for the results and then you're done and go back to do the next POST. Repeat until all POSTs are done.
What you may need to take into account is that you may have to deal with cookies and possibly to follow a redirect from the POST. A good approach is to record a "manual session" as done with a browser (use firebug or LiveHTTPHeaders etc) and then use that recording to help you repeat the same thing with a HTTP client.
A decent tutorial to get some starting up details on this kind of work can be found here: http://curl.haxx.se/docs/httpscripting.html
You could also use JMeter to run all the posts. You may use the CSV input to set the 10000 strings. Then you save the result as xml and extract the necessary data.
Related
I'm inputting text in a texbox which is working fine. But there is an existing watermark in the textbox. The entered text in the textbox is inputted as a watermark, hence clicking on 'Next' button results in an error.
Can somebody help on how to enable textfield before entering text using Blue Prism tool?
Can you sandkey event(like SAPCE key) into that field it may cause the watermark to clear it self?
How do you clear that filed in the first palce?
quoting directly from the browser automation guide which you should've read and consumed before being a developer:
Using a Write stage to write to an HTML element such as a text field
does not always work properly. For example, you might try writing to a
username field, only to see a message appear on the web site saying
something like “Please enter a username”, even though you can see that
the value has been correctly written to the field. This can happen
because the data validation functions used by the web form might be
expecting keystrokes, and the write stage has “fooled” it. To get
around this, you will need to use a Navigate stage to call the Send
Keys Action instead of using a Write stage. Some websites have maximum
character limits imposed on some text fields. Using a Write stage can
sometimes fool the website into allowing too many characters into the
field, because the Write stage “sets” the field value rather than
keying characters into it. This is important to bear in mind because
if a field has been filled with more characters than the website would
usually allow, the website could produce an error when you try to
submit or post the data.
to get around this go ahead and use sendkeys, those are likely the best option to push past this validation tool issue on the website.
I want to create a new tab and send data to it in a POST request from my chrome extension. I could use a GET request, but the data I am going to send could be arbitrarily long, and I want to JSON encode it(which also means I cannot use forms). I've only found two questions on SO about this, but both of them are talking about using a form, which is not desirable.
The reason I want to do this is because I want to request further input from the user before I do what I do with the data. I am totally lost here and have no idea how to do this, hence I can not add any code samples I've tried.
If I cannot do this with a URL, I could inject a script into the page and have a popup, but that is something of a last resort.
Any ideas on how this could be done?
I have a website where I can input a list of strings and it'll display the results of each in the same format (basically a table).
What I want to do is to be able to save the results as well as their corresponding parameters (the input string that I searched) and output them into a file to analyze later. So basically capture my input and the output it returns. It's kind of like, if I search "stack" on google, I want my output file to be "stack" and all the displayed results from the search.
I've done some research on web and screen scraping, but I can't find anything that fits my needs. I looked into the curl function in php, but it looks like it can only get the contents of a specific URL, which I don't have since I'll be repeating the searches frequently.
I also looked into the HTML Agility Pack and HttpWatch, but they don't seem to be able to extract contents this dynamically.
I was wondering if there are any ideas or tips that I could use. I was thinking maybe a plugin or application that I could write that captures the parameters of my request (input strings) and the results sent from the server, but I'm not really sure how to do this, any tips? Or maybe there's an existing one that I wasn't able to find?
Thanks in advance!
I'm developing a webapp that will need to download the html form a website and then iterate through the code and try to find a specific but ever changing value (in our case it will be the price for the product).
For this, I was thinking about asking the user (upon installation and setup) to provide the system with a few lines of html from the page (that has the price) and then from then on, every time we need to fetch the price we would try to search for those lines and find the price.
Now, I believe this is a horrible and slow way of doing this and since there are no rules and the html can be totally different from one website to another (even the same website might change) I couldn't find a better way.
One improvement that I thought about was to iterate through the first time and record the line at which we find the code. Once found, the subsequent times we would then start from a few lines before the expected location and start the search. Any Thoughts on how I can improve on this?
I posted this question on https://cstheory.stackexchange.com/ but they commented that it's not on topic and that I should post it here.
I have the code for the above and if needed I can post it, I'm simply thinking that there must be a better, faster way of doing this.
This is actually something I tried for a project recently (using BeautifulSoup and Python). The solution that worked for me was to workout CSS selectors (which can map to jQuery selectors) that targeted the elements that contained the values I was looking for. In my case I was able to narrow down the full document to just the elements that contained what I was looking for but if you couldn't get exactly what you where after you could combine this with some extra lactic like test to see if it looks like a price (via regex) or test what it is next to.
Need a way to pass a value between to pages using URL query strings if possible. However everytime I add "?customquery=customvalue" at the end it ends up to the 404 page of the website.
I want to basically make it look like this.
https://example.com/somedepartment/sample.nsf/page/hello+world?customquery=customvalue
hello+world is a document that is equivalent to a webpage.
I tried this plus a javascript that collects the strings after the number sign and it works.
https://example.com/somedepartment/sample.nsf/page/hello+world#customvalue
However, I couldn't use the hash sign because they told me not to use it and use another unique symbol instead. I am not aware of any symbols that could work the same with hash sign. If there is, please enlighten me.
Apparently, I was able to find an answer.
https://example.com/somedepartment/sample.nsf/page/hello+world?OpenDocument&RandomParam=sample
Now I could pass values by means of this format. Basically it has to be preceded by "OpenDocument" parameter before putting custom ones.
This documentation also helps: http://www.ibm.com/developerworks/lotus/library/ls-Domino_URL_cheat_sheet/