Web scraping from a Google Chrome extension - google-chrome-extension

I've started to develop a Chrome extension to navigate and perform actions on a website. Until now the extension is able to receive a couple of parameters and check a set of radio-buttons, fill in a few inputs of a form and then submit it.
What I want to do now is to repeat the process, but I'm stuck when the page is reloaded. And I don't know how can I do to make the script react to the finish of the request.
The workflow I want to achieve is the following (is for automatically copying a certain object):
Popup side
Enter the number of the Master object to copy
Enter the base name of the copies (example Mod, so the I can iterate and add mod1, mod2, modn)
Enter the number of copies
Background side
Select master
Select standard options
Fill in inputs
Submit form
Wait for the page to complete the request and continue to the next copy. (here I need help)
The problem is on the repetition, the rest is taking care of. I assume that must be a way of dealing with requests. Any ideas?
By the way I'm doing it all with the extension and tabs methods of Google Chrome plus JavaScript and jQuery.

Ok, i´m going to answer the question myself based on Matthew Getner´s comment. The chrome.webRequest.onCompleted was the solution to the problem. With this method I was able to wait for the request to be completed and start over with the process. And with the messegaes methods I´ve achieved the comunication between the background and the extension itself. So I finally was able to filled a form, send it, and repeat. This way I´ve made a kind of robot to help a co-worker with a lame repetitive task on a aged web plataform.

Related

Kentico Form - Edit Form Action and Field Names

I have a form built in Kentico and want to change the form action to point to a marketing automation vendor website (eloqua) to process the form. I noticed that there's no way to change the Action on the Form app, so I thought of using js to replace the action dynamically. I'm not sure whether it'll work, but another immediate problem I have is the field name/id generated by Kentico is so long (> 70 characters - e.g. p$lt$ctl02$pageplaceholder$p$lt$ctl03$On_lineForm$viewBiz$Company$txtText) and exceeds the length allowed by the vendor. Is there a way to shorten those names/ids?
To clarify, the reason I need to edit the name because the vendor allows to map (copy/paste) html name to whatever name it's using. I've tried pasting the whole string from one Kentico field and got the error message of exceeding character limit.
First part of the question, how do i direct a form to submit to another site?
In ASP.Net, the only way is to use javascript to alter the 's action url. Since you want to be careful where you edit this, i would use jQuery to replace the "Submit" button on your form with a javascript function that will alter the form before submission (so you don't mess up other postbacks)
$("button.MySubmitButton").click(function() {
$("form").attr("action", "http://TheVendor.com/PostLocation");
});
Next question is to alter the field names. If you absolutely MUST have form element IDs sub 70 characters, you will have to use Javascript again and it will break any postback-related functionality of the elements, so make sure the form is "as is" before doing it.
Again there is security concern because when you post to another location, you are sending ALL the data, including hidden asp.net inputs that contain viewstates and the like. You may want to take the time to 'eliminate' right before you submit any field that you don't want sent to the other site.
Something like this (Test it out a bit first though)
$("input:not(id*='txtName'):not(id*='txtEmail'),select:not(id*='ddlQuestion')").remove();
As #trevor-j-fayas points out, you can use javascript to point a form action to another url. While this does work you may end up writing a lot of javascript to not only point the form to a new url but also to do some data massaging (changing id's, doing url formatting, etc) before sending it to the target.
Additionally you lose some of the benefits of using a Kentico Form because the data never actually gets submitted back to Kentico such as email alerts.
I have worked in a similar scenario where were we sending data to Eloqua but instead of doing it client side we did it from the server by using either:
The OnOnAfterSave event on the BizForm control itself
The global BizForm submit hook BizFormInfo.TYPEINFO.Events.Insert.After
After the form is submitted to Kentico, our custom hook code runs which sends the data to Eloqua. In either hook you can fully access the Form metadata, field names, and submitted values. You can then craft an HTTP POST request and submit it asynchronously using a class such as HttpClient.
Is not a good idea from the architecture stand point and most likely not going to work without opening a huge hole in their vendor web site security. First of all how are they going to process the from if they don't know the field names, what if form fields change etc. Secondly you going to run into hell of trouble trying to submit form one site to another etc. What if the vendor site is not responding etc.
What you need to do is submit the form back to kentico web site i.e. process it on kentico web site and send email notification with results to marketing automation vendor website (the easy way for now) and redirect user to vendor web site.
Redirection and email - you can do out of the box without any programming. Actually to do all the above requires no programming and you get all the information recorded on your Kentico site.

waiting for the website to change something

I am a student and in the school website, what I want to do is that I want to busy wait on the certain URL and check if the class i want to register for is open or not. I was wondering if there was a way to constantly check on the website(busy waiting or otherwise) to see if the class is open or not. There is a table Rem where it shows the number of places remaining in the User Interface.
Also what language would you use to solve this problem?
Yes you can. but for that you will probably need to create a script that fetches the value of data from that table.
So something like web scraping should work.
I would definately use php for this stuff.
Google web scraping and you can code the script.
I am not sure if this is the exact thing that will help you, but what you need to do is something similar - See Here

Chrome extension: Identifying "actual" tabs from webNavigation events

I'm trying to identify if a tab for which I received "webNavigation.onCompleted" is an "actual" tab.
As described in the documentation:
http://developer.chrome.com/extensions/webNavigation.html
A short quote from the docs:
A note about tab IDs :
Not all navigating tabs correspond to actual tabs in Chrome's UI, e.g., a tab that is being pre-rendered. Such tabs are not accessible via the tabs API nor can you request information about them via webNavigation.getFrame or webNavigation.getAllFrames. Once such a tab is swapped in, an onTabReplaced event is fired and they become accessible via these APIs.
So now when I get an event I encounter 2 problems when trying to figure out if the tab is "actual":
The scenario as described in the documentation, however I see that onTabReplaced can called before/after onCompleted,
or even not called even though I expect it to be called since I dont have the tab id in a list of tabs I create in tabs.onCreate,
and later the page is displayed in a tab.
Pages that are pre-rendered in Google chrome pages (e.g google for "ford"), and eventually dont become to show on tabs.
In case you're not familiar with pre-rendering:
https://support.google.com/chrome/answer/1385029?hl=en
I consider these cases to be different because I see in difference in pre-rendered pages,
that end u to be tads, and the ones who are not used eventually.
One page is eventually "navigated to", while the other on is just "cached".
But this difference can not be detected by using the webNavigation api, as far I as could find.
So my question - is there any way to distinguish (hopefully using the webNaviagtion api),
between an "actual" tab, a pre-rendered page which is going to be displayed as tab,
and a pre-rendered page that is going stay in cache (for now) ?
Just 2 clarifications, about solutions that supposedly I can use, but they look to me "not good",
at least at this stage.
The docs say "Such tabs are not accessible via the tabs API".
so technically I can try an use this API, get an exception, and this way figure out it's "pre-renderd" tab,
but I dont like my code to use exceptions for this, so I'm looking for a "clean" way.
in webNavigation.onComplete, I can set a timeout for 1 second,
and then if in this one second, onTabReplaced, I can figure out that this tab was
moved to be an "actual" tab. and not stayed in "cache".
But during this 1 second, or any duration I choose, there can be a lot of other events,
and even the tab can be closed.
So again, I'm looking for a "clean" solution.
Thanks for your help.

Xpages dropdown menu resubmission on SSJS submission using submit value property

Just for reference before mentioning the problem I would like to say that I have asked to same question on IBM Lotus forum(http://www-10.lotus.com/ldd/ndseforum.nsf/xpTopicThread.xsp?documentId=2AD7C8F89D8930E685257BD50022A9E9) and I have not received any reply for the same in last 2 weeks.
So, I have a typical xpage dropdown menu with say around 40-50 leaf nodes, Every leaf node submits a specific value (using submitValue property) which is then evaluated and action is performed. However, the problem is that, after the action is performed if I try to refresh the page i get the browser notification for re-submission (I believe that it is some kind of programming error by me, however I don't know how to resolve it.)
One example to make it more clear:
I have a delete node inside the dropdown. It basically deletes the selected entries from the view(generated using repeat control) and then refreshes the view content. But at this point of time if I try to refresh the page then I get the notification of re-submission by the browser. If I accept it, then it tries to delete it again (which I prevent it but still this shouldn't happen) and if I don't accept it, it just doesn't refresh the page.
Any reply would be appreciated.
Thank you in advance. Hoping to hear some suggestions. (Please let me know if event he code is required)
I believe you need to implement the Post/Redirect/Get pattern to avoid re-posting on refresh.
Here's a solution by Tommy Valand for XPages for that pattern: http://dontpanic82.blogspot.dk/2010/06/xpages-avoid-saving-duplicate-documents.html

What's the best way to prevent multiple submission in XPages?

What's is the best way to prevent multiple submission when I'm using XPages?
For "classic web" solution is below.
How to Block Multiple Submissions of the Same Document from the Web
http://www-304.ibm.com/support/docview.wss?uid=swg21089865
Using jQuery, the solution is below.
http://www.norio.be/blog/2008/09/using-jquery-prevent-multiple-form-submissions
But I don't know the way in Xapges. How to prevent it in XPages or Dojo?
It is best not to apply such client-side techniques, because combined with an unreliable internet connection the user may find that the submission fails but be unable to retry.
A much more robust solution is to deduplicate on the server side, which can be done in a variety of ways; these are some that come to mind:
Define the semantics of your form contents so that it doesn't matter if you receive two requests (e.g. if it is updating a record, then a second update just changes nothing).
If you have seen the exact same submission before (compare all the relevant fields), ignore it.
Generate a serial number when you send the form to the client. Don't accept submissions that have a serial number you've seen before — or do something useful; for example, if it is a blog posting or comment form, then a second submission should be treated as an edit to the post created by the first submission.
You can prevent a submit by using XSP.addQuerySubmitListener and return false. This is not the easiest function to get working. So I suggest you take a look at the function in the book 'Xpages Portable Command Guide' or try my project multiple file uploader on OpenNTF. Download the project source code here.

Resources