how to provide request_url with specification while web scraping using python - python-3.x

I'm on the web page with the url=x
The url of that particular web page doesn't change after giving my preferences(like selecting options,..) or after clicking the button on that web page.
Problem:
Before performing the above mentioned actions i will not be displayed with any data; but post actions the web page displays the data.
Context:
I'm trying to scrape data from a web page using python
And i'm struck at providing the request_url with the above mentioned specification
if i'm providing request_url=x it is fetching no data because i have provided no specifications
How to provide those specifications while requesting with the url?
kindly address the specification of pressing the button also

Sounds like you're trying to scrape data through real navigation actions, like filling form data and clicking on buttons and/or posting some data, considering whatever javascript scripts contains in the page, but you don't have the specifications to post the data.
My approach would be automating a real browser using selenium, finding the button via xpath or id and calling a click function to it.
driver.get("http://www.google.com")
# Assume the button has the ID "submit" :)
driver.find_element_by_id("submit").click()

Related

How to make Powerapps work with Param function in Web Part

I am trying to create a powerapps web part that will open a specific screen from a canvas app using parameters. I have added the web part and added the power apps web link to it. When I enter &parametername=parameter nothing shows up. I can enter the web link and parameters directly into a browser window (tried in Chrome and Edge) and the app will display properly. I can also created a button in a web part and added the link to it and it will also display the app when the button is clicked. The app is working properly with no errors, it will just not show up in the web part when the parameters are added to the link (If no parameters are added to the link, everything will display fine).
I am trying to create a powerapps web part that will open a specific screen from a canvas app using parameters. I have added the web part and added the power apps web link to it. When I enter &parametername=parameter nothing shows up. I can enter the web link and parameters directly into a browser window (tried in Chrome and Edge) and the app will display properly. I can also created a button in a web part and added the link to it and it will also display the app when the button is clicked. The app is working properly with no errors, it will just not show up in the web part when the parameters are added to the link (If no parameters are added to the link, everything will display fine).
I was struggling with this myself; just figured it out.
You need to put the full URL with parameters in to the "App web link or ID" property for the PowerApp web part. Be careful because the first parameter needs to be prepended with ? not &. The ampersand is used for additional, following parameters after the first parameter.
Note that I have omitted my TLD from this screenshot, however you will want to substitute your own here.
The example URL syntax given by Microsoft is https://apps.powerapps.com/play/{App ID}?{Query}. For me that URL looks something like this:
https://apps.powerapps.com/play/f813580f-3182-90b0-b3a9-781cfb288808?linkId=checkIn
I have one URL for checkIn and another for checkOut that I can use to display different forms.
My primary mistake was that I was only pasting my appID into the "App web link or ID" field rather than the entire URL. Remember, you can edit your Canvas Form, click on "Share" and get the Web Link from this page. This contains the full URL, minus your custom parameters. Play around with the full URL plus parameters until you get something that the web part will accept.

Is there a way in which a background video keeps playing when you navigate through a static website?

this website is not using ajax to update sections of the site with content, its a static site and therefore moves from one html page to the other in the classic way.
each page has a video element embedded which plays the same video.
clearly this means each time a user clicks to navigate to a different page, the video will restart.
what approaches can be taken to enable the video to maintain its position no matter which page the user visits ?
I understand each time a user clicks a hyperlink to navigate to a different page, a new http request is performed and there is the request and retrieval of this html page.
I'm curious whether I can achieve what I'm looking for without using AJAX. be interested to hear from you all on this.

Programatically retrieve the updated og:image of a page in an extension

There seem to be many pages where the the og:image does not change as I keep browsing from one page to another. The og:image always points to the first (landing) page. This is true of youtube videos, for instance. Of course, reloading the page provides the correct og:image
I am wondering if there is a way, within a custom extension on Chrome and Safari, to force refresh the og:image data without affecting user experience?
Dynamically updated sites (aka AJAX sites) change only a portion of the page with the new content on intra-site navigation. The meta information in head element like og:image isn't updated usually.
A universal workaround for any site with AJAX navigation would be to make a XMLHttpRequest for the current URL, convert the response into DOM via DOMParser API and extract the og:image tag.
Or you can write site-specific code and try to find an internal variable or element that contains the og:image. It requires some reverse-engineering, and your code would break on site changes.

Can I preload content in a Webbrowser control?

I have a VC++ MFC dialog application with a web browser ActiveX control. I have "Next" and "Prev" buttons using which I let the user navigate through a list of pre-defined URLs, whose content are shown in the web browser control. Since I know the list of URLs at the start itself, I would like to pre-load the content in some way while the user is looking at one page, so that when they click on "Next", the content has already been fetched and can be shown to the user instead of waiting for the page to load. I did not find any documentation on how to do that so far. Does anyone have any ideas on how to achieve this? I was thinking of having a second invisible web browser control where I pre-load the next URL, but it would be tricky to handle the user clicking Next when the next URL is still loading in the other browser.

problem with Infopath Form services

I want to have some url for my infopath form and give that link to user instead of going to form library and click new...how is it possible?
Also when I created Infopath form I have given some text fields and button, I don't want the Infopath services buttons(Save,save as...) how I cna remove those?
Any suggestions would really be appreciated...Thanks
You can publish an InfoPath form to a network share and give your users a link to it (or use a file:// url on a webpage). Another option is to figure out the URL to the form within form services (whats the actual URL when they click on the new button) and send out that link as the official URL. Both options have advantages (and disadvantages) so make sure to evaluate them both with respect to what you are trying to accomplish and the environment (ie - are you just trying to save your users a click or is there some other reason you need the 'new' URL).
1.) Format your URL like this:
https://SPSITE/sites/_layouts/FormServer.aspx?xsnLocation=https://SPSITE/sites/FORMLIBRARY/forms/template.xsn?DefaultItemOpen=1
2.) You found already! ; )

Resources