How to scraping the link such as http://bitly.is/heretohelp in tweet and then open the link automatically?
Tweepy gives you access to the Tweet text and conveniently exposing the urls, hashtags, for example to get the URLs within a Tweet:
tweet = api.get_status(id='000001')
print(tweet.entities['urls'])
for url in tweet.entities['urls']:
# t.co url
print(url['url'])
# original url
print(url['expanded_url'])
Once you have the URLs you can decide to do what you want (scrape the target url or open in a browser tab - if you have a web app for example)
Related
I started this android webView project as my first android studio project awhile ago, I was slow with it and kept adding features little by little.
Now I want to develop the project into a web browser worthy of play store publication.
I have selected no action bar in my style, I am using an editText to load URL already but when a text is entered which is not an URL, it gives error.
So, I want to solve this by enabling search function when no URL is entered in edit text but regular text
i.e presently in editText google.com facebook.com or any URL loads but typing facebook, news etc in edit text shows error "WEB PAGE NOT AVAILABLE the web page https://google/ could not be loaded because net::ERR_NAME_NOT_RESOLVED"
I want to fix this, to enable google search or any other search engine in such case.
I do hope my question is understandable and well detailed?
Welcome in stack
to open URL with web View you must pass a full URL like https://stackoverflow.com and or stackoverflow.com
when you want to make query in some keys like Facebook or news
you can use google search as example with this key to get full link of this result
so when user input text you must check if this is url or not
So if is URL and will return true just pass it to web View
if not will return false,and pass it as query word to google and then show on web view
You can check if string is url or not by this code refer to this
bool isUrl= URLUtil.isValidUrl(text);
String url;
if(isUrl)
url = text;
else{
//this url well open in web view as google search
url = "https://www.google.com/search?q="+text.replace(" ", "%20");
}
//now you have url to open in your web view
i hope this help
I am trying to make a web scraper. I would like to get the destination URL from a query URL. But it redirects many times.
This is my URL:
https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO
Destination url should be:
https://www.jw.org/ins/library/videos/#ins/mediaitems/VODOrgLegal/pub-jwb_201812_16_VIDEO
But I am getting https://www.jw.org/ins/library/videos/?item=pub-jwb_201812_16_VIDEO&appLanguage=INS this as the redirected URL.
I tried this code:
import requests
url = 'https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO'
s = requests.get(url)
print(s.url)
The redirect is made using JavaScript
It is not a server redirect so requests is not following it.
You can get the URL using Selenium
from selenium import webdriver
import time
browser = webdriver.Chrome()
url = 'https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO'
browser.get(url)
time.sleep(5)
print (browser.current_url)
browser.quit()
Outputs
https://www.jw.org/ins/library/videos/#ins/mediaitems/VODOrgLegal/pub-jwb_201812_16_VIDEO
If you are building a scraper I would suggest you check out scrapy-splash https://github.com/scrapy-plugins/scrapy-splash or requests-html https://github.com/psf/requests-html
You can do this pretty easily using requests:
import requests
destination = requests.get("http://doi.org/10.1080/07435800.2020.1713802")
#this link redirects the user to another link with a research paper of a given DOI code
print(destination.url)
#this returns "https://www.tandfonline.com/doi/full/10.1080/07435800.2020.1713802", the redirect of the initial doi.org link
So when opening the dynamic link I get the error- Requested URL must be a parseable URI, but possibly incomplete to be a DynamicLink.
I have the app store link attached to it. How do I fix this?
Say my dynamic link is https://example.page.link and my URL is https://example.page.link/abcd.
DynamicLink dynamicLink = FirebaseDynamicLinks.getInstance().createDynamicLink()
.setLink(Uri.parse("https://exmaple.page.link/"))
.setDomainUriPrefix("https://example.page.link/")
.setAndroidParameters(new DynamicLink.AndroidParameters.Builder().build())
.buildDynamicLink();
dynamicLinkUri = dynamicLink.getUri();
But when I add abcd in setDomainUriPrefix() then the post in the app doesn't open, though the link opens in the browser.
DynamicLink dynamicLink = FirebaseDynamicLinks.getInstance().createDynamicLink()
.setLink(Uri.parse("https://exmaple.page.link/"))
.setDomainUriPrefix("https://example.page.link/abcd")
.setAndroidParameters(new DynamicLink.AndroidParameters.Builder().build())
.buildDynamicLink();
And when I receive this link, the url always shows as the playstore link, hence it doesn't open the app post. How can I fix this?
dynamicLinkUri = dynamicLink.getUri();
How can I add credential(username and password) and then fetch the URL of that particular website in python?
I need help to implement this in my project
Example: suppose the first page is the login page and when we provide the credential and validate that and then we are re-directed into the homepage and then pick the url of that homepage
You can use package os
import os
url = os.environ['HTTP_HOST']
uri = os.environ['REQUEST_URI']
return url+uri
I need to download some external urls to a string in a firefox addon.
How can I access the content of external urls for parsing?
To find out the url of the current tab in a firefox addon you can do:
var localString = gBrowser.currentURI.spec;
Then you can parse localString the way you want.