In Firefox extension, we can do:
var _prefService = Components.classes["#mozilla.org/preferences-service;1"].getService(Components.interfaces.nsIPrefBranch);
var httpHandler = Cc["#mozilla.org/network/protocol;1?name=http"].getService(Ci.nsIHttpProtocolHandler);
setCharPref("general.useragent.override",httpHandler.userAgent + " OurUAToken/1.0");
To add "OurUAToken/1.0" at the end of User-Agent string.
How can we duplicate this behavior in Google Chrome?
Not sure if someone still looking for the solution but the chrome.webRequest API suggested earlier is quite stable now.
chrome.webRequest.onBeforeSendHeaders.addListener(
function (details) {
for (var i = 0; i < details.requestHeaders.length; ++i) {
if (details.requestHeaders[i].name === 'User-Agent') {
details.requestHeaders[i].value = details.requestHeaders[i].value + ' OurUAToken/1.0';
break;
}
}
return { requestHeaders: details.requestHeaders };
},
{ urls: ['<all_urls>'] },
['blocking', 'requestHeaders']
);
One of the chrome extensions, Requestly already has similar implementation to allow overriding User Agent string for any website opened in the browser.
For more info, please visit blog: https://medium.com/#requestly_ext/switching-user-agent-in-browser-f57fcf42a4b5
The extension is also available for Firefox. Visit http://www.requestly.in for details.
You can use the WebRequest api: http://code.google.com/chrome/extensions/trunk/experimental.webRequest.html
Unfortunately, it's still in Experimental stage. I think it will be released as stable with Chrome version 17
Related
I cannot find an Instagram Reels related API. Does anyone know if there is one or if there will be one?
Currently the Facebook developer documentation does not mention anything about Instagram Reels.
https://developers.facebook.com/docs/instagram-api/overview/
UPDATED ANSWER:
Reels are now supported via the API; it became generally available on July 6, 2022. Please see https://developers.facebook.com/blog/post/2022/06/27/introducing-reels-apis-to-instagram-platform/.
ORIGINAL ANSWER:
In the Instagram API docs, under Limitations, it currently says "Reels are not supported."
Reels it is IG Media with type "VIDEO".
And you can fetch single "Reels" if you know his Facebook Media ID (It is not Media ID from instagram)
If reels published like stories you can get media's list from stories endpoint(GET graph.facebook.com/{ig-user-id}/stories). You will get array of medias' ids.
"data": [
{
"id": "{ig-media-id}"
},
...
]
}
Then you can fetch information from single media object endpoint (GET /{ig-media-id})
So at current moment you can get reels by api only if they published like story.
Anyway reels is not supported in some countries and you can see reels only from stories in this countries.
UPDATE
Reels also available when you fetch user media from business-discovery(GET {ig-user-id}?fields=business_discovery.username(instagramm_user_name){media{id,permalink,media_type,media_url}}
) or user media (GET /{ig-user-id}/media).
If media's permalink look likes https://www.instagram.com/reel/... and media_type VIDEO then is it reels.
Not sure, but I did some digging around an Instagram .ipa file and found these url scheme parameters.
instagram://reels_home
instagram://reels_share
Both of which if clicked on ios will take you to the reels feed. My intentions are to open the reels camera (or the story camera with the mode set to reels), but i guess currently we can only access the reels feed.
It seems they were included in the GET /{ig-user-id}/media and the GET {ig-user-id}?fields=business_discovery.username(instagramm_user_name) endpoints at the beginning but they removed them later. According to the docs they are not supported at the moment. Further confirmed in this bug report. It seems it is not the first time they include nodes that are not supposed to be included (IGTV and now reels).
Naa,
Instagram doesn't provide any API for Reels yet, since the reel feature is still not available in many countries but this going to be available soon.
So the question arises here, how we can get the reels data via API?
Well, the answer is you can't But! if you are in favour to do so some scraping thing then the answer is definitely Yes!
But How?
We all know scraping has lots of efforts, due to things get changed very frequent. and if you don't care about those below are sample PHP script to fetch reels data in JSON.
<?php
//!IMPORTANT
$sessionid = ["Cookie: sessionid=YOUR SESSION ID HERE"];
// REELS SHORT CODE FROM URL
$shortcode = null;
$response = [];
/**
* Get a web file (HTML, XHTML, XML, image, etc.) from a URL. Return an
* array containing the HTTP server response header fields and content.
*/
function get_web_page($url)
{
$user_agent = 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_CUSTOMREQUEST => "GET", //set request type post or get
CURLOPT_POST => false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
# CURLOPT_COOKIEFILE => "cookie.txt", //set cookie file
# CURLOPT_COOKIEJAR => "cookie.txt", //set cookie jar
CURLOPT_HTTPHEADER => $sessionid, // sending manually set cookie
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
if ((isset($_GET['q'])) && !empty($_GET['q'])) {
$shortcode = $_GET['q'];
$reels_url = "https://www.instagram.com/p/$shortcode/?__a=1";
// read a web page and check for errors:
$result = get_web_page($reels_url);
if ($result['errno'] != 0) {
$response = json_encode(['error' => "bad url, timeout, redirect loop"]);
}
if ($result['http_code'] != 200) {
$response = json_encode(['error' => "no page, no permissions, no service"]);
}
if ($result['http_code'] === 200) {
$response = $result['content'];
}
}
// JSON OUTPUT OR ERROR HERE
header('Content-Type: application/json');
echo $response;
How to use this PHP script?
Save the above script in a PHP file e.g. fetch-reels.php and run it like
http://localhost/fetch-reels.php?q={SHORT-CODE}
Reels URL e.g.
https://www.instagram.com/p/COlKxQLAM11
https://www.instagram.com/reel/COlKxQLAM11/?igshid=12f6j9a1dfx2x
As mentioned earlier Instagram doesn't provide direct apis but you can take help of Instagrapi its really awesome and easy to use. Note : This Instagram Private API wrapper is in python.
Here's how to use it -
Install the lib :
python -m pip install instagrapi
Code
from instagrapi import Client
cl = Client()
#enter your username and password
cl.login('username', 'password')
#you can replace 10 with whatever amount of reels you want to fetch
reel = cl.explore_reels(amount = 10)
print(reel)
That's it !!!
Instagram just released reels support (Available to everyone from July 7th 2022).
More here
https://developers.facebook.com/docs/instagram-api/guides/content-publishing/
Instagram Reels API is now generally available:
https://developers.facebook.com/blog/post/2022/06/27/introducing-reels-apis-to-instagram-platform/
You can make a scheduled post. For example using JavaScript and uploading a video to the Reels API[1]:
const access_token = "Js82ks92jald"; // The access token given from FB
const instagram_user_id = "12345"; // The IG user's ID
const reelUrl = "https://www.website.com/reel.mp4";
const caption = "This is the best real ever #Reels4Real";
const postUrl = `https://graph.facebook.com/${instagram_user_id}/media?video_url=${encodeURIComponent(reelUrl)}&caption=${encodeURIComponent(caption)}&access_token=${access_token}&media_type=REELS&share_to_feed=true&thumb_offset=2000`;
const postOptions = {
method: "POST",
headers: {
"Content-Type": "application/json",
},
};
fetch(postUrl, postOptions)
.then((res) => res.json())
.then(json => console.log(json))
.catch(console.error);
Note the key fields of media_type=REELS, video_url, and caption.
[1] https://www.ayrshare.com/instagram-reels-api-how-to-post-videos-to-reels-using-a-social-media-api/
Update (14-July-2022): Meta (Facebook) launched support for Reels via their official Graph API. Read more about it here: https://developers.facebook.com/docs/instagram-api/reference/ig-user/media#reel-specifications
Meta just release Reels API for the Instagram Platform.
https://developers.facebook.com/blog/post/2022/06/27/introducing-reels-apis-to-instagram-platform/
I´m trying to use a headless browser for crawling purposes to add SEO features in a open source project i´m developing.
The project sample site is deployed via Azure Websites.
I tried several ways to get the task working using different solutions like Selenium .NET (PhantomJSDriver, HTMLUnitDriver, ...) or even standalone PhantomJs .exe file.
I´m using a headless browser because the site is based in DurandalJS, so it needs to execute scripts and wait for a condition to be true in order to return the generated HTML. For this reason, can´t use things like WebClient/WebResponse classes or HTMLAgilityPack which use to work just fine for non-javascript sites.
All the above methods works in my devbox localhost environment but the problem comes when uploading the site to Azure Websites. When using standalone phantomjs the site freezes when accessing the url endpoint and after a while return a HTTP 502 error. In case of using Selenium Webdriver i´m getting a
OpenQA.Selenium.WebDriverException: Unexpected error. System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:XXXX
I think the problem is with running .exe files in Azure and not with the code. I know it´s possible to run .exe files in Azure CloudServices via WebRole/WebWorkers but need to stay in Azure Websites for keep things simple.
It´s possible to run a headless browser in Azure Websites? Anyone have experience with this type of situation?
My code for the standalone PhantomJS solution is
//ASP MVC ActionResult
public ActionResult GetHTML(string url)
{
string appRoot = Server.MapPath("~/");
var startInfo = new ProcessStartInfo
{
Arguments = String.Format("{0} {1}", Path.Combine(appRoot, "Scripts\\seo\\renderHTML.js"), url),
FileName = Path.Combine(appRoot, "bin\\phantomjs.exe"),
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true,
StandardOutputEncoding = System.Text.Encoding.UTF8
};
var p = new Process();
p.StartInfo = startInfo;
p.Start();
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
ViewData["result"] = output;
return View();
}
// PhantomJS script
var resourceWait = 300,
maxRenderWait = 10000;
var page = require('webpage').create(),
system = require('system'),
count = 0,
forcedRenderTimeout,
renderTimeout;
page.viewportSize = { width: 1280, height: 1024 };
function doRender() {
console.log(page.content);
phantom.exit();
}
page.onResourceRequested = function (req) {
count += 1;
//console.log('> ' + req.id + ' - ' + req.url);
clearTimeout(renderTimeout);
};
page.onResourceReceived = function (res) {
if (!res.stage || res.stage === 'end') {
count -= 1;
//console.log(res.id + ' ' + res.status + ' - ' + res.url);
if (count === 0) {
renderTimeout = setTimeout(doRender, resourceWait);
}
}
};
page.open(system.args[1], function (status) {
if (status !== "success") {
//console.log('Unable to load url');
phantom.exit();
} else {
forcedRenderTimeout = setTimeout(function () {
//console.log(count);
doRender();
}, maxRenderWait);
}
});
and for the Selenium option
public ActionResult GetHTML(string url)
{
using (IWebDriver driver = new PhantomJSDriver())
{
driver.Navigate().GoToUrl(url);
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
IWebElement myDynamicElement = wait.Until<IWebElement>((d) =>
{
return d.FindElement(By.CssSelector("#compositionComplete"));
});
var content = driver.PageSource;
driver.Quit();
return Content(content);
}
}
Thanks!!
You cannot execute exe files in the shared website environment, either you have to use the web services or you have to set up a proper (azure) virtual machine.
The free shared website service is really basic, and won't cut it when you need more advanced functionality.
See this question and accepted answer for a more elaborated answer: Can we run windowservice or EXE in Azure website or in Virtual Machine?
I am not sure about shared and basic website environment but i am successfully run ffmpeg.exe from standart website environment. Despite that still phantomjs and even chromedriver itself is not working.
However i am able run Firefox driver successfully. In order to do that
I copied latest firefox directory from my local to website and below code worked well.
var binary = new FirefoxBinary("/websitefolder/blabla/firefox.exe");
var driver = new FirefoxDriver(binary, new FirefoxProfile());
driver.Navigate().GoToUrl("http://www.google.com");
I want to include facebook login functionality in my chrome extension. I have included
connect.facebook.net/en_US/all.js file in popup.html. But not working.
i m using code
window.fbAsyncInit = function() {
// init the FB JS SDK
FB.init({
appId: 'APP_ID', // App ID from the App Dashboard
//channelUrl : '//www.example.com/', // Channel File for x-domain communication
status: true, // check the login status upon init?
cookie: true, // set sessions cookies to allow your server to access the session?
xfbml: true // parse XFBML tags on this page?
});
// Additional initialization code such as adding Event Listeners goes here
console.log(FB);
};
// Load the SDK's source Asynchronously
(function(d, debug) {
var js, id = 'facebook-jssdk', ref = d.getElementsByTagName('script')[0];
if (d.getElementById(id)) {
return;
}
js = d.createElement('script');
js.id = id;
js.async = true;
js.src = "http://connect.facebook.net/en_US/all" + (debug ? "/debug" : "") + ".js";
ref.parentNode.insertBefore(js, ref);
}(document, /*debug*/false));
but getting error
Refused to load the script 'http://connect.facebook.net/en_US/all.js' because it violates the following Content Security Policy directive: "script-src 'self' chrome-extension-resource:".
I use the following code and it works, without damaging the CSP
function loginfacebook()
{
chrome.windows.create(
{
'url': "https://www.facebook.com/dialog/oauth?client_id=yourclientid&redirect_uri=https://www.facebook.com/connect/login_success.html&scope=email&response_type=token"
}, null);
chrome.tabs.query({active:true}, function(tabs){
tabid = tabs[0].id;
chrome.tabs.onUpdated.addListener(function(tabid, tab)
{
var cadena = tab.url;
if (cadena != null)
{
var resultado = cadena.match(/[\\?&#]access_token=([^&#])*/i);
}
if (resultado != null)
{
token = resultado[0];
token = token.substring(14);
storagetoken(token);
};
});
As the error message says, the Facebook script isn't CSP-compliant. I haven't looked at that script, but if you can't modify it and fix the CSP issues, you have a couple options in general for dealing with such scripts:
Put it in a sandboxed iframe.
Put the script in a <webview>.
Unfortunately, the point of that Facebook script is likely to set a cookie after FB authentication, and that cookie would stay in either the iframe or the webview, so neither of these approaches will end up with the required cookie in your main app. You'll have to figure out a way to transmit the product (cookie) of the FB login operation to your app, likely through postMessage. If you do that legwork and succeed, please post your results somewhere, such as in a sample app on GitHub.
My extensions makes excessive use of chrome.tabs.captureVisibleTab(), however if the user is browsing in pages that I don't have permissions on the call fails with the following error message in the console:
Error during tabs.captureVisibleTab: Cannot access contents of url "chrome://settings/extensions#". Extension manifest must request permission to access this host.
While this doesn't interfere with the normal flow of my extension, I was wondering if these is a way to check for permissions before calling chrome.tabs.captureVisibleTab() to prevent my extension from seeming unprofessional
Did you try handling the error with a try...catch statement? Something like this:
try {
// If any error rises here
chrome.tabs.captureVisibleTab();
} catch(e) {
// Supress the error and go here
}
Just get extension's manifest info and then:
for(var i = 0, permissions = {}; i < chrome.manifest.permissions.length; i++){
// map array to object for later use
permissions[chrome.manifest.permissions[i]] = true;
}
if(permissions.hasOwnProperty('tabs')){
// capture
}
UPDATE:
Recently (dev channel) Chrome devs added new powerfull API - chrome.permissions (docs). So you can check if your extension have particular permission by:
chrome.permissions.contains({
permissions: ['tabs'],
origins: ['http://www.google.com/']
}, function(result) {
if (result) {
// The extension has the permissions.
} else {
// The extension doesn't have the permissions.
}
});
What is the best way to record statistics on the number of visitors visiting my site that have set their browser to block ads?
Since programs like AdBlock actually never request the advert, you would have to look the server logs to see if the same user accessed a webpage but didn't access an advert. This is assuming the advert is on the same server.
If your adverts are on a separate server, then I would suggest it's impossible to do so.
The best way to stop users from blocking adverts, is to have inline text adverts which are generated by the server and dished up inside your html.
Add the user id to the request for the ad:
<img src="./ads/viagra.jpg?{user.id}"/>
that way you can check what ads are seen by which users.
You need to think about the different ways that ads are blocked. The first thing to look at is whether they are running noscript, so you could add a script that would check for that.
The next thing is to see if they are blocking flash, a small movie should do that.
If you look at the adblock site, there is some indication of how it does blocking:
How does element hiding work?
If you look further down that page, you will see that conventional chrome probing will not work, so you need to try and parse the altered DOM.
AdBlock forum says this is used to detect AdBlock. After some tweaking you could use this to gather some statistics.
setTimeout("detect_abp()", 10000);
var isFF = (navigator.userAgent.indexOf("Firefox") > -1) ? true : false,
hasABP = false;
function detect_abp() {
if(isFF) {
if(Components.interfaces.nsIAdblockPlus != undefined) {
hasABP = true;
} else {
var AbpImage = document.createElement("img");
AbpImage.id = "abp_detector";
AbpImage.src = "/textlink-ads.jpg";
AbpImage.style.width = "0";
AbpImage.style.height = "0";
AbpImage.style.top = "-1000px";
AbpImage.style.left = "-1000px";
document.body.appendChild(AbpImage);
hasABP = (document.getElementById("abp_detector").style.display == "none");
var e = document.getElementsByTagName("iframe");
for (var i = 0; i < e.length; i++) {
if(e[i].clientHeight == 0) {
hasABP = true;
}
}
if(hasABP == true) {
history.go(1);
location = "http://www.tweaktown.com/supportus.html";
window.location(location);
}
}
}
}
I suppose you could compare the ad prints with the page views on your website (which you can get from your analytics software).