I am running a web-app on a Raspberry Pi in chromium that should be running 24/7. The main issue is that it runs out of memory and displays "He's dead Jim". I am wondering if someone can help me to:
Direct me to a chromium extension that will reload/ reboot the browser if memory runs out
Supply a possible cron job to detect when memory is running out and reboot the browser if that's the case
The aim is to keep chromium running everyday without human intervention. So any additional methods/ideas would be appreciated.
Thanks in advance!
I actually found the culprit to be a few ajax request that each ran every few seconds to check if the server is still up or not(it's a long story but has to be done).
Then I found a small memory-saving solution online: to put all the ajax requests in a variable and then clear the variable after use(I also cleared unused java-script variables application-wide). Here is an example below:
function getData(){
var request = $.ajax({
url : "/someurl",
type : "HEAD",
dataType : "json",
success : function(data) {
//use your data
}
error: function(){
//doSomething
},
cache : false
});
//HERE IS THE HACK! :)
data = null;
request.onreadystatechange = null;
request.abort = null;
request = null;
}
setTimeout(function(){
getData();
}, 0.05 * 60 * 1000)
}
P.S I found the code online.
Related
I am in a situation where I have a CRON task on google app engine (using flex environment) that just dies after some time, but I have no trace WHY (checked the GA Logs, nothing, tried try/catch, and explicitly log it - no error).
I have explicitly verified that if I create a cron task that runs for 8 minutes (but doesn't do much - just sleeps and updates database every second), it will run successfully. This is just to prove that CRON jobs can at least run 8 minutes if not more. & I have set up the Express & NodeJS combo up correctly.
This is all fine, but seems that my other cron job dies in 2-3 minutes, so quite fast. It is hitting some kind of limit, but I have no idea how to control for it, or even what limit it is, so all I can do is speculate.
I will tell more about my CRON task. It is basically rapidly querying MongoDB database where every query is quite fast. I've tried the same code locally, and there are no problems.
My speculation is that I am somehow creating too many MongoDB requests at once, and potentially running out of something?
Here's a pseudocode (just to describe what kind of scale data we're talking about - the numbers and flow are exactly the same):
function q1() {
return await mongoExecute(async (db) => {
const [l1, l2] = await Promise.all([
db.collection('Obj1').count({uid1: c1, u2action: 'L'}),
db.collection('Obj1').count({uid2: c2, u1action: 'L'}),
]);
return l1+l2;
});
}
for(let i = 0; i < 8000; i++) {
const allImportantInformation = Promise.all([
q1(),
q2(),
q3(),
.....
q10()
])
await mongoDb.saveToServer(document);
}
It is getting somewhere around i=1600 before the CRON job just dies without any explanation. The GA Cron Job panel clearly says the JOB has failed.
Here is also my mongoExecute (which is just a separate module that caches the db object, which hopefully is the correct practice in order to ensure that mongodb pooling works correctly.)
import { MongoClient, Db } from 'mongodb';
let db = null;
let promiseInProgress = null;
export async function mongoExecute<T> (executor: (instance: Db) => T): Promise<T | null> {
if (!db) {
if (!promiseInProgress) {
promiseInProgress = new Promise(async (resolve, reject) => {
const tempDb = await MongoClient.connect(process.env.MONGODB_URL);
resolve(tempDb);
});
}
db = await promiseInProgress;
}
try {
const value = await executor(db);
return value;
} catch (error) {
console.log(error);
return null;
}
}
What would be the solution? My idea is to basically ensure less requests are made at once (so all the promises would be sequential, and potentially add sleep between each cycle in the FOR.
I don't understand because it works fine up until some specific point (and quite big point, it's definitely different amount, sometimes it is 800, sometimes 1200, etc).
Is there any "running out of TCP connections" scenario happening? Theoretically we shouldn't run out of anything because we don't have much open at any given point.
It seems to be working if I throw 200ms wait between each cycle & I suspect I can figure out solution, all the items don't have to be updated in the same CRON execution, but it is a bit annoying, and I would like to know what's going on.
Is the garbage collector not catching up fast enough, why exactly is GA silently failing my cron task?
I discovered what the bug is, and fixed it accordingly.
Let me rephrase it; I have no idea what the bug was, and having no errors at any point was discouraging, however I managed to fix (lucky guess) whatever was happening by updating my nodejs mongodb driver to the latest version (from 2.xx -> 3.1.10).
No sleeps needed in my code anymore.
I am developing an application targetting mobile devices, so I have to consider bad network connectivity. In one use case, I need to reduce the timeout for a request, because if no network is available, that's okay, and I'd fall back to default data immediately, without having the user wait for the HTTP response.
I found that HttpMixin.MakeWebRequest() has a timeout parameter (with default=null) but DownloadUrl() never makes use of it, so the forementioned function always waits for up to 15 seconds:
request.Timeout(timeout ?? TimeSpan.FromSeconds(15),
BlobCache.TaskpoolScheduler).Retry(retries);
So actually I do not have the option to use a different timeout, or am I missing something?
Thanks for considering a helpful response.
So after looking at the signature for DownloadUrl in
HttpMixin.cs
I saw what you are talking about and am not sure why it is there but, it looks like the timeout is related to building the request and not a timeout for the request itself.
That being said, in order to set a timeout with a download, you have a couple options that should work.
Via TPL aka Async Await
var timeout = 1000;
var task = BlobCache.LocalMachine.DownloadUrl("http://stackoverflow.com").FirstAsync().ToTask();
if (await Task.WhenAny(task, Task.Delay(timeout)) == task) {
// task completed within timeout
//Do Stuff with your byte data here
//var result = task.Result;
} else {
// timeout logic
}
Via Rx Observables
var obs = BlobCache.LocalMachine
.DownloadUrl("http://stackoverflow.com")
.Timeout(TimeSpan.FromSeconds(5))
.Retry(retryCount: 2);
var result = obs.Subscribe((byteData) =>
{
//Do Stuff with your byte data here
Debug.WriteLine("Byte Data Length " + byteData.Length);
}, (ex) => {
Debug.WriteLine("Handle your exceptions here." + ex.Message);
});
Problem
I have a NodeJS Server with the request module.
I use requests pipe() for serving files.
Sometimes, the app throws an exception, all downloads cancel and I have to restart the app:
Out of memory: Kill process 9342 (nodejs) score 793 or sacrifice child
Killed process 9342 (nodejs) total-vm:1333552kB, anon-rss:410648kB, file-rss:0kB
I wrote another script which restarts the server automatically (with childprocess & fork) when it ends unexpectedly, which sometimes throws this error:
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
Server data
RAM: 500MB (I know that this is not much, but it's cheap)
Ubuntu 12.04.5 LTS
NodeJS version: v0.10.36
Assumptions
Too much downloads in parallel
Something wrong with pipe related to the RAM
Regarding 1:
When somebody downloads a big file a bit of it is loaded in the RAM (I know nothing about this, say 20MB at once, please correct me if I'm too wrong). When 400MB is available and 20 current downloads with the same download speed, the server crashes because he can't load more than 400MB at once in the RAM.
Regarding 2:
In addition to pipe() I use the following code to track current & canceled downloads:
req.on("close", function() {
currentDownloads--;
});
The pipe() doesn't close properly and the RAM it used doesn't get cleared.
Questions
If any of my assumptions should be right, how could I fix it?
If not, what could it be or rather where could be the cause (is it NodeJS/request module/is my code wrong or bad, are there better methods)?
Full Code
var currentDownloads = 0;
app.post("/", function (req, res) {
var open = false;
req.on("close", function () {
if (open) {
currentDownloads--;
open = false;
}
});
request.get(url)
.on("error", function (err) {
log("err " + err);
if (open) {
currentDownloads--;
open = false;
}
})
.on("response", function () {
open = true;
currentDownloads++;
})
.pipe(res);
});
I'm trying to create a pool of Phantom Webdrivers [using webdriverjs] like
var driver = new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
Once the pool gets populated [I see n-number of phantom processes spawned], I try to do driver.get [using different drivers in the pool] of different urls expecting them to work parallely [as driver.get is async].
But I always see them being done sequentially.
Can't we load different urls parallely via different web driver instances?
If not possible in this way how else could I solve this issue?
Very Basic Impl of my question would look like below
var Webdriver = require('selenium-webdriver'),
function getInstance() {
return new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
}
var pool = [];
for (var i = 0; i < 3; i++) {
pool.push(getInstance());
}
pool[0].get("http://mashable.com/2014/01/14/outdated-web-features/").then(function () {
console.log(0);
});
pool[1].get("http://google.com").then(function () {
console.log(1);
});
pool[2].get("http://techcrunch.com").then(function () {
console.log(2);
});
PS: Have already posted it here
Update:
I tried with selenium grid with the following setup; as it was mentioned that it can run tests parallely
Hub:
java -jar selenium/selenium-server-standale-2.39.0.jar -hosost 127.0.0.1 -port 4444 -role hub -nodeTimeout 600
Phantom:
phantomjs --webdriver=7777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=7877 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=6777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
Still I see the get command getting queued and executed sequentially instead being parall. [But gets properly distributed across 3 instances]
Am I still missing something out?
Why is it mentioned "scale by distributing tests on several machines ( parallel execution )" in the doc?
What is parallel as per the hub? I'm getting clueless
I guess I got the issue..
Basically https://code.google.com/p/selenium/source/browse/javascript/node/selenium-webdriver/executors.js#39 Is synchronous and blocking operation [atleast the get].
Whenever the get command is issued node's main thread get's stuck there. No further code execution.
A little late but for me it worked with webdriver.promise.createFlow.
You just have to wrap your code in webdriver.promise.createFlow() { ... }); and it works for me! Here's an example from Make parallel requests to a Selenium Webdriver grid. All thanks to the answerer there...
var flows = [0,1,2,3].map(function(index) {
return webdriver.promise.createFlow(function() {
var driver = new webdriver.Builder().forBrowser('firefox').usingServer('http://someurl:44111/wd/hub/').build();
console.log('Get');
driver.get('http://www.somepage.com').then(function() {
console.log('Screenshot');
driver.takeScreenshot().then(function(data){
console.log('foo/test' + index + '.png');
//var decodedImage = new Buffer(data, 'base64')
driver.quit();
});
});
});
});
I had the same issues, I finally got around the problem using child_process.
The way my app is setup is that I have many tasks that does different things, and that needs to run simultaneously (each of those use a different driver instance), obviously it was not working.
I now start those tasks in a child_process (which will run a new V8 process) and it does run everything in parallel.
I've a simple nodejs server that is started automatically.
It uses express to host the endpoint, which is started with a simple app.listen(port); command.
Since I've an automatic startup, I'd like to shutdown the server after an idle period - say 3 mins.
I've coded it manually just using the function below, which is called on each app.post:
//Idle timer
var timer;
function resetIdleTimer() {
if (timer != null) clearTimeout(timer);
timer = setTimeout(function () {
logger.info('idle shutdown');
process.exit();
}, 3 * 60 * 1000);
}
This seems a little crude though, so I wondered if there is an neater way (some sort of timer within express maybe).
Looking in the express docs I didn't see an easy way to configure this.
Is there a neater way to have this idle shutdown implemented?
app.listen() returns a wrapped HTTP server (as can be seen here in the source), on which you can then the .close() method.
var app = express();
var server = app.listen(port);
setTimeout(function() {
server.close();
}, 3 * 60 * 1000);
This will prevent the server from accepting new connection. When it has stopped serving existing connections, it will gracefully stop. This will then stop Nodejs entirely.
Edit: You might also find this GitHub issue relevant.
Take a look at forever . You can require it as a module into your application and it provides you with some functions that can help you achieve what you are looking for (such as forever.stop(index) which terminates the node process running at that index. Before terminating the process, you could retrieve the list of processes and manipulate the strings in order to get the uptime. Then, I would monitor the time that passes between server calls. If there is a gap of 3 minutes between requests, I would call forever.stop() in order to terminate the process.
I dont think it's "crude" to use your timer solution; I would take a slightly different tack:
app.timeOutDate = new Date().valueOf() + 1000*60*3; // 3 minutes from now, in ms
function quitIfTimedout(req, res, next){
if(new Date().valueOf() > app.timeOutDate){
logger.info('idle shutdown');
process.exit();
} else {
app.timeOutDate = new Date().valueOf() + 1000*60*3; //reset
next();
}
};
app.all('*', quitIfTimedout);
however this wont actually quit after 3 minutes, it would instead quit on the next request after 3 minutes. so that might not solve your problem