Sails with AWS XRAY - node.js

How in the world is one supposed to install AWS XRAY with Sails?
I'm attempting to translate the installation instructions to Sails' preferred ways of using Express middleware, but I'm falling flat on my face.
Most people will instantly start with "use config/http.js" to configure middleware. Well, that doesn't work in my case, because my API is consumed exclusively with Sails.io (sockets), so the http middleware config is never used.
So now, the logical step is to use policies. Well, if you've read the XRAY instructions, you know that they are trying to capture ALL requests to the app, which requires "start" and "stop" function calls, before and after routes have been configured. So, policies don't work.
So, my next step was to attempt it in the app.js, and the config/bootstrap.js files, to no avail, probably because I can't easily get the Express instance Sails is using. So, is it even possible with Sails' current config options? Anyone have any clue how to accomplish this?

To anyone that should stumble upon this, attempting to integrate AWS X-Ray into Sails.js:
I finally got it working, by building a project hook for it. If someone is ambitious enough, they are more then welcome to make it an installable hook.
IMPORTANT NOTES
The hook is designed to only run when the environment variable AWS_XRAY === 'yes'. This is a safety trap, to prevent local and CI machines from running XRAY.
The hook taps into the "before" part of the route setup. What this means is: "before routes are instantiated, use this middleware".
This code is setup to ignore the route "/_ping" (for X-Ray, it'll let the request complete as normal), which is used
for ELB health checks. These do not need to be logged on X-Ray, they
are just a waste of money. I HIGHLY recommend you read through this
code, and adjust as needed. Especially the req.headers.host and
req.connection "fixes". This was the only way I could get X-Ray to
work, without changing the repo's code (still can't find the Github
repo for it).
The req.connection.encrypted injection is just to have X-Ray report the URL as https. It's not important, unless you want your
traces to reflect the correct URL.
Because we use CloudFlare, there are additional catches to collect the end-user's IP address for requests. This should have no affect if you don't use CF, and should not require any modification. But, I have to ask, WHY aren't use using CF?
This has only gotten me so far, and I can only see basic data about
requests in the X-Ray console. I can't yet see database queries, or
other services that are in use.
RESULTS MAY VARY
Don't forget!
npm i aws-xray-sdk --save.
To install and run the X-Ray Daemon
This is the code I put together api/hooks/setup-aws-xray.js:
var AWSXRay = require('aws-xray-sdk');
module.exports = function setupAwsXray(sails){
var setupXray = false;
function injectXrayIfRequested(req, res, next){
if (
setupXray
&& !req.segment
&& req.path !== '/_ping'
) {
req.headers.host = (sails.config.environment === 'production')
? 'myapp.com'
: 'dev.myapp.com';
req.connection = {
remoteAddress: req.headers['http_cf_connecting_ip']
|| req.headers['HTTP_CF_CONNECTING_IP']
|| req.headers['X-Real-IP']
|| req.ip,
encrypted: true
};
AWSXRay.express.openSegment()(req, res, next); // not a mistake
} else {
next();
}
}
// This just allows us to get a handle on req.segment.
// This is important if you want to add annotations / metadata.
// Despite AWS's documentation, you DO NOT need to close segments
// when using manual mode and express.openSegment, it will
// do this for you automatically.
AWSXRay.enableManualMode();
return {
configure: function(){
if (process.env.AWS_XRAY && process.env.AWS_XRAY === 'yes') {
setupXray = true;
AWSXRay.setDefaultName('myapp_' + sails.config.environment);
}
},
routes: {
before: {
'/*': injectXrayIfRequested
}
}
};
};

Related

Register new route at runtime in NodeJs/ExpressJs

I want to extend this open topic: Add Routes at Runtime (ExpressJs) which sadly didn't help me enough.
I'm working on an application that allows the creation of different API's that runs on NodeJs. The UI looks like this:
As you can see, this piece of code contains two endpoints (GET, POST) and as soon as I press "Save", it creates a .js file located in a path where the Nodejs application is looking for its endpoints (e.g: myProject\dynamicRoutes\rule_test.js).
The problem that I have is that being that the Nodejs server is running while I'm developing the code, I'm not able to invoke these new endpoints unless I restart the server once again (and ExpressJs detects the file).
Is there a way to register new routes while the
NodeJs (ExpressJs) is running?
I tried to do the following things with no luck:
app.js
This works if the server is restarted. I tried to include this library (express-dynamic-router, but not working at runtime.)
//this is dynamic routing function
function handleDynamicRoutes(req,res,next) {
var path = req.path; //http://localhost:8080/api/rule_test
//LoadModules(path)
var controllerPath = path.replace("/api/", "./dynamicRoutes/");
var dynamicController = require(controllerPath);
dynamicRouter.index(dynamicController[req.method]).register(app);
dynamicController[req.method] = function(req, res) {
//invocation
}
next();
}
app.all('*', handleDynamicRoutes);
Finally, I readed this article (#NodeJS / #ExpressJS: Adding routes dynamically at runtime), but I couldn't figure out how this can help me.
I believe that this could be possible somehow, but I feel a bit lost. Anyone knows how can I achieve this? I'm getting a CANNOT GET error, after each file creation.
Disclaimer: please know that it is considered as bad design in terms of stability and security to allow the user or even administrator to inject executable code via web forms. Treat this thread as academic discussion and don't use this code in production!
Look at this simple example which adds new route in runtime:
app.get('/subpage', (req, res) => res.send('Hello subpage'))
So basically new route is being registered when app.get is called, no need to walk through routes directory.
All you need to do is simply load your newly created module and pass your app to module.exports function to register new routes. I guess this one-liner should work just fine (not tested):
require('path/to/new/module')(app)
Is req.params enough for you?
app.get('/basebath/:path, (req,res) => {
const content = require('content/' + req.params.path);
res.send(content);
});
So the user can enter whatever after /basepath, for example
http://www.mywebsite.com/basepath/bergur
The router would then try to get the file content/bergur.js
and send it's contents.

ArangoDB with custom JS

Hello is there any way to use JS environment built in ArangoDB to execute custom JS? I'd like to set up path to my JS files which would be executed instead of foxx application files.
Via GitHub: https://github.com/arangodb/arangodb/issues/1723#issuecomment-183289699
You are correct that modules are cached independently of the routing
cache. Clearing the module cache (or preventing a module from being
cached) is currently not supported.
The actions mechanism is really only intended as an internal API and
only supported for backwards compatibility with early ArangoDB
versions and some edge cases.
As you may have noticed while digging through the ArangoDB source
code, Foxx provides a per-service module cache which is cleared
whenever a Foxx service is reloaded. I would strongly encourage you to
see whether Foxx fits your use case before continuing to dig into the
actions mechanism.
It's actually possible to create a Foxx service with just two files (a
manifest and a controller file) and without using repositories or
models (you can just use the same APIs available in actions).
You just need a controller file like this (e.g. ctrl.js):
'use strict';
const Foxx = require('org/arangodb/foxx');
const ctrl = new Foxx.Controller(applicationContext);
ctrl.get('/', function (req, res) {
res.send('Hello World');
});
with a manifest.json like this:
{
"name": "my-foxx",
"version": "0.0.0",
"controllers": "ctrl.js",
"defaultDocument": "",
"engines": {"arangodb": "^2.8.0"}
}
You can then mount the service (upload a zip bundle) at a path like
/db and access it:
curl http://localhost:8529/_db/_system/db
The upcoming 3.0 release will remove a lot of the existing conceptual
overhead of Foxx which will hopefully make it even easier to get
started with it.
Yes, this can be done with User Actions. Foxx was created as a more comfortable alternative and is likely a better choice for non-trivial applications. The documentation can be intimidating but Foxx services can actually be very lightweight and simple (see my other answer). If you really don't want to use Foxx for this, here's how to do it manually:
First create a virtual module in the _modules system collection:
var db = require('org/arangodb').db;
db._modules.save({
path: '/db:/ownTest',
content: `
exports.do = function (req, res, options, next) {
res.body = 'test';
res.responseCode = 200;
res.contentType = 'text/plain';
};
`
});
Then create a route that uses it:
db._routing.save({
url: '/ourtest',
action: {
controller: 'db://ownTest'
}
});
Finally, tell ArangoDB to update its routing cache so it notices the new route:
require('internal').reloadRouting();
If you install your JavaScript module to the js/common/ or the js/server/ directory you can use the module name (e.g. myOnDiskModule) instead of the virtual module name "db://owntest" in the controller.
For smaller modules you can just define the function inline using callback instead of controller:
db._routing.save({
url: '/hello/echo',
action: {
callback: `
function (req, res) {
res.statusCode = 200;
res.body = require('js-yaml').safeDump({
Hello: 'World',
are: 'you here?'
});
}
`
}
});
Remember to always update the routing cache after changes to the routing collection:
require('internal').reloadRouting();
Note: the callback implementation in 2.8 has a bug that will be fixed in 2.8.3. If you want to apply the fix manually, it's in commit b714dc5.

Access Previously-Defined Middleware

Is there a way to access or delete middleware in connect or express that you already defined on the same instance? I have noticed that under koa you can do this, but we are not going to use koa yet because it is so new, so I am trying to do the same thing in express. I also noticed that it is possible with connect, with somewhat more complicated output, but connect does not have all the features I want, even with middleware.
var express = require('express');
var connect = require('connect');
var koa = require('koa');
var server1 = express();
var server2 = connect();
var server3 = koa();
server1.use(function express(req, res, next) {
console.log('Hello from express!');
});
server2.use(function connect(req, res, next) {
console.log('Hello from connect!');
});
server3.use(function* koa(next) {
console.log('Hello from koa!');
});
console.log(server1.middleware);
// logs 'undefined'
console.log(server2.middleware);
// logs 'undefined'
console.log(server2.stack);
logs [ { route: '', handle: [Function: connect] } ]
console.log(server3.middleware);
// logs [ [Function: koa] ]
koa's docs say that it added some sugar to its middleware, but never explicitly mentions any sugar, and in particular does not mention this behavior.
So is this possible in express? If it is not possible with the vanilla version, how hard would it be to implement? I would like to avoid modifying the library itself. Also, what are the drawbacks for doing this, in any of the 3 libraries?
EDIT:
My use case is that I am essentially re-engineering gulp-webserver, with some improvements, as that plugin, and all others like it, are blacklisted. gulp is a task runner, that has the concept of "file objects", and it is possible to access their contents and path, so I basically want to serve each file statically when the user goes to a corresponding URL in the browser. The trouble is watching, as I need to ensure that the user gets the new file, and not the old version. If I just add an app.use each time, the server would see the file as it is originally, and never get to the middleware with the new version.
I don't want to restart the server every time a file changes, though I will if I can find no better way, so it seems I need to either modify the original middleware on the fly (not a good idea), delete it, or add it to the beginning instead of the end. Either way, I first need to know where it "lives".
You might be able to find what your looking for in server1._router.stack, but it's not clear what exactly you're trying to do (what do you mean "access"?).
In any case, it's not a good idea to do any of these, since that relies strictly on implementation, and not on specification / API. As a result any and all assumptions made regarding the inner implementation of a library is eventually bound to break. You will eventually have to either rewrite your code (and "reverse engineer" the library again to do so), or lock yourself to a specific library version which will result in stale code, with potential bugs and vulnerabilities and no new features / improvements.

How to measure Sails.js requests/respond time

I would like to see in the console the time it takes for an HTTP request to be responded. Kind of like express.js does.
GET api/myurl/ 210ms 200
I run sails debug but this doesn't show much.
I have node-inspector running but it seems this lets me inspect the JavaScript objects at runtime but not this particular thing.
Is there a configuration in Sails I can enable or a NPM module I can install to find out this time between request and response?
If you want to measure total response time (including view generation and pushing data to socket) you can use req._startTime defined in startRequestTimer middleware or use response-time middleware which gives much more accurate results.
This middleware adds a X-Response-Time header to all HTTP responses so you can check it both on client and server side.
module.exports.http = {
middleware: {
order: [
'responseTimeLogger',
// ...
],
// Using built-in variable defined by Sails
responseTimeLogger: function (req, res, next) {
req.on("end", function() {
sails.log.info('response time: ' + new Date() - req._startTime + 'ms');
});
next();
},
// Using response-time middleware
responseTimeLogger: function (req, res, next) {
req.on("end", function() {
sails.log.info('response time: ' + res.getHeader('X-Response-Time'));
});
require('response-time')()(req, res, next);
}
}
}
#abeja's answer is great, but a little out of date. In case of anyone still seek the answer, I put the latest version below:
// Using response-time middleware
responseTimeLogger: function(req, res, next) {
res.on("finish", function() {
sails.log.info("Requested :: ", req.method, req.url, res.get('X-Response-Time'));
});
require('response-time')()(req, res, next);
}
A little explanation:
Listen to finish event of res object, because when req emit end event, res header is not set yet.
Recent Node.js change the API getHeader to get.
btw, my reputation is 0, so I can't post comment to the #abeja's answer :)
In addition to #abeja answer (very nice answer though), if you are looking for production application monitoring, you might be interested in New Relic.
The tool helps you to measure (among other things) response time, throughput, error rates etc.
It also measures database performance (in my case it is MongoDB) and help you easily investigate bottlenecks.
On top of that it allows you to measure application performance from a browser (including connection time, DOM rendering time etc.)
New Relic setup is very easy - a few steps to have your stats running (more here: Installing and maintaining Node.js). For development purposes you can also have free of charge account with 24hr data retention.
I hope that will also help.
For a more generic statistics solution (which could be used in your Sails.js project as well) there's a great series of tutorials on digitalocean on using StatsD with Graphite to measure anything (and everything) on your server: https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd
There's also a nice StatsD node.js client called Lynx: https://github.com/dscape/lynx along with an express middleware for Lynx to measure counts and response times for express routes: https://github.com/rosskukulinski/lynx-express (Sails is fully compatible with Express/Connect middleware)
I'm planning to use this soon on my sails project before going to production, will update this with more details when I do, let me know if you go ahead before I do..
I've created a Sails Hook that eases the whole process. Just install it using npm install sails-hook-responsetime --save and it will add X-Response-Time to both HTTP and Socket requests.
https://www.npmjs.com/package/sails-hook-responsetime

Express request is called twice

To learn node.js I'm creating a small app that get some rss feeds stored in mongoDB, process them and create a single feed (ordered by date) from these ones.
It parses a list of ~50 rss feeds, with ~1000 blog items, so it's quite long to parse the whole, so I put the following req.connection.setTimeout(60*1000); to get a long enough time out to fetch and parse all the feeds.
Everything runs quite fine, but the request is called twice. (I checked with wireshark, I don't think it's about favicon here).
I really don't get it.
You can test yourself here : http://mighty-springs-9162.herokuapp.com/feed/mde/20 (it should create a rss feed with the last 20 articles about "mde").
The code is here: https://github.com/xseignard/rss-unify
And if we focus on the interesting bits :
I have a route defined like this : app.get('/feed/:name/:size?', topics.getFeed);
And the topics.getFeed is like this :
function getFeed(req, res) {
// 1 minute timeout to get enough time for the request to be processed
req.connection.setTimeout(60*1000);
var name = req.params.name;
var callback = function(err, topic) {
// if the topic has been found
if (topic) {
// aggregate the corresponding feeds
rssAggregator.aggregate(topic, function(err, rssFeed) {
if (err) {
res.status(500).send({error: 'Error while creating feed'});
}
else {
res.send(rssFeed);
}
},
req);
}
else {
res.status(404).send({error: 'Topic not found'});
}};
// look for the topic in the db
findTopicByName(name, callback);
}
So nothing fancy, but still, this getFeed function is called twice.
What's wrong there? Any idea?
This annoyed me for a long time. It's most likely the Firebug extension which is sending a duplicate of each GET request in the background. Try turning off Firebug to make sure that's not the issue.
I faced the same issue while using Google Cloud Functions Framework (which uses express to handle requests) on my local machine. Each fetch request (in browser console and within web page) made resulted in two requests to the server. The issue was related to CORS (because I was using different ports), Chrome made a OPTIONS method call before the actual call. Since OPTIONS method was not necessary in my code, I used an if-statement to return an empty response.
if(req.method == "OPTIONS"){
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Headers', 'Content-Type');
res.status(204).send('');
}
Spent nearly 3hrs banging my head. Thanks to user105279's answer for hinting this.
If you have favicon on your site, remove it and try again. If your problem resolved, refactor your favicon url
I'm doing more or less the same thing now, and noticed the same thing.
I'm testing my server by entering the api address in chrome like this:
http://127.0.0.1:1337/links/1
my Node.js server is then responding with a json object depending on the id.
I set up a console log in the get method and noticed that when I change the id in the address bar of chrome it sends a request (before hitting enter to actually send the request) and the server accepts another request after I actually hit enter. This happens with and without having the chrome dev console open.
IE 11 doesn't seem to work in the same way but I don't have Firefox installed right now.
Hope that helps someone even if this was a kind of old thread :)
/J
I am to fix with listen.setTimeout and axios.defaults.timeout = 36000000
Node js
var timeout = require('connect-timeout'); //express v4
//in cors putting options response code for 200 and pre flight to false
app.use(cors({ preflightContinue: false, optionsSuccessStatus: 200 }));
//to put this middleaware in final of middleawares
app.use(timeout(36000000)); //10min
app.use((req, res, next) => {
if (!req.timedout) next();
});
var listen = app.listen(3333, () => console.log('running'));
listen.setTimeout(36000000); //10min
React
import axios from 'axios';
axios.defaults.timeout = 36000000;//10min
After of 2 days trying
you might have to increase the timeout even more. I haven't seen the express source but it just sounds on timeout, it retries.
Ensure you give res.send(); The axios call expects a value from the server and hence sends back a call request after 120 seconds.
I had the same issue doing this with Express 4. I believe it has to do with how it resolves request params. The solution is to ensure your params are resolved by for example checking them in an if block:
app.get('/:conversation', (req, res) => {
let url = req.params.conversation;
//Only handle request when params have resolved
if (url) {
res.redirect(301, 'http://'+ url + '.com')
}
})
In my case, my Axios POST requests were received twice by Express, the first one without body, the second one with the correct payload. The same request sent from Postman only received once correctly. It turned out that Express was run on a different port so my requests were cross origin. This caused Chrome to sent a preflight OPTION method request to the same url (the POST url) and my app.all routing in Express processed that one too.
app.all('/api/:cmd', require('./api.js'));
Separating POST from OPTIONS solved the issue:
app.post('/api/:cmd', require('./api.js'));
app.options('/', (req, res) => res.send());
I met the same problem. Then I tried to add return, it didn't work. But it works when I add return res.redirect('/path');
I had the same problem. Then I opened the Chrome dev tools and found out that the favicon.ico was requested from my Express.js application. I needed to fix the way how I registered the middleware.
Screenshot of Chrome dev tools
I also had double requests. In my case it was the forwarding from http to https protocol. You can check if that's the case by looking comparing
req.headers['x-forwarded-proto']
It will either be 'http' or 'https'.
I could fix my issue simply by adjusting the order in which my middlewares trigger.

Resources