filebeat json logs to elasticsearch max size - node.js

I am sending the logs produced by my nodejs app using winston
This is my configuration filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/log/application/*.log
json.keys_under_root: true
output.elasticsearch:
hosts: ["elasticsearch.staging:9200"]
setup.kibana:
host: "kibana.staging:5601"
Logs are written in for example /var/log/application/app-bank.log
I have something like that
{"message":"Application started","level":"info","application":"app-bank","environment":"staging"}
{"message":"Header x-website-name not found","stack":"Error: Header x-website-name not found\n at context.httpApp.use (/usr/src/app/controller/index.js:30:39)\n at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)\n at /usr/src/app/node_modules/express/lib/router/index.js:284:7\n at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)\n at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)\n at context.httpApp.use (/usr/src/app/controller/index.js:25:9)\n at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)\n at /usr/src/app/node_modules/express/lib/router/index.js:284:7\n at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)\n at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)\n at session (/usr/src/app/node_modules/express-session/index.js:468:7)\n at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)\n at /usr/src/app/node_modules/express/lib/router/index.js:284:7","level":"error","application":"app-bank","environment":"staging"}
The first log is received my elasticsearch, but not the second one, I can not see any log, the problem should be because of the stack key I do not know if it because of the format of the stack or the log is passing a certain size.

I fixed it adding
json.add_error_key: true

Related

ERROR AppError: Can't find /bundle.js.map on this server

I'm using parcel bundle in my node app but the Axios did not work and it shows me these errors:
also,bundle.js.map can not be found.
ERROR ?? AppError: Can't find /favicon.ico on this server!
at G:\NODE-APP\my-node.js\exp-2\exp\app.js:94:8
at Layer.handle [as handle_request] (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\layer.js:95:5)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:137:13)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:131:14)
at Route.dispatch (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\route.js:112:3)
at Layer.handle [as handle_request] (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\layer.js:95:5)
at G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\index.js:281:22
at param (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\index.js:354:14)
at param (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\index.js:365:14)
at Function.process_params (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\index.js:410:3)
at next (G:\NODE-APP\my-node.js\exp-2\exp\node_modules\express\lib\router\index.js:275:10)
image of errors
Your requesting favicon.ico image(the small image on top of your app). and it seems that you don't have it/you're requesting it in the wrong place.

Error while compiling ejs, the error is on local system

Can anyone help me with below error.
I am getting this error on local but on server its working fine.
Have tried changing include ../PATH/files.ejs to include('../PATH/files.ejs') this removes error but not able to print content on that file.
SyntaxError: Unexpected token . in /PATH/index.ejs while compiling ejs
If the above error is not helpful, you may want to try EJS-Lint:
https://github.com/RyanZim/EJS-Lint
Or, if you meant to create an async function, pass `async: true` as an option.
at new Function (<anonymous>)
at Template.compile (/PATH/node_modules/ejs/lib/ejs.js:626:12)
at Object.compile (/PATH/node_modules/ejs/lib/ejs.js:366:16)
at handleCache (/PATH/node_modules/ejs/lib/ejs.js:215:18)
at tryHandleCache (/PATH/node_modules/ejs/lib/ejs.js:254:16)
at View.exports.renderFile [as engine] (/PATH/node_modules/ejs/lib/ejs.js:459:10)
at View.render (/PATH/node_modules/express/lib/view.js:135:8)
at tryRender (/PATH/node_modules/express/lib/application.js:640:10)
at Function.render (/PATH/node_modules/express/lib/application.js:592:3)
at ServerResponse.render (/PATH/node_modules/express/lib/response.js:1012:7)
at ServerResponse.res.render (/PATH/node_modules/express-ejs-layouts/lib/express-layouts.js:77:18)
at res_data (/PATH/front_system.js:248:17)
at /PATH/front_system.js:218:13
at Layer.handle [as handle_request] (/PATH/node_modules/express/lib/router/layer.js:95:5)
at next (/PATH/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/PATH/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/PATH/node_modules/express/lib/router/layer.js:95:5)
at /PATH/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/PATH/node_modules/express/lib/router/index.js:335:12)
at next (/PATH/node_modules/express/lib/router/index.js:275:10)
at jsonParser (/PATH/node_modules/body-parser/lib/types/json.js:110:7)
at Layer.handle [as handle_request] (/PATH/node_modules/express/lib/router/layer.js:95:5)
Resolved by changing:
From
<% include ../PATH/files.ejs %>
To
<%- include('../PATH/files.ejs') %>

Parameter error with a script tag and alert

I am getting the following error showing up in my logger.
URIError: Failed to decode param '/%db<script>alert(539043);</script>/'
at decodeURIComponent (<anonymous>)
at decode_param (/usr/src/app/node_modules/express/lib/router/layer.js:172:12)
at Layer.match (/usr/src/app/node_modules/express/lib/router/layer.js:123:27)
at matchLayer (/usr/src/app/node_modules/express/lib/router/index.js:574:18)
at next (/usr/src/app/node_modules/express/lib/router/index.js:220:15)
at app.use (/usr/src/app/app.js:83:9)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
I think this is someone trying to probe my site for some vulnerability. This occurs 2 times every week like clockwork.
Around the same time I get the following error too:
URIError: Failed to decode param '/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd'
at decodeURIComponent (<anonymous>)
at decode_param (/usr/src/app/node_modules/express/lib/router/layer.js:172:12)
at Layer.match (/usr/src/app/node_modules/express/lib/router/layer.js:123:27)
at matchLayer (/usr/src/app/node_modules/express/lib/router/index.js:574:18)
at next (/usr/src/app/node_modules/express/lib/router/index.js:220:15)
at app.use (/usr/src/app/app.js:83:9)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
Is there some sort of way to prevent this? Should I be concerned about this? Is there a standard way of dealing with parameters that are not legitimate.

How to output the process.stderr.write to winston

Using winston loggers to write contents into the files but that works only when customlogger.error is used. If the node is outputting some reference error like below
ReferenceError: aksbd is not defined
at /home/nigilan/Desktop/homepagelogger/app.js:53:20
at Layer.handle [as handle_request] (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/layer.js:95:5)
at next (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/layer.js:95:5)
at /home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/index.js:335:12)
at next (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/index.js:275:10)
at /home/nigilan/Desktop/homepagelogger/app.js:38:5
at Layer.handle [as handle_request] (/home/nigilan/Desktop/homepagelogger/node_modules/express/lib/router/layer.js:95:5)
How to use winston to store the errors like above ?
P.S. Logging the unhandled exception is nodejs server is working fine.
`
function formatArgs(args){
return [util.format.apply(util.format, Array.prototype.slice.call(args))];
}
console.info = function(){
customlogger.info.apply(customlogger, formatArgs(arguments));
};
console.warn = function(){
customlogger.warn.apply(customlogger, formatArgs(arguments));
};`
Here you can also use the default logger of winston instead of your custom logger. The gist code below has the perfect answer for this.

LESS files not getting compiled

Below is a snippet from my node.js file
I am trying to compile code from LESS into CSS.
Am I missing anything
app.use(lessMiddleware({
src: path.join(__dirname, '/public/stylesheets/styles', 'less'),
dest: path.join(__dirname, '/public/stylesheets/css'),
prefix : '/stylesheets',
}));
When I refresh my index.html, it throws me below error
TypeError: Arguments to path.join must be strings
at f (path.js:204:15)
at Object.filter (native)
at Object.exports.join (path.js:209:40)
at C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\less-middleware\lib\mi
ddleware.js:161:27
at Layer.handle [as handle_request] (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\n
ode_modules\express\lib\router\layer.js:95:5)
at trim_prefix (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\l
ib\router\index.js:312:13)
at C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\lib\router\ind
ex.js:280:7
at Function.process_params (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modul
es\express\lib\router\index.js:330:12)
at next (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\lib\rout
er\index.js:271:10)
at expressInit (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\l
ib\middleware\init.js:33:5)
TypeError: Arguments to path.join must be strings
at f (path.js:204:15)
at Object.filter (native)
at Object.exports.join (path.js:209:40)
at C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\less-middleware\lib\mi
ddleware.js:161:27
at Layer.handle [as handle_request] (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\n
ode_modules\express\lib\router\layer.js:95:5)
at trim_prefix (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\l
ib\router\index.js:312:13)
at C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\lib\router\ind
ex.js:280:7
at Function.process_params (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modul
es\express\lib\router\index.js:330:12)
at next (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\lib\rout
er\index.js:271:10)
at expressInit (C:\SrkOwnRepo\Hotaaal\HotelUI10892515\node_modules\express\l
ib\middleware\init.js:33:5)
Please provide a workaround or any other middleware.
The less-middleware function has a different interface than the one you showed in the code snippet.
It looks like this lessMiddleware(source, options), where source is a string and options is an object. And it breaks somewhere around here when it tries to to call path.join on source, in your case an object.
You can check out the documentation of the middleware for the accepted parameters in options.

Resources