I don't want to see a log for every request the server receives when I'm testing (it makes reading the results much harder). Is there a simple way to start up Node so that it doesn't do that?
I'm referring the the lines that look like this just to be perfectly clear:
127.0.0.1 - - [Mon, 07 Jan 2013 15:59:52 GMT] "GET / HTTP/1.1" 200 1039 "-" "Mozilla/5.0 Chrome/10.0.613.0 Safari/534.15 Zombie.js/1.4.1"
NodeJS does not do this automatically.
Assuming you are using express, you need to remove the logger middleware. Remove this line:
app.use(express.logger());
Related
I have noticed numerous entries in Tomcat's local_access_log for various resources coming from IP address 127.0.0.1. These are clearly attempts to hack in. For example, here is a request to get access to the "manager" app:
127.0.0.1 - - [30/Apr/2015:13:35:13 +0000] "GET /manager/html HTTP/1.1" 401 2474
here is another one:
127.0.0.1 - - [30/Apr/2015:21:23:37 +0000] "POST /cgi-bin/php?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%22%79%65%73%22+%2D%64+%63%67%69%2E%66%69%78%5F%70%61%74%68%69%6E%66%6F%3D%31+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1" 404 1016
When decoded, the URL is this:
127.0.0.1 - - [30/Apr/2015:21:23:37 0000] "POST /cgi-bin/php?-d allow_url_include=on -d safe_mode=off -d suhosin.simulation=on -d disable_functions="" -d open_basedir=none -d auto_prepend_file=php://input -d cgi.force_redirect=0 -d cgi.redirect_status_env="yes" -d cgi.fix_pathinfo=1 -d auto_prepend_file=php://input -n HTTP/1.1" 404 1016
There are lots of such entries, all from IP address 127.0.0.1. Obviously, since this is the address of localhost, I can't block it. More over, I am not sure if there is something that I can do about it. Is there possibly an exploit that should be patched up? For instance, is there a version of Tomcat that has a related vulnerability? I am running Tomcat 8.
Much thanks for any advice!
UPDATE: thanks for the suggestion about a proxy. Turned out that httpd was indeed installed and not surprisingly, there are suspicious request. For example:
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir HTTP/1.0
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir%20c:\\ HTTP/1.0
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir%20c:\\ HTTP/1.0
This is not a windows system so cmd.exe has not place for it...
If you have a proxy server running on your computer, that will often receive requests and then call the primary server using the localhost (127.0.0.1) interface.
This could explain why you're logging these requests.
Due to application requirements, I have an externally accessible CouchDB instance. I would like to see what IP addresses are attempting to authenticate with my database. By checking the couchdb.log file, I can see failed authentication attempts. They look similar to this.
[Mon, 29 Sep 2014 13:43:32 GMT] [info] [<0.28472.7>] 127.0.0.1 - - GET
/offline_master/ 401
However, no matter where I connect from, it seems that the IP address that is logged is always 127.0.0.1. Am I mis-understanding how this works? I would really like to see the IP address that is attempting to connect.
The 127.0.0.1 is the address couchDB is bound to. It's there because you can set up couchdb to respond differently depending on what host name is being used.
The only way to get the client ip address is by turning the logging level to "debug". You can do this in the configuration page in futon.
You get records like this (client IP is on 1st line):
[Tue, 30 Sep 2014 00:14:27 GMT] [debug] [<0.451.4>] 'GET' / {1,1} from "192.168.1.52"
Headers: [{'Accept',"*/*"},
{'Host',"localhost:5984"},
{'User-Agent',"curl/7.30.0"}]
[Tue, 30 Sep 2014 00:14:27 GMT] [debug] [<0.451.4>] OAuth Params: []
[Tue, 30 Sep 2014 00:14:27 GMT] [info] [<0.451.4>] 127.0.0.1 - - GET / 200
Be careful with this. The debug logs are extremely verbose. It doesn't take long to fill up a hard drive.
It is possible to set log levels by module. The module you need to set is couch_httpd. Set the default for the rest to "error" or "fatal".
See: 3.6.2 Per module logging
Is it possible to write in the couchdb server log (the one defined by default.ini or local.ini in [log]) from a couchapp? (But from somewhere else than a view)
If that's not possible, maybe there's a workaround which would allow to log successful or unsuccessful authentication attemps in the couchdb server log? I'd like to process this server side and would like to avoid logging all httpd activity and grepping for user logging patterns, which doesn't seem to be easy or pretty...
Cheers,
Jun
A year later I find that it was in fact possible to log from views (or lists or any Javascript Design Doc functions) using the log() function: http://docs.couchdb.org/en/1.6.1/query-server/javascript.html#log
log(message)
Log a message to the CouchDB log (at the INFO level).
Arguments:
message – Message to be logged
function(doc){
log('Procesing doc ' + doc['_id']);
emit(doc['_id'], null);
}
After the map function has run, the following line can be found in CouchDB logs (e.g. at /var/log/couchdb/couch.log):
[Sat, 03 Nov 2012 17:38:02 GMT] [info] [<0.7543.0>] OS Process #Port<0.3289> Log :: Processing doc 8d300b86622d67953d102165dbe99467
Who would have guessed :)
I'm pretty sure you can't write to couch.log from a view, it's a sandboxed system.
Getting a record of connections to the server is possible though. Here's a dump from my couch.log, with an HTTP error in there:
/
[Sat, 13 Sep 2014 08:18:57 GMT] [info] [<0.160.0>] Opening index for db: test idx: _design/ivet sig: "f6b64ef8593e23cac644c13b895b7607"
[Sat, 13 Sep 2014 08:18:57 GMT] [info] [<0.121.0>] 127.0.0.1 - - GET /test/_design/ivet/_view/medicationWHP/foobar?include_docs=true 200
[Sat, 13 Sep 2014 08:18:57 GMT] [info] [<0.121.0>] 127.0.0.1 - - GET /test/_design/ivet/_view/medicationWHP/foobar?include_docs=true 500
[Sat, 13 Sep 2014 08:18:57 GMT] [error] [<0.121.0>] httpd 500 error response:
{"error":"json_encode","reason":"{bad_term,{key,null}}"}
[Sat, 13 Sep 2014 08:19:05 GMT] [info] [<0.36.0>] Apache CouchDB has started on http://127.0.0.1:5984/
You can see it has the VERB PATH CODE format for each line, so you can filter that for whatever you need. (Unauthorized is 401) You can also access the log through /_log. Details on that are here:
http://docs.couchdb.org/en/latest/api/server/common.html#log
To get all that information, you'll need to have the log level set to info. You can do this at the config screen in futon.
To do it server-side, you'd probably need to use node.js or something like that. Just have it consume the /_log endpoint, and filter each line by the HTTP response code.
I am an experienced Java developer picking up Node.js and making the shift to the asynchronous model. Most things are going fine except for logging. I cannot find anything similar to log4j and NDCs in Java while developing in Node.js with express.
My goal is to have each log statement automatically prepend the following information:
[2013-11-07 11:17:04.615 serverScript INFO 7036 192.168.7.209]
This includes the timestamp, name of the js file writing this statement (for modularized node apps), the debug level, the process ID (running clusters), and the client's IP address.
I can get it to write these when initially coming into my request handler, but without propagating a bunch of parameters to every called function, the logger statements inside the subroutines don't have the info. I know I can create an instance of my logger inside each js file that initializes its name, but I've yet to figure out a solution for the IP address of the client. For longer running requests, the address I set in my logger gets overwritten when the next request comes in, so the IPs that are logged get crossed.
I've looked at winston but have not been able to solve this issue even with it. Has anyone accomplished this? It is very useful tracking field issues when you can filter by IP to view only one user's activity.
[edit: test from parameter passing solution until I learn the syslog way]
[2013-11-07 14:29:28.641 server INFO 7527 192.168.7.209] Got request from 192.168.7.209 for /ionmed/executeQuery?
[2013-11-07 14:29:28.641 router INFO 7527 192.168.7.209] About to route a request for /ionmed/executeQuery, method=POST
[2013-11-07 14:29:28.642 router INFO 7527 192.168.7.209] getting POSTed data
[2013-11-07 14:29:28.642 router INFO 7527 192.168.7.209] POST params: {"sqlQuery":"select sleep(10)","sessionStart":"1383852558799","rand":"0.5510970998368581","jsessionid":"117DBAA89F599D923AF80D4AB171BDDF"}
[2013-11-07 14:29:28.642 requestHandlers INFO 7527 192.168.7.209] 'query' was called.
[2013-11-07 14:29:28.642 requestHandlers INFO 7527 192.168.7.209] select sleep(10)
[2013-11-07 14:29:30.673 server INFO 7527 192.168.7.217] Got request from 192.168.7.217 for /
[2013-11-07 14:29:30.673 router INFO 7527 192.168.7.217] About to route a request for /, method=GET
[2013-11-07 14:29:30.673 router INFO 7527 192.168.7.217] No request handler found for /; serving as file
[2013-11-07 14:29:30.673 router INFO 7527 192.168.7.217] Request handler 'serveFile' was called to get: /index.html
[192.168.7.217 Thu, 07 Nov 2013 19:29:30 GMT] HTTP/1.1 GET "/node/" 200 "Mozilla/5.0 (iPod; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
[2013-11-07 14:29:33.578 server INFO 7527 192.168.7.217] Got request from 192.168.7.217 for /
[2013-11-07 14:29:33.578 router INFO 7527 192.168.7.217] About to route a request for /, method=GET
[2013-11-07 14:29:33.578 router INFO 7527 192.168.7.217] No request handler found for /; serving as file
[2013-11-07 14:29:33.579 router INFO 7527 192.168.7.217] Request handler 'serveFile' was called to get: /index.html
[192.168.7.217 Thu, 07 Nov 2013 19:29:33 GMT] HTTP/1.1 GET "/node/" 200 "Mozilla/5.0 (iPod; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
[2013-11-07 14:29:38.644 requestHandlers INFO 7527 192.168.7.209] sending response
[192.168.7.209 Thu, 07 Nov 2013 19:29:38 GMT] HTTP/1.1 POST "/node/ionmed/executeQuery?" 200 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0"
[2013-11-07 14:29:41.540 server INFO 7527 192.168.7.217] Got request from 192.168.7.217 for /
[2013-11-07 14:29:41.541 router INFO 7527 192.168.7.217] About to route a request for /, method=GET
[2013-11-07 14:29:41.541 router INFO 7527 192.168.7.217] No request handler found for /; serving as file
[2013-11-07 14:29:41.541 router INFO 7527 192.168.7.217] Request handler 'serveFile' was called to get: /index.html
[192.168.7.217 Thu, 07 Nov 2013 19:29:41 GMT] HTTP/1.1 GET "/node/" 200 "Mozilla/5.0 (iPod; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
[2013-11-07 14:29:45.146 server INFO 7527 192.168.7.209] RLz6tmJ7KTH2R16VCVTX: bye {"user":"1"}
[2013-11-07 14:29:45.176 server INFO 7527 192.168.7.209] RLz6tmJ7KTH2R16VCVTX: disconnected
Now I just need to figure out how to get the express request logger to be in the same line entry format as my internal logger until it is all moved to rsyslog.
I got into this same problem some time ago, and finally I could spend sometime researching it. The #ibash approach and his post put me in the lead to solve the problem I had (thanks for your help). I only walked some steps more in order to print in the logs automatically a unique id per request.
In your case you can add origin and destination IP and all information needed to each request, using same approach and print it automatically in all logs.
My approach:
- As #ibash explained, I used continuation-local-storage to share information among all the modules per request. So I generate a unique id per request and store it in a namespace created with this library
- I wrapped the Winston library (in a very simple way) in order to recover the information from the namespace shared and override all Winston methods I use adding to the string the unique Id. Obviously in your case you should add all the info you need and you have stored previously in the namespace of the library.
As the problem was a little complex to explain to people no familiarize with all these things, I wrote it down in a post with a clear example that you can reuse if you want. Winston wrap could be really useful:
Express.js: Logging info with global unique request ID – Node.js
I hope you can reuse my code and perhaps in the future Express implements a solution for this.
These instructions were from an Ubuntu 12.04 distribution I set up, but they should apply pretty closely to RHEL, Fedora, CentOS, etc.
Rsyslog is a system logging utility you can use to log messages from any program on a Linux machine. First you need to find your rsylog configuration information. You can do that with the following command:
sudo find / -name rsyslog.conf
If you can't find the configuration file, you can list the service running to see if rsyslog is even on your machine with the following command:
service --status-all
Now open the file it finds and do the following:
Comment out the line $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
Uncomment $ModLoad imtcp
Uncomment $InputTCPServerRun and specify port number 1514, going to use 1514 b/c Ubuntu 12.04 rsyslog has a problem dropping permissions if I use port 514, other distributions don't have similar issues and you could keep default port #. I get around this by using iptables to reroute port 514 traffic to 1514
Change $FileCreateMode 0640 to 0644
Now I created a file named /etc/rsyslog.d/10.conf (this is a secondary configuration file for rsyslog where we can filter message, name log files, etc) and added the following to it:
$template DailyPerHostLogs,"/var/log/MyLogFile_%$YEAR%_%$MONTH%_%$DAY%.log"
#:msg,contains,"MsgName" -?DailyPerHostLogs
*.* -?DailyPerHostLogs
&~
This file creates a new file for each day and finds any message sent with MsgName in the text and puts it into the daily file and then removes it from the queue to be logged by any other log requests so we don't double log it.
Now you can reboot the machine you are working on and it all should be working. You can check this by looking for files in /var/log as defined in 10.conf above. Hit the logger from the command line by issuing the following commands:
logger this is from the command line
echo "this is from the tcp port" > /dev/tcp/127.0.0.1/1514
You should see both those lines pop up in the log file. If you get that, then let's move on to the node module that will be able to hit the log.
var net = require('net');
var client = net.connect({port: this.1514}, function(){ console.log("Open"); });
client.write(' ' + "sMsgName: What"+ ' ' + "hath" + ' ' + "God wrought?" + '\n');
//Do everything else your program needs. . .
The '\n' on the write tells rsyslog we are done with this line. Also, you will need to prepend a space for the filtering to work: http://www.rsyslog.com/log-normalization-and-the-leading-space/
The devil is always in the details with a setup like this, but I think this will get you most of the way there and google searching will get you the rest of the way.
Answering this as I just wrote a post on how to use continuation-local-storage to save a "transaction id" with every log (without manually propagating it). You can do the same for the client ip, process id, etc.
Follow this post: https://datahero.com/blog/2014/05/22/node-js-preserving-data-across-async-callbacks/
But instead of just saving a transaction id, you'll want these as well:
request.connection.remoteAddress and process.pid
Let me know if you have any questions here or there, and I'll answer them.
I'm currently testing LocomotiveJS and have a very simple "hello world" app up
I thought I'd set up the connect favicon middleware, but when I visit any route ('/' for example) I get the following in the console:
127.0.0.1 - - [Tue, 17 Jul 2012 21:41:25 GMT] "GET / HTTP/1.1" 200 1491 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11"
Error: Forbidden
at SendStream.error (/Users/alex/Desktop/LocoTest/node_modules/express/node_modules/send/lib/send.js:142:16)
at SendStream.pipe (/Users/alex/Desktop/LocoTest/node_modules/express/node_modules/send/lib/send.js:307:52)
at Object.static (/Users/alex/Desktop/LocoTest/node_modules/express/node_modules/connect/lib/middleware/static.js:78:8)
at next (/usr/local/lib/node_modules/locomotive/node_modules/express/node_modules/connect/lib/http.js:204:15)
at pass (/usr/local/lib/node_modules/locomotive/node_modules/express/lib/router/index.js:219:24)
at Router._dispatch (/usr/local/lib/node_modules/locomotive/node_modules/express/lib/router/index.js:280:5)
at Object.middleware [as handle] (/usr/local/lib/node_modules/locomotive/node_modules/express/lib/router/index.js:45:10)
at next (/usr/local/lib/node_modules/locomotive/node_modules/express/node_modules/connect/lib/http.js:204:15)
at multipart (/Users/alex/Desktop/LocoTest/node_modules/express/node_modules/connect/lib/middleware/multipart.js:52:61)
at module.exports (/Users/alex/Desktop/LocoTest/node_modules/express/node_modules/connect/lib/middleware/bodyParser.js:57:9)
127.0.0.1 - - [Tue, 17 Jul 2012 21:41:25 GMT] "GET /favicon.ico HTTP/1.1" 403 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11"
Any idea what's causing this?
It looks like the send middleware is considering this request malicious because it's not the root and contains "..". I'm not sure why /favicon.ico would cause isMalicious to return true though. I'd suggest debugging into isMalicious and examining this._root and this.path. Could either be some bug in send or favicon middleware or perhaps your code isn't configuring them. Can't help with that unless you post some code.
What version of Express and Connect were pulled in as dependencies? I encountered a bug with express#3.0.0beta6, which I think has been fixed in beta7 (though I haven't confirmed that myself). I'm running on express#3.0.0beta4, which I know works, and you can revert to that by doing:
$ npm uninstall express
$ npm install express#3.0.0beta4
The master branch of Locomotive is using the Express 3.x betas, which have generally been pretty solid. I'm looking forward to that being stable, and pushing out new releases.