I am outputting content of a log via netcat to an application over the network. I don't know if what I'm doing is the most efficient, especially when I notice the netcat session becomes non-responsive. I have to stop netcat and start it again for the application to work again.
The command I run is:
/bin/tail -n1 -f /var/log/custom_output.log | /bin/nc -l -p 5020 --keep-open
This needs to run like this 24/7. Is this the most efficient way of doing it? How can I improve on it so I don't have to restart the process daily?
EDIT
So I realised that when the log is being rotated, netcat is locked onto a file that's not longer being written to. I can deal with this easily enough.
The question still stands. Is this the best way to do something like this?
It's been 6 years, but maybe someone will come in handy.
To account for log rotation, use tail with the -F flag.
nc (aka netcat) variant
LOG_FILE="/var/log/custom_output.log"
PORT=5020
tail -n0 -F "$LOG_FILE" | nc -k -l -p $PORT
Notes:
Flag -k in nc is analog to --keep-open in "the OpenBSD rewrite of netcat";
Multiple clients can connect to nc at the same time, but only the first one will be receive appended log lines;
tail will run immediately, so it will collect appended log lines even if no client is connected. Thus, the first client can receive some buffered data - all log lines that have been appended since tail was run.
socat variant
LOG_FILE="/var/log/custom_output.log"
PORT=5020
socat TCP-LISTEN:$PORT,fork,reuseaddr SYSTEM:"tail -n0 -F \"$LOG_FILE\" </dev/null"
Note: here socat will fork (clone itself) on each client connection and start a separate tail process. Thus:
Each connected client will receive appended log lines at the same time;
Clients will not receive any previously buffered by tail strings.
additional
You can redirect stderr to stdout in the tail process by adding 2>&1 (in both variants). In this case, clients will receive auxiliary message lines, e.g.:
tail: /var/log/custom_output.log: file truncated;
tail: '/var/log/custom_output.log' has become inaccessible: No such file or directory - printed when the log file has been removed or renamed, only if -F is used;
tail: '/var/log/custom_output.log' has appeared; following new file - printed when a new log file is created, only if -F is used.
Related
If I open an nc in listening on Terminal1
nc -l 35555
And perform a connection in this way on Terminal2
mkfifo /tmp/f
cat /tmp/f | nc 127.0.0.1 35555 > /tmp/f
Now I have few questions
On Terminal2 you can see something in waiting. What is that?
Is it the cat process or the nc process?
In which order are they executed?
If I write a string on Terminal1, that string just come back.
Why cat /tmp/f | nc 127.0.0.1 35555 > /tmp/f doesn't create a cycle?
The received string comes out from nc and goes in /tmp/f. Because of the cat that string begins input of nc, sent back to Terminal1 and again in /tmp/f etc. etc.
Why isn't the cat of a fifo a blocking operations? Technically it not end, so why the nc is executed?
Thanks.
Is it the cat process or the nc process?
It's both. cat is waiting for something to be written to the FIFO. nc is waiting for something to be written to its stdin or to receive something over the network socket.
In which order are they executed?
All commands in a pipeline are run concurrently.
Why cat /tmp/f | nc 127.0.0.1 35555 > /tmp/f doesn't create a cycle?
The first terminal is not sending back what it received from the network, it just prints it to the terminal. When you type something on terminal 1 it gets written to the network once. nc on terminal 2 writes it to /tmp/f. Then it reads it and sends it to the network. nc on terminal 1 reads it and prints it. That's the end.
-Why isn't the cat of a fifo a blocking operations? Technically it not end, so why the nc is executed?
Commands in a pipeline don't wait for the previous one to exit. That would make it impossible to write code like
tail -f filename | grep word
since tail -f never exits (until you kill it with Ctl-c, but that also kills grep). All the processes run, which allows the later programs to process partial output of the earlier programs.
So nc doesn't wait for cat to exit. It runs at the same time, and probably uses select() or epoll() to listen for input from both stdin and the network socket at the same time (it might also use separate threads for each). So while cat is blocking on the FIFO, it can read the network data from terminal 1, and then write it to the FIFO.
I want to run a game server inside my Ubuntu machine. I want to run it in the background and write the live output of that process inside a log file. I tried using nohup and running the game server using "&" at the end but I couldn't make it work the way I wanted.
Then I started reading about named pipes and actually gave it a go. I made a simple script that in theory should work. But, of course I am missing something.
First, I made a pipe using the mkfifo command.
mkfifo testpipe
Then I created a small script:
#!/bin/bash
./mta-server64 > pipe &
pid=$!
echo $pid // so I know the pid of the process
cat < pipe > log.txt &
(Note: I wrote this code from memory.)
The code works only when there is an error and the process stops. It actually records the game console error. But when the game server is running I get no output in the log file.
I want to read the output (stdout and stderr if I am not mistaken) of a process running in background and record it those inside a log file.
I also thought about using screen as it logs everything inside a file but I would prefer not using it if there is a better solution.
EDIT:
First of all: thank you for the interest you had in helping me. In the same way, I have to apologize for only giving scarce details about what I intend to do with this small project and for my limited understanding of stdout and stderr.
Let's go to the first base.
I want to run a game server named Multi Theft Auto (https://multitheftauto.com/). This is GTA San Andreas but multiplayer.
I can easily run this game server in my Ubuntu server by calling the executable ./mta-server-64. After calling it the game server console appears:
[|] MTA: San Andreas :: 0/32 players :: 196 resources :: 125 fps (25)
MTA:BLUE Server for MTA:SA
==================================================================
= Multi Theft Auto: San Andreas v1.5.6 [64 bit]
==================================================================
= Server name : Default MTA Server
= Server IP address: auto
= Server port : 22884
=
= Log file : /root/mta/mods/deathmatch/logs/server.log
= Maximum players : 32
= HTTP port : 22564
= Voice Chat : Disabled
= Bandwidth saving : Medium
==================================================================
[09:49:07] Resource 'mapmanager' requests some acl rights. Use the command 'aclrequest list mapmanager'
[09:49:07] Resources: 196 loaded, 0 failed
[09:49:07] Starting resources...
[09:49:07] Server minclientversion is now 1.5.6-9.16588.0
[09:49:07] INFO: MAPMANAGER: Some important ACL permissions are missing. To ensure the correct functioning of Mapmanager, please write: aclrequest allow mapmanager all
[09:49:07] Gamemode 'play' started.
[09:49:07] Authorized serial account protection is enabled for the ACL group(s): `Admin` See http://mtasa.com/authserial
[09:49:07] WARNING: <owner_email_address> not set
[09:49:07] Server started and is ready to accept connections!
[09:49:07] To stop the server, type 'shutdown' or press Ctrl-C
[09:49:07] Type 'help' for a list of commands.
[09:49:07] Querying MTA master server... success! (Auto detected IP:xxx.xxx.xxx.xxx)
I am using the following script to run the process in the background and (try to) get the live output from:
#!/bin/bash
newport=$(shuf -i 22003-22900 -n 1)
newip=$(shuf -i 22003-22900 -n 1)
rm -rf ~/server/*
cp -r /home/user*/ftp/server/mtaserver/serverfiles/* ~/server
sed -i "s/<httpport>[0-9][0-9][0-9][0-9][0-9]<\/httpport>/<httpport>$newport<\/httpport>/g" ~/server/mods/deathmatch/mtaserver.conf
sed -i "s/<serverport>[0-9][0-9][0-9][0-9][0-9]<\/serverport>/<serverport>$newip<\/serverport>/g" ~/server/mods/deathmatch/mtaserver.conf
~/server/mta-server64 2>&1 | tee -a outfile &
mta_pid=$!
echo $mta_pid
sleep 6
pkill $mta_pid
(Note: Because of some technical problems I had to add the first few lines of script which automatically replace the game files with new ones and also replace the existing ports with random ones.)
This script starts the server and tries to log the output of the process. The process is automatically killed after few seconds so there is only one instance of the game server at any given time.
THE ISSUE:
This script only logs the output if there is an error. I still cannot get the live output of the process when it is still running. Maybe this is an issue with the game server but truly believe there should be a way to make it work the way I intend.
I believe you want to use tee command to split the pipe output to log file.
I suggest you read this article and these answers 1 2.
Usually this is enough nohup somecommand > somecommand.log 2>&1 & then, tail -F somecommand.log to follow the logs.
After 2 days I finally figured out a way to make it work (the way I intended to work, without taking in consideration any major security/performance risks).
Reading the comments made me realize I was attacking the wrong point. The stdout of the game server is buffered, thus making it impossible to log it into a log file using the methods I tried when I posted my question At least this is what I came to understand).
I did some research on how to run the application without having the stdout buffered: https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux
My code now:
stdbuf -o0 ~/server/mta-server64 >> pipe &
cat < pipe | tee -a outfile &
After creating the named pipe it executes the game server inside that pipe and then appends the stdout into the log file.
The stdbug -o0 command disables the stdout buffering (as noted in the link above).
This works for me and I cannot guarantee it will work for anybody else. I am still not aware if disabling the buffering is a safe approach to my issue but for now it is what I need.
I want to have tcpdump write raw packet data into a file and also display packet analysis into standard output as the packets are captured (by analysis I mean the lines it displays normally when -w is missing).
Can anybody please tell me how to do that?
Here's a neat way to do what you want:
tcpdump -w - -U | tee somefile | tcpdump -r -
What it does:
-w - tells tcpdump to write binary data to stdout
-U tells tcpdump to write each packet to stdout as it is received, rather than buffering them and outputting in chunks
tee writes that binary data to a file AND to its own stdout
-r - tells the second tcpdump to get its data from its stdin
Since tcpdump 4.9.3 4.99.0, the --print option can be used:
tcpdump -w somefile --print
Wednesday, December 30, 2020, by mcr#sandelman.ca, denis and fxl.
Summary for 4.99.0 tcpdump release
[...]
User interface:
[...]
Add --print, to cause packet printing even with -w.
tcpdump ${ARGS} &
PID=$!
tcpdump ${ARGS} -w ${filename}
kill $PID
If you want a way to do it without running tcpdump twice, consider:
sudo tcpdump port 80 -w $(tty) | tee /tmp/output.txt
From the interactive command prompt you could use $TTY instead of $(tty) but in a script the former wouldn't be set (though I'm not sure how common it is to run tcpdump in a script).
Side-note: it's not very Unix-y the way tcpdump by default makes you write to a file. Programs should by default write to stdout. Redirection to a file is already provided by the shell constructs. Maybe there's a good reason tcpdump is designed this way but I don't know what that is.
I am trying to listen continuously to port 5000 and print the first line of message but the program is stopping after one iteration. I am using the following command for it -
nc -k -l 5000 | head -n 1
I have used -k option with nc but still the program stops after one iteration. What is the problem in this command?
It is not so simple. The second nc could be started only after the full run of the previous. But it can't be done, because nc has a bug: it can't detect if the remote side closes its socket. Thus, the first nc never stops, thus the second can't be started.
The developers of the nc say, that it is an inherent problem of the TCP protocol, my opinion is that they haven't right [but it is only my opinion].
If you want to do practically a script listening on a TCP port, a working solution were to put this head -1 into a script, and calling this script from an inetd.
Some trickier hack could work as well, for example, killing the first nc after the first line, so:
nc -k -l 5000|(read;echo $REPLY;killall -9 nc)
Imho this tools aren't enough stable for productive environment, although they can be very funny. :-)
Running in bash under Ubuntu:
I have a source that generates me some output, but not straight away. Let's assume it is a first netcat listening on a socket: netcat -l 12345.
And I would like to pipe it to an outgoing netcat (connecting over TCP), e.g. netcat -l 12345 | netcat localhost 54321. But the tricky bit is, that I know there is nothing listening for that incoming connection on localhost 54321 when I run the command, but I know there will be one when the first actual character arrives through the pipe.
So my question is: is there a way either:
to delay the execution of the outgoing netcat until the first character arrives into the pipe, or
to delay the outgoing netcat from trying to establish the TCP connection until it receives the first character on its standard input? (no straight option for that in man, switching to UDP is not acceptable)
Thanks in advance!
Edit: In reality, the source is more complex than a netcat, namely it is a listening netcat piped through all sort of stream modification.
Using the research you already did and that I commented to (by not knowing it was an answer to your own question), here is the full delayed_netcat.sh:
#!/bin/bash
read line
netcat "${#}" < <(echo $line ; cat)
This first waits for a line of input and later prepends that line using a simple echo to the "newly generated" input to the actual netcat. The rest of stdin is just redirected using cat which slurps it from stdin and adds it to the input of netcat. It also supports passing commandline options and arguments to the "real" netcat.
The usage is as follows:
netcat -l 12345 | cmd1 | cmd2 | ... | ./delayed_netcat.sh localhost 54321
The netcat is delayed till the first line is read. If you really want to start it after the first character is read the parts with read and echo need some rewrite.
Port Forwarding or Port Mapping with netcat:
ncat -l -p 12345 -c 'ncat localhost 54321'
Using socat:
socat TCP4-LISTEN:12345 TCP4:localhost:54321
This command exits after the first connection is done.
I have found an answer to my question, but it is awful... so still looking for something better.
netcat -l 12345 | gawk '(NR==1){print""}{print;fflush()}' | ./delayed_netcat.sh
where ./delayed_netcat.sh:
#!/bin/sh
read line
netcat localhost 12345
So the read line delays the netcat localhost 12345 by waiting for and consuming the first input line, and I use gawk '(NR==1){print""}{print;fflush()}' to insert an empty line just before the first record... I'm sure there is room for much improvement to that!