Using logstash 2.4 (I have my reasons) on Ubuntu 16.04
root#logbox:/etc/logstash/conf.d# ls -al
total 16
drwxrwxr-x 2 root root 4096 Nov 2 19:53 .
drwxrwxr-x 3 root root 4096 Nov 2 15:46 ..
-rwxrwxrwx 1 root root 277 Nov 2 19:52 01_01_input.conf
-rwxrwxrwx 1 root root 604 Nov 2 19:48 03_02_output_pa_http.conf
root#logbox:/etc/logstash/conf.d# cat *.conf
input {
stdin {}
file {
path => "/usr/share/logstash/files/production_input.txt"
start_position => "beginning"
# codec => plain { charset => "ISO-8859-1" }
codec => json
# add_field => [ 'redis_db', '10' ]
}
}
output {
stdout {}
}
root#logbox:/etc/logstash/conf.d# ls -al /usr/share/logstash/files/production_input.txt
-rwxrwxrwx 1 root root 11910 Nov 2 16:09 /usr/share/logstash/files/production_input.txt
However...
vagrant#logbox:/etc/logstash/conf.d$ sudo tail -f /var/log/logstash/logstash*
Sending logstash logs to /var/log/logstash/logstash.log.
==> /var/log/logstash/logstash.log <==
{:timestamp=>"2018-11-02T19:59:52.947000+0000", :message=>"Pipeline main started"}
==> /var/log/logstash/logstash.stdout <==
{:timestamp=>"2018-11-02T19:59:52.947000+0000", :message=>"Pipeline main started"}
The file is not printed in stdout...I have tried both codecs (json and plain)
edit: actually it might be the case the file has been read only once; does this have to do with sincedb?
how do I force logstash to read it again? isn't start_position => beginning enough?
start_position is used the first time the file is read.
If you want to force logstash to read it again, you need to add in your file input plugin conf this parameter :
sincedb_path => "/dev/null"
Please read this for further explanation
How to force Logstash to reparse a file?
Related
I am trying to find a specific file/folder based on directory name . There are one thing i could not understand which is tabcmd -> ../bin.2012....
When i did a ls -lrt on a specific path. for e.g
cmd : ls -lrt /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251
it will list as
lrwxrwxrwx. 1 root root 33 Dec 23 07:56 atrdiag -> ../bin.20212.21.1217.2251/atrdiag
lrwxrwxrwx. 1 root root 32 Dec 23 07:56 tabcmd -> ../bin.20212.21.1217.2251/tabcmd
lrwxrwxrwx. 1 root root 39 Dec 23 07:56 serveractutil -> ../bin.20212.21.1217.2251/serveractutil
lrwxrwxrwx. 1 root root 34 Dec 23 07:56 odbcinst -> ../bin.20212.21.1217.2251/odbcinst
lrwxrwxrwx. 1 root root 31 Dec 23 07:56 iusql -> ../bin.20212.21.1217.2251/iusql
lrwxrwxrwx. 1 root root 30 Dec 23 07:56 isql -> ../bin.20212.21.1217.2251/isql
lrwxrwxrwx. 1 root root 37 Dec 23 07:56 custactutil -> ../bin.20212.21.1217.2251/custactutil
lrwxrwxrwx. 1 root root 29 Dec 23 07:56 tsm -> ../bin.20212.21.1217.2251/tsm
The objective is to find tabcmd path dynamically as this path will change from time to time. "I could not use whereis command as it is not registered as a system path/variable"
Alternatively, what i did was
TEST=`find /opt/tableau -type d -name "tabcmd"`
echo "Tabcmd path is : $TEST"
However with the cmd i tried ,, it is returning empty
it returned "Tabcmd path is : "
Update :
when i tried using command below
TABCMD_PATH=`find /opt/tableau/tableau_server -name "tabcmd"`
echo $TABCMD_PATH
/opt/tableau/tableau_server/packages/bin.20212.21.1217.2251/tabcmd /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251/tabcmd
this it will print 2 times instead of 1 and if this is the approach i should take ?
It is empty because you are finding for a directory called tabcmd, you have to find by file -type f
TEST=`find /opt/tableau -type d -name "tabcmd"`
echo "Tabcmd path is : $TEST"
And the following print two path because the file is in both paths, the file link and the symbolic link:
TABCMD_PATH=`find /opt/tableau/tableau_server -name "tabcmd"`
echo $TABCMD_PATH
/opt/tableau/tableau_server/packages/bin.20212.21.1217.2251/tabcmd /opt/tableau/tableau_server/packages/customer-bin.20212.21.1217.2251/tabcmd
You have to find the file excluding the symbolic link:
TABCMD_PATH=`find /opt/tableau/tableau_server -type f -name "tabcmd"`
echo $TABCMD_PATH
I would like to do a ssh connection to a remote host, execute a given command, and attach the standard streams (output, input, error) to the local process (sys.stdout, etc..) in an asynchronous/interactive way.
I can execute the first command and I can see the output, but if I try to run it again, there is no output because the streams are closed from the first execution.
See the example below:
import asyncio
import asyncssh
import sys
def run_command(command):
async def run():
async with asyncssh.connect(host='localhost') as ssh:
await ssh.run(
command=command,
stdout=sys.stdout,
stderr=sys.stderr,
stdin=sys.stdin
)
try:
asyncio.get_event_loop().run_until_complete(run())
except (OSError, asyncssh.Error) as exc:
sys.exit('SSH connection failed: ' + str(exc))
if __name__ == "__main__":
print(sys.stdout)
run_command("ls -la") # this one works fine
print(sys.stdout) # sys.stdout is already closed
run_command("ls -la") # this one won't print either
Output
$ python example.py
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
total 296
drwxr-xr-x 40 ********** ********** 4096 may 4 12:43 .
drwxr-xr-x 3 root root 4096 ene 8 17:33 ..
drwx------ 3 ********** ********** 4096 ene 10 15:44 .foo
drwxrwxr-x 4 ********** ********** 4096 mar 31 15:48 .bar
drwxr-xr-x 2 ********** ********** 4096 ene 10 14:44 .bash_rc
-rw------- 1 ********** ********** 65912 may 4 12:43 .bash_history
-rw-r--r-- 1 ********** ********** 220 ene 8 17:33 .bash_logout
...
What would be the best approach to attach the standard streams in order to communicate to the process ?
Thanks in advance,
I'm working with Yocto to create an embedded linux distribution for an ARM device (i.MX 6Quad Processors).
I've configured the list of desired locales with the variable:
IMAGE_LINGUAS = "de-de fr-fr en-gb en-gb.iso-8859-1 en-us en-us.iso-8859-1 zh-cn"
As result I've obtained a file systems that contains the following folders:
root#lam_icu:/usr/lib/locale# cd /usr/share/locale/
root#lam_icu:/usr/share/locale# ls -la
total 0
drwxr-xr-x 6 root root 416 Nov 17 2016 .
drwxr-xr-x 30 root root 2056 Nov 17 2016 ..
drwxr-xr-x 4 root root 296 Nov 17 2016 de
drwxr-xr-x 3 root root 232 Nov 17 2016 en_GB
drwxr-xr-x 4 root root 296 Nov 17 2016 fr
drwxr-xr-x 4 root root 296 Nov 17 2016 zh_CN
and:
root#lam_icu:/usr/share/locale# cd /usr/lib/locale/
root#lam_icu:/usr/lib/locale# ls -la
total 0
drwxr-xr-x 9 root root 640 Mar 13 2017 .
drwxr-xr-x 32 root root 40000 Mar 13 2017 ..
drwxr-xr-x 3 root root 1016 Mar 13 2017 de_DE
drwxr-xr-x 3 root root 1016 Mar 13 2017 en_GB
drwxr-xr-x 3 root root 1016 Mar 13 2017 en_GB.ISO-8859-1
drwxr-xr-x 3 root root 1016 Mar 13 2017 en_US
drwxr-xr-x 3 root root 1016 Mar 13 2017 en_US.ISO-8859-1
drwxr-xr-x 3 root root 1016 Mar 13 2017 fr_FR
drwxr-xr-x 3 root root 1016 Mar 13 2017 zh_CN
Which is the encoding of all non ISO-8859-1 locales? Can I assume that "en_GB" or "en_US" use the UTF-8 encoding?
I've tried to open the "LC_IDENTIFICATION" file, the result is:
Hc�������������cEnglish locale for the USAFree Software
Foundation,
Inc.http://www.gnu.org/software/libc/bug-glibc-locales#gnu.orgEnglishUSA1.02000-06-24en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000en_US:2000UTF-8
At the end of the file there is something that recalls "UTF-8". Is this enough to assume that the encoding is UTF-8?
How to check if a locale is UTF-8?
LC_IDENTIFICATION doesn't tell you much:
LC_IDENTIFICATION - this is not a user-visible category, it contains information about the locale itself and is rarely useful for users or developers (but is listed here for completeness sake).
You'd have to look at the complete set of files.
There appears to be no standard command-line utility for doing this, but there is a runtime call (added a little later than the original locale functions). Here is a sample program which illustrates the function nl_langinfo:
#include <stdio.h>
#include <locale.h>
#include <langinfo.h>
int
main(int argc, char **argv)
{
int n;
for (n = 1; n < argc; ++n) {
if (setlocale(LC_ALL, argv[n]) != 0) {
char *code = nl_langinfo(CODESET);
if (code != 0)
printf("%s ->%s\n", argv[n], code);
else
printf("?%s (nl_langinfo)\n", argv[n]);
} else {
printf("? %s (setlocale)\n", argv[n]);
}
}
return 0;
}
and some output, e.g., by foo $(locale -a):
aa_DJ ->ISO-8859-1
aa_DJ.iso88591 ->ISO-8859-1
aa_DJ.utf8 ->UTF-8
aa_ER ->UTF-8
aa_ER#saaho ->UTF-8
aa_ER.utf8 ->UTF-8
aa_ER.utf8#saaho ->UTF-8
aa_ET ->UTF-8
aa_ET.utf8 ->UTF-8
af_ZA ->ISO-8859-1
af_ZA.iso88591 ->ISO-8859-1
af_ZA.utf8 ->UTF-8
am_ET ->UTF-8
am_ET.utf8 ->UTF-8
an_ES ->ISO-8859-15
an_ES.iso885915 ->ISO-8859-15
an_ES.utf8 ->UTF-8
ar_AE ->ISO-8859-6
ar_AE.iso88596 ->ISO-8859-6
ar_AE.utf8 ->UTF-8
ar_BH ->ISO-8859-6
ar_BH.iso88596 ->ISO-8859-6
The directory names you're referring to are often (but not required) to be the same as encoding names. That is the assumption made in the example program. There was a related question in How to get terminal's Character Encoding, but it has no useful answers. One is interesting though, since it asserts that
locale charmap
will give the locale encoding. According to the standard, that's not necessarily so:
The command locale charmap gives the name used in localedef -f
However, localedef attaches no special meaning to the name given in the -f option.
localedef has a different option -u which identifies the codeset, but locale (in the standard) mentions no method for displaying this information.
As usual, implementations may (or may not) treat unspecified features in different ways. The GNU C library's documentation differs in some respects from the standard (see locale and localedef), but offers no explicit options for showing the codeset name.
I want to forward an SMS using gammu-smsd RunOnReceive.
That is the script I want to run (/var/spool/gammu/forward.sh) and it goes perfectly if I run it from a sudoer or using sudo -u gammu -g gammu /var/spool/gammu/forward.sh
#!/bin/bash
SMS_MESSAGES=1
for i in `seq $SMS_MESSAGES`
do
number="SMS_${i}_NUMBER"
text="SMS_${i}_TEXT"
eval "gammu-smsd-inject TEXT my_number_goes_here -text \"${!number}: ${!text}\""
done
And here is the problem I am experiencing:
Thu 2015/01/29 23:08:57 gammu-smsd[2549]: Starting run on receive: /var/spool/gammu/forward.sh IN20150130_000850_00_+37368214400_00.txt
Thu 2015/01/29 23:08:57 gammu-smsd[2154]: Process failed with exit status 2
Output of ls -l /etc/gammu-smsdrc /var/spool/gammu/ /usr/bin/gammu-smsd*:
-rw-r--r-- 1 root root 457 Jan 29 22:44 /etc/gammu-smsdrc
-rwxrwxrwx 1 root root 14336 Jun 10 2012 /usr/bin/gammu-smsd
-rwxrwxrwx 1 root root 51164 Jun 10 2012 /usr/bin/gammu-smsd-inject
-rwxrwxrwx 1 root root 9972 Jun 10 2012 /usr/bin/gammu-smsd-monitor
/var/spool/gammu/:
total 24
drwxrwxrwx 2 gammu gammu 4096 Jan 28 16:02 error
-rwxrwxrwx 1 gammu gammu 189 Jan 29 22:13 forward.sh
drwxrwxrwx 2 gammu gammu 4096 Jan 29 23:08 inbox
-rw-rw-r-- 1 gammu gammu 3702 Jan 29 23:08 log
drwxrwxrwx 2 gammu gammu 4096 Jan 29 23:07 outbox
drwxrwxrwx 2 gammu gammu 4096 Jan 29 23:07 sent
What happens if I just do ./forward.sh (not root) - so all is OK:
gammu-smsd-inject[2606]: Created outbox message OUTC20150029_231213_00_my_number_here_sms0.txt
Written message with ID /var/spool/gammu/outbox/OUTC20150029_231213_00_my_number_here_sms0.txt
Here is my /etc/gammu-smsdrc
# Configuration file for Gammu SMS Daemon
[gammu]
port = /dev/ttyUSB0
connection = at
[smsd]
service = files
logfile = /var/spool/gammu/log
debuglevel = 2
commtimeout = 1
sendtimeout = 15
statusfrequency = 0
outboxformat = unicode
transmitformat = unicode
RunOnReceive = /var/spool/gammu/forward.sh
inboxpath = /var/spool/gammu/inbox/
outboxpath = /var/spool/gammu/outbox/
sentsmspath = /var/spool/gammu/sent/
errorsmspath = /var/spool/gammu/error/
ps -fe | grep gammu:
gammu 2154 1 0 23:05 ? 00:00:02 /usr/bin/gammu-smsd --daemon --user gammu --pid /var/run/gammu-smsd.pid
cubie 2644 2403 0 23:20 pts/0 00:00:00 grep gammu
Please, help
I had the same problem and I solved it this way:
First add gammu user to sudoers, with no password:
type: $ sudo visudo
and add: gammu ALL=(ALL) NOPASSWD: ALL
Then run gammu-smsd as root user:
in /etc/init.d/gammu-smsd
change USER=gammu to USER=root
save it and don't forget to restart daemon: service gammu-smsd restart
In RunOnReceive script add sudo in front of gammu-smsd-inject:
e.g.: sudo gammu-smsd-inject TEXT my_tel_num -text "Hello world!"
I hope this will work for you too!
P.S.: I use Gammu version 1.31.90.
I am a newbie as far as node.js is concerned.I wrote the following code to pipeline two linux commands.
This is my nodejs code:
var spawn = require('child_process').spawn,
ls = spawn('ls',['-lh','/usr']),
grep = spawn('grep',['bin']);
/*
ls.stdout.on('data',function(data){
console.log('stdout: '+data);
});
*/
ls.stdout.on('data',function(data){
console.log(""+data);
grep.stdin.write(data);
});
ls.stderr.on('data',function(data){
console.log('stderr: '+data);
});
ls.on('exit',function(code){
console.log('Exit code '+code);
grep.stdin.end();
})
// ------------------------------------
grep.stdout.on('data',function(data){
console.log('stdout: '+data);
});
grep.stderr.on('data',function(data){
console.log('stderr: '+data);
});
Now this code fails sometimes and runs sometimes.I'm getting confused now.
When it fails,it says:
Exit code 0
total 160K
drwxr-xr-x 2 root root 68K Oct 12 12:54 bin
drwxr-xr-x 2 root root 4.0K Jun 20 19:58 games
drwxr-xr-x 54 root root 4.0K Sep 24 17:52 include
drwxr-xr-x 252 root root 44K Oct 2 21:53 lib
drwxr-xr-x 10 root root 4.0K Apr 28 19:16 local
drwxr-xr-x 2 root root 12K Sep 18 15:51 sbin
drwxr-xr-x 362 root root 12K Sep 28 17:58 share
drwxr-xr-x 5 root root 4.0K Jul 7 23:39 src
events.js:66
throw arguments[1]; // Unhandled 'error' event
^
Error: This socket is closed.
at Socket._write (net.js:517:19)
at Socket.write (net.js:509:15)
at Socket.<anonymous> (/home/rajat/nodexperiments/full-spawn.js:13:13)
at Socket.EventEmitter.emit (events.js:88:17)
at Pipe.onread (net.js:395:14)
And when it runs,it says:
total 160K
drwxr-xr-x 2 root root 68K Oct 12 12:54 bin
drwxr-xr-x 2 root root 4.0K Jun 20 19:58 games
drwxr-xr-x 54 root root 4.0K Sep 24 17:52 include
drwxr-xr-x 252 root root 44K Oct 2 21:53 lib
drwxr-xr-x 10 root root 4.0K Apr 28 19:16 local
drwxr-xr-x 2 root root 12K Sep 18 15:51 sbin
drwxr-xr-x 362 root root 12K Sep 28 17:58 share
drwxr-xr-x 5 root root 4.0K Jul 7 23:39 src
Exit code 0
stdout: drwxr-xr-x 2 root root 68K Oct 12 12:54 bin
drwxr-xr-x 2 root root 12K Sep 18 15:51 sbin
Any Ideas?
This is quite clearly a race condition, because node.js is a highly parallel environment you have found a really nice example to demonstrate this.
ls.on('exit',function(code){
console.log('Exit code '+code);
grep.stdin.end();
})
ls fires the above event, before it has finished writing to grep, and then closes the socket that's being used to communicate. The hint your output gives you is that the exit(0) message coming from ls appears once at the top of the output, and once just above the error message.
You shouldn't be closing grep's stdin channel here.
But what about using .exec() instead of spawn on a bash script that does something like
#/bin/bash mybashscript
ls $1 | grep bin
and then using the JavaScript closure, such as
child = exec('mybashscript',['-lh','/usr']
function (error, stdout, stderr) {
console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);
if (error !== null) {
console.log('exec error: ' + error);
}
});
With the events your code ordering is more difficult to read.