I need remove duplicated lines based on field between parenthesis (Ex: (265394673718132736)) but removing the shorter line.
Example:
SERVER: 1 - (265394673718132736) - NO - ['OK', 'GROUP1']
SERVER: 2 - (284906813495967745) - NO - ['OK', 'GROUP1']
SERVER: 3 - (184387362225258496) - NO - ['OK', 'GROUP2']
SERVER: 4 - (118642771161645056) - NO - ['OK', 'GROUP1', 'SAR']
SERVER: 4 - (118642771161645056) - NO - ['OK', 'GROUP1']
SERVER: 5 - (234329090943877122) - NO - ['OK', 'GROUP4', 'SAR']
SERVER: 5 - (234329090943877122) - NO - ['OK', 'GROUP4', 'SAR', 'NO']
SERVER: 6 - (287039745190658069) - NO - ['OK', 'GROUP6']
SERVER: 7 - (280378736145072130) - NO - ['OK', 'GROUP3']
Desidered result:
SERVER: 1 - (265394673718132736) - NO - ['OK', 'GROUP1']
SERVER: 2 - (284906813495967745) - NO - ['OK', 'GROUP1']
SERVER: 3 - (184387362225258496) - NO - ['OK', 'GROUP2']
SERVER: 4 - (118642771161645056) - NO - ['OK', 'GROUP1', 'SAR']
SERVER: 5 - (234329090943877122) - NO - ['OK', 'GROUP4', 'SAR', 'NO']
SERVER: 6 - (287039745190658069) - NO - ['OK', 'GROUP6']
SERVER: 7 - (280378736145072130) - NO - ['OK', 'GROUP3']
EDIT:
Tried with:
cat test | cut -f1 -d ":" --complement | sort -u -t'-' -k2,2
But I need remove the shorter line, not random.
awk '{a[$4]=length(a[$4])<length?$0:a[$4]}END{for(x in a)print a[x]}' file
does the job.
Note that, the order of lines in output is not preserved.
Related
Using
sed -ne '/13:40:47/,/13:41:48/p' ABCD.log | wc -l
on terminal right now to get the result.
Sample log format:
10.0.0.0 - - [29/Apr/2020:00:00:54 -0400]
10.0.0.0 - - [29/Apr/2020:00:00:59 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:03 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:05 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:09 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:11 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:18 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:24 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:33 -0400]
I am looking to get output as below if searched between 10.0.0.0 - - [29/Apr/2020:00:01:05 -0400] and 10.0.0.0 - - [29/Apr/2020:00:01:24 -0400].
10.0.0.0 - - [29/Apr/2020:00:01:09 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:11 -0400]
10.0.0.0 - - [29/Apr/2020:00:01:18 -0400]
Just read the file's line and return the lines b/w the two matching lines (start & end).See Files I/O and Loops.
This logic is not working :
start_time = input("Enter start time: ")
end_time = input("Enter end time: ")
def between(l1,start_time,end_time):
l2 = []
for i in l1:
if(i > start_time and i < end_time):
l2.append(i)
return l2
I'm having an issue with postfix on a raspberry (debian jessie).
I have looked on the web on how to force postfix to send emails on port 587 but nothing work for me.
When I send email via the mail command , the mail.log indicates that postfix is trying to connect to the hotmail mailserver on port 25.
I have the same configuration on a VPS and everything work perfectly. I already checked the firewall configuration of my ISP and my NAT setting are well configured to redirect 25, 587 ... on my raspberry.
Can you help me to force postfix using port 587 ?
Than you.
My config :
## Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - - smtpd
#smtp inet n - - - 1 postscreen
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
submission inet n - - - - smtpd
# -o syslog_name=postfix/submission
# -o smtpd_tls_security_level=encrypt
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - - - - smtpd
# -o syslog_name=postfix/smtps
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#628 inet n - - - - qmqpd
pickup unix n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr unix n - n 300 1 qmgr
#qmgr unix n - n 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
relay unix - - - - - smtp
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
# lmtp cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
# mailbox_transport = lmtp:inet:localhost
# virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus unix - n n - - pipe
# user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix - n n - - pipe
# flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}
dovecot unix - n n - - pipe
flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/dovecot-lda -f ${sender} -d ${recipient}
Main.conf
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mydomain.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = localhost.com, , localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination
virtual_transport = dovecot
dovecot_destination_recipient_limit = 1
A short extract of mail.log
Feb 21 14:25:22 raspberrypi postfix/qmgr[32705]: 61DE36021A: from=<hugo#raspberrypi>, size=315, nrcpt=1 (queue active)
Feb 21 14:25:41 raspberrypi postfix/smtp[32708]: connect to mx2.hotmail.com[65.55.92.184]:25: Connection timed out
Feb 21 14:25:41 raspberrypi postfix/smtp[32709]: connect to mx2.hotmail.com[65.54.188.72]:25: Connection timed out
Feb 21 14:25:52 raspberrypi postfix/smtp[32734]: connect to mx3.hotmail.com[65.55.33.119]:25: Connection timed out
From what I understand of this problem, to force Postfix to use submission to send e-mail you should define this in main.cf. I don't see anything related in your example, that's why Postfix still send on port 25 (mail.log).
master.cf defines daemons/listeners run by Postfix, so you have enabled submission to reach your mail server, but have not configured it to send via submission.
In additional for my question
Here is example of log:
10.10.10.10 - - [21/Mar/2016:00:00:00 +0000] "GET /example?page=&per_page=100&scopes= HTTP/1.1" 200 769 "-" "" "1.1.1.1"
10.10.10.10 - - [21/Mar/2016:00:00:00 +0000] "GET /example?page=&per_page=500&scopes= HTTP/1.1" 200 769 "-" "" "1.1.1.1"
10.10.10.10 - - [21/Mar/2016:00:00:00 +0000] "GET /example?page=&per_page=100&scopes= HTTP/1.1" 200 769 "-" "" "1.1.1.1"
11.11.11.11 - - [21/Mar/2016:00:00:00 +0000] "GET /example?page=&per_page=10&scopes= HTTP/1.1" 200 769 "-" "" "1.1.1.1"
12.12.12.12 - - [21/Mar/2016:00:00:00 +0000] "GET /example?page=&per_page=500&scopes= HTTP/1.1" 200 769 "-" "" "1.1.1.1"
13.13.13.13 - - [21/Mar/2016:00:00:00 +0000] "GET /example HTTP/1.1" 200 769 "-" "" "1.1.1.1"
With following command
awk --re-interval '/per_page=[0-9]{3}/{cnt[$1]++} END{for (ip in cnt) print ip, cnt[ip]}' file
I can get counted and groupped result of each IPs witch cosist per_page >= 100 in parameters:
12.12.12.12 1
10.10.10.10 3
How I can modify it for output with per_page parameter value? For example (any format):
12.12.12.12 - per_page-500 - 1
10.10.10.10 - per_page-100 - 2
10.10.10.10 - per_page-500 - 1
awk to the rescue!
$ awk --re-interval -v OFS=' - ' '
match($0,/per_page=[0-9]{3}/){cnt[$1 OFS substr($0, RSTART,RLENGTH)]++}
END{for (ip in cnt) print ip, cnt[ip]}' file
12.12.12.12 - per_page=500 - 1
10.10.10.10 - per_page=500 - 1
10.10.10.10 - per_page=100 - 2
I have a file with values separated by tabs. When a value is not present I put a '-' in the corresponding field.
Each line begins with an identifier. I'm simply searching for lines corresponding to given identifier and, using grep on a machine A (Linux) from two different machine (B and C) and two different results appear. In particular from one of the machines some consecutives '-' are missing.
The two machines are one with linux ubuntu (B) and the other with MAC OSX (C).
Here is an example:
INPUT FILE:
comp10034_c0_seq1 281 - UniRef90_B7GCX2 276 3e-29 640 98.220640569395 13.90625 Predicted_protein Phaeodactylum_tricornutum - - GO:0006200 ATP_catabolic_process GO:0005524 ATP
binding GO:0016020 membrane pfam00005 138-230 1.00e-09 - - - 93 - 0 0.136126 0
comp10036_c0_seq1 315 - - - - - - - - - - - - - - -- - - - - - - - - 77 + 2 0.00277103 0
comp10037_c0_seq1 350 - - - - - - - - - - - - - - -- - - - - - - - - 77 + 2 0.738719 0
comp6261_c0_seq1 1227 - UniRef90_K0R0D8 519 1e-82 186 42.2982885085575 98.9247311827957 Uncharacterized_protein Thalassiosira_ oceanica - - - - - - - - - - - - - -- 350 + 1 0.0034993 0
GREP FROM MACHINE B
grep 'comp6261_c0_seq1' file.txt
RESULT:
comp6261_c0_seq1 1227 - UniRef90_K0R0D8 519 1e-82 186 42.2982885085575 98.9247311827957 Uncharacterized_protein Thalassiosira_oceanica - - - - - - - - - - - - - -- 350 + 1 0.0034993 0
GREP FROM MACHINE C
grep 'comp6261_c0_seq1' file.txt
RESULT:
comp6261_c0_seq1 1227 - UniRef90_K0R0D8 519 1e-82 186 42.2982885085575 98.9247311827957 Uncharacterized_protein Thalassiosira_oceanica - 350 + 1 0.0034993 0
P.S.
Here in the forum tabs are not viewable so I chosen to write words separated by spaces.
Either your input files are different on each machine, or your input file contains control characters that are interpreted differently on each machine. Run diff and cat -v on your input files to discover which is true.
If the files are identical, perhaps the grep isn't. Check to see if your grep is a link, alias or builtin (a shell function). Try running:
which grep
`which grep` 'comp6261_c0_seq1' file.txt
I'd like to manipulate a variable in Groovy from within the closure of each, like this:
def stringTest = ''
def foo = ['one', 'two', 'three']
foo.each {
stringTest.concat(it)
}
println stringTest
But this gives me the following error:
| Error 2013-03-13 15:26:12,330 [http-bio-8080-exec-2] ERROR
errors.GrailsExceptionResolver - NoSuchMethodError occurred when
processing request: [GET] /Reporting-Web/reporting/show/1
reporting.web.AppFiguresService$_getProductIDs_closure2.(Ljava/lang/Object;Ljava/lang/Object;Lgroovy/lang/Reference;)V.
Stacktrace follows: Message: Executing action [show] of controller
[com.xyz.reporting.ReportingController] caused exception: Runtime
error executing action Line | Method
->> 195 | doFilter in grails.plugin.cache.web.filter.PageFragmentCachingFilter
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | 63 | doFilter in grails.plugin.cache.web.filter.AbstractFilter | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker |
918 | run in '' ^ 680 | run . . in java.lang.Thread
Caused by ControllerExecutionException: Runtime error executing action
->> 195 | doFilter in grails.plugin.cache.web.filter.PageFragmentCachingFilter
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | 63 | doFilter in grails.plugin.cache.web.filter.AbstractFilter | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker |
918 | run in '' ^ 680 | run . . in java.lang.Thread
Caused by InvocationTargetException: null
->> 195 | doFilter in grails.plugin.cache.web.filter.PageFragmentCachingFilter
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | 63 | doFilter in grails.plugin.cache.web.filter.AbstractFilter | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker |
918 | run in '' ^ 680 | run . . in java.lang.Thread
Caused by NoSuchMethodError:
reporting.web.Foo$_getProductIDs_closure2.(Ljava/lang/Object;Ljava/lang/Object;Lgroovy/lang/Reference;)V
->> 77 | getProductIDs in reporting.web.Foo$$ENzya8Hg
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | 45 | show in com.xyz.reporting.Foo | 195 | doFilter in grails.plugin.cache.web.filter.PageFragmentCachingFilter | 63 |
doFilter in grails.plugin.cache.web.filter.AbstractFilter | 895 |
runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 |
run in '' ^ 680 | run . . in java.lang.Thread
I'm quite new to Groovy, any help would be great here!
This works...
def stringTest = ''
def foo = ['one', 'two', 'three']
foo.each {
stringTest += it
}
println stringTest
the function "concat()" returns a string:
stringTest = stringTest.concat(it)
To modify the iterator in the closure is not possible.
//Edit
The error msg is a grails error, while the controller can not open the function "show()"
Java strings are immutable. You can collect the concatened string:
def stringTest = ''
def foo = ['one', 'two', 'three']
stringTest = foo.collect { stringTest + it }.join()
assert stringTest == "onetwothree"