How to clean up masscan output (-oG) - linux

I have a problem with the output produced by the masscan utility with the -oG options ("grep-able" output); for instance, it outputs this:
# Masscan 1.0.3 scan initiated Wed Jun 4 01:35:02 2014
# Ports scanned: TCP(3;21-23,) UDP(0;) SCTP(0;) PROTOCOLS(0;)
Host: 192.168.100.19 () Ports: 2222/open/tcp////
Host: 192.168.100.13 () Ports: 2222/open/tcp////
Host: 192.168.100.16 () Ports: 443/open/tcp////
Host: 192.168.100.8 () Ports: 21/open/tcp////
Host: 192.168.100.5 () Ports: 22/open/tcp////
Host: 192.168.100.5 () Ports: 443/open/tcp////
Host: 192.168.100.16 () Ports: 80/open/tcp////
Host: 192.168.100.19 () Ports: 22/open/tcp////
Host: 192.168.100.7 () Ports: 80/open/tcp////
Host: 192.168.100.8 () Ports: 80/open/tcp////
Host: 192.168.100.12 () Ports: 2222/open/tcp////
Host: 192.168.100.13 () Ports: 22/open/tcp////
# Masscan done at Wed Jun 4 01:35:16 2014
The above is neither very readable nor easy to understand.
How can I use Linux command-line utilities, e.g. sed, awk, or grep, to output something as follows, using the file above?
Host: 192.168.100.5
Ports: 22, 443
Host: 192.168.100.7
Ports: 80
Host: 192.168.100.8
Ports: 21, 80
Host: 192.168.100.12
Ports: 2222
Host: 192.168.100.13
Ports: 2222, 22
......
As you can see, the output is much more readable in this layout:
sorted by IP address, with all associated ports listed below, consolidated across multiple input lines with the same IP address.

Try this:
awk -F' +|/' '
!/\s*#/ { # ignore comment lines
# Add the port on the current line to the associative array
# element for the IP address on the current line.
ips[$2] = ips[$2] (ips[$2] == "" ? $5 : ", " $5)
}
END {
# Enumerate all IPs and the ports for each.
# Since the IPs will be listed in no specific order, the output
# is piped as a _single_ line to "sort" in order to sort by IP address,
# and then expanded into 2 lines via "tr".
for (ip in ips) {
printf "Host: %s#Ports: %s#\n", ip, ips[ip] | \
"sort -t. -n -k 1.6,1 -k 2,2 -k 3,3 -k 4,4 | tr # \"\n\""
}
}
' file
This solution properly sorts the output by IP address and separates the ports with commas.
By contrast, for a given IP address, the port numbers are listed in the order they were encountered in the input (as in the sample output data in the question).

Related

ERROR: for logstash "host" network_mode is incompatible with port_bindings

I was trying to create a listener for SNMP traps in Logstash using the sanmptrap input plugin in Docker. But while running the container it is showing this error.
This is the docker-compose file
version: '3.8'
services:
logstash_audiocode_snmptrap_listener:
container_name: logstash_audiocode_snmptrap_listener
image: docker.elastic.co/logstash/logstash:7.16.3
# command: bash -c "bin/logstash-plugin install logstash-input-websocket && logstash"
user: "root:root"
ports:
- 1063:1063
network_mode: host
restart: unless-stopped
volumes:
- ./config/:/usr/share/logstash/config/:ro
- ./pipeline/:/usr/share/logstash/pipeline/:ro
- ./audiocodes_yaml/:/usr/share/logstash/audiocodes_yaml/:ro
This is the listener
input {
snmptrap {
port => 1063
community => ["IPTtrapcommunity"]
yamlmibdir => "/usr/share/logstash/audiocodes_yaml/"
# type => "snmp_trap"
codec => json
}
}
filter {
csv {
separator => ","
}
}
output {
stdout {codec => rubydebug}
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "aoudiocodes_trap_listener"
}
}
I have tried it without this.
ports:
- 1063:1063
At that time it is running but not listening to any data.
When I used the tcpdump to check whether the traps are coming or not, the traps are coming and I am able to see them using this cmd.
tcpdump -i eth0 port 1063

How to set default service port/protocol with firewalld module using Ansible?

I'm looking for a method to set default port/protocol in my Ansible task.
# Add port in firewalld
- name: Open port in firewalld
firewalld:
port: "{{ foo_setted_port_var | default (8080/tcp) }}"
state: enabled
permanent: yes
immediate: yes
zone: "{{ setted_firewalldzone_var | default (public) }}"
when I put on line port : [...] default (8080/tcp) [...]
output is : The error was: 'tcp' is undefined
when I put on line port : [...] default (8080) [...]
output is : improper port format (missing protocol?)
When I precise in my foo_setted_port_var 8080/tcp it works. However I really want to set my port by default in my task code before in my var file.
How to fix it?
According providing default values and firewalld module – Manage arbitrary ports/services with firewalld the syntax should be
# Add port in firewalld
- name: Open port in firewalld
firewalld:
port: "{{ foo_setted_port_var | default('8080/tcp') }}"
state: enabled
permanent: yes
immediate: yes
zone: "{{ setted_firewalldzone_var | default('public') }}"
Without the quotes (') the term tcp or public would be interpreted as variable name and tried to be looked up for content.
You can see this from the error message
'tcp' is undefined
and observe the behavior with the following test.
---
- hosts: localhost
become: false
gather_facts: false
vars:
PUBLIC: "public"
tasks:
- name: Show values
debug:
msg:
- "{{ TXT | default(PUBLIC) }}"
- "{{ TXT | default('PUBLIC') }}"
resulting into an output of
TASK [Show values] ******
ok: [localhost] =>
msg:
- public
- PUBLIC
The other error message
improper port format
relates to Parameter: port which needs to be string and not int.

How to find and replace string using groovy script

I need to search some text from file and replace with other string using Groovy script. I am explaining my file below.
test.yml:
devices:
test-server:
type: test1
os: test
tacacs:
username: admin
passwords:
tacacs: admin
connections:
defaults:
class: unicon.Unicon
cli:
protocol: ssh
ip: 1.1.1.1
port: 2024
rest:
protocol: http
ip: 1.1.1.1
port: 8080
username: admin
password: admin
RFS1:
type: test
os: test
tacacs:
username: admin
passwords:
tacacs: admin
connections:
defaults:
class: unicon.Unicon
cli:
protocol: ssh
ip: 1.1.1.1
port: 2024
rest:
protocol: http
ip: 4.4.4.4
port: 8080
username: admin
password: admin
RFS2:
type: test
os: test
tacacs:
username: admin
passwords:
tacacs: admin
connections:
defaults:
class: unicon.Unicon
cli:
protocol: ssh
ip: 1.1.1.1
port: 2024
rest:
protocol: http
ip: 6.6.6.6
port: 8080
username: admin
password: admin
Here I need to search the IP which is under devices:/test-server:/connections:/cli:/ip: 1.1.1.1 with some new charcter like ip:10.10.10.10 using groovy script. I am using below code.
def myFile = new File("test.yml")
def fileText = myFile.text
fileText = (fileText =~ /ip:1.1.1.1/).replaceFirst("ip:10.10.10.10")
myFile.write(fileText)
Here my issue is its replacing the required string in whole file where ip:1.1.1.1 is present but I need to replace under devices:/test-server:/connections:/cli:/ip: 1.1.1.1. Please help me to resolve this issue.
A better way to do this is to simply do YAML parsing, manipulating the object, and saving back to the file.
Here's an example using Jackson:
#Grab(group='com.fasterxml.jackson.dataformat',
module='jackson-dataformat-yaml',
version='2.12.2')
def myFile = new File("test.yml")
def om = new com.fasterxml.jackson.databind.ObjectMapper(
new com.fasterxml.jackson.dataformat.yaml.YAMLFactory());
def value = om.readValue(myFile, Map)
value['devices']['test-server']['connections']['cli']['ip'] = '10.10.10.10'
That replaces the value in the in-memory object. You can then just save that back to a file, with something like:
om.writeValue(myFile, value)

NestJS and TypeORM fail to connect my local Postgres database. Claims my database does not exist, even tho it does

I have NestJS application that uses TypeORM to connect to my local database. I create database with shell script:
#!/bin/bash
set -e
SERVER="my_database_server";
PW="mysecretpassword";
DB="my_database";
echo "echo stop & remove old docker [$SERVER] and starting new fresh instance of [$SERVER]"
(docker kill $SERVER || :) && \
(docker rm $SERVER || :) && \
docker run --name $SERVER -e POSTGRES_PASSWORD=$PW \
-e PGPASSWORD=$PW \
-p 5432:5432 \
-d postgres
# wait for pg to start
echo "sleep wait for pg-server [$SERVER] to start";
SLEEP 3;
# create the db
echo "CREATE DATABASE $DB ENCODING 'UTF-8';" | docker exec -i $SERVER psql -U postgres
echo "\l" | docker exec -i $SERVER psql -U postgres
After that, it logs databases:
Then I fire up my application, and I encounter error "error: database "my_database" does not exist"
I use following code to connect to database:
static getDatabaseConnection(): TypeOrmModuleOptions {
console.log(require('dotenv').config())
return {
type: 'postgres',
host: "127.0.0.1",
port: 5432,
username: 'postgres',
password: 'mysecretpassword',
database: 'my_database',
entities: ['dist/**/*.entity{.ts,.js}'],
synchronize: true,
};
}
Any ideas where do I go wrong?
When connecting to a docker instance, you should usually use the service name. In this case I guess it is my_database_server as host parameter.
return {
type: 'postgres',
host: "my_database_server",
port: 5432,
username: 'postgres',
password: 'mysecretpassword',
database: 'my_database',
entities: ['dist/**/*.entity{.ts,.js}'],
synchronize: true,
};
"localhost" isn't address of your docker container. Which address uses docker you can look running command:
$ docker inspect {your_container_name}
for me is: 172.17.0.2
Try enable SSL, adding next configuration lines:
ssl: true,
extra: { ssl: { rejectUnauthorized: false } }
Try using localhost instead of 127.0.0.1

Inexplicable node.js http throwing connect ECONNREFUSED (IPv6?)

I am running node.js as follows:
> http = require('http')
> http.get('http://myhost.local:8080',
function (res) { console.log("RES" + res) }
).on('error', function (e) { console.log("Error:", e) })
> uri = require('url').parse("http://myhost.local:8080")
{ protocol: 'http:',
slashes: true,
auth: null,
host: 'myhost.local:8080',
port: '8080',
hostname: 'myhost.local',
hash: null,
search: null,
query: null,
pathname: '/',
path: '/',
href: 'http://myhost.local:8080/' }
> http.get(uri,
function (res) { console.log("RES" + res) }
).on('error', function (e) { console.log("Error:", e) })
An error is thrown for both the implicit and explicitly parsed URI and I get the following output for both:
Error: { [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' }
The host myhost.local is an alias for localhost in /etc/hosts, being:
127.0.0.1 localhost myhost.local myhost
255.255.255.255 broadcasthost
::1 localhost myhost.local myhost
fe80::1%lo0 localhost myhost.local myhost
EDIT: I tried virtually every permutation for the hosts file, including the most obvious:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost myhost.local myhost
fe80::1%lo0 localhost
EDIT I should also mention that I have tried this on more than one Mac now.
Although it seems this is a rather common error, I have seen no useful explanations or workarounds. Here are some notable related facts:
Running $ wget http://myhost.local:8080 works as expected, so it isn't a firewall problem.
Running $ telnet myhost.local 8080 and then manually GET'ing the url works fine, so it's not a weird HTTP problem.
I have no trouble using node.js to connect to other hosts e.g. http://www.google.com
I expect the useful system information would include:
$ node -v
v0.9.11
$ uname -a
Darwin myhost.local 12.2.1 Darwin Kernel Version 12.2.1:
Thu Oct 18 12:13:47 PDT 2012; root:xnu-2050.20.9~1/RELEASE_X86_64 x86_64
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.8.2
BuildVersion: 12C3104
$ sudo netstat -nalt | grep LISTEN | grep 8080
tcp6 0 0 ::1.8080 *.* LISTEN
Does anyone have any idea what is going on here, and what a fix might be?
I'm going to post this here in case somebody else has the problem.
Bert Belder, Node.js mailing list:
On your system "myhost.local" resolves to three different addresses
(127.0.0.1, ::1, and fe80::1). Node prefers ipv4 over ipv6 so it'll
try to connect to 127.0.0.1. Nothing is listening on 127.0.0.1:8080 so
the connect() syscall fails with ECONNREFUSED. Node doesn't retry with
any of the other resolved IPs - it just reports the error to you. A
simple solution would be to replace 'localhost' by the intended
destination ip address, '::1'.
Whether this behavior is right is somewhat open for debate, but this
is what causes it.
Bert
This stemmed from an issue with Node (though there are ways to work around it), as per the discussion on nodejs/Google Groups, as #alessioalex alluded in his answer. A useful comment per Bert Belder:
there should be a getaddrinfo wrapper that returns more that just the first result
For example,
> require('dns').lookup('myhost.local', console.log)
{ oncomplete: [Function: onanswer] }
> null '127.0.0.1' 4
This is the first of multiple getaddrinfo results passed to Node. It seems that nodejs only uses the first item of the getaddrinfo call. Bert and Ben Noordhuis agreed in the Groups discussion that there should be a way to return more than just the first result with the getaddrinfo wrapper.
Contrast python, which returns all results from getaddrinfo:
>>> import socket
>>> socket.getaddrinfo("myhost.local", 8080)
[(30, 2, 17, '', ('::1', 8080, 0, 0)),
(30, 1, 6, '', ('::1', 8080, 0, 0)),
(2, 2, 17, '', ('127.0.0.1', 8080)),
(2, 1, 6, '', ('127.0.0.1', 8080)),
(30, 2, 17, '', ('fe80::1%lo0', 8080, 0, 1)),
(30, 1, 6, '', ('fe80::1%lo0', 8080, 0, 1))]
does this work?
var http = require('http');
var options = {
host: 'myhost.local',
port: 8080,
path: '/'
};
http.get(options, function (res) {
console.log("RES" + res)
}).on('error', function (e) {
console.log("Error:", e)
});

Resources