Am looking for a way to restrict the output of curl command
For example when using curl to check if port is open on server, just want to restrict the output to first lines to confirm that port is open
curl -v host:1521
want to just display first 3 lines of output
*About to connect to
*Trying host ..connected
* Connected to host
Why not to pipe it to head?
curl -v host:1521 | head -n3
where -n3 means 3 lines from top.
EDIT:
As discussed in comments you use -v option to capture headers etc. which are printed on stderr instead of stdout so head doesn't affect it. You have to redirect stderr to stdout and after that operate on:
curl -v www.example.com 2>&1 | grep Connected
This will return * Connected to www.example.com (IP_ADDRESS_HERE) port 443 (#0) if connected successfully and nothing otherwise.
Related
I've tried using netcat in various ways.
The common just exits
echo "Response from server" | nc -l 127.0.0.1 8080
I get the following error in browser net::ERR_CONNECTION_REFUSED
nc -l localhost 8080 < temp.resp
I just simply want to print some text from nectar to browser and maybe later pass some headers along with it too. But right now I am not able to figure out what is going wrong.
I've looked in the man page and tried -w and -I options but none of them are working.
I have a command which out outputs certain data which i store in a ext file using a '>>' command.Now Instead of doing that I want to have a socket or a port on any server which will catch the output of the command.Basically i want to output all my script data to a socket or url which ever is possible.
Any help in this direction is most welcomed.
You can use socat to listening on a port 12345 and echo any data sent to it like this:
socat -u TCP-LISTEN:12345,keepalive,reuseaddr,fork STDOUT
If you want to capture it to a file as well (file.log), you can use the same command with tee:
socat -u TCP-LISTEN:12345,keepalive,reuseaddr,fork STDOUT | tee file.log
You can run your program to output to bash's TCP virtual device:
./prog > /dev/tcp/localhost/12345
If you don't want to use bash magic then you can also use socat to send the data:
./prog | socat - TCP-CONNECT:localhost:12345
The above example assume you are running your program and "logger" on the same system but you can replace "localhost" with the hostname or address of the system you wish to send to (where the socat is listening).
How list from the command line URLs requests that are made from the server (an *ux machine) to another machine.
For instance, I am on the command line of server ALPHA_RE .
I do a ping to google.co.uk and another ping to bbc.co.uk
I would like to see, from the prompt :
google.co.uk
bbc.co.uk
so, not the ip address of the machine I am pinging, and NOT an URL from servers that passes my the request to google.co.uk or bbc.co.uk , but the actual final urls.
Note that only packages that are available on normal ubuntu repositories are available - and it has to work with command line
Edit
The ultimate goal is to see what API URLs a PHP script (run by a cronjob) requests ; and what API URLs the server requests 'live'.
These ones do mainly GET and POST requests to several URLs, and I am interested in knowing the params :
Does it do request to :
foobar.com/api/whatisthere?and=what&is=there&too=yeah
or to :
foobar.com/api/whatisthathere?is=it&foo=bar&green=yeah
And does the cron jobs or the server do any other GET or POST request ?
And that, regardless what response (if any) these API gives.
Also, the API list is unknown - so you cannot grep to one particular URL.
Edit:
(OLD ticket specified : Note that I can not install anything on that server (no extra package, I can only use the "normal" commands - like tcpdump, sed, grep,...) // but as getting these information with tcpdump is pretty hard, then I made installation of packages possible)
You can use tcpdump and grep to get info about activity about network traffic from the host, the following cmd line should get you all lines containing Host:
tcpdump -i any -A -vv -s 0 | grep -e "Host:"
If I run the above in one shell and start a Links session to stackoverflow I see:
Host: www.stackoverflow.com
Host: stackoverflow.com
If you want to know more about the actual HTTP request you can also add statements to the grep for GET, PUT or POST requests (i.e. -e "GET"), which can get you some info about the relative URL (should be combined with the earlier determined host to get the full URL).
EDIT:
based on your edited question I have tried to make some modification:
first a tcpdump approach:
[root#localhost ~]# tcpdump -i any -A -vv -s 0 | egrep -e "GET" -e "POST" -e "Host:"
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
E..v.[#.#.......h.$....P....Ga .P.9.=...GET / HTTP/1.1
Host: stackoverflow.com
E....x#.#..7....h.$....P....Ga.mP...>;..GET /search?q=tcpdump HTTP/1.1
Host: stackoverflow.com
And an ngrep one:
[root#localhost ~]# ngrep -d any -vv -w byline | egrep -e "Host:" -e "GET" -e "POST"
^[[B GET //meta.stackoverflow.com HTTP/1.1..Host: stackoverflow.com..User-Agent:
GET //search?q=tcpdump HTTP/1.1..Host: stackoverflow.com..User-Agent: Links
My test case was running links stackoverflow.com, putting tcpdump in the search field and hitting enter.
This gets you all URL info on one line. A nicer alternative might be to simply run a reverse proxy (e.g. nginx) on your own server and modify the host file (such as shown in Adam's answer) and have the reverse proxy redirect all queries to the actual host and use the logging features of the reverse proxy to get the URLs from there, the logs would probably a bit easier to read.
EDIT 2:
If you use a command line such as:
ngrep -d any -vv -w byline | egrep -e "Host:" -e "GET" -e "POST" --line-buffered | perl -lne 'print $3.$2 if /(GET|POST) (.+?) HTTP\/1\.1\.\.Host: (.+?)\.\./'
you should see the actual URLs
A simple solution is to modify your '/etc/hosts' file to intercept the API calls and redirect them to your own web server
api.foobar.com 127.0.0.1
I need to follow a log file on linux machine and stream the updates of log file over http port to a remote machine. I have written a command with the combination of "tail" and "curl".
To test it initially, i used "tail -n", it works well and posts data successfully to remote machine. Below is the command.
$tail -n 200 /path/to/logfile/file1.log | curl --data-binary #- http://remotemachineIP:9000
Now, When i try to run the same command with "tail -f", it's not posting any data over http even though the log file is updated multiple times. Below is the command
$tail -f --follow=name /path/to/logfile/file1.log | curl --data-binary #- http://remotemachineIP:9000
As per my understanding, "tail -f" is not conveying my "curl" command that "input feed is complete over stdin(#-)". Any help on how to rectify this issue?
Thanks in advance
curl will make a single HTTP POST request with the piped data. What you want to do instead is to continuously send the data.
Assuming that by "HTTP port" you actually meant TCP there is a way using netcat:
Remote
nc -l 9000
Local
tailf /path/to/log/file | nc remote_ip 9000
I am using netcat utility on linux to receive outputs from a program on a windows machine. My problem being that the program on the windows machine does not always give an output.
How can i check that either a connection has been made to netcat ?
What i am doing till now is "nc -l -v 9103 > output" then i check the size of output, the problem this poses is that netcat only write to a file after a certain buffer size has been reached or a new line char is encountered, so some cases evne though a connection has been made the file size is detected as zero.
How can i check if someone has made a connection with netcat.
I tried using
nc -l -v -e someprog.exe 9103 > output
but my netcat doesnt seem to support this
below are the options i have
$ nc -h
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port]
[-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version]
[-x proxy_address[:port]] [hostname] [port[s]]
Command Summary:
-4 Use IPv4
-6 Use IPv6
-D Enable the debug socket option
-d Detach from stdin
-h This help text
-i secs Delay interval for lines sent, ports scanned
-k Keep inbound sockets open for multiple connects
-l Listen mode, for inbound connects
-n Suppress name/port resolutions
-p port Specify local port for remote connects
-r Randomize remote ports
-s addr Local source address
-T ToS Set IP Type of Service
-C Send CRLF as line-ending
-t Answer TELNET negotiation
-U Use UNIX domain socket
-u UDP mode
-v Verbose
-w secs Timeout for connects and final net reads
-X proto Proxy protocol: "4", "5" (SOCKS) or "connect"
-x addr[:port] Specify proxy address and port
-z Zero-I/O mode [used for scanning]
Port numbers can be individual or ranges: lo-hi [inclusive]
verbose mode will write connectivity to stderr, and you can redirect stderr to a file, the verbose log has something like
connect to [xxx] from [xxxx]
try
nc -l -v -p 9103 -k 1> output 2>connect.log
and monitor connect.log for connectivity
if you don't use -k , netcat quits after 1st connection.
If you can upgrade your copy of netcat: the modern versions (1.10, for one) have an option to execute a program (or a shell command) upon connect. Otherwise, you can make the netcat think it runs in a terminal (to disable buffering of stdout), by using for example script (it just saves everything on stdin/stdout/stderr in the given file). Or use logging features of screen and tmux.