I have links .php how do I substitute my values in all parameters using curl post?
Provided that I do not know what the parameters are in these php links, curl should determine for itself what parameters are in the post request and substitute my values.
If I know the parameter, then I can send it to the links like this:
while read p; do
curl $p -X POST --connect-timeout 18 --cookie "" --user-agent "" -d "parametr=helloworld" -w "%{url}:%{time_total}s\n"
done < domain.txt > output.txt
And if I do not know the parameters, what should I do? How to make curl automatically substitute values into parameters? For example, the value: "hello world" provided that I did not know "parameter"
It's simply not possible. curl is a client program and has no way of knowing or finding out which request parameters are supported by a server or which are not.
Unless of course, the API is properly documented and available as an OpenAPI/Swagger specification for example. If it isn't, you're out of luck.
Related
I want to download a url in a remote host using ssh, i was using exec(), it was working:
const cmd = `mkdir -p /home/username/test; wget --no-check-certificate -q -U \"\" -c \"${url}\" -O /home/username/test/img.jpg`;
const out = execSync(`ssh -o ConnectTimeout=8 -o StrictHostKeyChecking=no -p 2356 username#${ip} '${cmd}'`);
But it's usafe to use the url variable this way, the value of this variable is from user input, so i found some posts on stackoverflow saying that i need to use spawn:
const url = 'https://example.com/image.jpg';
const out = spawnSync('ssh', [
'-o', 'ConnectTimeout=8',
'-o', 'StrictHostKeyChecking=no',
'-p', '2356',
`username#${ip}`,
`mkdir -p /home/username/test; wget --no-check-certificate -q -U "" -c "${url}" -O /home/username/test/img.jpg`,
]);
What about if const url = 'https://example.com/image.jpg"; echo 5; "';, the echo will be executed, could someone tell me how to execute this code in a safe way?
There are two aspect. First, you are correct that execSync is unsafe. To quote from the documentation:
Never pass unsanitized user input to this function. Any input containing shell metacharacters may be used to trigger arbitrary command execution.
A solution is to use execSpawn, as you pointed out, for example, like in this related answer.
However, in your example, you are calling ssh and passing it a text, which will be executed by the shell on the remote system. Because of that, you are still vulnerable to the attack, as you showed in the example. But note that it is no longer a NodeJs related exploit, but an exploit on ssh and the shell.
To mitigate the attack, I would recommend to concentrate on the remote server. Instead of sending it a command over ssh, which it should trust and execute in the shell, you could provide a clear defined API from the server. In the HTTP interface, you can accept input and do a proper validation (instead of simplify trusting it). The advantage is that you do not need to deal with the subtleties of the shell.
If you are forced to work with ssh, you could validate the URL and only if it is safe, forward it to the server. It is still not ideal from a security perspective. First, the remote server will need to trust you completely (often it is better to avoid that and instead validate as locally as possible). Second, the validation itself is not straightforward. You will need to decide if a string looks like an URL (e.g. by using new URL(url)), but the more difficult aspect is to make sure that no exploits slip through. I don't have a concrete example, but I would be cautious to assume all strings that pass the URL parser will be safe to execute in a shell environment.
In summary, if possible try to avoid ssh with passing shell command in that situation (input data controlled by an attacker). Instead prefer a normal API like a HTTP interface (or other text or binary protocols). If it is not possible, try hard to sanitize the data before sending it out. Maybe you know in advance how a URL will look like (e.g. the list of allowed hostnames, allowed paths, etc). But realize that there might be hidden examples that you will overlook, and never underestimate the creativity of an attacker.
I want to fetch all records (from Solr) with a timestamp older than 30 days via cURL command.
What I have tried:
curl -g "http://localhost:8983/solr/input_records/select?q=timestamp:[* TO NOW/DAY-30DAYS]"
I do not understand why this does not work but it does not fetch anything. It simply returns nothing. If I replace '[* TO NOW/DAY-30DAYS]' with an actual value, it will retrieve that record.
Additional relevant information, this is how to delete all records older than 30 days (it works). Again, I do not want to delete, rather just fetch the data.
curl -g "http://localhost:8983/solr/input_records/update?commit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>timestamp:[* TO NOW/DAY-30DAYS]</query></delete>"
Thanks in advance!
This error is happening because you don't have proper URL encoding for your request. Most likely the problem is spaces - need to replace them with %20, same applies to other symbols
Try this:
curl -g "http://localhost:8983/solr/input_records/select?q=timestamp:[*%20TO%20NOW/DAY-30DAYS]
A further addition to Mysterion's answer,
Since you are doing this using curl, you are facing the issue of the URL encoding.
If you just mention
http://localhost:8983/solr/input_records/select?q=timestamp:[* TO NOW/DAY-30DAYS]
in your browser (chrome or others)
the Url encoding is automatically handled by the browser and you would get your response as expected.
curl --data "<xml>" --header "Content-Type: text/xml" --request PROPFIND url.com
By reading the curl man page I could not understand how the above commandline is using --data option.
Question:
What does above commandline do ?
Why doesn't man page describe this usage? If it does then what did I not understand after reading the man page?
The --data flag is for defining POST data.
The command sends a POST with contents <xml> and MIME type text/xml. However, with the flag --request, you are setting the HTTP method from POST to PROPFIND and sending the request to url.com.
I also did manage to find the --data flag in the manual:
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP server,
in the same way that a browser does when a user has filled in an HTML
form and presses the submit button. This will cause curl to pass the
data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. To post data purely binary, you should instead use the --data-binary option. To URL-encode the
value of a form field you may use --data-urlencode.
If any of these options is used more than once on the same command
line, the data pieces specified will be merged together with a
separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would
generate a post chunk that looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file
name to read the data from, or - if you want curl to read the data
from stdin. The contents of the file must already be URL-encoded.
Multiple files can also be specified. Posting data from a file named
'foobar' would thus be done with --data #filename. When --data is told
to read from a file like that, carriage returns and newlines will be
stripped out.
I am using Apache Stanbol. It works for enhancing the text, however when I tried sentiment analysis and sentence detection, it doesn't work.
I tried this code
curl -v -X POST -H "Accept: text/plain" -H "Content-type: text/plain; \
charset=UTF-8" --data "Some text for analysis" \
"http://localhost:8081/enhancer/engine/sentiment-wordclassifier"
But it gives blank { } output, I tried changing the header attributes but no luck.
am I missing something? Do I need to do some configuration first?
I even tried adding analyzer in the enhancer chain but the same blank output, also tried REST API for opennlp-sentence, but it didn't work.
I guess you are sending data to the wrong endpoint... usually calls to the enhancer need to be done to all chains:
http://host/stanbol/enhancer
or to a concrete chain:
http://host/stanbol/enhancer/chain/<name>
The enhancer results couldn't be serialized as plain text, but in any of the RDF serialization supported by Stanbol. So the Accept header would need to be any of those, text/turtle for instance.
Further details at the documentation: http://stanbol.apache.org/docs/trunk/components/enhancer/#RESTful_API
Here is the question.
Given the url http://www.example.com, can we read the first N bytes out of the page?
using wget, we can download the whole page.
using curl, there is -r, 0-499 specifies the first 500 bytes. Seems solve the problem.
You should also be aware that many HTTP/1.1 servers do not have this feature enabled, so that when you attempt to get a range, you'll instead get the whole document.
using urlib in python. similar question here, but according to Konstantin's comment, is that really true?
Last time I tried this technique it failed because it was actually impossible to read from the HTTP server only specified amount of data, i.e. you implicitly read all HTTP response and only then read first N bytes out of it. So at the end you ended up downloading the whole 1Gb malicious response.
So the problem is that how can we read the first N bytes from the HTTP server in practice?
Regards & Thanks
You can do it natively by the following curl command (no need to download the whole document). According to the curl man page:
RANGES
HTTP 1.1 introduced byte-ranges. Using this, a client can request to get only one or more subparts of a specified document. curl
supports this with the -r flag.
Get the first 100 bytes of a document:
curl -r 0-99 http://www.get.this/
Get the last 500 bytes of a document:
curl -r -500 http://www.get.this/
`curl` also supports simple ranges for FTP files as well.
Then you can only specify start and stop position.
Get the first 100 bytes of a document using FTP:
curl -r 0-99 ftp://www.get.this/README
It works for me even with a Java web app deployed to GigaSpaces.
curl <url> | head -c 499
or
curl <url> | dd bs=1 count=499
should do
Also there are simpler utils with perhaps borader availability like
netcat host 80 <<"HERE" | dd count=499 of=output.fragment
GET /urlpath/query?string=more&bloddy=stuff
HERE
Or
GET /urlpath/query?string=more&bloddy=stuff
You should also be aware that many
HTTP/1.1 servers do not have this
feature enabled, so that when you
attempt to get a range, you'll instead
get the whole document.
You will have to get the whole web anyways, so you can get the web with curl and pipe it to head, for example.
head
c, --bytes=[-]N
print the first N bytes of each file; with the leading '-', print all
but the last N bytes of each file
I came here looking for a way to time the server's processing time, which I thought I could measure by telling curl to stop downloading after 1 byte or something.
For me, the better solution turned out to be to do a HEAD request, since this usually lets the server process the request as normal but does not return any response body:
time curl --head <URL>
Make a socket connection. Read the bytes you want. Close, and you're done.