I'm looking for a way to include the backend name (as configured by the backend vcl option), that a request was sent to (or response received from; either way) in varnishncsa log output.
When running varnishncsa in client mode, i.e. logging the frontend requests/responses.
The log format is here: https://varnish-cache.org/docs/trunk/reference/varnishncsa.html#format
I've tried to find an "Extended variable" (%{X}x) that would satisfy this option, without luck so far. Varnish version is 6.x.
Do I need to set a custom request or response header, to match on, for this first??
You can use an extended variable to get the job done.
varnishncsa from a backend request
Here's an example where I retrieve the backend name for backend requests:
varnishncsa -b -F "%{VSL:BackendOpen[2]}x"
The VSL tag you need is the BackendOpen tag that would return the following output in varnishlog:
- BackendOpen 32 boot.bla 192.168.224.2 8080 192.168.224.3 53908
As you can see the backend name is the second field, hence the BackendOpen[2] expression.
varnishncsa from a client request
In the client thread the backend is hinted, but not opened.
If you want to access the hinted backend, you'll need to have access to the req.backend_hint variable. You could use std.log() to log this value as a VCL_Log tag.
Here's the VCL example:
vcl 4.0;
import std;
backend default {
.host="1.2.3.4";
.port="80";
}
sub vcl_recv {
std.log("Backend: "+ req.backend_hint);
}
And here's the varnishncsa command the would look for this value:
varnishncsa -c -F "%{VSL:VCL_Log:Backend}x"
Conclusion
Of course you still need to tune your varnishncsa command a bit to include the right fields and filter on the right parameters, but this one should hopefully answer your question.
It's up to you to look for the backend name at client request side or at the backend request side.
Related
I am using Docusign's REST API to create and send envelopes. I've included eventNotifications with requireAcknowledgment as true to get requests from Docusign whenever there's status change. I used ngrok while development and testing and everything worked as expected.
I've moved the project online and have edited the eventNotification's url to live url with https and that's when all the callbacks are getting logged in failed section in Docusign's admin panel.
The error message shown in admin panel is -
'https://xxx.xxxxxxx.com/webhook.php :: Error - The remote server
returned an error: (400) Bad Request.'
I've downloaded the failed request's xml body and tried sending a request through postman and it worked as expected. Iv'e tried everything to debug this error and have not found error at my end.
Edit:
The code that I've tried with is the same code from DocuSign's webhook sample page -
$data = file_get_contents('php://input');
$xml = simplexml_load_string ($data, "SimpleXMLElement", LIBXML_PARSEHUGE);
$envelope_id = (string)$xml->EnvelopeStatus->EnvelopeID;
$time_generated = (string)$xml->EnvelopeStatus->TimeGenerated;
$files_dir = getcwd() . '/' . $this->xml_file_dir;
if(! is_dir($files_dir)) {mkdir ($files_dir, 0755);}
$envelope_dir = $files_dir . "E" . $envelope_id;
if(! is_dir($envelope_dir)) {mkdir ($envelope_dir, 0755);}
$filename = $envelope_dir . "/T" .
str_replace (':' , '_' , $time_generated) . ".xml"; // substitute _ for : for windows-land
$ok = file_put_contents ($filename, $data);
if ($ok === false) {
error_log ("!!!!!! PROBLEM DocuSign Webhook: Couldn't store $filename !");
exit (1);
}
// log the event
error_log ("DocuSign Webhook: created $filename");
if ((string)$xml->EnvelopeStatus->Status === "Completed") {
// Loop through the DocumentPDFs element, storing each document.
foreach ($xml->DocumentPDFs->DocumentPDF as $pdf) {
$filename = $this->doc_prefix . (string)$pdf->DocumentID . '.pdf';
$full_filename = $envelope_dir . "/" . $filename;
file_put_contents($full_filename, base64_decode ( (string)$pdf->PDFBytes ));
}
}
I've also tried with simple code that just sets header to 200
http_response_code(200);
Sorry you're having so much trouble with the webhook feature. Hopefully this answer will be of assistance.
1. Try a test PHP program to check connectivity, etc:
<?php
header('Content-Type: text/html');
echo "OK!";
$h = fopen('/tmp/listener_access.log', 'a');
if ($h) {
$now = DateTime::createFromFormat('U.u', microtime(true), new DateTimeZone('UTC'));
fwrite ($h, $now->format ("Y-m-d_l.h.i.s.v "));
fwrite($h, $_SERVER["REMOTE_ADDR"] . " " . $msg . "\n");
fclose($h);
} else {
error_log ("### Could not open log file!");
}
Store it as ok.php in your web server directory. Then try (from a browser on a different network) to reach https://yourserver.company.com/ok.php You should see "ok" in the browser window.
Then use the same url in your eventNotification section of your Envelopes::create call to see if it all works. You should see a success in the Connect log, etc.
2. Stepwise debugging of your PHP listener
There are a number of issues to rule-out (as diagnosticians say) with your php listener app.
Basic connectivity with the listener. Since DocuSign is receiving a 400 error from your app, connectivity is ok.
Platform errors with your app. This is a problem with your PHP software stack setup that is causing the stack (not your PHP app, per se) to reject the request.
The usual indicator for this is your web server's error log. Have you looked at it? What is on the logging line where you see the 400 response from your server to DocuSign? If you don't see the 400 response to DocuSign then something is wrong with your web server's setup.
A common platform error with PHP and other stacks when default settings are used is maximum_post_size_exceeded. This is happens if you have requested that DocuSign include your envelope's documents in the notification message.
A good test is to temporarily change your envelope create code to not include documents in the notification messages.
Fixes: a good fix is to not include the envelope documents in the notification message. Instead, after the message is received, make separate API calls to retrieve the documents.
The other fix is to increase the maximum post body size. You may need to increase it both in the PHP settings and in the underlying web server's settings too. (By the way, which web server are you using?)
You are processing the incoming notification in the response thread of your server/php app. This really isn't the best technique (I will be updating the DocuSign example page in the future with this information.)
The best technique is to use the following pattern:
1. Receive the incoming notification message from DocuSign
2. Put the message onto a reliable FIFO queue
3. Respond to DocuSign with a 200 response.
Then, in a separate thread of execution, a
different software application "works off" the
entries in the queue.
Your web server, for example, may be timing out your PHP program. This probably isn't happening in your case, but others may experience this.
I think I would, next, add more debugging statements to your PHP program to try and understand what is happening to it. You can either do this with error-log
or copy the technique from my example (above) and write to a file in /tmp. (Assuming a Linux server.)
The other option is to increase the debugging level of the php stack itself, or of your web server, or both.
3. Last thoughts
400 in the DocuSign logs usually indicates that DocuSign reached your server and it returned a 400 status code. This can be confirmed by examining your server log (regular or error), there should be a matching entry.
If there isn't an entry in your server log, but the "ok.php" program from above does work, then it would be time to comment out big chunks of code from your PHP program, and then do another test from DocuSign. Eventually, using a binary-search technique (See step 8 in the article), you find the code that is causing the problem.
Commenting out code as part of a binary-search to find the bug is a very common, and powerful debugging technique.
HTH, Larry
I implemented a Rest service that creates an Employee. In the response message I want to dynamically set the HTTP Location header with the newly created Employee resource Uri.
The below code is working fine and I am able to see the value in Location header as expected. However I have the Uri hardcoded in the EmpService and I want it to be dynamic. How do I extract/pass Uri information to the EmpService bean?
Config.xml
<int-http:inbound-gateway
request-channel="httpPostChannel"
reply-channel="responseChannel"
path="/emp"
supported-methods="POST"
message-converters="converters"
request-payload-type="com.samples.jaxb.Employee"/>
<int:service-activator ref="empService" method="post"
input-channel="httpPostChannel" output-channel="responseChannel"/>
EmpService.java
public Message<Employee> post (Message<Employee> msg) {
Employee emp = empDao.createEmployee(msg.getPayload());
return MessageBuilder.withPayload(emp)
.setHeader(org.springframework.http.HttpHeaders.LOCATION, "http://localhost:8080/RestSample/emp/" + emp.getEmpId())
.build();
}
Actually even right now your URI is dynamic:
"http://localhost:8080/RestSample/emp/" + emp.getEmpId()
OTOH you always can inject it via setter or #Value property during application start from some external property.
Or you even can do that extracting some property/header from the incoming Message.
However I guess you would like to know the host and port you are ran on.
The host you can know via InetAddress.getLocalHost().
The port you can extract via an appropriate ServletContainer vendor API, e.g. for Tomcat: Get the server port number from tomcat with out a request.
With Spring Boot you can just use #LocalServerPort:
* Annotation at the field or method/constructor parameter level that injects the HTTP
* port that got allocated at runtime. Provides a convenient alternative for
* <code>#Value("${local.server.port}")</code>.
Although... I guess this one should be enough for:
.setHeader(org.springframework.integration.http.HttpHeaders.REQUEST_URL,
request.getURI().toString())
I mean that your incoming Message after <int-http:inbound-gateway> has header set. In my test case with Spring Boot and random Tomcat port it looks like:
"http_requestUrl" -> "http://localhost:64476/service/?name=foo"
I am using DocuSign connect to update the state of my app after an event happens on a document.
I have set up my account like so:
At the moment my "URL to Publish" looks something similar to https://key:secret#example.herokuapp.com. However when I look in the logs I always seem to receive something similar to:
error: Exception in EnvelopeIntegration.RunIntegration: key :: https://key:secret#example.herokuapp.com/webhook :: Error - The remote server returned an error: (401) Unauthorized
When I copy the Envelope Data into a file locally (complete-webhook.xml) and I run the following command through the command line it seems to run successfully:
curl -i -X POST -d #complete-webhook.xml https://key:secret#example.herokuapp.com/webhook
Has anybody got any ideas as to the reason why this could be happening?
When you use a url such as https://username:password#example.com/, your client takes the username:password part of the url and uses it to create an Authorization: Basic header.
You can try it yourself, create a requestb.in and then use the curl command
curl -X POST -d "fizz=buzz" http://username:password#requestb.in/12345
# where 12345 is your requestb.in address
The result on requestb.in:
A request to just /12345 (the incoming url does not include the username or password)
The request includes the header Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
If you put dXNlcm5hbWU6cGFzc3dvcmQ= through a base64 decoder, you get username:password
Answer, at this point, the Connect system does not support sending basic authentication information when it calls listeners. I have filed an internal feature request.
Work-around
Your listener url can include a query parameter that serves as a password. eg. `example.com/webhook/?pw=9e47a953-c105-44c5-ba5c-4bb77d63694d
Then, in your listener, simply reject any request that does not include the pw query parameter and the value that you chose.
In its requests to your listener, the Connect system will use any query parameters that you originally set when you added the Connect subscription.
The app I'm working on has a controller that issues templates to the front end (single page app). It's very basic, and simply consists of
#path = params[:path]
render template: "templates/#{#path}", layout: nil
Here my concern however is the direct use of the users input. Everything about this to me feels like it can be attacked with something as simple as path traversal. The route for this is
get "/templates/:path.html" => "templates#file", constraints: { path: /.+/ }, defaults: { format: 'html' }
I've tried multiple things to attempt a path traversal attack, such as
request /templates/path/to/../somewhere/else.html
request /templates?path=/path/to/../../something.rb
request /templates/index.html?path=/path/to/../../config/something.html
request /templates/path/../../../file.html
Fortunately, I haven't had any success with this. The requests that just start with /templates and don't specify anything after it, don't match the route thanks to the constraint so that is good.
It seems as though when that route is matched, rails doesn't allow you to override the path parameter through a url parameter, so I don't seem to be able to inject it there.
The ones that interest are the first and last examples above, where rails seems to internally be changing the requested URL before invoking the routes file. When I request /templates/path/to/../somewhere/else.html, my console output shows a request for /templates/path/somewhere/else.html. When I make a request for /templates/path/../../../file.html, the log shows a request for /file.html.
Am I missing something somewhere that will leave the app open to security issues, or is this just rails being sensible and protecting itself for me?
UPDATE
I've done some more digging, and if I try doing some URL encoding then I can cause the server to simply not respond at all. If I request /templates/%2e%2e%2f%2e%2e%2f%2e%2e%2ffresult.html then I just get an empty response with a connection: close header.
I assume that the parameter parser higher up in the rack is checking all urls for this type of attack? Regardless, my original question still stands. Am I missing something here?
Using ServiceStack 3.9.2x.
Out of the box (and as seen on the metadata page) service stack comes with built-in support for a bunch of content types - xml, json, jsv, etc. What is the best way to tell ServiceStack to limit the set of supported content. For example my service only knows how to speak JSON and I don't want ServiceStack to honor requests that sport "Content-type: application/xml" or "Accept: application/xml" headers. In said case I would like ServiceStack to respond with a 406 (Not Acceptable) response, ideally including in the body the set of supported content (per HTTP 1.1 spec).
Also how does ServiceStack decide what the default type of the content is for requests that do not sport an Accept or Content-Type header (I think I am seeing it render HTML now)? Is there a way to tell ServiceStack to assume a specific content type in these cases?
See this answer to find out how to set the default content type in ServiceStack: ServiceStack default format
You can use a Request Filter to detect the requested content type with:
httpReq.ResponseContentType
In your global request filter you can choose to allow it (do nothing) or write directly to the response, e.g. 406 with list of supported content as you wish.
ServiceStack order of operations
The Implementation architecture diagram shows a visual cue of the order of operations that happens in ServiceStack. Where:
EndointHostConfig.RawHttpHandlers are executed before anything else, i.e. returning any ASP.NET IHttpHandler by-passes ServiceStack completely.
The IAppHost.PreRequestFilters gets executed before the Request DTO is deserialized
Request Filter Attributes with Priority < 0 gets executed
Then any Global Request Filters get executed
Followed by Request Filter Attributes with Priority >= 0
Action Request Filters (New API only)
Then your Service is executed
Action Response Filters (New API only)
Followed by Response Filter Attributes with Priority < 0
Then Global Response Filters
Followed by Response Filter Attributes with Priority >= 0
Any time you close the Response in any of your filters, i.e. httpRes.Close() the processing of the response is short-circuited and no further processing is done on that request.