How to manage logstash output-http return codes? I'm using logstash-output-http to post data to a restful webservice. Since http may result in application failure (code >= 400) how can i handle this scenario via logstash ?
my configuration is the following:
output {
http {
url => "http://bla:8080/restful/endpoint"
http_method => "post"
format => message
content_type => "application/json; charset=UTF-8"
message => '{"json":"here"}'
}
}
suppose you have a code = 500 and you want to eventually deliver such message, how you would do that?
The code seems to imply that you can look in your log file.
There's an issue to retry them.
Related
I have a server written in node.js. The client sends a get request by fetch with the url of the mp3 file that is in the files on the server. My goal is to send the mp3 file back to the client so that it can be played. I wrote something like this:
if (req.url.indexOf(".mp3") != -1) {
fs.readFile(__dirname + decodeURI(req.url), function (error, data) {
res.setHeader("Access-Control-Allow-Origin", "*");
res.writeHead(200, {
"Content-type": "audio/mpeg",
});
res.write(data);
res.end();
})
}
but I get this error: Uncaught (in promise) SyntaxError: Unexpected token I in JSON at position 0
Also, here it is client side:
fetch("http://localhost:3000/static/mp3/" + value, { method: "get" })
.then((response) => response.json())
.then((data) => (this.song = data));
document.getElementById("audio_src").src =
"http://localhost:3000/" + this.song;
In the client, you're calling response.json(), but the response you're getting back is NOT json. The data you're getting back is binary. Perhaps you should be calling response.blob()?
But, then you're trying to put the binary data into a URL as text. And, you're not handling the asynchronous nature of fetch() properly either. No, this is not the way to do things. You could create a data encoded URL, but there's really no point in doing it that way since whatever audio element you're using the the HTML page can fetch the MP3 from the URL by itself.
I might suggest something simpler in the client:
document.getElementById("audio_src").src = "http://localhost:3000/static/mp3/" + value;
And, let the browser's html tag go get the MP3 for you. I'm assuming that the element represented by audio_src is something that knows how to play MP3 audio sources on it's own. If so, that means you just give it the URL and it will go fetch it and play it on its own.
I want to send multiple files to Github repository via nodejs. Tried several approaches and end up using node-rest-client module. Tried below code send a sample file to repository called 'metadata'. But after post I am getting error message "Request forbidden by administrative rules. Please make sure your request has a User-Agent header"...please let me know if anyone faced this error before and get rid of it.
convertval = "somedata";
var dataObj = {
"message": "my commit message",
"committer": {
"name": "Scott Chacon",
"email": "ravindra.devagiri#gmail.com"
},
"content": "bXkgbmV3IGZpbGUgY29udGVudHM="
}
debugger;
var Client = require('node-rest-client').Client;
var client = new Client()
var args = {
data: dataObj,
headers: { 'Content-Type': 'application/json' },
};
client.post("https://api.github.com/repos/metadata/contents", args, function (data, response) {
console.log("file send: True : " + data);
});
According to the REST API:
All API requests MUST include a valid User-Agent header. Requests with
no User-Agent header will be rejected.
First of all, you need to define 'User-Agent' with value 'request' in your request header. Refer to this link.
Second, endpoint you are trying to call might require authentication. Generate a personal token from here, add that token in your request header, 'Authorization': 'token '.
If you're using Git extensively in your code, I suggest you to use this - Nodegit.
Edit:
I don't think sending multiple files in a single request is possible in 'Contents' endpoints group (link).
You can checkout Git Data API (as discussed here).
I have two URLs (due to security concern i will explain by using dummy)
a> https://xyz.company.com/ui/api/token
b> https://xyz.company.com/request/transaction?date=2016-01-21&token=<tokeninfo>
When you hit url mentioned in point 'a' it will generate a token let it be a string of 16 characters
Then that token should be used in making second request of point 'b' in token param
Updated
The second url response is important to me i.e is a JSON response, I need
to filter the json data and extract required data and output it to standard
output and elastic search.
is there any way of doing so in logstash using plugin "http_poller" or any other plugins.
Note : these request urls should be executed one after another, i.e point 'a' url should be executed first and point 'b' url should be executed next after receiving new token.
Please suggest.
Yes, it's possible with a mix of an http_poller input and an http output.
Here is the config I came up with:
input {
# 1. trigger new token requests every hour
http_poller {
urls => {
token => "https://xyz.company.com/ui/api/token"
}
interval => 3600
add_field => {"token" => "%{message}"}
}
}
filter {
}
output {
# 2. call the API
http {
http_method => "get"
url => "https://xyz.company.com/request/transaction?date=2016-01-21&token=%{token}"
}
}
UPDATE
If you want to be able to get the content of the API call and store it in ES, you need a hybrid solution. You need to set up a cron that will call a script that runs the two HTTP calls and stores the results in a file and then you can let logstash tail that file and forward the results to ES.
Shell script to put on cron:
#!/bin/sh
# 1. Get the token
TOKEN=$(curl -s -XGET https://xyz.company.com/ui/api/token)
# 2. Call the API with the token and append JSON to file
curl -s -XGET "https://xyz.company.com/request/transaction?date=2016-01-21&token=$TOKEN" >> api_calls.log
The above script can be set on cron using crontab (or similar), there are plenty of examples out there on how to achieve this.
Then the logstash config can be very simple. It just needs to tail the api_calls.log file and send the document to ES
input {
file {
path => "api_calls.log"
start_position => "beginning"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "my_index"
document_type" => "my_type"
}
stdout {
codec => "rubydebug"
}
}
I am trying to use logstash to poll the Uber API (just as an example) to get data from an API and (at the moment) just output to console. Cannot seem to get custom headers to work:
input {
http_poller {
urls => {
test2 => {
# Supports all options supported by ruby's Manticore HTTP client
method => get
url => "https://api.uber.com/v1/products?latitude=37.7759792&longitude=-122.41823"
headers => {
Authorization => "P8YegbF53nWPZST5xX0ZlktVnufXYYQa01Dy0ocm"
}
}
}
request_timeout => 60
interval => 60
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
metadata_target => "http_poller_metadata"
}
}
output {
stdout {}
}
If I understand the concept right, that should be the place to put the header but since its not basic auth I cannot use the Auth flag in the sample.
I have gotten the URL of an image using Alamofire and SwityJSON to parse the response obtained from an API. Later, on my code, I am trying to use AlamofireImage to set that image to an ImageView... I am trying to do the following:
let headers = ["Authorization": requestToken, "Content-Type": "image/jpg"]
print(foods[indexPath.row].image)
Alamofire.request(.GET, imageEndPoint, headers: headers)
.responseImage { response in
debugPrint(response)
print(response.request)
print(response.response)
if let image = response.result.value {
print("image downloaded: \(image)")
}
}
However, when debugging, I am getting the following errors
Request]: <NSMutableURLRequest: 0x7fa93e052e60> { URL: http://159.203.92.55:9000/api/media/image?path=./server/uploads/images/products/jOqmy768a5wN2tcPd07cPhVH.jpg }
[Response]: <NSHTTPURLResponse: 0x7fa93e05a0a0> { URL: http://159.203.92.55:9000/api/media/image?path=./server/uploads/images/products/jOqmy768a5wN2tcPd07cPhVH.jpg } { status code: 200, headers {
Connection = "keep-alive";
"Content-Type" = "image/jpg";
Date = "Sun, 17 Jan 2016 16:28:38 GMT";
"Transfer-Encoding" = Identity;
"X-Powered-By" = Express;
} }
[Data]: 26224 bytes
[Result]: FAILURE: Error Domain=com.alamofire.error Code=-1016 "Failed to validate response due to unacceptable content type" UserInfo={NSLocalizedFailureReason=Failed to validate response due to unacceptable content type}
Optional(<NSMutableURLRequest: 0x7fa93e052e60> { URL: http://159.203.92.55:9000/api/media/image?path=./server/uploads/images/products/jOqmy768a5wN2tcPd07cPhVH.jpg })
Obviously there is an error due to the content type but I am not sure how to solve it:
[Result]: FAILURE: Error Domain=com.alamofire.error Code=-1016 "Failed to validate response due to unacceptable content type" UserInfo={NSLocalizedFailureReason=Failed to validate response due to unacceptable content type}
Because of this error, I can't go ahead and just set the image to the imageView
cell.thumbnailImageView.image = myImage
When I try to do the same requests using Chrome's Postman, I get the image without any issues.
Suggestions anyone? thanks
UPDATE: I found a way that is far from what I wanted but works
Alamofire.request(.GET, foods[indexPath.row].image, headers: headers)
.response { (request, response, data, error) in
debugPrint(response)
cell.thumbnailImageView.image = UIImage(data: data!, scale: 1)
}
this is vanilla Alamofire, so I still haven't been able to use AlamofireImage which provides better support for images. I got this method from the following post: How to load image in swift using Alamofire (skywinder's answer)
So I noticed the answer from the server was "Content-Type" = "image/jpg"; but as the user #Gomines pointed out, the correct content type for alamofireImage is actually "Content-Type": "image/jpeg"
Then I went back to AlamofireImages docs and read this
If the image you are attempting to download is an invalid MIME type not in the list, you can add custom acceptable content types using the addAcceptableImageContentTypes extension on the Request type.
So I just had to add this line of code before the AlamofireImage request:
Request.addAcceptableImageContentTypes(["image/jpg"])
The problem was that I was not matching the same content type that the server was responding with. However, that line solves the issue because now AlamofireImage will also accept the same type that the server is sending.
You made a mistake on the content type. It should be image/jpeg (http://www.sitepoint.com/web-foundations/mime-types-complete-list/).
let headers = ["Authorization": requestToken, "Content-Type": "image/jpeg"]
I hope this resolves the problem.