how to reuse the output of logstash - logstash

I am using http-filter plugin in filter-stage
filter {
http {
url => "someUrl"
verb => "GET"
target_body => "apiResponse"
target_headers => "apiResponseHeader"
}
}
and I want to send the result of http filter to kafka, but when http filter got error, it gives me error as below
(below error is intentionally caused: 404)
[2023-02-09T14:47:34,517][ERROR][logstash.filters.http ][main][45dd1acfcf38ba5088de03ee36672bc4b8f8046ff3853965393fa1c6b80f4cb8] error during HTTP request {:url=>"baseUrl", :code=>404, :response=>"{\"timestamp\":\"2023-02-09T14:47:34.507+00:00\",\"status\":404,\"error\":\"Not Found\",\"path\":\"someWrongUrl"}"
now, i want to re-process(parse) above error to
url: baseUrl
code: 404
is there any way to re-process the output or catch the error message in filter-stage?
thanks

Related

How to catch invalid url error in got module get stream?

I'm trying to using got module to get a file from a given url, but when the url is invalid, i can't catch the error with the error handler.
My code:
got.stream('https://google.com').on('error', error =>
{
console.log('error!!!')
})

How to return HTTP error response in TypeMoq

I'm trying to write a TypeScript unit test using TypeMoq that mocks an HTTP request and returns an error response. When returning a basic object as expected there is no problem, but when trying to return an HTTP error response then the test always fails due to an exception being thrown.
How can I write a mock setup using TypeMoq that returns an HTTP error response and doesn't throw an exception? If there is an HTTP response error code, then I want to set the "component.SomeProperty" property.
mock.setup(x => x.getHttpData()).throws(() => new Error('error'));
expect(component.SomeProperty).toBe('someValue');
Just tackled the same issue.
Got it working by returning a throwError():
import { throwError } from 'rxjs';
...
mock.setup(x => x.getHttpData()).returns(() => throwError(new Error('error')));

send error message back to browser in nodejs

I have a node API that is working fine when tested using postman.
But when I use this API in my angular project there occurs an error and browser don't get any response there it keep waiting for a response. When I go to console I see the error message.
How I can make that error message to be sent back to the browser with full stack trace
In general, you will need to catch that error, then populate http response object with it just the same as if you were sending successful response data back to the requestor.
Synchronous processing:
try {
// do my requested stuff
res.status(200).json({something:"returned"});
} catch(ex) {
res.status(500).json(ex);
};
Promises:
Promise.resolve()
.then(() => {
// do my requested stuff
// return my results stuff to the client
res.status(200).json({something:"returned"});
})
.catch((ex) => {
// return 500 error and exception data to the client
res.status(500).json(ex);
});
Also, as standard practice you should catch all errors, and at the very least, you should return a 500 to the browser res.status(500) so you don't leave it hanging when unexpected issues arise.
And, of course you can return html rather than json, and/or more info in the response.
Good luck.

What's the correct way of returning HTTP error codes in serverless lambda

I have a serverless lambda function written in Node.JS.
What is the best / correct way of returning error codes?
The pattern that I use right now (and it works!) is:
module.exports.endpoint = (event, context, callback) => {
const response = {
statusCode: 404,
body: JSON.stringify({ message: 'Hello World!' })
};
callback(null, response);
}
When I make a call, for example from POSTMAN, to my endpoint I get:
Status: 404 Not Found which is exactly what I'm expecting.
Moreover, in the logs I can see:
Serverless: GET / (λ: get)
Serverless: [404] {"statusCode":404,"body":"{\"message\":\"Hello World!\"}"}
That works well.
What bothers me is that I'm passing null as the error. Looking into a few other tutorials/examples I've found patterns like:
https://aws.amazon.com/blogs/compute/error-handling-patterns-in-amazon-api-gateway-and-aws-lambda/
https://serverless.com/framework/docs/providers/aws/events/apigateway/
callback ("the sky is falling!");
callback("[BadRequest] Validation error: Missing field 'name'");
callback("[404] Not Found");
callback(new Error('[404] Not found'));
callback(JSON.stringify(myErrorObj));
All of them make perfect sense and you can specify HTTP Status Code - yet what I'm getting is HTTP Status Code 200 in the end. When I look at the logs I can see that the error was followed straight after with 200:
Serverless: GET / (λ: get)
Serverless: Failure: the sky is falling!
Serverless: Replying 200
Serverless: GET / (λ: get)
Serverless: Failure: [BadRequest] Validation error: Missing field 'name'
Serverless: Replying 200
Serverless: GET / (λ: get)
Serverless: Failure: [404] Not Found
Serverless: Replying 200
Serverless: GET / (λ: get)
Serverless: Failure: [404] Not found
Serverless: Replying 200
Serverless: GET / (λ: get)
Serverless: Failure: {"errorType":"InternalServerError","httpStatus":500,"message":"An unknown error has occurred. Please try again."}
Serverless: Replying 200
In this place I've found following explanation:
https://github.com/serverless/serverless/issues/4119
If you want to respond with HTTP errors in that case, you have to
encode the HTTP error as successful Lambda response
with following example:
Sample 403:
callback(null, { statusCode: 403, body: "Forbidden", headers: { "Content-Type": "text/plain" } });
Sample 404:
callback(null, { statusCode: 400 });
So that's basically the same way I've. For the sake of completeness, I can add that there is also a plenty of examples that uses context.fail(result) or context.succeed(result) - but from what I gathered context is deprecated and shouldn't be used (even though it still works).
What is the point of using callback(error)?
If you want to respond with HTTP errors in that case, you have to encode the HTTP error as successful Lambda response
This type of error-handling is specific to API Gateway.
Just like in traditional Node web servers (e.g. express), you can throw any error using throw new Error('Invalid Payload') and the middleware usually converts it into an HTTP response with the correct response status.
In API Gateway Lambda, this can be written like this...
function createResponse(status, body) {
return {
headers: {
'Access-Control-Allow-Origin': '*'
},
statusCode: status,
body: JSON.stringify(body)
}
}
module.exports.endpoint = (event, context, callback) => {
try {
return callback(null, createResponse(200, processEvent(event)))
} except (e) {
console.error(e)
return callback(null, createResponse(500, {
error: 'Internal Server Error'
}))
}
}
Basically, it's a handled error. The lambda function succeeded but the request failed (maybe 400, 404, or 500).
You should be handling errors always, or else if your handler crashes (due to a runtime error or syntax error or any unhandled error), your user will get an unexpected response (a 500 or a 502) which you probably don't want.
Please remember that Lambda is not only used for API Gateway. callback(error) is used for non-API Gateway-triggered Lambdas.
For example, if you have an SNS-triggered Lambda, you can return callback('Any error message here') and it will let SNS know that it failed and so SNS can retry the invocation.

Logstash output-http return code

How to manage logstash output-http return codes? I'm using logstash-output-http to post data to a restful webservice. Since http may result in application failure (code >= 400) how can i handle this scenario via logstash ?
my configuration is the following:
output {
http {
url => "http://bla:8080/restful/endpoint"
http_method => "post"
format => message
content_type => "application/json; charset=UTF-8"
message => '{"json":"here"}'
}
}
suppose you have a code = 500 and you want to eventually deliver such message, how you would do that?
The code seems to imply that you can look in your log file.
There's an issue to retry them.

Resources