I have record in log file with records in JSON:
{"created":1610314252.6277733,"process":13,"levelname":"INFO","message":"Requesting url: http://smshost:8080/push with method: POST and params {'data': {'message': 'Record changed: go to:XXXX', 'resource_id': '1824908f-efae-474e-aa3e-579cabe67517', 'subscriber': '000000000000', 'service': 'SERVICE'}, 'json': None}","message_id":268562185}
Filebeat send this data to logstash
filebeat.prospectors:
- paths:
- /opt/apps/mobilizer/*.log
input_type: log
json.keys_under_root: true
json.add_error_key: true
json.message_key: log
Output data:
{
"#timestamp":"2021-01-11T09:53:48.062Z",
"#metadata":{"beat":"filebeat","type":"doc","version":"6.8.13"},
"levelname":"INFO",
"source":"/opt/apps/mobilizer/logs/test.json",
"offset":114,
"message":"Requesting url: http://smshost:8080/push with method: POST and params {'data': {'message': 'Record changed: go to:XXXX', 'resource_id': '1824908f-efae-474e-aa3e-579cabe67517', 'subscriber': '000000000000', 'service': 'SERVICE'}, 'json': None}",
"beat":{"name":"skif-HP-240-G7-Notebook-PC","hostname":"skif-HP-240-G7-Notebook-PC","version":"6.8.13"},
"host":{"name":"skif-HP-240-G7-Notebook-PC"},
"message_id":268562185,
"error":{"message":"Key 'log' not found","type":"json"},
"log":{"file":{"path":"/opt/apps/mobilizer/logs/test.json"}},
"process":13,"created":1.6103142526277733e+09
}
But I want to parse the "message" and get the dictionary data after "paran". (Either in logstash or filebeat ):
{
...
"message":"Requesting url: http://smshost:8080/push with method: POST and params ",
"data": {"message": "Record changed: go to:XXXX", "resource_id": "1824908f-efae-474e-aa3e-579cabe67517', "subscriber": "000000000000', "service": "SERVICE"},
"json": "None",
...
}
How can I do this? Where is it better to do in logstash or in filebeat?
Related
I created an application using AWS Amplify, and I want to create an endpoint in an API function that will mutate a field in GraphQL. Since its a nodejs API function, I cannot use the recommended libraries from the AWS documentation which use ES6, since the functions can only use ES5. Therefore I need to use Axios.
I created a graphql query:
const query = /* GraphQL */ `mutation updatePublication($id: ID!, $keywords: String) {
updatePublication(id: $id, keywords: $keywords){
id
keywords
}
}`
Next, I created the Axios request based on this StackOverflow question.
const data = await axios.post(
process.env.API_APPNAME_GRAPHQLAPIENDPOINTOUTPUT,
{
query: query,
variables: {
id: variableWithID,
keywords: "updated keywords!"
}
},
{
headers: {
'Content-Type': 'application/json',
'x-api-key': process.env.API_APPNAME_GRAPHQLAPIKEYOUTPUT
}
}
)
When I run the query, I get a status 200 from the server with the following errors:
data: null,
errors: [
{
path: null,
locations: [Array],
message: "Validation error of type MissingFieldArgument: Missing field argument input # 'updatePublication'"
},
{
path: null,
locations: [Array],
message: "Validation error of type UnknownArgument: Unknown field argument id # 'updatePublication'"
},
{
path: null,
locations: [Array],
message: "Validation error of type UnknownArgument: Unknown field argument keywords # 'updatePublication'"
}
]
}
Can anyone advise on what this means?
I am trying to Build a form submission handler for JS-free form submission.
But the api is not receiving the data sent by the html form.
I am following this documentation.
The signin.json.js file contains an export async function called post that takes in a request parameter and logs it to the console. It then returns a new Response object with the request parameter stringified as JSON and a status of 200.
The index.astro file contains an HTML form with an action of /api/signin.json and a method of post. It has two input fields, one of type text with the name text and a value of test, and the other of type submit.
Upon form submission, the output in the terminal shows the request object that was logged to the console. However, the output in the browser shows a JSON object with several properties, but none of them contain the data that was submitted in the form. It is unclear why the data from the form is not being received by the API. It could be a problem with the form itself, the way the data is being processed by the post function, or something else.
In the Astro JS directory my files are at
/pages/signin.json.js
/pages/index.astro
Code
// signin.json.js
export async function post({request}) {
console.log(request)
return new Response(JSON.stringify(request), {
status: 200,
});
}
// index.astro
---
---
<form action="/api/signin.json" method="post" >
<input type='text' name='text' value='test' />
<input type="submit" />
</form>
Output after submit
Terminal
Request {
size: 0,
follow: 20,
compress: true,
counter: 0,
agent: undefined,
highWaterMark: 16384,
insecureHTTPParser: false,
[Symbol(Body internals)]: {
body: <Buffer 74 65 78 74 3d 61 73 61>,
stream: Readable {
_readableState: [ReadableState],
_read: [Function: read],
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
[Symbol(kCapture)]: false
},
boundary: null,
disturbed: false,
error: null
},
[Symbol(Request internals)]: {
method: 'POST',
redirect: 'follow',
headers: {
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en',
'cache-control': 'max-age=0',
connection: 'keep-alive',
'content-length': '8',
'content-type': 'application/x-www-form-urlencoded',
cookie: 'io=NH8tzNimTmy0AtvFAAAA; asasa=asaa',
host: 'localhost:3000',
origin: 'http://localhost:3000',
referer: 'http://localhost:3000/login',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'same-origin',
'sec-fetch-user': '?1',
'sec-gpc': '1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
},
parsedURL: URL {
href: 'http://localhost:3000/api/signin.json',
origin: 'http://localhost:3000',
protocol: 'http:',
username: '',
password: '',
host: 'localhost:3000',
hostname: 'localhost',
port: '3000',
pathname: '/api/signin.json',
search: '',
searchParams: URLSearchParams {},
hash: ''
},
signal: null,
referrer: undefined,
referrerPolicy: ''
},
[Symbol(astro.clientAddress)]: '::1'
}
Browser
{
"size": 0,
"follow": 20,
"compress": true,
"counter": 0,
"highWaterMark": 16384,
"insecureHTTPParser": false
}
I verified that the form action and method are correct and match the expected endpoint and HTTP verb. In this case, the form action is /api/signin.json and the method is post, which seem to be correct based on the code provided.
that's my config file
// astro.config.mjs
import { defineConfig } from 'astro/config';
import netlify from "#astrojs/netlify/functions";
import svelte from "#astrojs/svelte";
// https://astro.build/config
export default defineConfig({
output: "server",
adapter: netlify(),
integrations: [svelte()]
});
Here's a way to handle forms:
// pages/index.astro
<form action="/api/signin" method="post" >
<input type='text' name='text' value='asa' />
<input type="submit" />
</form>
Now the endpoint
// pages/api/signin.js
export async function post({request}) {
const data = await request.formData(); // Here's the data sent by the form
const text = data.get('text'); // Here's how you get the value of a field
console.log(text);
return new Response(JSON.stringify(request), {
status: 200,
});
}
I have been using NodeJS library for querying the price of the chepeast flight for a route, for next 60 days. Even after switching to production environment, the API throws 404 error (details mentioned below) for some common routes like SFO to YYC.
Do let me know how I can resolve this.
NotFoundError {
response: Response {
headers: {
date: 'Tue, 19 Oct 2021 17:09:32 GMT',
'content-type': 'application/vnd.amadeus+json',
'content-length': '263',
connection: 'close',
'ama-internal-message-version': '14.1',
'ama-request-id': '00010UI9218IBV',
'ama-gateway-request-id': 'rrt-0b7b1036c139b8050-a-eu-4707-50043169-1',
'access-control-allow-headers': 'origin, x-requested-with, accept, Content-Type, Authorization',
'access-control-max-age': '3628800',
'access-control-allow-methods': '*',
server: 'Amadeus',
'access-control-allow-origin': '*'
},
statusCode: 404,
request: Request {
host: 'api.amadeus.com',
port: 443,
ssl: true,
scheme: 'https',
verb: 'GET',
path: '/v1/shopping/flight-dates',
params: [Object],
queryPath: '/v1/shopping/flight-dates?origin=SFO&destination=YYC&departureDate=2021-10-19%2C2021-12-
18&oneWay=true&viewBy=DATE',
bearerToken: '[REDACTED]',
clientVersion: '5.7.0',
languageVersion: '14.16.1',
appId: null,
appVersion: null,
headers: [Object]
},
body: '{"errors":[{"status":404,"code":6003,"title":"ITEM/DATA NOT FOUND OR DATA NOT EXISTING","detail":"No price result found"},{"status":404,"code":6003,"title" :"ITEM/DATA NOT FOUND OR DATA NOT EXISTING","detail":"No price results found for input combined criteria"}]}',
result: { errors: [Array] },
data: undefined,
parsed: true
},
description: [
{
status: 404,
code: 6003,
title: 'ITEM/DATA NOT FOUND OR DATA NOT EXISTING',
detail: 'No price result found'
},
{
status: 404,
code: 6003,
title: 'ITEM/DATA NOT FOUND OR DATA NOT EXISTING',
detail: 'No price results found for input combined criteria'
}
],
code: 'NotFoundError'
}
My code to query the data looks something like this:
const Amadeus = require("amadeus");
var client = new Amadeus({
hostname: "production",
clientId: FLIGHT_SEARCH_API_KEY,
clientSecret: FLIGHT_SEARCH_API_SECRET,
});
const response = await client.shopping.flightDates.get({
origin,
destination,
departureDate: `${currentDate},${lastDate}`,
oneWay: true,
viewBy: "DATE",
});
The Flight Inspiration Search & Flight Cheapest Date Search APIs are built on top of a pre-computed cache. The APIs compute every day the most trending options based on past searches and bookings and fill the cache, which means that the cache is dynamic. If you need to get access to a full inventory of Amadeus you need to use the live Flight Offers Search API.
am using emailjs to send mails in nodejs but the problem is, the callback is called twice, output is:
mail test
sending status...
err { Error: no connection has been established
at module.exports (D:\tests\node_modules\emailjs\smtp\error.js:33:14)
at SMTP.send (D:\tests\node_modules\emailjs\smtp\smtp.js:377:5)
at SMTP.command (D:\tests\node_modules\emailjs\smtp\smtp.js:414:8)
at SMTP.helo (D:\tests\node_modules\emailjs\smtp\smtp.js:428:8)
at ehlo (D:\tests\node_modules\emailjs\smtp\smtp.js:635:11)
at caller (D:\tests\node_modules\emailjs\smtp\smtp.js:76:12)
at command (D:\tests\node_modules\emailjs\smtp\smtp.js:507:5)
at caller (D:\tests\node_modules\emailjs\smtp\smtp.js:76:12)
at response (D:\tests\node_modules\emailjs\smtp\smtp.js:397:5)
at caller (D:\tests\node_modules\emailjs\smtp\smtp.js:76:12) code: 6, smtp: undefined, previous: null }
message Message {
attachments: [],
alternative:
{ data: '<html>i <i>hope</i> this works! </html>',
alternative: true,
charset: 'utf-8',
type: 'text/html',
inline: true },
header:
{ 'message-id': '<1535820394667.0.14872#Ali-pc>',
date: 'Sat, 01 Sep 2018 19:46:34 +0300',
from: 'portal#hudumakenya.go.ke',
to: 'hamisi.yusuf#ekenya.co.ke',
subject: '=?UTF-8?Q?testing_emailjs?=' },
content: 'text/plain; charset=utf-8',
text: 'i hope this works' }
sending status...
err { Error: timedout while connecting to smtp server
at module.exports (D:\tests\node_modules\emailjs\smtp\error.js:33:14)
at TLSSocket.timedout (D:\tests\node_modules\emailjs\smtp\response.js:65:5)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:106:13)
at TLSSocket.emit (events.js:208:7)
at TLSSocket.Socket._onTimeout (net.js:410:8)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5) code: 4, smtp: undefined, previous: null }
message Message {
attachments: [],
alternative:
{ data: '<html>i <i>hope</i> this works! </html>',
alternative: true,
charset: 'utf-8',
type: 'text/html',
inline: true },
header:
{ 'message-id': '<1535820394667.0.14872#Ali-pc>',
date: 'Sat, 01 Sep 2018 19:46:34 +0300',
from: 'portal#hudumakenya.go.ke',
to: 'hamisi.yusuf#ekenya.co.ke',
subject: '=?UTF-8?Q?testing_emailjs?=' },
content: 'text/plain; charset=utf-8',
text: 'i hope this works' }
and this is my code
var server = email.server.connect({
user: "",
password:"",
host: "",
tls: {ciphers: "SSLv3"},
ssl: false,
port: 587,
authentication: [ 'LOGIN' ],
// timeout: 20000
});
var message = {
text: "i hope this works",
from: "portal#hudumakenya.go.ke",
to: "hamisi.yusuf#ekenya.co.ke",
subject: "testing emailjs",
attachment: [
{
data: "<html>i <i>hope</i> this works! </html>",
alternative: true
}
]
};
console.log('mail test');
server.send(message, function(err, message) {
console.log('sending status...');
console.log();
console.log('err', err);
console.log('message', message);
});
the console.log('mail test') is just to check if the file is executed twice but turns out its called only once but the callback is called twice. Even more strange and confusing is the fact that even though its producing those errors, its still sending the email, can anyone help me out here because this is now driving me crazy, thanks.
EDIT
The package am using is emailjs
It seems CouchDB will randomly throw an error when trying to insert a user.
Here is the full response from CouchDB:
{ [Error: missing]
name: 'Error',
scope: 'couch',
status_code: 404,
'status-code': 404,
request:
{ method: 'POST',
headers:
{ 'content-type': 'application/json',
accept: 'application/json',
'X-CouchDB-WWW-Authenticate': 'Cookie',
cookie: [Object] },
uri: 'http://127.0.0.1:5984/_users',
body: '{"_id":"org.couchdb.user:fake_user","name":"fake_user","realname":"fake user","institution":"366a8e5ba861bdd9cad5cd318a002ee4","email":"fake_user#gmail.com","phone":"123-456-7890","type":"user","roles":[],"password":"123456","level":"user","unconfirmed":"true"}',
jar: false },
headers:
{ date: 'Tue, 21 Jan 2014 20:32:23 GMT',
'content-type': 'application/json',
'cache-control': 'must-revalidate',
'status-code': 404,
uri: 'http://127.0.0.1:5984/_users' },
errid: 'non_200',
error: 'not_found',
reason: 'missing',
description: 'missing',
stacktrace:
[ 'Error: missing',
' at Request._callback (/apps/arcapp/node_modules/nano/nano.js:304:39)',
' at Request.self.callback (/apps/arcapp/node_modules/nano/node_modules/request/request.js:129:22)',
' at Request.EventEmitter.emit (events.js:98:17)',
' at Request.<anonymous> (/apps/arcapp/node_modules/nano/node_modules/request/request.js:873:14)',
' at Request.EventEmitter.emit (events.js:117:20)',
' at IncomingMessage.<anonymous> (/apps/arcapp/node_modules/nano/node_modules/request/request.js:824:12)',
' at IncomingMessage.EventEmitter.emit (events.js:117:20)',
' at _stream_readable.js:872:14',
' at process._tickCallback (node.js:415:13)' ] }
This is obviously not very insightful. All I get is error: not_found, reason: missing, which doesn't tell me much. It only happens sometimes and I can't really tell what's triggering it. Any insight into this is much appreciated.
Here is the relevant code. I'm using nano, but it's just a basic post request to CouchDB.
var user = {
"_id": "org.couchdb.user:" + username,
"name": username,
"realname": first + " " + last,
"institution": institution,
"email": email,
"phone": phone,
"type": "user",
"roles": [],
"password": password,
"level": "user",
"unconfirmed": "true",
"verificationCode": verificationCode
};
nano.request({
db: '_users',
method: 'POST',
body: user
}, function(err, body) {
if(err) {
console.log("ERROR INSERTING USER");
console.log(err);
res.send(err.status_code + " " + err.reason, err.status_code);
return;
}
...
The problem here was that the cookie I was using to validate a user was not being properly unset after logout. So if you logged out and immediately tried to register, the database would authenticate the request as the logged in user. Since only db administrators are allowed access to the _users db, CouchDB would throw a '404 missing' error.
I was setting nano.config.cookie on login, so when I unset it on logout, it worked!
This error rings a bell but unfortunately I'm not 100% how/if I fixed it.
One thing I did notice you are including "roles:[]" I dont think this is allowed unless your post as an admin.
I haven't enough rep to comment but can you send the code that's making the request ?