Apollo Server timeout while waiting for stream data - node.js

I'm attempting to wait for the result of a stream with my Apollo Server. My resolver looks like this.
async currentSubs() {
try {
const stream = gateway.subscription.search(search => {
search.status().is(braintree.Subscription.Status.Active);
});
const data = await stream.pipe(new CollectObjects()).collect();
return data;
} catch (e) {
console.log(e);
throw new Meteor.Error('issue', e.message);
}
},
This resolver works just fine when the data stream being returned is small, but when the data coming in is larger, I'm getting a 503 (Service Unavailable). I looks like the timeout is happening around 30 seconds. I've tried increasing the timeout of my Express server with graphQLServer.timeout = 240000; but that hasn't made a difference.
How can I troubleshoot this & where is the 30 second timeout coming from? It only fails when the results take longer.
I'm using https://github.com/mrdaniellewis/node-stream-collect to collect the results from the stream.
Error coming in from the try catch:
I20180128-13:09:26.872(-7)? { proxy:
I20180128-13:09:26.872(-7)? { error: 'Post http://127.0.0.1:26474/graphql: net/http: request canceled (Client.Timeout exceeded while awaiting headers)',
I20180128-13:09:26.872(-7)? level: 'error',
I20180128-13:09:26.873(-7)? msg: 'Error sending request to origin.',
I20180128-13:09:26.873(-7)? time: '2018-01-28T13:09:26-07:00',
I20180128-13:09:26.873(-7)? url: 'http://127.0.0.1:26474/graphql' } }

Had this same problem and was a pretty simple solution. My calls were lasting a bit over 30 seconds and the default timeout was returning 503s as well so I increased that.
Assuming you're using apollo-engine (this may be true for some other forms of Apollo), you can set your engine configs like so:
export function startApolloEngine() {
const engine = new Engine({
engineConfig: {
stores: [
{
name: "publicResponseCache",
memcache: {
url: [environmentSettings.memcache.server],
keyPrefix: environmentSettings.memcache.keyPrefix
}
}
],
queryCache: {
publicFullQueryStore: "publicResponseCache"
},
reporting: {
disabled: true
}
},
// GraphQL port
graphqlPort: 9001,
origin: {
requestTimeout: "50s"
},
// GraphQL endpoint suffix - '/graphql' by default
endpoint: "/my_api_graphql",
// Debug configuration that logs traffic between Proxy and GraphQL server
dumpTraffic: true
});
engine.start();
app.use(engine.expressMiddleware());
}
Notice the part where I specify
origin: {
requestTimeout: "50s"
}
That alone is what fixed it for me. Hope this helps!
You can find more information about that here

Related

Cloud Run Readablestream takes 5 minutes to cancel enqueue

I created an adapter-node Sveltekit API endpoint, which streams quotes using a readable stream. When I quit the client route The streaming has to stop. This works fine in development using Sveltekit "npm run dev" (vite dev) or using a windows desktop container (node build).
onDestroy(async () => {
await reader.cancel(); // stop streaming
controller.abort(); // signal fetch abort
});
But when I build and deploy the node container on Google Cloud Run the streaming works fine. Except when I quit the client route: the API endpoint keeps on streaming. The log shows: enqueus for 5 more minutes followed by a delayed Readablestream cancel() on the API server.
Why this 5 minutes between the client cancel / abort and the cancel on the server?
The API +server.js
import { YahooFinanceTicker } from "yahoo-finance-ticker";
/** #type {import('./$types').RequestHandler} */
export async function POST({ request }) {
const { logging, symbols } = await request.json();
const controller = new AbortController();
const ticker = new YahooFinanceTicker();
ticker.setLogging(logging);
if (logging) console.log("api ticker", symbols);
const stream = new ReadableStream({
start(controller) {
(async () => {
const tickerListener = await ticker.subscribe(symbols);
tickerListener.on("ticker", (quote) => {
if (logging) console.log("api", JSON.stringify(quote, ["id", "price", "changePercent"]));
controller.enqueue(JSON.stringify(quote, ["id", "price", "changePercent"]));
});
})().catch(err => console.error(`api listen exeption: ${err}`));
},
cancel() { // arrives after 5 minutes !!!
console.log("api", "cancel: unsubscribe ticker and abort");
ticker.unsubscribe();
controller.abort();
},
});
return new Response(stream, {
headers: {
'content-type': 'text/event-stream',
}
});
}
Route +page.svelte
const controller = new AbortController();
let reader = null;
const signal = controller.signal;
async function streaming(params) {
try {
const response = await fetch("/api/yahoo-finance-ticker", {
method: "POST",
body: JSON.stringify(params),
headers: {
"content-type": "application/json",
},
signal: signal,
});
const stream = response.body.pipeThrough(new TextDecoderStream("utf-8"));
reader = stream.getReader();
while (true) {
const { value, done } = await reader.read();
if (logging) console.log("resp", done, value);
if (done) break;
... and more to get the quotes
}
} catch (err) {
if (!["AbortError"].includes(err.name)) throw err;
}
}
...
The behavior you are observing is expected, Cloud Run does not support client-side disconnects yet.
It is mentioned in this article, that
Cloud Run (fully managed) currently only supports server-side
streaming. Having only "server-side streaming" basically means when
the "client" disconnects, "server" will not know about it and will
carry on with the request. This happens because "server" is not
connected directly to the "client" and the request from the "client"
is buffered (in its entirety) and then sent to the "server".
You can also check this similar thread
It is a known issue, there is already a public issue exists for the same. You can follow that issue for future updates and also add your concerns there.

Performance issue nodejs connecting to Oracle Database by node-oracledb

I am facing the performance issue during load testing of the Rest API for Oracle Database developed by using Express and OracleDB nodejs modules.
I see the decrease in performance during load testing during increasing number of the requests per seconds to the developed API.This issue is reproduced on stored procedure and standard select requests to the database.
From database side I see the standard and stable response time for each of the requests. Requests are recieved by the database in the same time as they were initiated.
From application(reponse) side it looks like responses being puted in some kind queue and being readed by nodejs(OracleDB module) in packages.
With a standard response time in 0,5 seconds per requests, I might recieve 3-5 seconds response time on 10 request per second.
This issues is reproduced on different connection pool sizes(grater than number of requests per seconds).
At the moment I am stuck with possible options that might lead to the issue. What might be the reason for such behaviour? Or what options for diagnostic or turning is available for oracleDB module for the nodejs?
some code bellow:
creating the connection:
const init = async () => {
try {
Logger.info(`oracle instant client address ${process.env.LD_LIBRARY_PATH}`);
const pool = await oracledb.createPool(dbCredentials);
Logger.info('db connections pool created');
return pool;
} catch (err) {
Logger.error(`init() error: ${err.message}`);
}
};
let pool = await init();
route:
router.get('/test', async (req, res, next) => {
try {
const result = await testController(pool);
sendResponse(res, 200, result);
} catch (e) {
sendErrResponse(res, 500, e.message);
}
});
controller:
const testController = async (pool) => {
let connection;
try {
connection = await pool.getConnection();
} catch (error) {
return error;
}
try {
let items = { items: [{ itemSKU: 'xxxxx', itemQTY: 1234 }, { itemSKU: 'yyyyy', itemQTY: 123}] };
items = JSON.stringify(items);
const { rows: data } = await connection.execute(
'select shop_id, prod_id, qnt from TABLE(pkg_ecom.check(:items))',
{ items },
);
return data;
} catch (error) {
return error;
} finally {
if (connection) {
try {
await connection.close();
} catch (err) {
console.error(err);
}
}
}
};
The common problem with scaling node-oracledb connection load is Node.js thread starvation. Increase the value of the environment variable UV_THREADPOOL_SIZE before your application starts. See the documentation Connections, Threads, and Parallelism. On Linux, your package.json file might have:
"scripts": {
"start": "export UV_THREADPOOL_SIZE=10 && node index.js"
},
. . .
You can monitor pool usage by setting the enableStatistics attribute during pool creation and then calling pool.getStatistics() or pool.logStatistics(), see Connection Pool Monitoring. Look out for too many getConnection() requests being queued.
You should also read the node-oracledb case study Always Use Connection Pools — and How.

NodeJS to NodeJS http requests hang

I have 2 services: 1. worker 2.fetch documents
Both are Nodejs express services.
The worker uses Promise.all to send 48 http POST (dont ask why post and not GET) requests to the fetch service, the fetch service then fetches all 48 documents (each in a separate request) and sends them back to the worker.
Logs are showing the fetch service finished everything successfuly and the worker sending all 48 requests, but showing partial responses (sometimes 0, sometimes 15/48 sometimes 31/48 but never fully succeeds)
The worker keeps waiting for a response it seems, untill the 15 minute timeout for the job end and its moved to failed.
code examples:
worker.js (service 1 - NodeJS with express)
await Promise.all(docIDs.map(async (docID) => {
try {
logger.info(`Worker Get Document: Fetching ${docID.docID} transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
var document = await axios.post(getVariableValue("GET_DOCUMENT_SERVICE_URL"), docID, {
httpsAgent: new https.Agent({
rejectUnauthorized: false,
keepAlive: false
}),
auth: {
username: getVariableValue("WEBAPI_OPTIDOCS_SERVICE_USERNAME"),
password: getVariableValue("WEBAPI_OPTIDOCS_SERVICE_PASSWORD")
},
headers: {
"x-global-transaction-id": transactionId,
"timeStamp": timeStamp
}
});
logger.info(`Worker Get Document: Fetched ${docID.docID} Status: ${document.data.status}. transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
documents.push(document.data.content);
}
catch (err) {
const responseData = err.response ? err.response.data : err.message
logger.error(`Worker Get Document: Failed DocID ${docID.docID}, Error received from Get Document: ${responseData} - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
throw Error(responseData)
}
}));
fetch.js (service 2 - NodeJS with express)
module.exports = router.post('/getDocument', async (req, res, next) => {
try {
var transactionId = req.headers["x-global-transaction-id"]
var timeStamp = req.headers["timestamp"]
logger.info(`GetDocumentService: API started. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
var document = await getDocuemntService(req.body, transactionId, timeStamp);
var cloneDocument = clone(document)
logger.info(`GetDocumentService: Document size: ${JSON.stringify(cloneDocument).length / 1024 / 1024}. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
logger.info(`GetDocumentService: API Finished. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
res.status(200);
res.set("Connection", "close");
res.json({
statusDesc: "Success",
status: true,
content: cloneDocument
});
logger.info(`GetDocumentService: Response status: ${res.finished} . - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
}
catch (err) {
logger.error(`GetDocumentService: Error:${err.message} / Stack: ${err.stack}. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
res.status(500)
res.send("stack: " + err.stack + "err: " + err.message + " fromgetdocument")
}
});
So the logs from the get document are full and show success.
The logs from the worker show like this:
"fetching" X48
"fetched" X0, X15, X31 (three different attempts at fetching 48 docs)
I have tried changing keep-alive to true.
Anything else I might be missing? anyone knows why it hangs forever (atleast 15 minutes untill job gets "timed out")
Thanks :)
I only have guesses here but I feel they are strong guesses.
Axios has a default timeout of none but I'm gonna guess somewhere there is a timeout of 900 seconds. Edit: This timeout is very likely in the server side web server.
You are hammering the server with 48 request in a fraction of a second. I'm not surprised this is failing.
Their server is likely rate limiting you or just getting overloaded.
For testing limit your request to one at a time. When one finishes send the next.
Simple work queue example - Not tested
The addWork or doWork caller will need to await
let queue = []
const sleep = (ms)=> new Promise((resolve)=>setTimeout(resolve,ms))
async addWork(work, cb, sleepMS) {
if (Array.isArray(work) queue = [...queue, ...work]
else queue.push(work)
if (cb) return doWork(cb, sleepMS)
}
async doWork(cb, sleepMS) {
let work = queue.pop()
var document = await getDocuemntService(work)
cb(document)
if (queue.length > 0) {
if (sleepMS) await sleep(sleepMS)
return doWork(cb, sleepMS)
}
}
Once you confirm that one at a time works then you can look at a more complex work queue. I'd check NPM for something I'm sure there is something out there to handle this.

How do I output view compiler errors generated from Hapi.js views?

Coming from the world of express, if there was an error when compiling a Jade template, the error would output to the browser with details of the template error including line number. I would like to see this level of error detail in Hapi.js when the view template compiler encounters an error.
Instead, with Hapi.js, I receive a 500 Internal Server Error instead. The only output I see in the logs is the following:
150511/005652.910, [response], http://localhost:3000: get /abc123 {} 500 (24ms)
This is the basics of my setup.
var Hapi = require('hapi');
var server = new Hapi.Server();
server.connection({ port: 3000 });
server.views({
engines: { jade: require('jade') },
path: __dirname + '/views',
compileOptions: {
pretty: true,
debug: true,
compileDebug: true
}
});
server.route({
method: 'GET',
path: '/{name}',
handler: function (request, reply) {
reply.view('page', {name: request.params.name});
}
});
// I know putting my plugins in an array is not necessary but I like it.
var plugins = [{
register: Good,
options: {
reporters: [{
reporter: require('good-console'),
events: {
response: '*',
log: '*'
}
}]
}
}];
server.register(plugins, function (err) {
if (err) {
throw err; // something bad happened loading the plugin
}
server.start(function () {
server.log('info', 'Server running at: ' + server.info.uri);
});
});
Had the same problem this week too.
var server = new Hapi.Server({
debug: {
request: ['error']
}
});
It won't output to the client (you'll still get a 500 error), but the compilation error will show up in the terminal console.
One possible solution is to log on server 'request-error'. For instance:
server.on('request-error', function (request, err) {
//logs the object
server.log('error', err);
//logs the view compiler error line number and details
server.log('error', err.toString());
});
I would still prefer to see this in the browser (while in "development mode") in addition to the logs.
Because the rendering of the template happens after onPreResponse, it's not possible to catch the error at that point. A way of sending the error to the browser, albeit slightly hacky, is to do a dry run of the compilation inside an extension point, and then transmit the error to the browser at that point:
server.ext('onPreResponse', function (request, reply) {
var response = request.response;
if (response.variety === 'view') {
var source = response.source;
// Let's pre-render the template here and see if there's any errors
return server.render(source.template, source.context, function (err) {
if (err) {
return reply(err.message); // transmit the compile error to browser
}
reply.continue();
});
}
reply.continue();
});
Obviously this has a performance impact because you're rendering the view twice under normal conditions, so you'll want to disable it in production.

NodeJS How To Confirm MongoDB Driver Is Not Bkocking

I've asked some general questions around this topic before (node and blocking). This time the question is a little more specific.
Let's say I've got a node/express app which has a handle that is accepting HTTP requests (doesn't matter, say they're simple GETs).
And it has a separate handler which reads messages off of a RabbitMQ queue, as they arrive, and then does a read from Mongo (Mongo is on a different machine), followed by a write.
If Mongo was "very" busy, would/could that cause the HTTP handler to appear unavailable?
I'm using the Mongo native driver. I would think any blocking that is occurring while the Mongo driver waits for a response from the server would have Node happily accepting and handling HTTP requests, but I don't know for sure.
In a related scenario, swap-out a busy Mongo for a handler that reads a Rabbit message and PUTs a record into a "very" busy ElasticSearch. Will that cause issues with the HTTP handler?
I'd go straight to testing it, but that's a little tricky and gets expensive testing every time I'm not sure what the theory is. So I thought I'd ask.
Here's a (simplified) example of the code:
// HTTP handler...
app.post('/eventcapture/event', (req: express.Request, res: express.Response) => {
var evt: eventDS.IEvent = ('TypeID' in req.body) ? req.body : JSON.parse(req.body);
//create an id
evt._id = uuid.v4();
bus.Publish(evt)
.then((success) => {
res.jsonp(200, { success: true });
})
.catch((failReason:Error) => {
console.error('[ERROR] - Failure writing event: %s,%s', failReason.name, failReason.message);
logError(failReason, evt);
res.jsonp(500, { success: false, reason: failReason });
});
});
// We generically define additional handlers in an array, and then kick them off with a loop.
// Here we have one handler which reads an event, goes to mongo to get additional data which
// it adds into the event before publishing it back out. And a second handler which will catch
// these "augmented" events and push them into Mongo
var processes = [
{
enabled: true,
name: 'augmenter',
inType: 'EventCapture:RawEvent',
handler: (event: eventDS.IEvent) => {
console.log('[LOG] - augment event: %s', event._id);
Profile.FindOne({ _id: event.User.ProfileID })
.then((profile) => {
if (profile) {
console.log('[LOG] - found Profile: %s', profile._id);
event.User.Email = profile.PersonalDetail.Email;
//other values also...
//change the TypeID for publishing
event.TypeID = 'EventCapture:AugmentedEvent';
return event;
}
else throw new Error(util.format('unable to find profile: %s', event.User.ProfileID));
})
.then((augmentedEvent) => bus.Publish(augmentedEvent)) //publish the event back out
.catch((failReason:Error) => {
console.error('[ERROR] - failure publishing augmented event: %s, %s, %s', event._id, failReason.name, failReason.message);
logError(failReason, event);
});
}
},
{
enabled: true,
name: 'mongo',
inType: 'EventCapture:AugmentedEvent',
handler: (event: eventDS.IEvent) => {
console.log('[LOG] - push to mongo: %s', event.User.ProfileID);
Event.Save(event, { safe: true })
.then((success) => console.log('[LOG] - pushed to mongo: %s', event._id))
.catch((failReason:Error) => {
console.error('[ERROR] - failure pushing to mongo: %s, %s', event._id, failReason);
logError(failReason, event);
});
}
}
];
processes.forEach((process, idx, allProcesses) => {
if (process.enabled) {
bus.Subscribe(process.name, process.inType, process.handler);
}
});
No. This is the awesomeness of async programming. Node can do other things while it waits for mongodb to get back to it. You can assume that popular node modules like mongodb write things in an async fashion.
Here's a video that goes into a lot of detail about the event loop: http://vimeo.com/96425312?utm_source=nodeweekly&utm_medium=email
At the end of the day, things like the mongo driver are written using node's low level io and network libraries. These libraries enforce async flow. The author of a package would have to go out of her way to make it sync.

Resources