Nodejs Elastic benastalk refused to connect to upsteam/ upsteam prematurely closed - node.js

I am getting the following errors when running my application in elastic beanstalk: [error] 3636#0: *295 upstream prematurely closed connection while reading response header from upstream and [error] 3636#0: *295 connect() failed (111: Connection refused) while connecting to upstream Its strange because if I hit those routes independently it works fine. It only appears to error when firing those routes from my vuex action.
The following is the log from the AWS elastic beanstalk.
The following is the network tab when it hits my FFmpeg route:
The following is the generate video action as fired from vuex.
async [GENERATE_VIDEO]({state, rootState, dispatch, commit}){
const username = rootState.user.currentUser.username;
const s3Id = rootState.templates.currentVideo.stock_s3_id;
const type = rootState.dataClay.fileFormat || state.type;
const vid = new Whammy.fromImageArray(state.captures, 30);
vid.lastModifiedDate = new Date();
vid.name = "canvasVideo.webm";
const data = new FormData();
const id = `${username}_${new Date().getTime()}`;
data.append("id", id);
data.append("upload", vid);
const projectId = await dispatch(INSERT_PROJECT);
await dispatch(UPLOAD_TEMP_FILE, data);
const key = await dispatch(CONVERT_FILE_TYPE, { id, username, type, projectId});
const role = rootState.user.currentUser.role;
state.file = `/api/files/${key}`;
let message;
if(role!='banner'){
message =`<p>Your video is ready.</p> Download`;
} else {
message = `<p>Your video is ready. You may download your file from your banner account</p>`;
const resolution = rootState.dataClay.matrix[0];
await dispatch(EXPORT_TO_BANNER, { s3Id, fileUrl: key, extension: `.${type}`, resolution});
}
And here are the api routes called in the actions.
async [UPLOAD_TEMP_FILE]({ commit }, data) {
try {
const response = await axios.post("/api/canvas-editor/upload-temp", data);
return response.data;
} catch (error) {
console.log(error);
}
},
async [CONVERT_FILE_TYPE]({commit}, data) {
try{
const response = await axios.post("/api/canvas-editor/ffmpeg", data);
return response.data;
} catch(error){
console.log(error);
}
}
}
As I said all my routes work and the application runs as expected on localhost however when uploaded to aws I receive unexpected errors.

After some digging I found out that I did not set the ffmpeg path.
Once this was done it worked great.
const ffmpeg = require('fluent-ffmpeg');
const ffmpegPath = require('#ffmpeg-installer/ffmpeg').path;
ffmpeg.setFfmpegPath(ffmpegPath);
module.exports = ffmpeg;

Related

Delivering image from S3 to React client via Context API and Express server

I'm trying to download a photo from an AWS S3 bucket via an express server to serve to a react app but I'm not having much luck. Here are my (unsuccessful) attempts so far.
The Workflow is as follows:
Client requests photo after retrieving key from database via Context API
Request sent to express server route (important so as to hide the true location from the client)
Express server route requests blob file from AWS S3 bucket
Express server parses image to base64 and serves to client
Client updates state with new image
React Client
const [profilePic, setProfilePic] = useState('');
useEffect(() => {
await actions.getMediaSource(tempPhoto.key)
.then(resp => {
console.log('server resp: ', resp.data.data.newTest) // returns ����\u0000�\u0000\b\u0006\
const url = window.URL || window.webkitURL;
const blobUrl = url.createObjectURL(resp.data.data.newTest);
console.log("blob ", blobUrl);
setProfilePic({ ...profilePic, image : resp.data.data.newTest });
})
.catch(err => errors.push(err));
}
Context API - just axios wrapped into its own library
getMediaContents = async ( key ) => {
return await this.API.call(`http://localhost:5000/${MEDIA}/mediaitem/${key}`, "GET", null, true, this.state.accessToken, null);
}
Express server route
router.get("/mediaitem/:key", async (req, res, next) => {
try{
const { key } = req.params;
// Attempt 1 was to try with s3.getObject(downloadParams).createReadStream();
const readStream = getFileStream(key);
readStream.pipe(res);
// Attempt 2 - attempt to convert response to base 64 encoding
var data = await getFileStream(key);
var test = data.Body.toString("utf-8");
var container = '';
if ( data.Body ) {
container = data.Body.toString("utf-8");
} else {
container = undefined;
}
var buffer = (new Buffer.from(container));
var test = buffer.toString("base64");
require('fs').writeFileSync('../uploads', test); // it never wrote to this directory
console.log('conversion: ', test); // prints: 77+977+977+977+9AO+/vQAIBgYH - this doesn't look like base64 to me.
delete buffer;
res.status(201).json({ newTest: test });
} catch (err){
next(ApiError.internal(`Unexpected error > mediaData/:id GET -> Error: ${err.message}`));
return;
}
});
AWS S3 Library - I made my own library for using the s3 bucket as I'll need to use more functionality later.
const getFileStream = async (fileKey) => {
const downloadParams = {
Key: fileKey,
Bucket: bucketName
}
// This was attempt 1's return without async in the parameter
return s3.getObject(downloadParams).createReadStream();
// Attempt 2's intention was just to wait for the promise to be fulfilled.
return await s3.getObject(downloadParams).promise();
}
exports.getFileStream = getFileStream;
If you've gotten this far you may have realised that I've tried a couple of things from different sources and documentation but I'm not getting any further. I would really appreciate some pointers and advice on what I'm doing wrong and what I could improve on.
If any further information is needed then just let me know.
Thanks in advance for your time!
Maybe it be useful for you, that's how i get image from S3, and process image on server
Create temporary directory
createTmpDir(): Promise<string> {
return mkdtemp(path.join(os.tmpdir(), 'tmp-'));
}
Gets the file
readStream(path: string) {
return this.s3
.getObject({
Bucket: this.awsConfig.bucketName,
Key: path,
})
.createReadStream();
}
How i process file
async MainMethod(fileName){
const dir = await this.createTmpDir();
const serverPath = path.join(
dir,
fileName
);
await pipeline(
this.readStream(attachent.key),
fs.createWriteStream(serverPath + '.jpg')
);
const createFile= await sharp(serverPath + '.jpg')
.jpeg()
.resize({
width: 640,
fit: sharp.fit.inside,
})
.toFile(serverPath + '.jpeg');
const imageBuffer = fs.readFileSync(serverPath + '.jpeg');
//my manipulations
fs.rmSync(dir, { recursive: true, force: true }); //delete temporary folder
}

NodeJS Amazon AWS SDK S3 client stops working intermittently

I have NodeJS express web server that serves files from AWS S3. Most of the time this exact code works correctly and serves files for a wide verity of applications with large numbers of requests in Production. The NodeJS web server is running across multiple nodes on a docker swarm server.
After about 2-3 weeks this stops working. There is no response from S3Client GetObjectCommand, there no error returned or anything. This starts working again only after restarting the NodeJS Docker container.
I read the S3 SDK docs that indicate a that the SDK will retry automatically.
Each AWS SDK implements automatic retry logic.
Questions:
How can we make this code more resilient and not need a restart?
Is the error handling correct? I'm wondering why there is no seemingly no response or error returned at all in this situation.
Is it necessary to configure the re-try settings?
NodeJS version: node:lts-alpine
Module: #aws-sdk/client-s3
Controllers
AWS Controller
const consoleLogger = require('../logger/logger.js').console;
const { S3Client, GetObjectCommand } = require('#aws-sdk/client-s3');
const config = {
"credentials": {
"accessKeyId": "example",
"secretAccessKey": "example"
},
"endpoint": "example",
"sslEnabled": true,
"forcePathStyle": true
}
const s3client = new S3Client(config);
const awsCtrl = {};
awsCtrl.getObject = async (key) => {
// Get object from Amazon S3 bucket
let data;
try {
// Data is returned as a ReadableStream
data = await s3client.send(new GetObjectCommand({ Bucket: "example", Key: key }));
console.log("Success", data);
} catch (e) {
consoleLogger.error("AWS S3 error: ", e);
const awsS3Error = {
name: e.name || null,
status: e.$metadata.httpStatusCode || 500
};
throw awsS3Error;
}
return data;
}
module.exports = awsCtrl;
Files Controller
const queryString = require('query-string');
const consoleLogger = require('../logger/logger.js').console;
const httpCtrl = require('./http.ctrl');
const jwtCtrl = require('./jwt.ctrl');
const awsCtrl = require('./aws.ctrl');
filesCtrl.deliverFile = async (req, res) => {
/* Get object from AWS S3 */
let fileObjectStream;
try {
fileObjectStream = await awsCtrl.getObject(filePath);
} catch (e) {
consoleLogger.error(`Unable to get object from AWS S3`, e);
if (e.status && e.status === 404) {
result.error = `Not found`;
result.status = 404;
return res.status(result.status).json(result);
}
return res.status(e.status || 500).json(result);
}
const filename = lookupResponse.data.filename;
// Set response header: Content-Disposition
res.attachment(filename);
// API response object stream download to client
return fileObjectStream.Body.pipe(res);
}
API
const express = require('express');
const router = express.Router();
const filesCtrl = require('../../controllers/files.ctrl');
const filesValidation = require('../validation/files');
router.get('/:fileId', [filesValidation.getFile], (req, res, next) => {
return filesCtrl.deliverFile(req, res);
});

Amazon Lex SDK only running one command

I am attempting to use the AWS SDK version 3 and the Lex Runtime client V2. (Documentation)
What I don't understand is why can I not reuse my client instance to send multiple commands? See the code below. I am attempting to do a basic flow of putting a session, recognizing the text, and then deleting the session. When I comment the code out to run each command individually it works. Why does my application stop after running only the first client.send command? All the documentation ends up doing is creating a new client for each command because the examples are showing multiple services being used not the same service (example)
const {
LexRuntimeV2Client,
PutSessionCommand,
RecognizeTextCommand,
} = require("#aws-sdk/client-lex-runtime-v2");
const AWS = require("aws-sdk");
require("dotenv").config();
const sessionCustomId = "12345678";
const getLexClient = () => {
return new LexRuntimeV2Client({
credentials: new AWS.Credentials(
process.env.AWS_ACCESS_KEY,
process.env.AWS_SECRET_KEY
),
region: process.env.AWS_REGION,
});
};
const main = async () => {
await console.log(
`******* ENV VARS *********\n\n AWS_REGION : ${process.env.AWS_REGION}\n DEWEY_ALIAS_ID : ${process.env.DEWEY_ALIAS_ID}\n DEWEY_BOT_ID : ${process.env.DEWEY_BOT_ID}\n LOCAL_ID : ${process.env.LOCALE_ID}\n\n`
);
/******************************* PUT SESSION ***********************************/
// a client can be shared by different commands.
const client = await getLexClient();
const putSessionCommand = new PutSessionCommand({
accept: "text/plain; charset=utf-8",
botAliasId: process.env.DEWEY_ALIAS_ID,
botId: process.env.DEWEY_BOT_ID,
localeId: process.env.LOCALE_ID,
sessionState: {},
sessionId: sessionCustomId,
});
try {
const response = await client.send(putSessionCommand);
console.log("finished putting session\n");
console.log(response);
var chatSessionState = response.sessionState;
} catch (err) {
console.error(
"Error sending PutSessionCommand, error message: " + JSON.stringify(err)
);
}
/******************************* RECOGNIZE TEXT ***********************************/
//get new client
//var client = await getLexClient();
const recognizeTextCommand = new RecognizeTextCommand({
botAliasId: process.env.DEWEY_ALIAS_ID,
botId: process.env.DEWEY_BOT_ID,
localeId: process.env.LOCALE_ID,
sessionState: {},
sessionId: sessionCustomId,
text: "Create a case",
});
try {
const responseTwo = await client.send(recognizeTextCommand);
console.log("finished posting text input\n");
console.log(responseTwo);
} catch (err) {
console.error(
"Error sending RecognizeTextCommand, error message: " +
JSON.stringify(err)
);
}
/******************************* DELETE SESSION ***********************************/
const deleteSessionCommand = new DeleteSessionCommand({
botAliasId: process.env.DEWEY_ALIAS_ID,
botId: process.env.DEWEY_BOT_ID,
localeId: process.env.LOCALE_ID,
sessionId: sessionCustomId,
});
try {
const responseThree = await client.send(deleteSessionCommand);
console.log("finished deleting session\n");
console.log(responseThree);
} catch (err) {
console.error(
"Error sending DeletingSession, error message: " + JSON.stringify(err)
);
}
};
main();

Web3 smart contract instance is not able to listen Transfer events

I have a NFT deployed on the mainnet, and I want to listen any mint event by watching the Transfer event via web3js. But unfortunately I am not able to retrieve any Transfer event happening. But the thing is that, when I try to getPastEvents, I successfully retrieve the correct data so it is most likely not due to another part of the code.
Here is my relevant piece of code:
const Web3 = require('web3')
const nodeClient = require('node-rest-client-promise').Client()
const dotenv = require('dotenv')
dotenv.config()
const CONTRACT_ADDRESS = '0x8d4B648F7fAB1c72d1690b42693fb7525ce3025e'
const projectId = process.env.INFURA_KEY
const etherscanKey = process.env.ETHERSCAN_KEY
const etherscan_url = `http://api.etherscan.io/api?module=contract&action=getabi&address=${CONTRACT_ADDRESS}&apikey=${etherscanKey}`
async function getContractAbi() {
const etherscan_response = await nodeClient.getPromise(etherscan_url)
const CONTRACT_ABI = JSON.parse(etherscan_response.data.result);
return CONTRACT_ABI;
}
async function handleTransferEvent(event) {
try{
const fromAddress = event.returnValues.from
const tokenId = event.returnValues.tokenId
console.log("SOMEONE TRANSFERED NFT!")
if(fromAddress == '0x0000000000000000000000000000000000000000') {
console.log("Minted:\n", event.returnValues)
/* Do stuff */
}
}
catch(err) {
console.log(err)
console.log("ERROR WHILE HANDLING TRANSFER EVENT")
}
}
const init = async () => {
var web3 = new Web3('wss://mainnet.infura.io/ws/v3/' + projectId)
console.log("Connected to mainnet")
const CONTRACT_ABI = await getContractAbi()
console.log("Retrieved contract abi")
const contract = new web3.eth.Contract(
CONTRACT_ABI,
CONTRACT_ADDRESS
)
contract.events.Transfer({})
.on('data', handleTransferEvent)
.on('error', console.error)
console.log('Started listening minting events...')
}
init()
You can check the smart contract from https://etherscan.io/address/0x8d4b648f7fab1c72d1690b42693fb7525ce3025e#code
EDIT: I think problem might be related to calling listen event inside a function.

Intent handler failed with error: Buffer is not defined

I have built a Google Smart Home Action using the Local Fulfillment SDK as outlined in the following articles:
https://developers.google.com/assistant/smarthome/develop/local
https://github.com/actions-on-google/create-local-home-app
I am using UDP for device discovery and my Google Nest Hub can successfully scan and detect the virtual device running on my laptop as well as download the JS of the local app.
My Local Home SDK configuration is as follows - Local Home SDK Configuration.
When the Nest Hub executes the IDENTIFY intent of my app handler I am receiving the following error(s):
[smarthome.DeviceManager] Intent handler failed with error: Buffer is not defined
[smarthome.DeviceManager] Got a rejected promise Buffer is not defined
This appears to be a Node.JS error as opposed to something specific to the local SDK app itself. Below is the code of my local app.
/// <reference types="#google/local-home-sdk" />
import App = smarthome.App;
import Constants = smarthome.Constants;
import DataFlow = smarthome.DataFlow;
import Execute = smarthome.Execute;
import Intents = smarthome.Intents;
import IntentFlow = smarthome.IntentFlow;
const SERVER_PORT = 3388;
interface ILightParams {
on?: boolean,
brightness?: number
}
class LocalExecutionApp {
constructor(private readonly app: App) { }
identifyHandler(request: IntentFlow.IdentifyRequest):
Promise<IntentFlow.IdentifyResponse> {
console.log("IDENTIFY intent: " + JSON.stringify(request, null, 2));
const scanData = request.inputs[0].payload.device.udpScanData;
console.log("SCANDATA: " + JSON.stringify(scanData));
if (!scanData) {
const err = new IntentFlow.HandlerError(request.requestId, 'invalid_request', 'Invalid scan data');
return Promise.reject(err);
}
const localDeviceId = Buffer.from(scanData.data, 'hex');
console.log("ScanData Local Device: " + localDeviceId);
const response: IntentFlow.IdentifyResponse = {
intent: Intents.IDENTIFY,
requestId: request.requestId,
payload: {
device: {
// id: localDeviceId.toString(),
id: 'sample-device',
verificationId: localDeviceId.toString(),
}
}
};
console.log("IDENTIFY response: " + JSON.stringify(response, null, 2));
return Promise.resolve(response);
}
executeHandler(request: IntentFlow.ExecuteRequest):
Promise<IntentFlow.ExecuteResponse> {
console.log("EXECUTE intent: " + JSON.stringify(request, null, 2));
const command = request.inputs[0].payload.commands[0];
const execution = command.execution[0];
const response = new Execute.Response.Builder()
.setRequestId(request.requestId);
const promises: Promise<void>[] = command.devices.map((device) => {
console.log("Handling EXECUTE intent for device: " + JSON.stringify(device));
// Convert execution params to a string for the local device
const params = execution.params as ILightParams;
const payload = this.getDataForCommand(execution.command, params);
// Create a command to send over the local network
const radioCommand = new DataFlow.HttpRequestData();
radioCommand.requestId = request.requestId;
radioCommand.deviceId = device.id;
radioCommand.data = JSON.stringify(payload);
radioCommand.dataType = 'application/json';
radioCommand.port = SERVER_PORT;
radioCommand.method = Constants.HttpOperation.POST;
radioCommand.isSecure = false;
console.log("Sending HTTP request to the smart home device:", payload);
return this.app.getDeviceManager()
.send(radioCommand)
.then(() => {
const state = {online: true};
response.setSuccessState(device.id, Object.assign(state, params));
console.log(`Command successfully sent to ${device.id}`);
})
.catch((e: IntentFlow.HandlerError) => {
e.errorCode = e.errorCode || 'invalid_request';
response.setErrorState(device.id, e.errorCode);
console.error('An error occurred sending the command', e.errorCode);
});
});
return Promise.all(promises)
.then(() => {
return response.build();
})
.catch((e) => {
const err = new IntentFlow.HandlerError(request.requestId, 'invalid_request', e.message);
return Promise.reject(err);
});
}
/**
* Convert execution request into a local device command
*/
getDataForCommand(command: string, params: ILightParams): unknown {
switch (command) {
case 'action.devices.commands.OnOff':
return {
on: params.on ? true : false
};
default:
console.error('Unknown command', command);
return {};
}
}
}
const localHomeSdk = new App('1.0.0');
const localApp = new LocalExecutionApp(localHomeSdk);
localHomeSdk
.onIdentify(localApp.identifyHandler.bind(localApp))
.onExecute(localApp.executeHandler.bind(localApp))
.listen()
.then(() => console.log('Ready'))
.catch((e: Error) => console.error(e));
Any insight on why this error is occurring is greatly appreciated.
Cheers.
Reposting the answer in the comments:
Buffer is not a type that is directly supported in a browser environment. You can see this for yourself by trying to run something like x = new Buffer() in your web browser's dev console.
To support a class like Buffer, you can use a bundling tool like Webpack. In the official sample, you can see an example Webpack configuration. Other bundling tools can also be used, and examples can be found in the official Local Home scaffolding tool or can be called directly with an npm init command.
npm init #google/local-home-app app/ --bundler webpack
npm init #google/local-home-app app/ --bundler rollup
npm init #google/local-home-app app/ --bundler parcel

Resources