How to upload files to AWS S3 from NestJS ( Suggestion ) - node.js

im working in my reutilizable code, and i have for you
a FileService to Upload files to AWS S3 with NestJS.
The if conditional that you see in the code about cloudPlatform variable
is because this is not the final code.
Eventually this service will be able to upload to many cloud platforms like
GCP, Azure, AWS and FileSystem.
import { Injectable, Logger, NotFoundException, InternalServerErrorException } from '#nestjs/common';
import { ConfigService } from '#nestjs/config';
import { S3 } from 'aws-sdk';
import * as moment from 'moment';
import { v4 as uuid } from 'uuid';
import { IAWS_S3 } from './interfaces/aws-s3.interface';
import { Response } from 'express';
#Injectable()
export class FileService {
/**
*
* #variable cloudPlatform is the tecnology to save files in the cloud
* #variable awsConfig is the Amazon Web Service S3 configuration
* #S3 is the S3 to instanciate your credentials to use Amazon Web Service
*/
private logger = new Logger('FileService');
private cloudPlatform: string = this.configService.get('uploadFilesSettings').fileUploadPlatform;
private awsConfig: IAWS_S3;
private s3: S3;
constructor(
private readonly configService: ConfigService,
) {
if ( this.cloudPlatform === 'AWS_S3' ) {
this.awsConfig = this.configService.get<IAWS_S3>('aws');
this.s3 = new S3( this.awsConfig );
}
}
/**
* * Function to upload files sent from the body
* #param files of the request body in an object
* #returns an object with properties that are an string array
* like this obj = { prop1: ['a.jpg','b.jpg'], prop2: ['test.gif'] ...prop n }
*/
async uploadFiles<T, U>( files: T ): Promise<U> {
let props = Object.keys(files);
let arrPromisesPending = await this.getArrayToUpload<T>( props, files );
let arrPromisesDone = await Promise.all(
arrPromisesPending.map( async( file: Express.Multer.File ) => {
return {
fieldname: file.fieldname,
...( await this.uploadFileTo( file, this.cloudPlatform )),
}
})
)
return await this.getUploadedFiles<U>( arrPromisesDone, props );
}
/**
* * Function to get an array of objects of the files sent from the body
* * to make a Promise.all()
* #param props all properties ( fieldname in the form-data body ) of the files object
* #param files of the request body in an object
* #returns an array of objects that every object is the file that will be uploaded
*/
async getArrayToUpload<T>( props: string[], files: T ): Promise<Object[]> {
let promises: Object[] = [];
props.map(( prop: string ) => {
files[prop].map(( file: Express.Multer.File ) => {
promises.push( file );
});
})
return promises;
}
/**
* * Function that upload the file to cloud platform
* #param file object to upload
* #param cloudPlatform site where files will be uploaded
* #returns the response of cloudPlatform
*/
async uploadFileTo( file: Express.Multer.File, cloudPlatform: string ): Promise<any> {
let type: string = file.originalname.split(".").pop();
let genericFilename: string = `${ uuid() }-${ moment().format().split('T')[0] }`;
if ( cloudPlatform === 'AWS_S3' ) {
try {
return await this.s3.upload({
Bucket: this.awsConfig.bucketName,
Key: `${ genericFilename }.${ type }`,
Body: file.buffer,
ContentType: file.mimetype,
}).promise();
} catch ( error ) {
this.logger.error( error );
throw new InternalServerErrorException('Unexpected error, check server logs');
}
}
this.logger.error('Set .env variable "FILE_UPLOAD_PLATFORM"');
throw new InternalServerErrorException('Unexpected error, check server logs');
}
/**
* * Function to format the names of the files uploaded to the cloud
* * platform and return them in an object with the corresponding
* * properties and each of these will have an array of strings
*
* #param arrPromisesDone an array of objects that were uploaded
* #param props that's means fieldnames of the files
* #returns
* ! Example of returns -----------------------------------------------------------
* ! {
* ! avatar: ['image1.png','image2.jpg', 'image3.jpeg', 'image4.jpg'],
* ! background: ['image1.png','image2.jpg', 'image3.jpeg', 'image4.jpg'],
* ! }
*/
async getUploadedFiles<T>( arrPromisesDone: Object[], props: string[] ): Promise<T> {
let uploadedFiles: any = {};
props.map(( prop: string ) => {
uploadedFiles[prop] = [];
arrPromisesDone.map(( file: any ) => {
if ( file.fieldname === prop ) {
uploadedFiles[prop].push( file.key );
}
});
});
return uploadedFiles;
}
/**
*
* #param imageName name of the image
* #param res #Res() to response from Express
* #returns the image obtained from the cloud platform to client
*/
async getFile( imageName: string, res: Response ) {
const params = { Bucket: this.awsConfig.bucketName, Key: imageName };
if ( this.cloudPlatform === 'AWS_S3' ) {
try {
const data = await this.s3.getObject( params ).promise();
res.contentType( data.ContentType );
return res.send( data.Body );
} catch ( error ) {
if ( error.statusCode === 404 )
throw new NotFoundException(`image ${ imageName } don't exist`);
this.logger.error( error );
throw new InternalServerErrorException('Unexpected error, check server logs');
}
}
this.logger.error('Set .env variable "FILE_UPLOAD_PLATFORM"');
throw new InternalServerErrorException('Unexpected error, check server logs');
}
}

Related

How to stub nested dependecies with ts-sinon

I got a simple unit test with the following code:
my-pubsub.spec.ts
import * as tsSinon from 'ts-sinon';
import { myPubSubFunction } from './my-pubsub';
import * as sendEmail from './send-mail';
describe("Notifications PubSub tests", () => {
it("Should trigger audit", (done) => {
const today = new Date()
const data = {
( my data )
}
const spy = tsSinon.default.spy(sendEmail, "sendNotificationMessage")
const dataBuffer = Buffer.from(JSON.stringify(data))
// Call tested function and verify its behavior
myPubSubFunction(dataBuffer)
setTimeout(() => {
// check if spy was called
tsSinon.default.assert.calledOnce(spy)
done()
}, 100)
})
})
And my-pubsub.ts got a call to a function from send-mail with contains a function to set the Api key
import * as sgMail from '#sendgrid/mail';
sgMail.setApiKey(
"MyKey"
) // error in here
export function sendNotificationMessage(mailConfig: any) {
const defaultConfig = {
from: {
email: "noreply#mymail.com",
name: "my name",
},
template_id: "my template",
}
const msg = { ...defaultConfig, ...mailConfig }
return sgMail.send(msg)
}
However when running my tests I got the following error TypeError: sgMail.setApiKey is not a function
Edit: added a bit more code to the send-mail code.
Bellow you can find a bit more code about my-pubsub.ts
my-pubsub.ts
import * as admin from 'firebase-admin';
import * as functions from 'firebase-functions';
import moment = require('moment');
import { IModel, ModelType } from '../models/model.model';
import { sendNotificationMessage } from '../shared/send-mail';
const { PubSub } = require("#google-cloud/pubsub")
try {
admin.initializeApp()
} catch (e) {}
const db = admin.firestore()
const pubSubClient = new PubSub()
export const myPubSubTrigger = functions.pubsub
.topic("on-trigger")
.onPublish(async (message) => {
console.log("version 1")
const myMessage = Buffer.from(message.data, "base64").toString("utf-8")
const data: IModel = JSON.parse(myMessage)
( logic to create my object )
/**
* Send email
*/
const result: any = await sendNotificationMessage(myObject)
/**
* Check result
*/
if (result[0].statusCode === 202) {
await docRef.update({ emailSent: true })
}
( another publish to audit the action )
})
The problem is not with tests per se, but incorrect types definition of #sendgrid/mail:
// OK
import sgMail from "#sendgrid/mail";
import { default as sgMail2 } from "#sendgrid/mail";
console.log(sgMail === sgMail2);
sgMail.setApiKey("SG.key");
sgMail2.setApiKey("SG.key2");
// BROKEN
// type definition does not match runtime shape
import * as sgMailIncorrectlyTyped from "#sendgrid/mail";
console.log(sgMailIncorrectlyTyped, sgMailIncorrectlyTyped.setApiKey === undefined);
STACKBLITZ

I'm trying to show multiple signed urls from gcs to the client and I don't know how to change the console.log to something that works

I have a bucket in gcs that contains images so with this code from the server I managed to paginate them and get 10 in a request and at the same time generate 10 signed urls but I still don't know how to send these urls to the client to be able to show them on my web page
For now, I can only send the name of the objects with this code and the signed urls appear in the console
import { Injectable, Options, UseFilters } from '#nestjs/common';
import { AdminService } from 'src/firebase-admin/admin/admin.service';
#Injectable()
export class FilesService {
constructor(
private adminService: AdminService) {}
async get() {
let options = undefined;
options = {
projection: 'noAcl',
maxResults: 10,
};
return this.adminService.bucket.getFiles(options).then(async ([files]: any) => {
const fileNames = files.map((file: any) => file.name);
for (const fileName of fileNames) {
const [signedUrl] = await this.adminService.bucket.file(fileName).getSignedUrl({
version: 'v4',
expires: Date.now() + 1000 * 60 * 60,
action: 'read'
});
console.log(`The signed URL for ${fileName} is ${signedUrl}`);
}
return fileNames;
})
}
}
u

TypeError: intlProvider.getChildContext is not a function

I'm using injectIntl in my react functional component to achieve the localization.
I'm using enzyme/jest to do the unit test. I've copied the intl-test-helper file, but got error:
" TypeError: intlProvider.getChildContext is not a function "
I've tried other suggestions from stackflow:
1. remove mock file --- which I don't have mock file
2. use const { IntlProvider } = jest.requireActual("react-intl"); to force it use the actual one , not mock --- not working.
the react component is: WarningModal.jsx:
import { FormattedMessage, injectIntl } from "react-intl";
.......
const WarningModal = ({
.........
...props
}) => {
........
export default injectIntl(WarningModal);
the intlTestHelper.js file is :
* Components using the react-intl module require access to the intl context.
* This is not available when mounting single components in Enzyme.
* These helper functions aim to address that and wrap a valid,
* English-locale intl context around them.
*/
import React from "react";
import { IntlProvider, intlShape } from "react-intl";
import { mount, shallow } from "enzyme"; // eslint-disable-line import/no-extraneous-dependencies
/** Create the IntlProvider to retrieve context for wrapping around. */
function createIntlContext(messages, locale) {
const { IntlProvider } = jest.requireActual("react-intl");
const intlProvider = new IntlProvider({ messages, locale }, {});
const { intl } = intlProvider.getChildContext();
return intl;
}
/** When using React-Intl `injectIntl` on components, props.intl is required. */
function nodeWithIntlProp(node, messages = {}, locale = "en") {
return React.cloneElement(node, {
intl: createIntlContext(messages, locale)
});
}
/**
* Create a shadow renderer that wraps a node with Intl provider context.
* #param {ReactComponent} node - Any React Component
* #param {Object} context
* #param {Object} messages - A map with keys (id) and messages (value)
* #param {string} locale - Locale string
*/
export function shallowWithIntl(
node,
{ context } = {},
messages = {},
locale = "en"
) {
return shallow(nodeWithIntlProp(node), {
context: Object.assign({}, context, {
intl: createIntlContext(messages, locale)
})
});
}
/**
* Mount the node with Intl provider context.
* #param {Component} node - Any React Component
* #param {Object} context
* #param {Object} messages - A map with keys (id) and messages (value)
* #param {string} locale - Locale string
*/
export function mountWithIntl(
node,
{ context, childContextTypes } = {},
messages = {},
locale = "en"
) {
return mount(nodeWithIntlProp(node), {
context: Object.assign({}, context, {
intl: createIntlContext(messages, locale)
}),
childContextTypes: Object.assign({}, { intl: intlShape }, childContextTypes)
});
}
here how I use it to test:
import React from "react";
import { _WM as WarningModal } from "../components/WarningModal";
// import { shallow } from "enzyme";
import { mountWithIntl } from "../utils/intlTestHelper.js";
describe("<WarningModal />", () => {
const props = {
discardChanges: jest.fn(),
saveChanges: jest.fn(),
closeWarningModal: jest.fn(),
intl: { formatMessage: jest.fn() }
};
it("should have heading", () => {
const wrapper = mountWithIntl(<WarningModal {...props} />);
expect(wrapper.find(".confirm-title")).toBeTruthy();
});
});
error:
● <WarningModal /> › should have heading
TypeError: intlProvider.getChildContext is not a function
14 | const { IntlProvider } = jest.requireActual("react-intl");
15 | const intlProvider = new IntlProvider({ messages, locale }, {});
> 16 | const { intl } = intlProvider.getChildContext();
| ^
17 | return intl;
18 | }
19 |
at getChildContext (src/utils/intlTestHelper.js:16:33)
at createIntlContext (src/utils/intlTestHelper.js:23:11)
at nodeWithIntlProp (src/utils/intlTestHelper.js:60:16)
at Object.<anonymous> (src/tests/WarningModal.spec.js:29:21)
please shine some lights on this. Thank you.
In later versions of react-intl getChildContext has been deprecated and may generate this error. You can use the following instead:
import { createIntl } from 'react-intl';
const intl = createIntl({ locale: "en",
messages: {
message1: "Hello world"
}
});
React-Intl has replaced IntlProvider.getChildContext, with the createIntl for testing purpose, while migrating V2 to V3.
We've removed IntlProvider.getChildContext for testing and now you can use createIntl to create a standalone intl object outside of React and use that for testing purposes. See Testing with React Intl for more details
Here is the Link
So the working code for this is
For resolving this error, you have to create custom shallow component. Like as
import { createIntl } from 'react-intl';
const LocalLanguage = {
french:{},
arabic:{},
english:{}
}
const lang = getCurrentLanguage('en', LocalLanguage);
const intl = createIntl({ locale: 'en', lang }, {});
export const shallowWithIntl = (node) => {
return shallow(nodeWithIntlProp(node), { context: { intl } });
}
If this not helps, then you can define the following function, in your helper file.
const messages = require('./Lang/en.json') // en.json
const defaultLocale = 'en'
const locale = defaultLocale
export const intl = (component) => {
return (
<IntlProvider
locale={locale}
messages={messages}
>
{React.cloneElement(component)}
</IntlProvider>
);
}
And use it in your test files as below
const wrapper = mount(intl(<MobileRechargeComponent />));

How to unit test an AWS SDK method called in a separate method?

I am currently writing a piece of code which simply uploads a file to an S3 bucket. This is a serverless NodeJS project written in Typescript. For testing, I am using Mocha, Chai, and attempting to use SinonJS.
I have a class with a method which uploads a buffer as a file to S3:
import { S3 } from "aws-sdk"
import { S3UploadError } from "../errors/S3UploadError"
/**
* This class provides a means of interacting with S3
*
* #export
* #class AWSS3Manager
*/
export class AWSS3Manager {
/**
* Instance of S3
*
* #private
* #type {S3}
* #memberof AWSS3Manager
*/
private s3: S3
/**
* Creates an instance of AWSS3Manager.
*
* #memberof AWSS3Manager
*/
public constructor() {
this.s3 = new S3()
}
/**
* Upload a file to bucket on S3 by using a buffer
*
* #param {Buffer} body
* #param {string} bucketName
* #param {string} fileName
* #returns {Promise<PromiseResult<S3.PutObjectOutput, S3UploadError>>}
* #memberof AWSS3Manager
*/
public async upload(body: Buffer, bucketName: string, fileName: string) {
const params = {
Body: body,
Bucket: bucketName,
Key: fileName,
}
try {
return await this.s3.putObject(params).promise()
} catch (err) {
console.log(err)
throw new S3UploadError()
}
}
}
I'm not even sure if this is the right thing to be doing, but I want to test the upload method of the above class. And in my mind, to do that I need to mock the response from S3.putObject in order to ensure that the external library doesn't cause interference. However, no matter what I have tried I can't get it to work.
I have tried stubbing the S3 put object method, but to no avail:
chai.should()
const assert = chai.assert
const expect = chai.expect
const s3manager = new AWSS3Manager()
let sandbox: sinon.SinonSandbox
let spy
describe("AWSS3Manager behaves as expected", () => {
beforeEach(() => {
sandbox = sinon.createSandbox()
spy = sinon.spy()
})
afterEach(() => {
// Restore the default sandbox here
sandbox.restore()
})
it("Uploads a file correctly to S3", async () => {
const putObjectStub = sinon.stub(S3.prototype, "putObject")
putObjectStub.yields("ERROR", 'data')
//const uploadStub = sandbox.stub(s3manager, "upload").resolves("Yup")
//sandbox.stub(AWS, "S3").resolves('HEYY')
const test = await s3manager.upload(new Buffer("ddds"), "TestBucket", "Test")
assert(putObjectStub.called)
})
})
I have tried stubbing the S3 method, and also stubbing the upload method its self and seeing how many calls there was to putObject but I can't get nothing to work.
Can anyone help?
I ran into a similar situation attempting to stub out the putObject call on an S3 instance that I was then returning chained with .promise(). See below:
export class UploadService {
s3: S3
constructor() {
this.s3 = new S3({ region: 'us-west-2' })
}
upload({ data, path }: { data: string, path: string }) {
const uploadParams = {
Body: data,
Bucket: bucketName, // defined elsewhere
Key: path
}
return this.s3.putObject(uploadParams).promise()
}
}
I dug into the aws-sdk definitions and came up with this test setup--I don't love it because it's kinda exposing the guts of the sdk that you shouldn't have to think about but it passes the typescript compiler and gets the job done 🤷‍♂️
import { expect } from 'chai'
import * as sinon from 'sinon'
import { UploadService } from '../../src/services/upload_service'
import { Request, Service } from 'aws-sdk'
describe('UploadService', () => {
const uploadService = new UploadService()
let uploadStub: sinon.SinonStub
describe('#upload', () => {
beforeEach(() => {
uploadStub = sinon
.stub(uploadService.s3, 'putObject')
.returns(new Request(new Service(), 'put'))
})
it('uploads the data to the specified path in S3', async () => {
const data = '{"cool": "data"}'
const path = 'my/cool/file/path'
await uploadService.upload({ data, path })
expect(uploadStub).to.have.been.calledOnceWith({ data, path })
})
})
})
Hope that helps!

App Engine Node.js: how to link app logs and requests logs

I am using Node.js on App Engine Standard and Flexible.
In the logs viewer, is it possible to display application logs nested inside request logs?
Yes it is possible to correlate application logs and request logs. This is the end result in the Logs Viewer:
To achieve this you can either:
Use both the #google-cloud/trace-agent and #google-cloud/logging-bunyan modules in your application. When you do so, your logs are automatically annotated with the correct Trace ID (see docs for Bunyan).
Extract the trace ID from the request header
If you do not want to use the Trace module, you can extract the trace ID from the request header, use the following code to extract the traceId:
const traceHeader = req && req.headers ? req.headers['x-cloud-trace-context'] || '' : '';
const traceId = traceHeader ? traceHeader.split('/')[0] : '';
Then, you need to populate the trace attribute of your log entries. When using the Bunyan logger, use the following code:
logger.info({
'logging.googleapis.com/trace': `projects/${project}/traces/${traceId}`
}, 'your message');
I also faced the same issue sometime back and did some workaround to make it. But in the above-mentioned solution might not help in some use cases where you have don't req, res object reference.
So here the solution. it will group all the logs under the request log.
Also created -> NPM Module
File Name: correlate-logs.js
import bunyan from 'bunyan';
import { LOGGING_TRACE_KEY } from '#google-cloud/logging-bunyan';
import cls from 'cls-hooked';
import uuid from 'uuid/v1';
/**
* CreateLogger will return loggerContextMiddleware and log.
* Bind the loggerContextMiddleware on top to corelate other middleware logs. `app.use(loggerContextMiddleware);`
* then you can log like this anywhere `log.info('This is helpful to see corelated logs in nodejs.)` and it will show with-in reqeust log.
* #param {*} options
*/
export default function createLogger(projectId, bunyanLoggerOptions) {
if (!projectId || !bunyanLoggerOptions) throw new Error('Please pass the required fields projectId and bunyanLoggerOption');
const ns = cls.createNamespace(`logger/${uuid()}`); // To create unique namespace.
const logger = bunyan.createLogger(bunyanLoggerOptions);
/**
* Express Middleware to add request context to logger for corelating the logs in GCP.
* #param {*} req
* #param {*} res
* #param {*} next
*/
const loggerContextMiddleware = (req, res, next) => {
const traceHeader = (req && req.headers && req.headers['x-cloud-trace-context']) || '';
if (traceHeader) {
ns.bindEmitter(req);
ns.bindEmitter(res);
const traceId = traceHeader ? traceHeader.split('/')[0] : '';
const trace = `projects/${projectId}/traces/${traceId}`;
ns.run(() => {
ns.set('trace', trace);
next();
});
} else {
next();
}
};
/**
* Helper method to get the trace id from CLS hook.
*/
function getTrace() {
if (ns && ns.active) return ns.get('trace');
return '';
}
/**
* Simple wrapper to avoid pushing dev logs to cloud.
* #param {*} level
* #param {*} msg
*/
function printLog(level, ...msg) {
const trace = getTrace();
if (trace) { logger[level]({ [LOGGING_TRACE_KEY]: trace }, ...msg); } else { logger[level](...msg); }
}
/**
* Little wrapper to abstract the log level.
*/
const log = ['trace', 'debug', 'info', 'warn', 'error', 'fatal'].reduce((prev, curr) => ({ [curr]: (...msg) => printLog(curr, ...msg), ...prev }), {});
return { loggerContextMiddleware, log };
}
File Name: logger.js
import { LoggingBunyan } from '#google-cloud/logging-bunyan';
import createLogger from '../lib/corelate-logs';
import { getProjectId, ifDev } from './config';
// Creates a Bunyan Stackdriver Logging client
const loggingBunyan = new LoggingBunyan();
let loggerOption;
if (ifDev()) {
const bunyanDebugStream = require('bunyan-debug-stream'); // eslint-disable-line
loggerOption = {
name: 'my-service',
streams: [{
level: 'info',
type: 'raw',
stream: bunyanDebugStream({
forceColor: true,
}),
}],
serializers: bunyanDebugStream.serializers,
};
} else {
loggerOption = {
name: 'my-service',
level: 'info',
streams: [loggingBunyan.stream('info')],
};
}
const { loggerContextMiddleware, log } = createLogger(getProjectId() || 'dev', loggerOption);
export { loggerContextMiddleware, log };
Hope this helps somebody.

Resources