multer: How to process and manipulate files before saving? - node.js

I want to process images using sharp before saving so I don't need to save the file twice (one by multer and one by sharp). The best way I found is to save the file in memory with initializing multer with no argument:
const upload = multer()
and then giving the file buffer to sharp in the route handler:
await sharp(req.file.buffer)
.resize(500)
.jpeg({ quality: 50 })
.toFile(path)
);
If is there a better way let me know.
The better question would be:
Is there something like a hook in multer that calls before saving? so I can change the file content before saving it.

the process is start from fileFilter method then fileName of multer so you should start resize it in fileFilter save it to temporary directory then reszie image after resize you can use method .ToFile of sharp package it will save the destination inside .toFile("./upload")
file-mapper.ts (use for return another object)
export const fileMapper = ({ file, req }: FileMapper) => {
if (!file) {
return null!;
}
const fileName = editFilename(req.file);
const image_url = `${req.protocol}://${req.headers.host}/${req.file.destination}/${fileName}`;
resizeImage(req, fileName);
return {
filename: file.filename,
image_url,
path: file.path,
};
};
my resize-image.ts
import { Request } from 'express';
import path from 'path';
import sharp from 'sharp';
import fs from 'fs';
export const resizeImage = async (req: Request, fileName: string) => {
await sharp(req.file.path)
.resize({ height: 700, width: 700, position: 'center' })
.jpeg({ quality: 90, chromaSubsampling: '4:4:4' })
.toFile(path.resolve(req.file.destination, fileName))
.then((sharpOutPut) => {
if (sharpOutPut) fs.unlinkSync(req.file.path);
})
.catch((err) => {
throw err;
});
};
disk-storage.ts
const storage= diskStorage({
destination: 'upload/product',
filename: editFileName,
}),
fileFilter: imageFileFilter,
})
use sharp in fileFilter of multer
I'm using fileMapper because I want to receive new object for my db

Related

Storing and serving images in NestJS

I'm figuring out how to enable a local upload folder for a profile image.
Now, i've come as far as storing a file in the uploads folder. However, when I return that file it has no extension and its a JFIF binary file:
���� JFIF �� cmp3.10.3.3Lq4 0xa61382cc�� C
// etcetera...
The controller I've created is:
./src/controllers/user-profile.controller.ts
#Controller('user-profile')
export class UserProfileController {
#Post('/:id/upload-photo')
#UseInterceptors(FileInterceptor('image', { dest: './uploads' }))
uploadSinglePhoto(
#Param('id', ParseIntPipe) id: number,
#UploadedFile() image
) {
console.log('===> ', image);
return this.userProfileService.saveUserProfilePhotoLocation(id, image.path);
}
#Get('/:id/profile-photo')
getUserProfilePhoto(
): any {
// '15c924f42ffaa67b3f14a5be05f0a312' is the file name that is created by the upload
return new StreamableFile(createReadStream(join(process.cwd(), 'uploads', '15c924f42ffaa67b3f14a5be05f0a312')))
}
}
The image object in the console log is:
===> {
fieldname: 'image',
originalname: '526dad4edd250b689eeb1394c3c6eb41.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
destination: './uploads',
filename: '590b454b4315009660273deac082b4ed',
path: 'uploads\\590b454b4315009660273deac082b4ed',
size: 45842
}
Then I store the path to the database.
./src/services/user-profile.service.ts
#Injectable()
export class UserProfileService {
constructor(
#InjectRepository(UserProfileEntity)
private userProfileRepostory: Repository<UserProfileEntity>
) {}
async saveUserProfilePhotoLocation(id, path): Promise<UserProfileEntity> {
const result = await this.userProfileRepostory.createQueryBuilder()
.update(UserProfileEntity)
.set({
photo: path
})
.where({id})
.returning('*')
.execute();
return result.raw;
}
}
And finally I've configured the express server to be able to serve static files from the uploads folder:
./src/main.ts
async function bootstrap() {
const app = await NestFactory.create<NestExpressApplication>(AppModule);
app.useStaticAssets(join(__dirname, '..', 'uploads'), {
index: false,
prefix: 'uploads',
});
await app.listen('3000');
}
bootstrap();
Now when I do a get on the end point /:id/profile-photo it does not return me a rendered image. Ïnstead it returns a file with a random set of characters in it which is probably because of the encoding.
What should I do so that I can serve an jpeg file to my application?
What is happening here?
When setting the content header (thanks Jay McDoniel) to 'image/jpeg' and creating a readable stream it returns the image:
#Get('/:id/profile-photo')
getUserProfilePhoto(
#Res({ passthrough: true }) res: Response
): StreamableFile {
res.set({'Content-Type': 'image/jpeg'});
const imageLocation = join(process.cwd(), 'uploads', '15c924f42ffaa67b3f14a5be05f0a312');
const file = createReadStream(imageLocation);
return new StreamableFile(file);
}

Nestjs throw error and http status 400 when fileFilter in fileinterceptor finds an unsupported file extension

I am having difficulty handling an error in my application.
I want to create a FileInterceptor (multer) which checks if the file has an allowed file extension. The documentation shows this:
const allowedFileExtensions = ['.jpg', '.png'];
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (allowedFileExtensions.includes(extension)) {
callback(null, true);
} else {
// gives the 500 error
callback(new Error('Only images are allowed'), false);
}
}
}
This kinda works. But it has two flaws in my opinion. First, it returns a 500 error:
{
"statusCode": 500,
"message": "Internal server error"
}
It would be better to return a 400 error with a error message which explains why it failed.
Second, in the console of the Nest application it shows the stacktrace. I'd rather use the logging for this instead. (The application has middleware in place to pick up NestJS errors and log them automagically.)
So what I am trying is the following:
const allowedFileExtensions = ['.jpg', '.png'];
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (!allowedFileExtensions.includes(extension)) {
// crashes the application
throw new BadRequestException('Only images are allowed', `Bad request. Accepted file extensions are: ${allowedFileExtensions.toString()}`);
}
callback(null, true)
}
}
But this crashes the application. I am not able to upload a different image anymore after this.
Any idea?
Ah found it. The BadRequestException cannot be thrown outside the body of the function.
const allowedFileExtensions = ['.jpg', '.png'];
enum FileValidationErrors {
UNSUPPORTED_FILE_TYPE
}
#Post('/:id/upload-photo')
#UseInterceptors(
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (allowedFileExtensions.includes(extension)) {
callback(null, true);
} else {
// provide the validation error in the request
req.fileValidationError = FileValidationErrors.UNSUPPORTED_FILE_TYPE
callback(null, false);
}
}
}
)
)
uploadSinglePhoto(
#Param('id', ParseIntPipe) id: number,
#UploadedFile() image,
#Req() req // add the request property
): Promise<UserProfileEntity> {
// check here for a potential error
if (req?.fileValidationError === FileValidationErrors.UNSUPPORTED_FILE_TYPE) {
// if so, throw the BadRequestException
throw new BadRequestException('Only images are allowed', `Bad request. Accepted file extensions are: ${allowedFileExtensions.toString()}`);
}
this.logger.verbose(`Adding image ${image}`);
const imageUrlLocation = `${image.destination.substring(1)}/${image.filename}`;
return this.userProfileService.saveUserProfilePhotoLocation(id, imageUrlLocation);
}

how to store images in unique folder(using primary key) for each upload using multer in nest js

I have an application where i want to store multipart form data along with image. I want images to be stored in unique folder which is primary id. How can i do it.
Here i am saving multiform data in database. I want to use primary id return by below function to create folder to save its respective images.
async createFaceDetection(faceDetectionReqDto: FaceDetectionReqDto): Promise<Object> {
try {
const newFaceDetection = new face_detection({
Name: faceDetectionReqDto.Name
})
const savedMediaUpload = await this.FaceDetectionRepository.save(newFaceDetection);
return savedMediaUpload
}
catch (err) {
throw err;
}
}
how can i use primary id fetched by "savedMediaUpload" variable in multer
my component.controller.ts code
#UseInterceptors(
FilesInterceptor('Images', 20,
{
storage: diskStorage({
destination: async function (req, file, cb) {
const filePath = path.join(ROOT_DIR, FACE_DETECTION_PATH, {Primary-Id}) // (primary id
generated by above function.)
if (!existsSync(`${filePath}`)) { mkdirSync(filePath, { recursive: true }) }
cb(null, filePath);
},
filename: (_req, file, cb) => {
return cb(null, file.originalname);
},
}),
}),
)
I've done it like this, by appending generated UUID to the file name - may not be perfect but works good enough for me so far:
Controller:
#UseInterceptors(
FilesInterceptor('asset', 10, {
storage: diskStorage({
destination: UPLOADS_DIR,
filename: getUniqueFileName,
}),
limits: {
fileSize: MAX_FILE_SIZE_BYTES,
},
fileFilter: validateImageFile,
}),
)
#Post()
async createAsset(
#Body() assetCreateRequest: AssetCreateRequest,
#UploadedFile() asset: UploadedFileModel,
): Promise<AssetResponse[]> {
....
}
And getUniqueFileName looks like this:
import { extname } from 'path';
import { v4 as uuidv4 } from 'uuid';
import { UploadedFileModel } from '../../dto/internal/file-upload';
export const getUniqueFileName = (
_request: unknown,
file: UploadedFileModel,
callback: (error: Error | null, fileName: string) => void,
) => {
const name = file.originalname.split('.')[0];
const extension = extname(file.originalname);
const randomUuid = uuidv4();
callback(null, `${name}-${randomUuid}${extension}`);
};
Later on in the database, I have separate Asset and e.g. User tables where User references a record in the Asset table by foreign key, and Asset record contains all the necessary info - file name, path, URL etc.

Wrong API on file-upload using Cypress.io for E2E testing

I am testing a portal with Cypress.io, which has a file upload functionality.
But my file always failed to upload because the API call is going wrong.
Correct API Call:
**
POST 200 /etl/v1.0.0/datauploaderetl/spaces/etl_jyddc0tx/data-files
**
But when uploading through Cypress, the following is the URL:
**
POST 404 /etl/v1.0.0/datauploaderetl/data-files
**
As you can clearly see, the API is incorrect. I added the wait here, still, it doesn't work.
Following is the piece of code:
cy.fixture(fileName1).then(fileContent => {
cy.get('input[type="file"]').attachFile({
fileContent: fileContent.toString(),
fileName: fileName1,
mimeType: fileType
})
});
cy.waitUntil(() => cy.get(":nth-child(98) > .modal > .modal-lg > .modal-content > .modal-body")
.should('contain.text', 'Status: completed')
);
Please help!
At Command.js, add below code:
let LOCAL_STORAGE_MEMORY = {};
Cypress.Commands.add("saveLocalStorage", () => {
Object.keys(localStorage).forEach(key => {
LOCAL_STORAGE_MEMORY[key] = localStorage[key];
});
});
Cypress.Commands.add("restoreLocalStorage", () => {
Object.keys(LOCAL_STORAGE_MEMORY).forEach(key => {
localStorage.setItem(key, LOCAL_STORAGE_MEMORY[key]);
});
});
Then at the test case file, add below beforeEach and afterEach block respectively:
beforeEach(() => {
cy.restoreLocalStorage();
})
afterEach(() => {
cy.saveLocalStorage();
})
This will solve the issue where Cypress clears the "local storage" at the browser.
As per documentation this is the way to upload file :
cy.fixture('filepath').as('filetoupload')
cy.get('input[type=file]').then(function($input) {
// convert the logo base64 string to a blob
const blob = Cypress.Blob.base64StringToBlob(this.filetoupload, fileType)
$input.fileupload('add', { files: blob })
})
or
cy.fixture('filepath').as('filetoupload')
cy.get('input[type=file]').then(function(el) {
// convert the logo base64 string to a blob
const blob = Cypress.Blob.base64StringToBlob(this.filetoupload,fileType )
const file = new File([blob], '<path>', { type: fileType })
const list = new DataTransfer()
list.items.add(file)
const myFileList = list.files
el[0].files = myFileList
el[0].dispatchEvent(new Event('change', { bubbles: true }))
})
https://docs.cypress.io/api/utilities/blob.html#Image-Fixture

How to get rid of "TypeError: req.pipe is not a function" nestjs and fastify

I tried to upload file with NestJS/Fastify and typescript
this is main.ts
async function bootstrap() {
//file upload with fastify
const fastifyAdapter = new FastifyAdapter();
fastifyAdapter.register(fmp, {
limits: {
fieldNameSize: 100, // Max field name size in bytes
fieldSize: 1000000, // Max field value size in bytes
fields: 10, // Max number of non-file fields
fileSize: 100, // For multipart forms, the max file size
files: 1, // Max number of file fields
headerPairs: 2000, // Max number of header key=>value pairs
},
});
const app = await NestFactory.create<NestFastifyApplication>(
AppModule,
fastifyAdapter,
);
await app.listen(3000);
Logger.log('application started on http://localhost:3000', 'Bootstrap');
}
bootstrap();
and this is file.controller.ts
#Post()
#UseInterceptors(FileInterceptor('image'))
#ApiConsumes('multipart/form-data')
#ApiBody({
description: 'logo',
type: UploadFileDto,
})
uploadedFile(#UploadedFile() file) {
const response = {
originalname: file.originalname,
filename: file.filename,
};
return response;
}
after uploading a file to this action, code throw an exception like this
TypeError: req.pipe is not a function
at multerMiddleware (D:\R.Khodabakhshi\Repository\raimun-web\node_modules\multer\lib\make-middleware.js:176:9)
at Promise (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\platform-express\multer\interceptors\file.interceptor.js:15:81)
at new Promise ()
at MixinInterceptor.intercept (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\platform-express\multer\interceptors\file.interceptor.js:15:19)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:22:36
at Object.handle (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:20:56)
at LoggingInterceptor.intercept (D:\R.Khodabakhshi\Repository\raimun-web\dist\shared\logging.interceptor.js:28:21)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:22:36
at InterceptorsConsumer.intercept (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:24:24)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\router\router-execution-context.js:45:60
[Nest] 10928 - 2020-02-06 10:10:49 [ExceptionFilter] undefined undefined +587529ms
TypeError: req.pipe is not a function
at multerMiddleware (D:\R.Khodabakhshi\Repository\raimun-web\node_modules\multer\lib\make-middleware.js:176:9)
at Promise (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\platform-express\multer\interceptors\file.interceptor.js:15:81)
at new Promise ()
at MixinInterceptor.intercept (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\platform-express\multer\interceptors\file.interceptor.js:15:19)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:22:36
at Object.handle (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:20:56)
at LoggingInterceptor.intercept (D:\R.Khodabakhshi\Repository\raimun-web\dist\shared\logging.interceptor.js:28:21)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:22:36
at InterceptorsConsumer.intercept (D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\interceptors\interceptors-consumer.js:24:24)
at D:\R.Khodabakhshi\Repository\raimun-web\node_modules#nestjs\core\router\router-execution-context.js:45:60
how can I fix the problem???
You cannot use the FastifyAdapter with the FileInterceptor. It says so in the beginning of the docs. If you want to use Fastify and a file upload, you'll need to create your own interceptor for it.
Problem solved, as Jay McDaniel mentioned we couldn't use the FastifyAdapter with the FileInterceptor.
I resolved the problem with this little code.
import {
Controller,
Logger,
Post,
Req,
Res,
} from '#nestjs/common';
import * as fs from 'fs';
import * as path from 'path';
import * as pump from 'pump';
const logger = new Logger('FileController');
#ApiTags('File')
#Controller('api/file')
export class FileController {
#Post()
upload(#Req() req: any, #Res() reply: any): void {
const mp = req.multipart(
(field: any, file: any, filename: any, encoding: any, mimeType: any) => {
console.log('save file from request ---- ', field, filename, mimeType);
file.on('limit', () => logger.error('SIZE_LIMITED'));
const filePath = path.resolve('./'+filename);
const writeStream = fs.createWriteStream(filePath);
pump(file, writeStream);
writeStream.on('finish', () => {
reply.code(200).send();
});
},
(error: any) => {
if (error) {
logger.error(error);
reply.code(500).send();
}
},
);
mp.on('partsLimit', () => logger.error('MAXIMUM_NUMBER_OF_FORM_PARTS'));
mp.on('filesLimit', () => logger.error('MAXIMUM_NUMBER_OF_FILES'));
mp.on('fieldsLimit', () => logger.error('MAXIMUM_NUMBER_OF_FIELD'));
}
}
I hope this will help you too...

Resources