how to store images in unique folder(using primary key) for each upload using multer in nest js - node.js

I have an application where i want to store multipart form data along with image. I want images to be stored in unique folder which is primary id. How can i do it.
Here i am saving multiform data in database. I want to use primary id return by below function to create folder to save its respective images.
async createFaceDetection(faceDetectionReqDto: FaceDetectionReqDto): Promise<Object> {
try {
const newFaceDetection = new face_detection({
Name: faceDetectionReqDto.Name
})
const savedMediaUpload = await this.FaceDetectionRepository.save(newFaceDetection);
return savedMediaUpload
}
catch (err) {
throw err;
}
}
how can i use primary id fetched by "savedMediaUpload" variable in multer
my component.controller.ts code
#UseInterceptors(
FilesInterceptor('Images', 20,
{
storage: diskStorage({
destination: async function (req, file, cb) {
const filePath = path.join(ROOT_DIR, FACE_DETECTION_PATH, {Primary-Id}) // (primary id
generated by above function.)
if (!existsSync(`${filePath}`)) { mkdirSync(filePath, { recursive: true }) }
cb(null, filePath);
},
filename: (_req, file, cb) => {
return cb(null, file.originalname);
},
}),
}),
)

I've done it like this, by appending generated UUID to the file name - may not be perfect but works good enough for me so far:
Controller:
#UseInterceptors(
FilesInterceptor('asset', 10, {
storage: diskStorage({
destination: UPLOADS_DIR,
filename: getUniqueFileName,
}),
limits: {
fileSize: MAX_FILE_SIZE_BYTES,
},
fileFilter: validateImageFile,
}),
)
#Post()
async createAsset(
#Body() assetCreateRequest: AssetCreateRequest,
#UploadedFile() asset: UploadedFileModel,
): Promise<AssetResponse[]> {
....
}
And getUniqueFileName looks like this:
import { extname } from 'path';
import { v4 as uuidv4 } from 'uuid';
import { UploadedFileModel } from '../../dto/internal/file-upload';
export const getUniqueFileName = (
_request: unknown,
file: UploadedFileModel,
callback: (error: Error | null, fileName: string) => void,
) => {
const name = file.originalname.split('.')[0];
const extension = extname(file.originalname);
const randomUuid = uuidv4();
callback(null, `${name}-${randomUuid}${extension}`);
};
Later on in the database, I have separate Asset and e.g. User tables where User references a record in the Asset table by foreign key, and Asset record contains all the necessary info - file name, path, URL etc.

Related

Storing and serving images in NestJS

I'm figuring out how to enable a local upload folder for a profile image.
Now, i've come as far as storing a file in the uploads folder. However, when I return that file it has no extension and its a JFIF binary file:
���� JFIF �� cmp3.10.3.3Lq4 0xa61382cc�� C
// etcetera...
The controller I've created is:
./src/controllers/user-profile.controller.ts
#Controller('user-profile')
export class UserProfileController {
#Post('/:id/upload-photo')
#UseInterceptors(FileInterceptor('image', { dest: './uploads' }))
uploadSinglePhoto(
#Param('id', ParseIntPipe) id: number,
#UploadedFile() image
) {
console.log('===> ', image);
return this.userProfileService.saveUserProfilePhotoLocation(id, image.path);
}
#Get('/:id/profile-photo')
getUserProfilePhoto(
): any {
// '15c924f42ffaa67b3f14a5be05f0a312' is the file name that is created by the upload
return new StreamableFile(createReadStream(join(process.cwd(), 'uploads', '15c924f42ffaa67b3f14a5be05f0a312')))
}
}
The image object in the console log is:
===> {
fieldname: 'image',
originalname: '526dad4edd250b689eeb1394c3c6eb41.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
destination: './uploads',
filename: '590b454b4315009660273deac082b4ed',
path: 'uploads\\590b454b4315009660273deac082b4ed',
size: 45842
}
Then I store the path to the database.
./src/services/user-profile.service.ts
#Injectable()
export class UserProfileService {
constructor(
#InjectRepository(UserProfileEntity)
private userProfileRepostory: Repository<UserProfileEntity>
) {}
async saveUserProfilePhotoLocation(id, path): Promise<UserProfileEntity> {
const result = await this.userProfileRepostory.createQueryBuilder()
.update(UserProfileEntity)
.set({
photo: path
})
.where({id})
.returning('*')
.execute();
return result.raw;
}
}
And finally I've configured the express server to be able to serve static files from the uploads folder:
./src/main.ts
async function bootstrap() {
const app = await NestFactory.create<NestExpressApplication>(AppModule);
app.useStaticAssets(join(__dirname, '..', 'uploads'), {
index: false,
prefix: 'uploads',
});
await app.listen('3000');
}
bootstrap();
Now when I do a get on the end point /:id/profile-photo it does not return me a rendered image. Ïnstead it returns a file with a random set of characters in it which is probably because of the encoding.
What should I do so that I can serve an jpeg file to my application?
What is happening here?
When setting the content header (thanks Jay McDoniel) to 'image/jpeg' and creating a readable stream it returns the image:
#Get('/:id/profile-photo')
getUserProfilePhoto(
#Res({ passthrough: true }) res: Response
): StreamableFile {
res.set({'Content-Type': 'image/jpeg'});
const imageLocation = join(process.cwd(), 'uploads', '15c924f42ffaa67b3f14a5be05f0a312');
const file = createReadStream(imageLocation);
return new StreamableFile(file);
}

Nestjs throw error and http status 400 when fileFilter in fileinterceptor finds an unsupported file extension

I am having difficulty handling an error in my application.
I want to create a FileInterceptor (multer) which checks if the file has an allowed file extension. The documentation shows this:
const allowedFileExtensions = ['.jpg', '.png'];
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (allowedFileExtensions.includes(extension)) {
callback(null, true);
} else {
// gives the 500 error
callback(new Error('Only images are allowed'), false);
}
}
}
This kinda works. But it has two flaws in my opinion. First, it returns a 500 error:
{
"statusCode": 500,
"message": "Internal server error"
}
It would be better to return a 400 error with a error message which explains why it failed.
Second, in the console of the Nest application it shows the stacktrace. I'd rather use the logging for this instead. (The application has middleware in place to pick up NestJS errors and log them automagically.)
So what I am trying is the following:
const allowedFileExtensions = ['.jpg', '.png'];
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (!allowedFileExtensions.includes(extension)) {
// crashes the application
throw new BadRequestException('Only images are allowed', `Bad request. Accepted file extensions are: ${allowedFileExtensions.toString()}`);
}
callback(null, true)
}
}
But this crashes the application. I am not able to upload a different image anymore after this.
Any idea?
Ah found it. The BadRequestException cannot be thrown outside the body of the function.
const allowedFileExtensions = ['.jpg', '.png'];
enum FileValidationErrors {
UNSUPPORTED_FILE_TYPE
}
#Post('/:id/upload-photo')
#UseInterceptors(
FileInterceptor(
'image',
{
dest: './uploads',
fileFilter: (req, file, callback) => {
const extension = path.extname(file.originalname);
if (allowedFileExtensions.includes(extension)) {
callback(null, true);
} else {
// provide the validation error in the request
req.fileValidationError = FileValidationErrors.UNSUPPORTED_FILE_TYPE
callback(null, false);
}
}
}
)
)
uploadSinglePhoto(
#Param('id', ParseIntPipe) id: number,
#UploadedFile() image,
#Req() req // add the request property
): Promise<UserProfileEntity> {
// check here for a potential error
if (req?.fileValidationError === FileValidationErrors.UNSUPPORTED_FILE_TYPE) {
// if so, throw the BadRequestException
throw new BadRequestException('Only images are allowed', `Bad request. Accepted file extensions are: ${allowedFileExtensions.toString()}`);
}
this.logger.verbose(`Adding image ${image}`);
const imageUrlLocation = `${image.destination.substring(1)}/${image.filename}`;
return this.userProfileService.saveUserProfilePhotoLocation(id, imageUrlLocation);
}

Image uploaded works in localhost but not on server (s3)

Well, I have an option in the system where I can see some documents sent by the user. But the images appear as broken icons.
As "localhost" everything works great, but I can't see the images when I run straight from the website. (On the s3 everything is working normally, I can see the images.)
In the brownser console, an error is thrown about the route that the request is being made "api.mysite.com/the-image-example.png" with status code 404.
What I don't understand is why the request is being made there and not directly on amazon s3 (there I have the images loading normally). Someone cal help me with that??
How images appear:
This is the code
Upload config:
const tmpFolder = path.resolve(__dirname, '..', '..', 'tmp');
export const mimeTypesPhotos = [
'image/png',
'image/jpeg',
'image/bmp',
'image/webp',
];
export const mimeTypesVideos = ['video/webm', 'video/ogg', 'video/mp4'];
export const mimeTypesPng = ['application/pdf'];
const pngRoutes = ['/requests/test', '/test/:id/confirm'];
interface IUploadConfig {
driver: 's3' | 'disk';
tmpFolder: string;
uploadsFolder: string;
multer: multer.Options;
image: {
height: number;
width: number;
};
config: {
disk: unknown;
aws: {
bucket: string;
};
};
}
export default {
driver: process.env.STORAGE_DRIVER,
tmpFolder,
uploadsFolder: path.resolve(tmpFolder, 'uploads'),
multer: {
storage: multer.diskStorage({
destination: tmpFolder,
filename: (request, file, callback) => {
const fileHash = crypto.randomBytes(10).toString('hex');
const fileName = `${fileHash}-${file.originalname}`;
return callback(null, fileName);
},
}),
fileFilter: (request, file, callback) => {
const url = matchId(request.originalUrl);
const mimetypes = [...mimeTypesPhotos];
if (pngRoutes.includes(url)) {
mimetypes.push(...mimeTypesPng);
}
if (!mimetypes.includes(file.mimetype)) {
return callback(new Error("File doesn't supported"));
}
return callback(null, true);
},
},
image: {
width: Number(process.env.MAX_IMAGE_SIZE || '1024'),
height: Number(process.env.MAX_IMAGE_SIZE || '1024'),
},
config: {
disk: {},
aws: {
bucket: process.env.AWS_BUCKET || 'mybucket',
},
},
} as IUploadConfig;
Router file (where is calling the route i mean):
app.use('/files', express.static(uploadConfig.uploadsFolder));
The Media Entity:
#Column({ type: 'varchar' })
type: 'photo';
#Column()
media: string;
#Expose({ name: 'mediaUrl' })
getMediaUrl(): string | null {
if (!this.media) {
return null;
}
switch (uploadConfig.driver) {
case 'disk':
return `${process.env.APP_API_URL}/files/${this.media}`;
case 's3':
return `https://${uploadConfig.config.aws.bucket}.s3.amazonaws.com/${this.media}`;
default:
return null;
}
}
Step-1
we need to Enable CORS in the API gateway, so click on the resource and then click on the Action button and Enable CORS. In CORS settings the value of Access-Control-Allow-Headers is '' and Access-Control-Allow-Origin is '' , and leave other settings as it is. Click on Enable CORS and replace existing CORS headers.
Step-2
Now, go to the settings of that API on which you are working. In settings enable Binary Media Types. Set the value of that field like this / , and Save the changes.

multer: How to process and manipulate files before saving?

I want to process images using sharp before saving so I don't need to save the file twice (one by multer and one by sharp). The best way I found is to save the file in memory with initializing multer with no argument:
const upload = multer()
and then giving the file buffer to sharp in the route handler:
await sharp(req.file.buffer)
.resize(500)
.jpeg({ quality: 50 })
.toFile(path)
);
If is there a better way let me know.
The better question would be:
Is there something like a hook in multer that calls before saving? so I can change the file content before saving it.
the process is start from fileFilter method then fileName of multer so you should start resize it in fileFilter save it to temporary directory then reszie image after resize you can use method .ToFile of sharp package it will save the destination inside .toFile("./upload")
file-mapper.ts (use for return another object)
export const fileMapper = ({ file, req }: FileMapper) => {
if (!file) {
return null!;
}
const fileName = editFilename(req.file);
const image_url = `${req.protocol}://${req.headers.host}/${req.file.destination}/${fileName}`;
resizeImage(req, fileName);
return {
filename: file.filename,
image_url,
path: file.path,
};
};
my resize-image.ts
import { Request } from 'express';
import path from 'path';
import sharp from 'sharp';
import fs from 'fs';
export const resizeImage = async (req: Request, fileName: string) => {
await sharp(req.file.path)
.resize({ height: 700, width: 700, position: 'center' })
.jpeg({ quality: 90, chromaSubsampling: '4:4:4' })
.toFile(path.resolve(req.file.destination, fileName))
.then((sharpOutPut) => {
if (sharpOutPut) fs.unlinkSync(req.file.path);
})
.catch((err) => {
throw err;
});
};
disk-storage.ts
const storage= diskStorage({
destination: 'upload/product',
filename: editFileName,
}),
fileFilter: imageFileFilter,
})
use sharp in fileFilter of multer
I'm using fileMapper because I want to receive new object for my db

Merge 2 log files in nodejs

I have a file saved in my server, suppose its path is data/x.log. I upload a new file to the server:
import { Injectable } from '#angular/core';
import { HttpClient } from '#angular/common/http';
import { Observable } from 'rxjs';
import { IResponse } from 'src/app/interfaces/response.interface';
import { environment } from 'src/environments/environment';
const ENDPOINT = environment.apiUrl + '/log';
#Injectable()
export class HttpUploadService {
constructor(
private http: HttpClient,
) { }
public uploadLogFile(file: File): Observable<IResponse> {
const baseUrl: string = ENDPOINT;
const postData = new FormData();
postData.append('file', file);
return this.http.post<IResponse>(baseUrl, { postData });
}
}
This is also a log file. My order is to merge these 2 files to the same location data/x.log path.
The merge is like append. just add the second file content after the first one.
How can I do it my server code with node.js?
This is my controller for this:
const saveLogFile = async (req, res) => {
}
In my first approach I tried to do it with multer package:
const storage = multer.diskStorage({
destination(req, file, cb) {
cb(null, './data');
},
filename(req, file, cb) {
cb(null, `x.log`);
}
});
But as far as I know it will just override the file content with no merge.
How can I do it?
I haven't heard of upload libraries supporting append. You'd most likely have to do it manually without using a library that automatically stores files on disk.
You could, for example, use busboy to access the incoming file as a stream and then append it to your desired file. Once you get a hold of the incoming stream, you can append to an existing file with something like:
const fs = require("fs");
const destination = fs.createWriteStream("data/x.log", {
flags: "a" // "a" for append
});
incomingFile.pipe(destination);

Resources