Merge 2 log files in nodejs - node.js

I have a file saved in my server, suppose its path is data/x.log. I upload a new file to the server:
import { Injectable } from '#angular/core';
import { HttpClient } from '#angular/common/http';
import { Observable } from 'rxjs';
import { IResponse } from 'src/app/interfaces/response.interface';
import { environment } from 'src/environments/environment';
const ENDPOINT = environment.apiUrl + '/log';
#Injectable()
export class HttpUploadService {
constructor(
private http: HttpClient,
) { }
public uploadLogFile(file: File): Observable<IResponse> {
const baseUrl: string = ENDPOINT;
const postData = new FormData();
postData.append('file', file);
return this.http.post<IResponse>(baseUrl, { postData });
}
}
This is also a log file. My order is to merge these 2 files to the same location data/x.log path.
The merge is like append. just add the second file content after the first one.
How can I do it my server code with node.js?
This is my controller for this:
const saveLogFile = async (req, res) => {
}
In my first approach I tried to do it with multer package:
const storage = multer.diskStorage({
destination(req, file, cb) {
cb(null, './data');
},
filename(req, file, cb) {
cb(null, `x.log`);
}
});
But as far as I know it will just override the file content with no merge.
How can I do it?

I haven't heard of upload libraries supporting append. You'd most likely have to do it manually without using a library that automatically stores files on disk.
You could, for example, use busboy to access the incoming file as a stream and then append it to your desired file. Once you get a hold of the incoming stream, you can append to an existing file with something like:
const fs = require("fs");
const destination = fs.createWriteStream("data/x.log", {
flags: "a" // "a" for append
});
incomingFile.pipe(destination);

Related

how to prevent file upload when body validation fails in nestjs

I have the multipart form to be validated before file upload in nestjs application. the thing is that I don't want the file to be uploaded if validation of body fails.
here is how I wrote the code for.
// User controller method for create user with upload image
#Post()
#UseInterceptors(FileInterceptor('image'))
create(
#Body() userInput: CreateUserDto,
#UploadedFile(
new ParseFilePipe({
validators: [
// some validator here
]
})
) image: Express.Multer.File,
) {
return this.userService.create({ ...userInput, image: image.path });
}
Tried so many ways to turn around this issue, but didn't reach to any solution
Interceptors run before pipes do, so there's no way to make the saving of the file not happen unless you manage that yourself in your service. However, another option could be a custom exception filter that unlinks the file on error so that you don't have to worry about it post-upload
This is how I created the whole filter
import { isArray } from 'lodash';
import {
ExceptionFilter,
Catch,
ArgumentsHost,
BadRequestException,
} from '#nestjs/common';
import { Request, Response } from 'express';
import * as fs from 'fs';
#Catch(BadRequestException)
export class DeleteFileOnErrorFilter implements ExceptionFilter {
catch(exception: BadRequestException, host: ArgumentsHost) {
const ctx = host.switchToHttp();
const response = ctx.getResponse<Response>();
const request = ctx.getRequest<Request>();
const status = exception.getStatus();
const getFiles = (files: Express.Multer.File[] | unknown | undefined) => {
if (!files) return [];
if (isArray(files)) return files;
return Object.values(files);
};
const filePaths = getFiles(request.files);
for (const file of filePaths) {
fs.unlink(file.path, (err) => {
if (err) {
console.error(err);
return err;
}
});
}
response.status(status).json(exception.getResponse());
}
}

Nextjs next-session storage other than memory

I don’t find any resource or package to use another storage other than memory for next-session, where memory is not a suitable solution for production?
I want to use either file storage ( preferred ) or db .
you can store session for example in sqlite3.
install sqlite 'npm install connect-sqlite3'
Create a new file for example 'get-session.js' and write this:
import nextSession from "next-session";
import { expressSession, promisifyStore } from "next-session/lib/compat";
var SQLiteStore = require("connect-sqlite3")(expressSession);
export const getSession = nextSession({
name: "WIB_SESSION",
store: promisifyStore(
new SQLiteStore({ dir: "./tmp/", table: "wiberSessions" })
),
});
Create a new api endpoint and put this.
import { getSession } from "../../lib/get-session.js";
export default async function handler(req, res) {
const session = await getSession(req, res);
const data={hello:"hello im a data in session"}
session.myData= data
res.status(200).json({save:"session saved"});
}
export const config = {
api: {
externalResolver: true,
},
};
And now from your Page
import { getSession } from "../lib/get-session";
export async function getServerSideProps({ req, res }) {
const session = await getSession(req, res);
return {
props: {
dataInSession: session.myData,
},
};
}
Now you will have a cookie named "WIB_SESSION" with a value, and the data stored in a database SQLITE in ./tmp/wiberSessions.
Here you have a basic project with next.
Github
First time you load the page,
First Time
And when you press the button and reload the page you have the cooki and the value from the /tmp/sqlite.withcookie
Greetings

How to import fs on the server side

I'm trying to send an ssr page containing the contents of a txt file, however I'm stuck on the part of being able to import the fs.
I understand that I need to be on the server side to be able to import the fs, I tried to use this code, without success
// pages/index.js
import fs from 'fs'
export async function getServerSideProps(ctx) {
return {
props: {}
}
}
export default function Home() {
return (
<h1>...</h1>
)
}
You can use import() or require() syntax inside getServerProps.
// pages/index.js
export async function getServerSideProps(ctx) {
const fs = await import('fs');
// or
const fs = require('fs');
return {
props: {},
};
}
export default function Home() {
return <h1>...</h1>;
}

How to store, access and retrieve Image files from Nodejs-Express ( MongoDB ) to Angular8 FrontEnd,

This is an Employee Management System, Profile images of Employees are uploaded and stored in API server (defined path).
I implemented the following steps.
Step 1: API server Request - the sample API code
router.get('/public/users-images/*', function(req, res) {
var filepath = __dirname +'/..'+req.url
fs.exists(filepath, (exists) => {
if (exists) {
var filepath = filepath
} else {
var filepath = __dirname +'/../public/users-images/user-image.png'
}
})
var fileext = filepath.split('.')
fs.readFile(filepath, function(err, buffer){
// console.log(base64Image);
var base64Image = new Buffer(buffer, 'binary').toString('base64');
res.send({img:'data:image/'+fileext[fileext.length -1]+';base64,' + base64Image});
});
})
Step 2: Front end request from Angular8 using the pipe method,
import { Pipe, PipeTransform } from '#angular/core';
import { HttpClient } from '#angular/common/http';
#Pipe({
name: 'userImages'
})
export class UserImagesPipe implements PipeTransform {
constructor(
private http: HttpClient,
private authenticationService: AuthenticationService
) { }
transform(img_name: string): any {
this.http.get('/public/users-images/'+img_name).subscribe(result => {
// You could also cache your result
return result.img;
});
}
}
Now, My question is , How to access the images from API Server location and display it in the UI screen. I assume to use the below method.
Html template <img [src]="'user-image.png' | userImages" class="responsive-12">
I use PIPE method - Because the User List (User Information including the Profile images ) will be used in Search filter, Task Creation form etc., So I tried to implement it in common, And maybe In the future, there is a chance to implement it through CDN.
Am I doing it in the right way? Or will there be any vents for issues ?
You are overcomplicating it. Let the browser do the fetching of the image.
Simply output the correct path to the image in the view
<img [src]="'/public/users-images/' + username + '.png'" alt="" />
or if you want you could create a pipe for it, but only to output the correct path
import { Pipe, PipeTransform } from '#angular/core';
import { HttpClient } from '#angular/common/http';
#Pipe({
name: 'userImages'
})
export class UserImagesPipe implements PipeTransform {
constructor(
private authenticationService: AuthenticationService
) { }
transform(img_name: string): any {
return `/public/users-images/${img_name}.png`
}
}
<img [src]="username | userImages" alt="" />

NestJS upload using GraphQL [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
Is anyone has an example of how to upload a file in NestJs using GraphQl?
I can upload using given example via controller
https://github.com/nestjs/nest/issues/262#issuecomment-366098589,
but I couldn't find any comprehensive documentation how to upload using GrahpQL in NestJS
Apollo Server 2.0 should be able to do this now (packaged in nest), although I needed to install graphql-upload and import GraphQLUpload as I couldn't find the Upload type:
#Mutation(() => Image, { nullable: true })
async addImage(#Args({ name: 'image', type: () => GraphQLUpload }) image) {
// Do stuff with image...
}
At the time of this answer FileInterceptor is using multer and by converting ExecutionContext to http it uses getRequest and getResponse methods to provide req and res to multer.single which they are (req and res) undefined in GraphQL.
I have tried to get request from context using:
const ctx = GqlExecutionContext.create(context);
and there is req property in ctx but I can't find a way to use multer (yet).
Anyway, I made some changes to FileFieldsInterceptor to use it inside my project, but I may make pull request when I had time to clean it up:
import { Observable } from 'rxjs';
import {
NestInterceptor,
Optional,
ExecutionContext,
mixin,
} from '#nestjs/common';
import { GqlExecutionContext } from '#nestjs/graphql';
import { storeFile } from './storeFile';
interface IField {
name: string;
options?: any;
}
export function GraphqlFileFieldsInterceptor(
uploadFields: IField[],
localOptions?: any,
) {
class MixinInterceptor implements NestInterceptor {
options: any = {};
constructor(#Optional() options: any = {}) {
this.options = { ...options, ...localOptions };
}
async intercept(
context: ExecutionContext,
call$: Observable<any>,
): Promise<Observable<any>> {
const ctx = GqlExecutionContext.create(context);
const args = ctx.getArgs();
let storeFilesResult = await Promise.all(
uploadFields.map(uploadField => {
const file = args[uploadField.name];
return storeFile(file, {
...uploadField.options,
...this.options,
}).then(address => {
args[uploadField.name] = address;
return address;
});
}),
);
return call$;
}
}
const Interceptor = mixin(MixinInterceptor);
return Interceptor;
}
and store file is something like this (may not be used like this):
import uuid from 'uuid/v4';
import fs from 'fs';
import path from 'path';
const dir = './files';
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir);
}
export const storeFile = async (file, options): Promise<any> => {
// options is not doing anything right now
const { stream } = await file;
const filename = uuid();
const fileAddress = path.join(dir, filename + '.jpg');
return new Promise((resolve, reject) =>
stream
.on('error', error => {
if (stream.truncated)
// Delete the truncated file
fs.unlinkSync(fileAddress);
reject(error);
})
.pipe(fs.createWriteStream(fileAddress))
.on('error', error => reject(error))
.on('finish', () => resolve(fileAddress)),
);
};
In my Cats.resolvers.ts:
...
#Mutation()
#UseInterceptors(
GraphqlFileFieldsInterceptor([
{ name: 'catImage1' },
{ name: 'catImage2' },
{ name: 'catImage3' },
]),
)
async cats(
#Args('catImage1') catImage1: string,
#Args('catImage2') catImage2: string,
#Args('catImage3') catImage3: string,
){
console.log(catImage1) // will print catImage1 address
...
This implementation works perfectly with Node >= v14
package.json
Remove the fs-capacitor and graphql-upload entries from the resolutions section if you added them, and install the latest version of graphql-upload (v11.0.0 at this time) package as a dependency.
src/app.module.ts
Disable Apollo Server's built-in upload handling and add the graphqlUploadExpress middleware to your application.
import { graphqlUploadExpress } from "graphql-upload"
import { MiddlewareConsumer, Module, NestModule } from "#nestjs/common"
#Module({
imports: [
GraphQLModule.forRoot({
uploads: false, // disable built-in upload handling
}),
],
})
export class AppModule implements NestModule {
configure(consumer: MiddlewareConsumer) {
consumer.apply(graphqlUploadExpress()).forRoutes("graphql")
}
}
src/blog/post.resolver.ts (example resolver)
Remove the GraphQLUpload import from apollo-server-core and import from graphql-upload instead
import { FileUpload, GraphQLUpload } from "graphql-upload"
#Mutation(() => Post)
async postCreate(
#Args("title") title: string,
#Args("body") body: string,
#Args("attachment", { type: () => GraphQLUpload }) attachment: Promise<FileUpload>,
) {
const { filename, mimetype, encoding, createReadStream } = await attachment
console.log("attachment:", filename, mimetype, encoding)
const stream = createReadStream()
stream.on("data", (chunk: Buffer) => /* do stuff with data here */)
}
Source: https://github.com/nestjs/graphql/issues/901#issuecomment-780007582
Some other links that I found helpful:
https://stephen-knutter.github.io/2020-02-07-nestjs-graphql-file-upload/
For uploading files using postman Link
EDIT: As per Developia comment below, apollo-server now implements file upload. Should be preferred way.
Below, original answer, for reference.
One normally does not use GraphQL for upload. GraphQL is fancy "specification of API", meaning that in the end of the day, low level HTTP request and responses are translated to/from JSON objects (if you don't have custom transport).
One solution could be to define special endpoint in GraphQL schema like:
mutation Mutation {
uploadFile(base64: String): Int
}
Then client would convert binary data to base64 string, which would be handled accordingly on resolver side. This way, file will become part of JSON object exchanged between GraphQL client and server.
While this is might be suitable for small files, small number of operations, it is definitely not a solution for upload service.
try this
import { Resolver, Mutation, Args } from '#nestjs/graphql';
import { createWriteStream } from 'fs';
import {GraphQLUpload} from "apollo-server-express"
#Resolver('Download')
export class DownloadResolver {
#Mutation(() => Boolean)
async uploadFile(#Args({name: 'file', type: () => GraphQLUpload})
{
createReadStream,
filename
}): Promise<boolean> {
return new Promise(async (resolve, reject) =>
createReadStream()
.pipe(createWriteStream(`./uploads/${filename}`))
.on('finish', () => resolve(true))
.on('error', () => reject(false))
);
}
}
You could use the apollo-upload-server lib. Seems like the easiest thing to do, in my opinion. Cheers
You need to define an upload controller and add it in your app.module, this is an example of what a controller should be (back-end):
#Controller()
export class Uploader {
#Post('sampleName')
#UseInterceptors(FileInterceptor('file'))
uploadFile(#UploadedFile() file) {
// file name selection
const path = `desired path`;
const writeStream = fs.createWriteStream(path);
writeStream.write(file.buffer);
writeStream.end();
return {
result: [res],
};
}
}
And call your controller by fetch in the front-end:
fetch('controller address', {
method: 'POST',
body: data,
})
.then((response) => response.json())
.then((success) => {
// What to do when succeed
});
})
.catch((error) => console.log('Error in uploading file: ', error));

Resources