NodeJS create and POST file with GQL - node.js

I've been unable to figure out in NodeJS how to:
create a "file" in memory from a raw string; and,
how to POST that data to another server that expects a multipart/form-data payload.
Seems you cannot use the Blob or File classes in NodeJS.
I've read the pattern should be to use the Buffer class.
I still cannot get it to work with Buffers.
My GQL Datasoruce class looks something like:
const { RESTDataSource } = require('apollo-datasource-rest');
const FormData = require('form-data');
export default class MyDatasource extends RESTDataSource {
async postFileToServer({ string }) {
const inMemoryFile = Buffer.from(string, 'utf-8');
const myForm = new FormData();
myForm.append('file', inMemoryFile, 'file.txt');
const url = 'http://examnple.com';
const opts = { headers: { 'Content-Type': 'multipart/form-data' } };
return await this.post(url, myForm, opts);
}
}
The endpoint I want to hit works fine when I use Postman to make the API call with a file from my local machine. However, I need the GQL server to create the file from a raw string to afterwards call the example.com endpoint that is expecting a multipart/form-data.
The above example code always gives me an error of Status 400 and SyntaxError: Unexpected token - in JSON at position 0

File upload works for me using the apollo-datasource-rest package. Here is an example:
server.ts:
import { ApolloServer, gql } from 'apollo-server';
import MyDatasource from './datasource';
const typeDefs = gql`
type Query {
dummy: String
}
type Mutation {
upload: String
}
`;
const resolvers = {
Mutation: {
upload(_, __, { dataSources }) {
return dataSources.uploadAPI.postFileToServer({ str: '1234' });
},
},
};
const server = new ApolloServer({
typeDefs,
resolvers,
dataSources: () => {
return {
uploadAPI: new MyDatasource(),
};
},
});
const port = 3001;
server.listen(port).then(({ url }) => console.log(`🚀 Server ready at ${url}`));
datasource.ts:
import { RESTDataSource } from 'apollo-datasource-rest';
import FormData from 'form-data';
export default class MyDatasource extends RESTDataSource {
public async postFileToServer({ str }) {
const inMemoryFile = Buffer.from(str, 'utf-8');
const myForm = new FormData();
myForm.append('file', inMemoryFile, 'file.txt');
const url = 'http://localhost:3000/upload';
return this.post(url, myForm);
}
}
uploadServer.ts:
import multer from 'multer';
import express from 'express';
import path from 'path';
const upload = multer({ dest: path.resolve(__dirname, 'uploads/') });
const app = express();
const port = 3000;
app.post('/upload', upload.single('file'), (req, res) => {
console.log(req.file);
console.log(req.body);
res.sendStatus(200);
});
app.listen(port, () => {
console.log(`upload server is listening on http://localhost:${port}`);
});
The logs printed in the controller of /upload API:
{
fieldname: 'file',
originalname: 'file.txt',
encoding: '7bit',
mimetype: 'text/plain',
destination: '/Users/ldu020/workspace/github.com/mrdulin/apollo-graphql-tutorial/src/stackoverflow/63181608/uploads',
filename: '3cba4dded6089479ad495e2fb2daac21',
path: '/Users/ldu020/workspace/github.com/mrdulin/apollo-graphql-tutorial/src/stackoverflow/63181608/uploads/3cba4dded6089479ad495e2fb2daac21',
size: 4
}
[Object: null prototype] {}
source code: https://github.com/mrdulin/apollo-graphql-tutorial/tree/master/src/stackoverflow/63181608

Related

S3 upload with multer typescript nodejs - Cannot read properties of undefined (reading 'map')

I have created a route for uploading files to an S3 bucket, which is working perfectly. However, when I try and add it into my Accommodation controller with additional logic it is causing the error Cannot read properties of undefined (reading 'map'). I am using the exact same request in postman for each route.
Can anyone spot why this is happening?
My original logic for the upload controller:
import { Request, Response } from "express";
import catchBlock from "../utils/catchBlock";
import { s3Uploadv2 } from "../utils/s3Service";
const UploadController = {
Upload: async (req: Request, res: Response) => {
const files = req.files as Express.Multer.File[];
try {
const results = await s3Uploadv2(files);
res.send({ success: "successful", results });
} catch (e: unknown) {
catchBlock(e, res);
}
},
};
export default UploadController;
My accommodation upload:
UploadImages: async (req: Request, res: Response) => {
const accommodationId = req.params.id;
const accommodation = await accommodationSchema.findOne({
_id: accommodationId,
});
const files = req.files as Express.Multer.File[];
try {
const results = await s3Uploadv2(files);
console.log(results);
if (accommodation) {
results.forEach((file) => accommodation.photos.push(file.Location));
await accommodation.save();
res.send({ success: "successful", accommodation });
} else {
res.status(400).send("No such accommodation");
}
res.send({ success: "successful", results });
} catch (e: unknown) {
catchBlock(e, res);
}
},
S3 service:
import { S3 } from "aws-sdk";
import { v4 as uuid } from "uuid";
export interface Param {
Bucket: string;
Key: string;
Body: Buffer;
}
export const s3Uploadv2 = async (files: Express.Multer.File[]) => {
const s3 = new S3();
const params: Param[] = files.map((file) => {
return {
Bucket: process.env.AWS_BUCKET_NAME,
Key: `uploads/${uuid()}-${file.originalname}`,
Body: file.buffer,
};
});
const results = await Promise.all(
params.map((param) => s3.upload(param).promise())
);
return results;
};
Multer service:
import multer, { FileFilterCallback, MulterError } from "multer";
import { Request } from "express";
const storage = multer.memoryStorage();
const fileFilter = (
req: Request,
file: Express.Multer.File,
cb: FileFilterCallback
) => {
if (file.mimetype.split("/")[0] === "image") {
cb(null, true);
} else {
cb(new MulterError("LIMIT_UNEXPECTED_FILE"));
}
};
export const upload = multer({
storage,
fileFilter,
limits: { fileSize: 1000000, files: 5 },
});
Accommodation Route:
import express from "express";
import AccommodationController from "../controllers/accommodation";
const accommodationRouter = express.Router();
accommodationRouter.get("/", AccommodationController.All);
accommodationRouter.post(
"/create",
AccommodationController.CreateAccommodation
);
accommodationRouter.get(
"/users-accommodation",
AccommodationController.UsersAccommodation
);
accommodationRouter.post("/delete/:id", AccommodationController.Delete);
accommodationRouter.post("/upload/:id", AccommodationController.UploadImages);
export default accommodationRouter;
Upload Route:
import express from "express";
import UploadController from "../controllers/upload";
import { upload } from "../utils/multer";
const uploadRouter = express.Router();
uploadRouter.post("/", upload.array("file", 5), UploadController.Upload);
export default uploadRouter;
realised my mistake was not putting my upload middleware on the accommodation route.
changed this:
accommodationRouter.post("/upload/:id", AccommodationController.UploadImages);
to this:
accommodationRouter.post(
"/upload/:id",
upload.array("file", 5),
AccommodationController.UploadImages
);

SvelteKit endpoint: converting from Node/Express

New to SvelteKit and working to adapt an endpoint from a Node/Express server to make it more generic so as to be able to take advantage of SvelteKit adapters. The endpoint downloads files stored in a database via node-postgresql.
My functional endpoint in Node/Express looks like this:
import stream from 'stream'
import db from '../utils/db'
export async function download(req, res) {
const _id = req.params.id
const sql = "SELECT _id, name, type, data FROM files WHERE _id = $1;"
const { rows } = await db.query(sql, [_id])
const file = rows[0]
const fileContents = Buffer.from(file.data, 'base64')
const readStream = new stream.PassThrough()
readStream.end(fileContents)
res.set('Content-disposition', `attachment; filename=${file.name}`)
res.set('Content-Type', file.type)
readStream.pipe(res)
}
Here's what I have for [filenum].json.ts in SvelteKit so far...
import stream from 'stream'
import db from '$lib/db'
export async function get({ params }): Promise<any> {
const { filenum } = params
const { rows } = await db.query('SELECT _id, name, type, data FROM files WHERE _id = $1;', [filenum])
if (rows) {
const file = rows[0]
const fileContents = Buffer.from(file.data, 'base64')
const readStream = new stream.PassThrough()
readStream.end(fileContents)
let body
readStream.pipe(body)
return {
headers: {
'Content-disposition': `attachment; filename=${file.name}`,
'Content-type': file.type
},
body
}
}
}
What is the correct way to do this with SvelteKit without creating a dependency on Node? Per SvelteKit's Endpoint docs,
We don't interact with the req/res objects you might be familiar with from Node's http module or frameworks like Express, because they're only available on certain platforms. Instead, SvelteKit translates the returned object into whatever's required by the platform you're deploying your app to.
UPDATE: The bug was fixed in SvelteKit. This is the updated code that works:
// src/routes/api/file/_file.controller.ts
import { query } from '../_db'
type GetFileResponse = (fileNumber: string) => Promise<{
headers: {
'Content-Disposition': string
'Content-Type': string
}
body: Uint8Array
status?: number
} | {
status: number
headers?: undefined
body?: undefined
}>
export const getFile: GetFileResponse = async (fileNumber: string) => {
const { rows } = await query(`SELECT _id, name, type, data FROM files WHERE _id = $1;`, [fileNumber])
if (rows) {
const file = rows[0]
return {
headers: {
'Content-Disposition': `attachment; filename="${file.name}"`,
'Content-Type': file.type
},
body: new Uint8Array(file.data)
}
} else return {
status: 404
}
}
and
// src/routes/api/file/[filenum].ts
import type { RequestHandler } from '#sveltejs/kit'
import { getFile } from './_file.controller'
export const get: RequestHandler = async ({ params }) => {
const { filenum } = params
const fileResponse = await getFile(filenum)
return fileResponse
}

Troubles uploading files with Graphql

I'm using graphql and trying to upload file to the localhost server from react.
I was following this tutorial from apollo server official documentation: Enabling file uploads in Apollo Server
For some reason, image is corrupt (its empty) and I don't know how to fix it. I've tried number of things with no luck what so ever.
At this point, I'm desperate for any workaround or solution.
typedefs
const {gql} = require('apollo-server')
module.exports = gql`
type File {
url: String!
filename: String!
mimetype: String!
encoding: String!
}
type Mutation {
uploadFile(file: Upload!): File!
}
`
resolvers
const fs = require("fs");
const path = require("path");
module.exports = {
Query: {
test: () => 'Test completed',
files: () => files
},
Mutation: {
uploadFile: async (parent, {file}) => {
const {createReadStream, filename, mimetype, encoding} = await file
const stream = await createReadStream()
const pathName = path.join(__dirname, `../../public/images/${filename}`)
await stream.pipe(fs.createWriteStream(pathName))
return {
filename,
mimetype,
encoding,
url: `http://locahost:5000/images/${filename}`}
},
}
}
Apollo client setup
import React from 'react'
import App from './App'
import ApolloClient from 'apollo-client'
import {InMemoryCache} from 'apollo-cache-inmemory'
import {createUploadLink} from 'apollo-upload-client'
import {ApolloProvider} from '#apollo/react-hooks'
import {setContext} from 'apollo-link-context'
const httpLink = createUploadLink({
uri: 'http://localhost:5000/graphql/'
})
const authLink = setContext(() => {
const token = localStorage.getItem('jwtToken')
return {
headers: {
authorization: token ? `Bearer ${token}` : ''
}
}
})
const client = new ApolloClient({
link: authLink.concat(httpLink),
cache: new InMemoryCache()
})
export default (
<ApolloProvider client={client}>
<App/>
</ApolloProvider>
)
React code itself
import React from 'react'
import {useMutation, gql} from '#apollo/client'
const UPLOAD_FILE = gql`
mutation uploadFile($file: Upload!) {
uploadFile(file: $file) {
url
mimetype
filename
encoding
}
}
`
function UploadForm() {
const [uploadFile] = useMutation(UPLOAD_FILE, {
onCompleted: data => console.log(data)
})
const handleFileChange = e => {
const file = e.target.files[0]
if(!file) return
uploadFile({variables: {file}})
}
return (
<div>
<h2>Upload file</h2>
<input type='file' onChange={handleFileChange}/>
</div>
)
}
export default UploadForm
console output
uploadFile: {…}
​​
__typename: "File"
​​
encoding: "7bit"
​​
filename: "91f29c99.jpg"
​​
mimetype: "image/jpeg"
​​
url: "http://locahost:5000/images/91f29c99.jpg"
UPD: I've tried this approach. Now sometimes I get non-empty images, but they're still corrupt and unreadable. But most of the times its still empty.
uploadFile: async (_, {file}) => {
const {createReadStream, filename} = await file;
await new Promise((resolve, reject) =>
createReadStream()
.pipe(fs.createWriteStream(path.join(__dirname, `../../public/images/${filename}`))
.on("finish", () => resolve())
.on("error", reject))
)
return {
url
}
},

File upload from React Native ( expo ) to Node ( multer )

How can I upload a file( pdf, docs etc) from React Native using expo to the server using node. I've seen many examples for images using the expo image-picker api but I've come across none that uses document-picker or filesystem apis from expo. The expo file system documentation was a little hard to interpret for a beginner like me.
Thanks for the help. I was able to come up with a solution and I'll post it below so it can be of some use to whoever comes here in the future.
React Native
import React, { useState } from 'react';
import { Button, View } from 'react-native';
import * as DocumentPicker from 'expo-document-picker';
import * as FileSystem from 'expo-file-system';
const DocPicker = () => {
const [ doc, setDoc ] = useState();
const pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({ type: "*/*", copyToCacheDirectory: true }).then(response => {
if (response.type == 'success') {
let { name, size, uri } = response;
let nameParts = name.split('.');
let fileType = nameParts[nameParts.length - 1];
var fileToUpload = {
name: name,
size: size,
uri: uri,
type: "application/" + fileType
};
console.log(fileToUpload, '...............file')
setDoc(fileToUpload);
}
});
// console.log(result);
console.log("Doc: " + doc.uri);
}
const postDocument = () => {
const url = "http://192.168.10.107:8000/upload";
const fileUri = doc.uri;
const formData = new FormData();
formData.append('document', doc);
const options = {
method: 'POST',
body: formData,
headers: {
Accept: 'application/json',
'Content-Type': 'multipart/form-data',
},
};
console.log(formData);
fetch(url, options).catch((error) => console.log(error));
}
return (
<View>
<Button title="Select Document" onPress={pickDocument} />
<Button title="Upload" onPress={postDocument} />
</View>
)
};
export default DocPicker;
Node.js
const express = require('express')
const bodyParser = require('body-parser')
var multer = require('multer')
var upload = multer({ dest: 'uploads/' })
const app = express()
const fs = require('fs')
const http = require('http')
const port = 8000
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.get('/', (req,res) => {
res.json({
success: true
})
})
app.post('/', (req, res) => {
console.log(req.body)
res.status(200)
})
app.post('/upload', upload.single('document'),(req , res) => {
console.log(req.file, req.body)
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
Cheers!!!
If the solution given by #Anandhu doesn't work then try the above code like this.
import React, { useState } from 'react';
import { Button, View } from 'react-native';
import * as DocumentPicker from 'expo-document-picker';
import * as FileSystem from 'expo-file-system';
const DocPicker = () => {
const [ doc, setDoc ] = useState();
const pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({
type: "*/*",
copyToCacheDirectory: true })
.then(response => {
if (response.type == 'success') {
let { name, size, uri } = response;
/ ------------------------/
if (Platform.OS === "android" && uri[0] === "/") {
uri = `file://${uri}`;
uri = uri.replace(/%/g, "%25");
}
/ ------------------------/
let nameParts = name.split('.');
let fileType = nameParts[nameParts.length - 1];
var fileToUpload = {
name: name,
size: size,
uri: uri,
type: "application/" + fileType
};
console.log(fileToUpload, '...............file')
setDoc(fileToUpload);
}
});
// console.log(result);
console.log("Doc: " + doc.uri);
}
const postDocument = () => {
const url = "http://192.168.10.107:8000/upload";
const fileUri = doc.uri;
const formData = new FormData();
formData.append('document', doc);
const options = {
method: 'POST',
body: formData,
headers: {
Accept: 'application/json',
'Content-Type': 'multipart/form-data',
},
};
console.log(formData);
fetch(url, options).catch((error) => console.log(error));
}
return (
<View>
<Button title="Select Document" onPress={pickDocument} />
<Button title="Upload" onPress={postDocument} />
</View>
)
};
export default DocPicker;
There is a bug in the way the path was encoded, and the file:// scheme is missing.
This bug may be fixed in next release.
Try this Uploading pictures,documents and videos from your phone in your app with React Native, Expo
Here is an example, which also uses multer and express on the backend: https://github.com/expo/examples/tree/master/with-formdata-image-upload
That said, I'd recommend using FileSystem.uploadAsync instead of fetch and the background sessionType in order to support uploads while the app is backgrounded on iOS.

Using loopback4 and graphQL together

import {Lb4Application} from './application';
import {ApplicationConfig} from '#loopback/core';
import graphqlHTTP from 'express-graphql';
import {createGraphQLSchema} from 'openapi-to-graphql';
import {Oas3} from 'openapi-to-graphql/lib/types/oas3';
export {Lb4Application};
export async function main(options: ApplicationConfig = {}) {
const app = new Lb4Application(options);
await app.boot();
await app.start();
const url: string = <string>app.restServer.url;
console.log(`REST API Server is running at ${url}`);
const graphqlPath = '/graphql';
const oas: Oas3 = <Oas3>(<unknown>app.restServer.getApiSpec());
const {schema} = await createGraphQLSchema(oas, {
strict: false,
viewer: false,
baseUrl: url,
headers: {
'X-Origin': 'GraphQL',
},
tokenJSONpath: '$.jwt',
});
const handler: graphqlHTTP.Middleware = graphqlHTTP(
(request, response, graphQLParams) => ({
schema,
pretty: true,
graphiql: true,
context: {jwt: getJwt(request)},
}),
);
// Get the jwt from the Authorization header and place in context.jwt, which is then referenced in tokenJSONpath
function getJwt(req: any) {
if (req.headers && req.headers.authorization) {
return req.headers.authorization.replace(/^Bearer /, '');
}
}
app.mountExpressRouter(graphqlPath, handler);
console.log(`Graphql API: ${url}${graphqlPath}`);
return app;
}
I have taken this code from this github issue, and I still cannot seem to get it to run.
The error is get is
Error: Invalid specification provided
When i just use an express server, and run npx openapi-to-graphql --port=3001 http://localhost:3000/openapi.json --fillEmptyResponses The graphql is served correctly.
I need to get the example code running, as it seems to be the only way to pass JWT token headers correctly when using loopback4 and graphql together
This is how i solved it for anyone that needs help:
/* eslint-disable #typescript-eslint/no-explicit-any */
import {Lb4GraphqlPocApplication} from './application';
import {ApplicationConfig} from '#loopback/core';
const graphqlHTTP = require('express-graphql');
const {createGraphQLSchema} = require('openapi-to-graphql');
const fetch = require('node-fetch');
export {Lb4GraphqlPocApplication};
export async function main(options: ApplicationConfig = {}) {
console.log('hello world!')
const app = new Lb4GraphqlPocApplication(options);
await app.boot();
await app.start();
const url = app.restServer.url;
const graphqlPath = '/graphql';
console.log(`REST Server is running at ${url}`);
console.log(`Try ${url}/ping`);
// replace with process.env.{active-environment} once deployments setup
const openApiSchema = 'http://localhost:3000/openapi.json';
const oas = await fetch(openApiSchema)
.then((res: any) => {
console.log(`JSON schema loaded successfully from ${openApiSchema}`);
return res.json();
})
.catch((err: any) => {
console.error('ERROR: ', err);
throw err;
});
const {schema} = await createGraphQLSchema(oas, {
strict: false,
viewer: true,
baseUrl: url,
headers: {
'X-Origin': 'GraphQL',
},
tokenJSONpath: '$.jwt',
});
const handler = graphqlHTTP(
(request: any, response: any, graphQLParams: any) => ({
schema,
pretty: true,
graphiql: true,
context: {jwt: getJwt(request)},
}),
);
// Get the jwt from the Authorization header and place in context.jwt, which is then referenced in tokenJSONpath
function getJwt(req: any) {
if (req.headers && req.headers.authorization) {
return req.headers.authorization.replace(/^Bearer /, '');
}
}
app.mountExpressRouter(graphqlPath, handler);
console.log(`Graphql API: ${url}${graphqlPath}`);
return app;
}

Resources