How to use Nginx NJS with native nodejs modules and webpack? - node.js

There is a guide https://nginx.org/en/docs/njs/node_modules.html guide that describes how to use 'native' nodejs modules with njs.
I followed the guide until I did not understand what it means in bold:
Note that in this example generated code is not wrapped into function
and we do not need to call it explicitly. The result is in the "dist" directory:
$ cat dist/wp_out.js code.js > njs_dns_bundle.js
Let's call our code at the end of a file: <<<--- HERE
var b = set_buffer(global.dns);
console.log(b);
And execute it using node:
$ node ./njs_dns_bundle_final.js
Question is how do I include / require / import the webpack generated njs_dns_bundle.js in njs_dns_bundle_final.js which is the name of the Let's call our code at the end of a file since without it I get the error:
njs_dns_bundle_final.js:1
var b = set_buffer(global.dns);
ReferenceError: set_buffer is not defined
My code.js:
module.exports = {
hello: function set_buffer(dnsPacket) {
// create DNS packet bytes
var buf = dnsPacket.encode({
type: 'query',
id: 1,
flags: dnsPacket.RECURSION_DESIRED,
questions: [{
type: 'A',
name: 'google.com'
}]
})
return buf;
}
}
My njs_dns_bundle_final.js:
var myModule = require('./njs_dns_bundle');
var b = myModule.hello(global.dns);
console.log(b);
Node runs fine I think?!:
node ./njs_dns_bundle_final.js
<Buffer 00 01 01 00 00 01 00 00 00 00 00 00 06 67 6f 6f 67 6c 65 03 63 6f 6d 00 00 01 00 01>
NJS does not:
njs ./njs_dns_bundle_final.js
Thrown:
Error: Cannot find module "./njs_dns_bundle"
at require (native)
at main (njs_dns_bundle_final.js:1)
Thanks

Related

change multer uploaded file to ReadStream

I'm using nestjs, multer to read uploaded files.
file is well uploade via POST rest api.
I want to convert. this file to ReadableStream.
I want to avoid using write this files in disk and read again using createReadStream,
it would be better convert direct to ReadableStream using uploaded meta infos.
export function ApiFile(fieldName: string) {
return applyDecorators(UseInterceptors(FileInterceptor(fieldName)));
}
#Post("/file_upload")
#ApiFile('file')
create(
#Body() createNewsDto: CreateNewsDto,
#UploadedFile() file: Express.Multer.File,
) {
console.log({ file });
return this.myService.create(createNewsDto, file);
}
this is file meta data
{
file: {
fieldname: 'file',
originalname: 'screenshot.png',
encoding: '7bit',
mimetype: 'image/png',
buffer: <Buffer 59 10 4f 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 02 cf 00 00 02 1b 08 06 00 00 00 14 dd 73 8e 00 00 01 55 61 43 43 50 49 43 43 20 50 72 6f 66 69 ... 298432 more bytes>,
size: 298982
}
}
how can I achieve this?
I figured out how to send my files to remote server.
you need to use ReadableStream.from which change buffer file to ReadStream.
if your file meta info is below,
{
file: {
fieldname: 'file',
originalname: 'screenshot.png',
encoding: '7bit',
mimetype: 'image/png',
buffer: <Buffer 59 10 4f 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 02 cf 00 00 02 1b 08 06 00 00 00 14 dd 73 8e 00 00 01 55 61 43 43 50 49 43 43 20 50 72 6f 66 69 ... 298432 more bytes>,
size: 298982
}
}
you can convert this meta into stream
import { Readable} from 'stream';
import * as FormData from "form-data";
const formData = new FormData();
const stream = Readable.from(file.buffer);
formData.append("anyKeyValue", stream, {
filename: file.originalname,
contentType: file.mimetype
})
then send to remote server with content type multipart/form-data

Gulp image-min - How to continue after an error?

I am trying to run an image optimisation task on a folder with 1000s of images, unfortunately some of them are corrupted so the task keeps failing
I have tried to capture the error and continue, but its not working once it hits a corrupted image the task then aborts. Is there a way for this to continue running?
gulp.task('imgCompress', (done) => {
gulp.src(imgMedia)
.pipe(imagemin([
mozjpeg({
quality: 75
}),
pngquant({
quality: [0.65, 0.80]
})
]))
.on('error', gutil.log)
.pipe(gulp.dest(imgMediaDest))
done();
});
Error:
{ Error: write EPIPE
at WriteWrap.afterWrite [as oncomplete] (net.js:789:14)
errno: 'EPIPE',
code: 'EPIPE',
syscall: 'write',
stdout:
<Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff e1 2b 18 45 78 69 66 00 00 4d 4d 00 2a 00 00 00 08 00 0b 01 0f 00 02 00 00 00 06 00 00 ... >,
stderr: <Buffer >,
failed: true,
signal: null,
cmd:
'/private/var/www/website/m2/tools-media/node_modules/mozjpeg/vendor/cjpeg -quality 75',
timedOut: false,
killed: false,
message: 'write EPIPE',
name: 'Error',
stack:
'Error: write EPIPE\n at WriteWrap.afterWrite [as oncomplete] (net.js:789:14)',
__safety: undefined,
_stack: undefined,
plugin: 'gulp-imagemin',
showProperties: true,
showStack: false,
fileName:
'/private/var/www/website/m2/pub/media/catalog/product/1/i/1img_5232.jpg' }
You should be fine with gulp-plumber, which lets the stream continue after an error and even has an option to log error messages automatically without handling error events.
Also note that you should not call done after creating a stream, but after the stream has terminated (you could for example listen to finish events). But it's easier to simply return the stream from the task function.
const plumber = require('gulp-plumber');
gulp.task('imgCompress', () => {
return gulp.src(imgMedia)
.pipe(plumber({ errorHandler: true }))
.pipe(imagemin([
mozjpeg({
quality: 75
}),
pngquant({
quality: [0.65, 0.80]
})
]))
.pipe(gulp.dest(imgMediaDest))
});

How to save a FormData image file locally with "fs" in Node

I have a React front-end that allows user to send an image using FormData() to the node backend.
In the Express backend's controller, I am using multer as a middleware to grab the image into the files variable:
private initializeRoutes() {
this.router.post(`${this.path}/image`, multerUpload.array("filesToUpload[]"), this.saveImage);
}
In the backend's service, I am trying to save files[0] into the uploads folder:
public saveImage(files: any) {
console.log("OtherServiceLocal saveImage - files :");
console.log(files[0]); // see screenshot at bottom for output
let localPath = fs.createWriteStream("./uploads/image.png");
files[0].buffer.pipe(localPath);
}
But I am getting the error:
I tried piping file[0] and file[0].buffer to no avail, and I had difficulty understanding how to transform it into a stream even after some research.
This is the outputs for console.log(files[0]);
{
fieldname: 'filesToUpload[]',
originalname: 'Screen Shot 2021-09-08 at 3.42.48 PM.png',
encoding: '7bit',
mimetype: 'image/png',
buffer: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 03 85 00 00 02 c5 08 06 00 00 00 74 ff 46 78 00 00 00 01 73 52 47 42 00 ae ce 1c e9 00 00 00 62 ... 249405 more bytes>,
size: 249455
}
Please note that I am aware that you can use multer's upload method to save the image as a middleware directly in the router, but I can't to do this due to some requirement of my app.
Thanks,
file[0].buffer is an instance of Buffer.
So you can use fs.writeFile directly
fs.writeFile("./uploads/image.png", file[0].buffer, (err) => {
console.error(error)
})

Unable to get screenshot of window through winapi

Using the Node.js ffi-napi package, I'm attempting to get a buffer of bitmap data from a screenshot of a given window or desktop if no window handle is supplied. To that end, I'm trying to port the c++ example from the microsoft documentation and making the api calls through ffi-napi.
Even though all the api calls come back without errors, I only end up with a buffer filled with 0s. I've traced it down to a few potential pieces that might be causing it but I don't know which piece is incorrect and how to fix it.
Even though my call to BitBlt is returning true, the pixel values I get from calling GetPixel on my source window dc is returning correct values but when I call GetPixel on my memory dc, I get 0s.
My call to GetObjectA seems to be populating the BITMAP struct I created, (my structs are just extensions of buffers), except the last 8 bytes which is supposed to hold the pointer to the bitmap data is all 0s.
My call to GetDIBits is returning 1080 which is the correct number of rows it should have read from the bitmap data but the buffer I get back is all 0s.
My guess is that BitBlt didn't actually copy the desktop dc to the memory dc even though it's returning true and I don't know why that is. I've also tried running the calculator app and passing in the dc to the screenshot function as well and even though it returns the appropriate items, the buffer I get back is still all 0s.
Any help would be greatly appreciated!
This is my node.js code
function screenshot(hWnd = null) {
let hdcWindow = null;
let hdcMemDC = null;
let hbmScreen = null;
let hDIB = null;
try {
if (!hWnd) hWnd = user32.GetDesktopWindow();
console.log('hWnd', hWnd);
// Retrieve the handle to a display device context for the client area of the window.
hdcWindow = user32.GetDC(hWnd);
console.log('hdcWindow', hdcWindow);
const rcClient = new win32_structs.RECT();
user32.GetClientRect(hWnd, rcClient);
console.log('rcClient', rcClient);
// Create a compatible DC and bitmap
hdcMemDC = gdi32.CreateCompatibleDC(hdcWindow);
console.log('hdcMemDC', hdcMemDC);
hbmScreen = gdi32.CreateCompatibleBitmap(hdcWindow, rcClient.right - rcClient.left, rcClient.bottom - rcClient.top);
console.log('hbmScreen', hbmScreen);
const hPrevDC = gdi32.SelectObject(hdcMemDC, hbmScreen);
console.log('hPrevDC', hPrevDC);
// Bit block transfer into our compatible memory DC.
const bitBltRes = gdi32.BitBlt(hdcMemDC, 0, 0, rcClient.right - rcClient.left, rcClient.bottom - rcClient.top, hdcWindow, 0, 0, apiConstants.SRCCOPY);
const pixelWnd = gdi32.GetPixel(hdcWindow, 0, 0);
const pixelMem = gdi32.GetPixel(hdcMemDC, 0, 0);
console.log('pixelWnd', pixelWnd);
console.log('pixelMem', pixelMem);
console.log('bitBltRes', bitBltRes);
// Get the BITMAP from the HBITMAP
const bmpScreen = new win32_structs.BITMAP();
const getObjectRes = gdi32.GetObjectA(hbmScreen, bmpScreen.length, bmpScreen);
console.log('getObjectRes', getObjectRes);
console.log('bmpScreen.length', bmpScreen.length);
console.log('bmpScreen', bmpScreen);
const bi = new win32_structs.BITMAPINFOHEADER();
bi.biSize = bi.length;
bi.biWidth = bmpScreen.bmWidth;
bi.biHeight = bmpScreen.bmHeight;
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = apiConstants.BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
console.log('bi', bi);
const dwBmpSize = ((bmpScreen.bmWidth * bi.biBitCount + 31) / 32) * 4 * bmpScreen.bmHeight;
console.log('dwBmpSize', dwBmpSize);
// Starting with 32-bit Windows, GlobalAlloc and LocalAlloc are implemented as wrapper functions that
// call HeapAlloc using a handle to the process's default heap. Therefore, GlobalAlloc and LocalAlloc
// have greater overhead than HeapAlloc.
// hDIB = kernel32.GlobalAlloc(apiConstants.GHND, dwBmpSize);
// const lpBitmap = kernel32.GlobalLock(hDIB);
const lpBitmap = new Buffer.alloc(dwBmpSize);
// Gets the "bits" from the bitmap and copies them into buffer lpbitmap
const getDIBitsRes = gdi32.GetDIBits(hdcWindow, hbmScreen, 0, bmpScreen.bmHeight, lpBitmap, bi, apiConstants.DIB_RGB_COLORS);
console.log('getDIBitsRes', getDIBitsRes);
console.log('lpBitmap', lpBitmap);
for (const c of lpBitmap) {
if (c > 0) {
console.log(c);
break;
}
}
// clean up
if (hDIB != null) {
kernel32.GlobalUnlock(hDIB);
kernel32.GlobalFree(hDIB);
}
if (hbmScreen != null) gdi32.DeleteObject(hbmScreen);
if (hdcMemDC != null) gdi32.DeleteObject(hdcMemDC);
if (hdcWindow != null) user32.ReleaseDC(hWnd, hdcWindow);
return lpBitmap;
} catch (err) {
// clean up memory on errors
if (hDIB != null) {
kernel32.GlobalUnlock(hDIB);
kernel32.GlobalFree(hDIB);
}
if (hbmScreen != null) gdi32.DeleteObject(hbmScreen);
if (hdcMemDC != null) gdi32.DeleteObject(hdcMemDC);
if (hdcWindow != null) user32.ReleaseDC(hWnd, hdcWindow);
throw err;
}
}
Here's my console log:
hWnd 65552
hdcWindow 83954845
rcClient <Buffer#0x000001BE188E8670 00 00 00 00 00 00 00 00 80 07 00 00 38 04 00 00, _structProps: { left: { offset: 0, dataType: 'long' }, top: { offset: 4, dataType: 'long' }, right: { offset: 8, dataType: 'long'
}, bottom: { offset: 12, dataType: 'long' } }, left: 0, top: 0, right: 1920, bottom: 1080>
hdcMemDC 1375804638
hbmScreen 990189696
hPrevDC 8716303
pixelWnd { r: 231, g: 234, b: 237 }
pixelMem { r: 0, g: 0, b: 0 }
bitBltRes true
getObjectRes 32
bmpScreen.length 32
bmpScreen <Buffer#0x000001BE188C5CC0 00 00 00 00 80 07 00 00 38 04 00 00 00 1e 00 00 01 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00,
_structProps: { bmType: { offset: 0, dataType: 'long' }, bmWidth: { offset: 4, dataType: 'long' }, bmHeight: { offset: 8, dataType: 'long' }, bmWidthBytes: { offset: 12, dataType: 'long' }, bmPlanes: { offset: 16, dataType: 'uint' }, bmBitsPixel: { offset: 18, dataType: 'uint' }, pointerPadding: { offset: 20, dataType: 'long' }, bmBits: { offset: 24, dataType: 'ulonglong' } },
bmType: 0, bmWidth: 1920, bmHeight: 1080, bmWidthBytes: 7680, bmPlanes: 1, bmBitsPixel: 32, pointerPadding: 0, bmBits: 0n>
bi <Buffer#0x000001BE188C5540 28 00 00 00 80 07 00 00 38 04 00 00 01 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00,
_structProps: { biSize: { offset: 0, dataType: 'ulong' }, biWidth: { offset: 4, dataType: 'long' }, biHeight: { offset: 8, dataType: 'long' }, biPlanes: { offset: 12, dataType: 'uint' }, biBitCount: { offset: 14, dataType: 'uint' }, biCompression: { offset: 16, dataType: 'ulong' }, biSizeImage: { offset: 20, dataType: 'ulong' }, biXPelsPerMeter: { offset: 24, dataType: 'long' }, biYPelsPerMeter: { offset: 28, dataType: 'long' }, biClrUsed: { offset: 32, dataType: 'ulong' }, biClrImportant: { offset: 36, dataType: 'ulong' } },
biSize: 40, biWidth: 1920, biHeight: 1080, biPlanes: 1, biBitCount: 32, biCompression: 0, biSizeImage: 0,
biXPelsPerMeter: 0, biYPelsPerMeter: 0, biClrUsed: 0, biClrImportant: 0>
dwBmpSize 8298585
getDIBitsRes 1080
lpBitmap <Buffer#0x000001BE18AED040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 8298535 more bytes>
The OP mentioned they were able to get their code working in the end, with a simple fix to one of the constants. However, that doesn't help other readers so much, because they don't have the full code for it; so I thought I would try to reproduce the working code that they had. (thanks to #vincitego for the starting point!)
After a whole lot of trial and error, I was able to get my reproduction working, and have published the solution as a component in my Windows FFI library here: https://github.com/Venryx/windows-ffi
Usage:
import {VRect, CaptureScreenshot, GetForegroundWindowHandle} from "windows-ffi";
// First capture a screenshot of a section of the screen.
const screenshot = CaptureScreenshot({
windowHandle: GetForegroundWindowHandle(), // comment to screenshot all windows
rectToCapture: new VRect(0, 0, 800, 600),
});
// The image-data is now stored in the `screenshot.buffer` Buffer object.
// Access it directly (and cheaply) using the helper functions on `screenshot`.
for (let x = 0; x < 800; x++) {
console.log(`Pixel color at [${x}, 0] is:`, screenshot.GetPixel(x, 0).ToHex_RGB());
}
You can use the library as packaged, or, if you prefer, just reference its source-code and extract the parts you need. The main module for screenshot-capturing can be seen here.

Node.js: Download file from s3 and unzip it to a string

I am writing an AWS Lambda function which needs to download files from AWS S3, unzips the file and returns the content in the form of a string.
I am trying this
function getObject(key){
var params = {
Bucket: "my-bucket",
Key: key
}
return new Promise(function (resolve, reject){
s3.getObject(params, function (err, data){
if(err){
reject(err);
}
resolve(zlib.unzipSync(data.Body))
})
})
}
But getting the error
Error: incorrect header check
at Zlib._handle.onerror (zlib.js:363:17)
at Unzip.Zlib._processChunk (zlib.js:524:30)
at zlibBufferSync (zlib.js:239:17)
The data looks like this
{ AcceptRanges: 'bytes',
LastModified: 'Wed, 16 Mar 2016 04:47:10 GMT',
ContentLength: '318',
ETag: '"c3xxxxxxxxxxxxxxxxxxxxxxxxx"',
ContentType: 'binary/octet-stream',
Metadata: {},
Body: <Buffer 50 4b 03 04 14 00 00 08 08 00 f0 ad 6f 48 95 05 00 50 84 00 00 00 b4 00 00 00 2c 00 00 00 30 30 33 32 35 2d 39 31 38 30 34 2d 37 34 33 30 39 2d 41 41 ... >
}
The Body buffer contains zip-compressed data (this is identified by the first few bytes), which is not just plain zlib.
You will need to use some zip module to parse the data and extract the files within. One such library is yauzl which has a fromBuffer() method that you can pass your buffer to and get the file entries.

Resources