Extend writeable stream to become transform stream? - node.js

I'm trying to wrap my head around how you would do this, because you can't just inherit from transform and it doesn't seem like you can just inherit from this particular writeable stream.
Ideally it'd be something like:
const Writeable = require('Writeable');
class Transform extends Writeable {
constructor() {
super();
}
_transform(chunk, encoding) {
}
}
But I can just tell that won't work. The particular writeable stream I'm trying to inherit from would be the tapjs/tap-parser. I'd ideally be able to leverage the event listeners inside of it to parse TAP output.

Hmm, you wrote about Writable, but pasted code for Transform, not sure if you just made a mistake, but if you did this on purpose here is an example for overnighting Transform.
Where line is your data, that you can play with before passing it forward.
let Transform = require("stream").Transform;
class awesome_class_name extends Transform
{
constructor()
{
super()
}
_transform (line, encoding, processed) {
//
// Add the data that came in, to the output stream
//
this.push(line);
//
// We let system know that we finished processing the data.
//
processed();
}
}

I'm not sure about es6 syntax, but here is a traditional Transform stream that receives TAP output and outputs objects describing the various parsed parts:
const Transform = require('stream').Transform;
const inherits = require('util').inherits;
const Parser = require('tap-parser');
function MyTransform() {
const self = this;
this._parser = new Parser();
Transform.call(this, { readableObjectMode: true });
this._parser.on('complete', function(results) {
self.push({ type: 'complete', results });
}).on('assert', function(assert) {
self.push({ type: 'assert', assert });
}).on('comment', function(comment) {
self.push({ type: 'comment', comment });
}).on('plan', function(plan) {
self.push({ type: 'plan', plan });
}).on('version', function(version) {
self.push({ type: 'version', version });
}).on('bailout', function(reason) {
self.push({ type: 'bailout', reason });
}).on('extra', function(extra) {
self.push({ type: 'extra', extra });
});
}
inherits(MyTransform, Transform);
MyTransform.prototype._write = function(chunk, encoding, cb) {
this._parser.write(chunk, cb);
};
MyTransform.prototype._flush = function(cb) {
const self = this;
this._parser.end(function() {
self.push(null);
cb();
});
};

Related

How to test mongoose methods using sinon fakes?

I have the following arrangement of tests using sinon, mocha and chai:
type ModelObject = {
name: string;
model: typeof Categoria | typeof Articulo | typeof Usuario;
fakeMultiple: () => object[];
fakeOne: (id?: string) => object;
}
const models: ModelObject[] = [
{
name: 'categorias',
model: Categoria,
fakeMultiple: () => fakeMultiple({ creator: oneCategoria }),
fakeOne: oneCategoria
},
{
name: 'articulos',
model: Articulo,
fakeMultiple: () => fakeMultiple({ creator: oneArticulo }),
fakeOne: oneArticulo
},
{
name: 'usuarios',
model: Usuario,
fakeMultiple: () => fakeMultiple({ creator: oneUsuario }),
fakeOne: oneUsuario
}
];
const randomModel = models[Math.floor(Math.random() * models.length)];
describe(`v1/${randomModel.name}`, function () {
this.afterEach(function () {
sinon.restore();
});
context.only("When requesting information from an endpoint, this should take the Model of the requested endpoint and query the database for all the elements of that model", function () {
it.only(`Should return a list of elements of ${randomModel.name} model`, function (done) {
const fakes = randomModel.fakeMultiple();
const findFake = sinon.fake.resolves({ [randomModel.name]: fakes });
sinon.replace(randomModel.model, 'find', findFake);
chai.request(app)
.get(`/api/v1/${randomModel.name}`)
.end(
(err, res) => {
expect(res).to.have.status(200);
expect(res.body.data).to.be.an('object');
expect(res.body.data).to.have.property(randomModel.name);
expect(res.body.data[randomModel.name]).to.have.lengthOf(fakes.length);
expect(findFake.calledOnce).to.be.true;
done();
}
)
});
}}
I use this to test an endpoint that arbitrary returns information about a given model. In my controllers, I'm using a dynamic middleware to determine which model is going to be queried, for example, if the route consumed is "api/v1/categorias", it will query for Categorias model. If the route consumed is "api/v1/articulos", it will query for Articulos model, and so on.
To make the query, i use the following service:
import { Articulo } from '../models/articulo';
import { Usuario } from '../models/usuario';
import { Categoria } from '../models/categoria';
import logger from '../config/logging';
import { Model } from 'mongoose';
const determineModel = (model: string): Model<any> => {
switch (model) {
case 'articulos':
return Articulo;
case 'usuarios':
return Usuario;
case 'categorias':
return Categoria;
default:
throw new Error(`Model ${model} not found`);
}
};
export const getInformation = async (schema: string, page: number, limit: number) => {
try {
const model = determineModel(schema);
const data = await model.find().skip((page - 1) * limit).limit(limit);
const dataLength = await model.find().countDocuments();
return {
data,
total: dataLength,
};
} catch (err) {
logger.error(err);
console.log(err);
throw err;
}
};
The problem here lies when running my tests, it seems that is unable to run the .skip() and .limit() methods for my model.find()
error: model.find(...).skip is not a function
TypeError: model.find(...).skip is not a function
I think that I need to fake those methods, because when running the same test without skip and limit, it works as a charm. My problem lies in the fact that I don't know how to fake those, or to see if my guess is correct.
As a note, I have default params for the variables page and limit (1 and 15 respectively) so I'm not passing empty values to the methods.

Using AWS DynamoDBDocumentClient for Marshall/Unmarshall Complex Objects Throws an Error

I'm using AWS SDK (v3) in my NodeJS/Typescript application, specifically their DynamoDBDocumentClient to easily marshall/unmarshall my entities to reduce the amount of code needed to query the database.
As my entities are complex objects, meaning that an instance holds, for example, another class-type, or a array of them; I couldn't find any tutorials online to explain what I'm missing (maybe I'm not and that is how things need to be done) as the document-client makes me marshall them into the regular attribute-values structure which then makes the unmarshall into my entities not as simple as I would expect by their documentation...
Below is the function I've written (mostly copied) to get a client-connection to the database (it is maybe important to note that I'm working with DynamoDB local docker):
import { DynamoDBClient, DynamoDBClientConfig } from '#aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient } from '#aws-sdk/lib-dynamodb';
import { NodeHttpHandler } from '#aws-sdk/node-http-handler';
const getClient = (): DynamoDBDocumentClient => {
if (documentClient) {
return documentClient;
}
const configuration = {} as DynamoDBClientConfig;
configuration.region = 'localhost';
configuration.credentials = {
accessKeyId: 'accessKeyId',
secretAccessKey: 'secretAccessKey',
};
configuration.requestHandler = new NodeHttpHandler({ connectionTimeout: 1000, socketTimeout: 1000 });
configuration.maxAttempts = 2;
configuration.endpoint = 'http://localhost:8000';
client = new DynamoDBClient(configuration);
const marshallOptions = {
// Whether to automatically convert empty strings, blobs, and sets to `null`.
convertEmptyValues: false, // false, by default.
// Whether to remove undefined values while marshalling.
removeUndefinedValues: true, // false, by default.
// Whether to convert typeof object to map attribute.
convertClassInstanceToMap: false, // false, by default. <-- HERE IS THE ISSUE
};
const unmarshallOptions = {
// Whether to return numbers as a string instead of converting them to native JavaScript numbers.
wrapNumbers: false, // false, by default.
};
const translateConfig = { marshallOptions, unmarshallOptions };
documentClient = DynamoDBDocumentClient.from(client, translateConfig);
return documentClient;
};
And here is an abstraction of the PutCommand for an instance of an entity of complex-type:
const createEntity = async (instance: ComplexEntityType): Promise<ComplexEntityType> => {
const client = getClient();
try {
await client.send(
new PutCommand({
TableName: 'TableName',
Item: {
PK: instance.pk,
SK: instance.sk,
instance,
},
}),
);
} catch (error) {
console.log(error);
throw error;
}
return instance;
};
When I try to save an entity of that complex-type I get the error:
Unsupported type passed: [object Object]. Pass options.convertClassInstanceToMap=true to marshall typeof object as map attribute.
Which, as you can see, makes me set the convertClassInstanceToMap in my client-configuration to true, which then makes the simple unmarshall into the complex-type not that simple (needs to go through the attribute-values).
So, am I missing something?

How to pass a child Interface to a parent class?

I have this:
LocationController.ts
import {GenericController} from './_genericController';
interface Response {
id : number,
code: string,
name: string,
type: string,
long: number,
lat: number
}
const fields = ['code','name','type','long','lat'];
class LocationController extends GenericController{
tableName:string = 'location';
fields:Array<any> = fields;
}
const locationController = new LocationController();
const get = async (req, res) => {
await locationController._get(req, res);
}
export {get};
GenericController.ts
interface Response {
id : number
}
export class GenericController{
tableName:string = '';
fields:Array<any> = [];
_get = async (req, res) => {
try{
const id = req.body['id'];
const send = async () => {
const resp : Array<Response> = await db(this.tableName).select(this.fields).where('id', id)
if (resp[0] === undefined) {
// some error handling
}
res.status(status.success).json(resp[0]);
}
await send();
}catch (error){
// some error handling
}
}
}
What I want to do is to pass the Response interface from LocationController to the GenericController parent, so that the response is typed accurately depending on how the child class has defined it. Clearly it doesn't work like this since the interface is defined outside of the class so the parent has no idea about the Response interface in the LocationController.ts file.
I've tried passing interface as an argument in the constructor, that doesn't work. So is there a way I can make this happen? I feel like I'm missing something really simple.
Typically, generics are used in a situation like this. Here's how I'd do it:
interface Response {
id: number;
}
// Note the generic parameter <R extends Response>
export class GenericController<R extends Response> {
tableName: string = "";
fields: Array<any> = [];
_get = async (req, res) => {
try {
const id = req.body["id"];
const send = async () => {
// The array is now properly typed. You don't know the exact type,
// but you do know the constraint - R is some type of `Response`
let resp: Array<R> = await db(this.tableName).select(this.fields).where("id", id);
if (resp[0] === undefined) {
// some error handling
}
res.status(status.success).json(resp[0]);
};
await send();
} catch (error) {
// some error handling
}
};
}
import { GenericController } from "./_genericController";
interface Response {
id: number;
code: string;
name: string;
type: string;
long: number;
lat: number;
}
const fields = ["code", "name", "type", "long", "lat"];
// Here we tell the GenericController exactly what type of Response it's going to get
class LocationController extends GenericController<Response> {
tableName: string = "location";
fields: Array<any> = fields;
}
const locationController = new LocationController();
const get = async (req, res) => {
await locationController._get(req, res);
};
export { get };
If this is not enough and you wish to somehow know the exact response type you're going to get, I believe the only way is a manual check. For example:
import { LocationResponse } from './locationController';
// ... stuff
// Manual runtime type check
if (this.tableName === 'location') {
// Manual cast
resp = resp as Array<LocationResponse>
}
// ...
You could also check the form of resp[0] (if (resp[0].hasOwnProperty('code')) { ... }) and cast accordingly. There are also nicer ways to write this, but the basic idea remains the same.
Generally, a properly written class should be unaware of any classes that inherit from it. Putting child-class-specific logic into your generic controller is a code smell. Though as always, it all depends on a particular situation.

How to return a list of objects from Cypress Custom Commands in type script

I am using Cypress for my end to end Integration tests. I have a use case which involves returning a list of objects from Cypress Custom Commands and I have a difficulty in doing so. Here is my code pointer:
index.ts
declare global {
namespace Cypress {
interface Chainable<Subject> {
getTestDataFromElmoDynamoDB({locale, testType}): Cypress.Chainable<JQuery<expectedData[]>> // ??? not sure what return type should be given here.
}
}
}
Cypress.Commands.add('getTestDataFromDynamoDB', ({locale, testType}) => {
// expectedData is an interface declared. My use case is to return the list of this type.
let presetList: expectedData[]
cy.task('getTestDataFromDynamoDB', {
locale: locale,
testType: testType
}).then((presetData: any) => {
presetList = presetData;
// the whole idea here is to return presetList from cypress task
return cy.wrap(presetList) //??? not sure what should be written here
})
})
sampleSpec.ts
describe('The Sample Test', () => {
it.only('DemoTest', () => {
cy.getTestDataElmoDynamoDB({
locale: env_parameters.env.locale,
testType: "ChangePlan"
}).then((presetlist) => {
// not sure on how to access the list here. Tried wrap and alias but no luck.
presetList.forEach((preset: expectedData) => {
//blah blah blah
})
})
})
})
Did anyone work on similar use case before?
Thanks,
Saahith
Here My own command for doing exactly that.
Cypress.Commands.add("convertArrayOfAlliasedElementsToArrayOfInteractableElements", (arrayOfAlliases) => {
let arrayOfRecievedAlliasValues = []
for (let arrayElement of arrayOfAlliases) {
cy.get(arrayElement)
.then(aelement =>{
arrayOfRecievedAlliasValues.push(aelement)
})
}
return cy.wrap(arrayOfRecievedAlliasValues)
})
The way I do it is to pass it in an array and cy.wrap the array, Because it lets you chain the command with an interactable array.
The key point is - it has to be passed as array or object, because they are Reference types, and in cypress it is hard to work with let/var/const that are value types.
You can also allias the cy.wrapped object if you like.
The way to use it in code is:
cy.convertArrayOfAlliasedElementsToArrayOfInteractableElements(ArayOfElements)
What you asked for can be implemented as follows, but I do not know what type expectedData is, so let's assume that expectedData:string [], but you can replace string[] with your type.
plugins/index.ts
module.exports = (on: any, config: any) => {
on('task', {
getDataFromDB(arg: {locale: string, testType: string}){
// generate some data for an example
const list: string[] = [];
list.push('a', 'b');
return list;
},
});
};
commands.ts
declare global {
namespace Cypress {
interface Chainable<Subject> {
getTestDataElmoDynamoDB(arg: {locale: string, testType: string}): Cypress.Chainable<string[]>
}
}
}
Cypress.Commands.add('getTestDataElmoDynamoDB', (arg: {locale: string, testType: string}) => {
let presetList: string[] = [];
cy.task('getDataFromDB', arg)
.then((presetData?: string[]) => {
expect(presetData).not.be.undefined.and.not.be.empty;
// if the data is incorrect, the code will break earlier on expect, this line for typescript compiler
if (!presetData || !presetData.length) throw new Error('Present data are undefined or empty');
presetList = presetData;
return cy.wrap(presetList); // or you can return cy.wrap(presetData)
});
});
db.spec.ts
describe('Test database methods', () => {
it('When take some test data, expect that the data was received successfully ', () => {
cy.getTestDataElmoDynamoDB({ locale: 'someEnvVar', testType: 'ChangePlan' })
.then((list) => {
expect(list).not.empty.and.not.be.undefined;
cy.log(list); // [a,b]
// You can interact with list here as with a regular array, via forEach();
});
});
});
You can also access and receive data from cy.task directly in the spec file.
describe('Test database methods', () => {
it('When take some test data, expect that the data was received successfully ', () => {
cy.task('getDataFromDB', arg)
.then((list?: string[]) => {
expect(list).not.be.empty.and.not.be.undefined;
cy.log(list); // [a,b] — the same list as in the version above
});
});
});

NodeJS streams not awaiting async

I have run into an issue when testing NodeJS streams. I can't seem to get my project to wait for the output from the Duplex and Transform streams after running a stream.pipeline, even though it is returning a promise. Perhaps I'm missing something, but I believe that the script should wait for the function to return before continuing. The most important part of the project I'm trying to get working is:
// Message system is a duplex (read/write) stream
export class MessageSystem extends Duplex {
constructor() {
super({highWaterMark: 100, readableObjectMode: true, writableObjectMode: true});
}
public _read(size: number): void {
var chunk = this.read();
console.log(`Recieved ${chunk}`);
this.push(chunk);
}
public _write(chunk: Message, encoding: string,
callback: (error?: Error | null | undefined, chunk?: Message) => any): void {
if (chunk.data === null) {
callback(new Error("Message.Data is null"));
} else {
callback();
}
}
}
export class SystemStream extends Transform {
public type: MessageType = MessageType.Global;
public data: Array<Message> = new Array<Message>();
constructor() {
super({highWaterMark: 100, readableObjectMode: true, writableObjectMode: true});
}
public _transform(chunk: Message, encoding: string,
callback: TransformCallback): void {
if (chunk.single && (chunk.type === this.type || chunk.type === MessageType.Global)) {
console.log(`Adding ${chunk}`);
this.data.push(chunk);
chunk = new Message(chunk.data, MessageType.Removed, true);
callback(undefined, chunk); // TODO: Is this correct?
} else if (chunk.type === this.type || chunk.type === MessageType.Global) { // Ours and global
this.data.push(chunk);
callback(undefined, chunk);
} else { // Not ours
callback(undefined, chunk);
}
}
}
export class EngineStream extends SystemStream {
public type: MessageType = MessageType.Engine;
}
export class IOStream extends SystemStream {
public type: MessageType = MessageType.IO;
}
let ms = new MessageSystem();
let es = new EngineStream();
let io = new IOStream();
let pipeline = promisify(Stream.pipeline);
async function start() {
console.log("Running Message System");
console.log("Writing new messages");
ms.write(new Message("Hello"));
ms.write(new Message("world!"));
ms.write(new Message("Engine data", MessageType.Engine));
ms.write(new Message("IO data", MessageType.IO));
ms.write(new Message("Order matters in the pipe, even if Global", MessageType.Global, true));
ms.end(new Message("Final message in the stream"));
console.log("Piping data");
await pipeline(
ms,
es,
io
);
}
Promise.all([start()]).then(() => {
console.log(`Engine Messages to parse: ${es.data.toString()}`);
console.log(`IO Messages to parse: ${io.data.toString()}`);
});
Output should look something like:
Running message system
Writing new messages
Hello
world!
Engine Data
IO Data
Order Matters in the pipe, even if Global
Engine messages to parse: Engine Data
IO messages to parse: IO Data
Any help would be greatly appreciated. Thanks!
Note: I posted this with my other account, and not this one that is my actual account. Apologies for the duplicate.
Edit: I initially had the repo private, but have made it public to help clarify the answer. More usage can be found on the feature/inital_system branch. It can be run with npm start when checked out.
Edit: I've put my custom streams here for verbosity. I think I'm on a better track than before, but now getting a "null" object recieved down the pipeline.
As the documentation states, stream.pipeline is callback-based doesn't return a promise.
It has custom promisified version that can be accessed with util.promisify:
const pipeline = util.promisify(stream.pipeline);
...
await pipeline(...);
After some work of the past couple of days, I've found my answer. The issue was my implementation of the Duplex stream. I have since changed the MessageSystem to be a Transform stream to be easier to manage and work with.
Here is the product:
export class MessageSystem extends Transform {
constructor() {
super({highWaterMark: 100, readableObjectMode: true, writableObjectMode: true});
}
public _transform(chunk: Message, encoding: string,
callback: TransformCallback): void {
try {
let output: string = chunk.toString();
callback(undefined, output);
} catch (err) {
callback(err);
}
}
}
Thank you to #estus for the quick reply and check. Again, I find my answer in the API all along!
An archived repository of my findings can be found in this repository.

Resources