I'm proofing out an integration test with NestJS/KafkaJS.
I have everything implemented except the function on the event listener (consumer) for the topic I'm emitting to is not being called.
I read somewhere you can't consume a message until the consumer event GROUP_JOIN has completed, not sure if this is right, and/or how I could get my e2e test to wait for this to happen?
Here's the setup of the e2e test -
describe('InventoryController(e2e)', () => {
let app: INestApplication
let client: ClientKafka
beforeAll(async () => {
const moduleFixture: TestingModule = await Test.createTestingModule({
imports: [InventoryModule, KafkaClientModule],
}).compile()
app = moduleFixture.createNestApplication()
await app.connectMicroservice({
transport: Transport.KAFKA,
options: {
client: {
clientId: 'test_clientid',
brokers: process.env.KAFKA_BROKERS.split(' '), // []
ssl: true,
sasl: {
mechanism: 'plain',
username: process.env.KAFKA_CLUSTER_APIKEY,
password: process.env.KAFKA_CLUSTER_SECRET,
},
},
consumer: {
groupId: 'test_consumerids',
},
},
})
await app.startAllMicroservices()
await app.init()
client = moduleFixture.get<ClientKafka>('test_name')
await client.connect()
await app.listen(process.env.port || 3000)
})
afterAll(async () => {
await app.close()
await client.close()
})
it('/ (GET)', async () => {
return request(app.getHttpServer()).get('/inventory/kafka-inventory-test')
})
it('Emits a message to a topic', async () => {
await client.emit('inventory-test', { foo: 'bar' })
})
The client is emitting the message fine,
In my controller I have the event handler for the topic 'inventory-test'
#EventPattern('inventory-test')
async consumeInventoryTest(
// eslint-disable-next-line
#Payload() inventoryMessage: any,
#Ctx() context: KafkaContext,
): Promise<void> {
console.log('inventory-test consumer')
}
I have also logged the microservice with the app.getMicroservices() method and can see under the messageHandlers object it has 'inventory-test' which returns a function
server: ServerKafka {
messageHandlers: Map(1) { 'inventory-test' => [Function] }
Also the message handler is working when I run the app locally.
I've been searching google a lot and the docs for both kafkajs and nest, there isn't a lot of info out there
Thanks for any advice/help!
I actually finally solved this, you need to await a new promise after your client emits a message for the handler to have time to read it.
Related
I'm testing a service in my NestJs application which calls a factory to receive an object with a method up(), which I'm mocking like this:
export const mockExecutorFactory = {
getExecutor: jest.fn().mockImplementation(() => {
return {
up: jest.fn().mockImplementation((_updatedInstance, _parameters) => {
console.log('****************************');
return Promise.resolve({ successful: true });
}),
};
}),
};
In my describe block for the service test I initialize the executor like this:
let executor;
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
InstanceService,
{ provide: getRepositoryToken(Instance), useValue: mockInstanceRepository },
ExecutorFactory,
],
})
.overrideProvider(ExecutorFactory)
.useValue(mockExecutorFactory)
.compile();
service = module.get<InstanceService>(InstanceService);
executor = module.get<ExecutorFactory>(ExecutorFactory).getExecutor();
});
And the test its self is written like this:
it('should create a new instance with default settings', async () => {
// Check if instance created correctly
expect(
await service.createInstance(
MOCK_INSTANCE_CREATE_PARAMS.solutionId,
MOCK_INSTANCE_CREATE_PARAMS.instanceType,
MOCK_INSTANCE_CREATE_PARAMS.orgId,
MOCK_INSTANCE_CREATE_PARAMS.parameters,
MOCK_INSTANCE_CREATE_PARAMS.customPrice,
true,
MOCK_INSTANCE_CREATE_PARAMS.req,
),
).toEqual({ instance: MOCK_INSTANCE_DEFAULT });
console.log(executor);
expect(executor.up).toHaveBeenCalled();
});
Essentially, the createInstance() runs through some logic and at the end is supposed to call the executor's up() function.
From the terminal output (photo below) I can see by the console logs that the the up() is being called, but the test fails... Any idea why?
I have a Node 14 server which is initialized like this:
import express, { Express } from 'express';
import kafkaConsumer from './modules/kafkaConsumer';
async function bootstrap(): Promise<Express> {
kafkaConsumer();
const app = express();
app.get('/health', (_req, res) => {
res.send('ok');
});
return app;
}
export default bootstrap;
kafkaConsumer code:
import logger from './logger.utils';
import KafkaConnector from '../connectors/kafkaConnector';
// singleton
const connectorInstance: KafkaConnector = new KafkaConnector('kafka endpoints', 'consumer group name');
// creating consumer and producer outside of main function in order to not initialize a new consumer producer per each new call.
(async () => {
await connectorInstance.createConsumer('consumer group name');
})();
const kafkaConsumer = async (): Promise<void> => {
const kafkaConsumer = connectorInstance.getConsumer();
await kafkaConsumer.connect();
await kafkaConsumer.subscribe({ topic: 'topic1', fromBeginning: true });
await kafkaConsumer.run({
autoCommit: false, // cancel auto commit in order to control committing
eachMessage: async ({ topic, partition, message }) => {
const messageContent = message.value ? message.value.toString() : '';
logger.info('received message', {
partition,
offset: message.offset,
value: messageContent,
topic
});
// commit message once finished all processing
await kafkaConsumer.commitOffsets([ { topic, partition, offset: message.offset } ]);
}
});
};
export default kafkaConsumer;
You can see that in the kafkaConsumer module there's an async function which is called at the begging to initialize the consumer instance.
How can I guarantee that it successfully passed when importing the module?
In addition, when importing the module, does this mean that the kafkaConsumer default function, is automatically called? won't it cause the server to be essentially stuck at startup?
Would appreciate some guidance here, thanks in advance.
Twicked the Kafka initalzation, and tested with local Kafka. Everything works as expected.
export const produce = async (topic: string, url: string): Promise<void> => {
await producer.connect();
await producer.send({
topic,
compression: CompressionTypes.GZIP,
messages: [{ key: `key-${Math.random() * 100000}`, value: url }],
});
await producer.disconnect();
};
I have a send message kafka like this how can I test this function with jest, because that if we want to run jest we must run service Kafka and zookeeper. what could I do ?
I had try something like :
describe("Helper kafka", () => {
describe("produce", () => {
it("Pass 1 url should return 1", () => {
expect(produce("testing", "URL")).toHaveBeenCalled();
});
});
});
It's failed because the service is not running.
I'm trying to a test a async function of a service in nestJS.
this function is async... basically get a value (JSON) from database (using repository - TypeORM), and when successfully get the data, "transform" to a different class (DTO)...
the implementation:
async getAppConfig(): Promise<ConfigAppDto> {
return this.configRepository.findOne({
key: Equal("APPLICATION"),
}).then(config => {
if (config == null) {
return new class implements ConfigAppDto {
clientId = '';
clientSecret = '';
};
}
return JSON.parse(config.value) as ConfigAppDto;
});
}
using a controller, I checked that this worked ok.
Now, I'm trying to use Jest to do the tests, but with no success...
My problem is how to mock the findOne function from repository..
Edit: I'm trying to use #golevelup/nestjs-testing to mock Repository!
I already mocked the repository, but for some reason, the resolve is never called..
describe('getAppConfig', () => {
const repo = createMock<Repository<Config>>();
beforeEach(async () => {
await Test.createTestingModule({
providers: [
ConfigService,
{
provide: getRepositoryToken(Config),
useValue: repo,
}
],
}).compile();
});
it('should return ConfigApp parameters', async () => {
const mockedConfig = new Config('APPLICATION', '{"clientId": "foo","clientSecret": "bar"}');
repo.findOne.mockResolvedValue(mockedConfig);
expect(await repo.findOne()).toEqual(mockedConfig); // ok
const expectedReturn = new class implements ConfigAppDto {
clientId = 'foo';
clientSecret = 'bar';
};
expect(await service.getAppConfig()).toEqual(expectedReturn);
// jest documentation about async -> https://jestjs.io/docs/en/asynchronous
// return expect(service.getAppConfig()).resolves.toBe(expectedReturn);
});
})
the expect(await repo.findOne()).toEqual(mockedConfig); works great;
expect(await service.getAppConfig()).toEqual(expectedReturn); got a timeout => Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout;
using debug, I see that the service.getAppConfig() is called, the repository.findOne() too, but the .then of repository of findOne is never called.
Update: I'm trying to mock the repository using #golevelup/nestjs-testing, and for some reason, the mocked result don't works on service.
If I mock the repository using only jest (like code below), the test works... so, I think my real problem it's #golevelup/nestjs-testing.
...
provide: getRepositoryToken(Config),
useValue: {
find: jest.fn().mockResolvedValue([new Config()])
},
...
So, my real problem is how I'm mocking the Repository on NestJS.
For some reason, when I mock using the #golevelup/nestjs-testing, weird things happens!
I really don't found a good documentation about this on #golevelup/nestjs-testing, so, I gave up using it.
My solution for the question was to use only Jest and NestJS functions... the result code was:
Service:
// i'm injecting Connection because I need for some transactions later;
constructor(#InjectRepository(Config) private readonly configRepo: Repository<Config>, private connection: Connection) {}
async getAppConfig(): Promise<ConfigApp> {
return this.configRepo.findOne({
key: Equal("APPLICATION"),
}).then(config => {
if (config == null) {
return new ConfigApp();
}
return JSON.parse(config.value) as ConfigApp;
})
}
Test:
describe('getAppConfig', () => {
const configApi = new Config();
configApi.key = 'APPLICATION';
configApi.value = '{"clientId": "foo", "clientSecret": "bar"}';
beforeEach(async () => {
const module = await Test.createTestingModule({
providers: [
ConfigAppService,
{
provide: getRepositoryToken(Config),
useValue: {
findOne: jest.fn().mockResolvedValue(new
Config("APPLICATION", '{"clientId": "foo", "clientSecret": "bar"}')),
},
},
{
provide: getConnectionToken(),
useValue: {},
}
],
}).compile();
service = module.get<ConfigAppService>(ConfigAppService);
});
it('should return ConfigApp parameters', async () => {
const expectedValue: ConfigApp = new ConfigApp("foo", "bar");
return service.getAppConfig().then(value => {
expect(value).toEqual(expectedValue);
})
});
})
some sources utilized for this solution:
https://github.com/jmcdo29/testing-nestjs/tree/master/apps/typeorm-sample
I think expect(await repo.findOne()).toEqual(mockedConfig); works because you mocked it, so it returns right away.
In the case of expect(await service.getAppConfig()).toEqual(expectedReturn);, you did not mock it so it is probably taking more time, thus the it function returns before the Promise resolved completely.
The comments you posted from jest documentation should do the trick if you mock the call to getAppConfig().
service.getAppConfig = jest.fn(() => Promise.resolve(someFakeValue))
or
spyOn(service, 'getAppConfig').and.mockReturnValue(Promise.resolve(fakeValue))
This answer from #roberto-correia made me wonder if there must be something wrong with the way we are using createMock from the package #golevelup/nestjs-testing.
It turns out that the reason why the method exceeds the execution time has to do with the fact that createMock does not implement the mocking, and does not return anything, unless told to do so.
To make the method work, we have to make the mocked methods resolve something at the beginning of the test:
usersRepository.findOneOrFail.mockResolvedValue({ userId: 1, email: "some-random-email#email.com" });
A basic working solution:
describe("UsersService", () => {
let usersService: UsersService;
const usersRepository = createMock<Repository<User>>();
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
UsersService,
{
provide: getRepositoryToken(User),
useValue: usersRepository,
},
}).compile();
usersService = module.get(UsersService);
});
it("should be defined", () => {
expect(usersService).toBeDefined();
});
it("finds a user", async () => {
usersRepository.findOne.mockResolvedValue({ userId: 1, email: "some-random-email#email.com" });
expect(await usersRepository.findOne()).toBe({ userId: 1, email: "some-random-email#email.com" });
});
});
In integration tests I am using the following snippets to create connection
import {Connection, createConnection} from 'typeorm';
// #ts-ignore
import options from './../../../ormconfig.js';
export function connectDb() {
let con: Connection;
beforeAll(async () => {
con = await createConnection(options);
});
afterAll(async () => {
await con.close();
});
}
I am trying to unit test a class which calls typeorm repository in one of its method and without call that helper function connectDb() above I get the following error which is expected of course.
ConnectionNotFoundError: Connection "default" was not found.
My question is how can I mock connection. I have tried the following without any success
import typeorm, {createConnection} from 'typeorm';
// #ts-ignore
import options from "./../../../ormconfig.js";
const mockedTypeorm = typeorm as jest.Mocked<typeof typeorm>;
jest.mock('typeorm');
beforeEach(() => {
//mockedTypeorm.createConnection.mockImplementation(() => createConnection(options)); //Failed
mockedTypeorm.createConnection = jest.fn().mockImplementation(() => typeorm.Connection);
MethodRepository.prototype.changeMethod = jest.fn().mockImplementation(() => {
return true;
});
});
Running tests with that kind of mocking gives this error
TypeError: decorator is not a function
Note: if I call connectDb() in tests everything works fine. But I don't want to do that since it takes too much time as some data are inserted into db before running any test.
Some codes have been omitted for simplicity. Any help will be appreciated
After a bunch of research and experiment I've ended up with this solution. I hope it works for someone else who experienced the same issue...
it does not need any DB connection
testing service layer content, not the DB layer itself
test can cover all the case I need to test without hassle, I just need to provide the right output to related typeorm methods.
This is the method I want to test
#Injectable()
export class TemplatesService {
constructor(private readonly templatesRepository: TemplatesRepository) {}
async list(filter: ListTemplatesReqDTO) {
const qb = this.templatesRepository.createQueryBuilder("tl");
const { searchQuery, ...otherFilters } = filter;
if (filter.languages) {
qb.where("tl.language IN (:...languages)");
}
if (filter.templateTypes) {
qb.where("tl.templateType IN (:...templateTypes)");
}
if (searchQuery) {
qb.where("tl.name LIKE :searchQuery", { searchQuery: `%${searchQuery}%` });
}
if (filter.skip) {
qb.skip(filter.skip);
}
if (filter.take) {
qb.take(filter.take);
}
if (filter.sort) {
qb.orderBy(filter.sort, filter.order === "ASC" ? "ASC" : "DESC");
}
return qb.setParameters(otherFilters).getManyAndCount();
}
...
}
This is the test:
import { SinonStub, createSandbox, restore, stub } from "sinon";
import * as typeorm from "typeorm";
describe("TemplatesService", () => {
let service: TemplatesService;
let repo: TemplatesRepository;
const sandbox = createSandbox();
const connectionStub = sandbox.createStubInstance(typeorm.Connection);
const templatesRepoStub = sandbox.createStubInstance(TemplatesRepository);
const queryBuilderStub = sandbox.createStubInstance(typeorm.SelectQueryBuilder);
stub(typeorm, "createConnection").resolves((connectionStub as unknown) as typeorm.Connection);
connectionStub.getCustomRepository
.withArgs(TemplatesRepository)
.returns((templatesRepoStub as unknown) as TemplatesRepository);
beforeAll(async () => {
const builder: TestingModuleBuilder = Test.createTestingModule({
imports: [
TypeOrmModule.forRoot({
type: "postgres",
database: "test",
entities: [Template],
synchronize: true,
dropSchema: true
})
],
providers: [ApiGuard, TemplatesService, TemplatesRepository],
controllers: []
});
const module = await builder.compile();
service = module.get<TemplatesService>(TemplatesService);
repo = module.get<TemplatesRepository>(TemplatesRepository);
});
beforeEach(async () => {
// do something
});
afterEach(() => {
sandbox.restore();
restore();
});
it("Service should be defined", () => {
expect(service).toBeDefined();
});
describe("list", () => {
let fakeCreateQueryBuilder;
it("should return records", async () => {
stub(queryBuilderStub, "skip" as any).returnsThis();
stub(queryBuilderStub, "take" as any).returnsThis();
stub(queryBuilderStub, "sort" as any).returnsThis();
stub(queryBuilderStub, "setParameters" as any).returnsThis();
stub(queryBuilderStub, "getManyAndCount" as any).resolves([
templatesRepoMocksListSuccess,
templatesRepoMocksListSuccess.length
]);
fakeCreateQueryBuilder = stub(repo, "createQueryBuilder" as any).returns(queryBuilderStub);
const [items, totalCount] = await service.list({});
expect(fakeCreateQueryBuilder.calledOnce).toBe(true);
expect(fakeCreateQueryBuilder.calledOnce).toBe(true);
expect(items.length).toBeGreaterThan(0);
expect(totalCount).toBeGreaterThan(0);
});
});
});
cheers!