I am implementing kafka adapter using NestJS adapters.
Within this adapter, I need to be able to publish and subscribe to Kafka topics. The adapter class is not a controller, so I can't add #EventPattern() methods there.
I can inject kafkaClient and be able to only send messages:
export class KafkaAdapter extends IoAdapter {
constructor(
private readonly app: INestApplicationContext,
private kafkaClient: ClientKafka,
) {
super(app);
}
createIOServer(port: number, options?: any): any {
const server: Server = super.createIOServer(port, options);
const client = this.kafkaClient;
server.adapter(function (nsp) {
// using kafkaClient to send/emit messages
});
return server;
}
}
But it seems I can't subscribe to Kafka topics with this kafkaClient. The consumer is created but not exposed to a public KafkaClient interface. So I forced to initialize another custom consumer, which is creating a separate connection to Kafka, and I don't like it.
Any way to subscribe to Kafka topics outside of controllers?
Related
Im currently implementing a Microservice CQRS architecture with NestJs.
Microservices currently talk through RabbitMQ with each other.
I now want to introduce a BFF.
Problem example: SignUp
When the user signs up, the bff calls the procedure on the auth microservice which dispatches the SignUpUserCommand.
After checking and validating, the bff gets a 201 response (Commands don't return information). The UserSignedUpEvent is triggered inside auth microservice and now persists the user into the database and generates an access and refresh token.
How can the BFF now receive the event result (tokens)? I know it's a practice to let the bff listen to events. Does this mean that the bff must be able to uniquely identify the exact event which corresponds the the api call?
What is a good way to implement the event listening?
You should decide, are u going to use commands or events. It depends on which type of communication you try to achieve, request/response via message broker or publish/subscribe.
Request/Response Approach with Commands:
From Nestjs docs
To enable the request-response message type, Nest creates two logical channels - one is responsible for transferring the data while the other waits for incoming responses.
I don't have good experience with RMQ and mostly I'm using Kafka, but the approach is the same. if you put SignUpUserCommand in registration topic, you get the reply in registration.replay topic. Nest will automatically take care of it .
From Bff you can use:
this.client.send(topic, payload) // Observable is returned and you can subscribe to it.
Auth microservice:
#MessagePattern('registration')
async signUp(data: ...)
Publish/Subscribe Approach with Events:
Bff
async signUp(){
this.client.emit('registration', payload)
}
#EventPattern('registration.complete')
async signUpComplete(data: ...)
Auth microservice:
#EventPattern('registration')
async signUp(data: ...)
async signUpComplete(){
this.client.emit('registration.complete', payload)
}
Just see the difference between send and emit in Nestjs docs.
Thank you for your response/s!
Yes I did it like that now:
The BFF calls the Microservice (RPC) via the rabbit message queue:
// BFF Application / authentication.controller.ts
constructor(private readonly authenticationService: AuthenticationService) {}
#Post('/local/signup')
public async signupLocal(#Body() signUpDto: SignUpDto): Promise<AuthTokensDto> {
return await this.authenticationService.signUpUser(signUpDto);
}
// BFF Application / authentication.service.ts
constructor(
#Inject(RmqChannels.AUTH_SERVICE) private readonly authClient: ClientProxy,
private readonly apiService: ApiService,
) {}
signUpUser(signUpUserDto: SignUpDto): Promise<AuthTokensDto> {
return this.apiService.requestTo<AuthTokensDto>(
this.authClient, // Which microservice to talk to
signUpUserDto, // Payload for the rpc
RmqRpcPatterns.SIGN_UP_USER, // The rpc signature
RmqApiEventPatterns.SIGNED_UP_USER, // Wait for this event to get response
);
}
Inside the BFF Application there is a api.service.ts and an api.controller.ts.
The api.service.ts just forwards the communication. The api.controller.ts just handles the api events and adds them to the event observable:
// BFF Application / api.controller.ts
constructor(private apiService: ApiService, private rmqService: RmqService) {}
#EventPattern(RmqApiEventPatterns.EVENT)
async handleEvent(
#Payload() payload: RmqEventPayload,
#Ctx() ctx?: RmqContext): Promise<void> {
this.apiService.pushEvent(payload);
if (ctx) this.rmqService.ack(ctx);
return;
}
// Note: Its listening for 'api_event' pattern
// The exact Api Event Type is inside the RmqEventPayload (signed_up, user_updated,...)
Now the delicate part: The api.service.ts makes any type of rpc and listens for a correlating api event (rabbit event).
// BFF Application / api.service.ts
private events: Subject<RmqEventPayload> = new Subject();
private events$ = this.events.asObservable();
public pushEvent(event: RmqEventPayload): void {
this.events.next(event);
}
requestTo<T>(
client: ClientProxy,
dto: any,
rpc: RmqRpcPatterns,
apiEvent: RmqApiEventPatterns,
): Promise<T> {
return new Promise(async (resolve, reject) => {
// Generating requestId to identify upcoming api event
const requestId = uuidv4();
// Creating the rpc payload
const payload: RmqRpcPayload = {
requestId,
dto,
};
// Waiting for the associated api event of the to be scheduled rpc call
this.notifyWhen<T>(apiEvent, requestId).then((result) => {
resolve(result);
});
// Invoking the remote procedure
const result = await lastValueFrom(client.send(rpc, payload));
// It result is not true (error in command execution), throw the error it returned
if (result !== true) {
// If the result is a known exception code, throw it
if (Object.values(ExceptionTypesCode).includes(result)) {
reject(ExceptionUtils.ToHttpException(result));
} else {
// else just throw the unknown result
reject(result);
}
}
});
}
public notifyWhen<T>(
pattern: RmqApiEventPatterns,
requestId: string,
): Promise<T> {
// Returning a promise which waits for the associated api event and parses the result dto
return firstValueFrom(
this.events$.pipe(
filter(
(apiEvent: RmqEventPayload) =>
apiEvent.eventPattern === pattern &&
apiEvent.requestId === requestId,
),
map((apiEvent: RmqEventPayload) => apiEvent.dto),
),
);
}
The Microservice RPC returns true when no (rpc) exception was thrown in the command validation process (auth, integrity,...).
When some other Exception was thrown, it translates it to a http exception and throws it
When the async data access/manipulation on the microservice is done, an Rabbit Event is emitted. In the example of the sign up process:
// Microservice Application / signed_up.handler.ts
constructor(
#Inject(RmqChannels.API_SERVICE) private readonly client: ClientProxy,
) {}
// ... done
const payload: RmqEventPayload = {
requestId: event.requestId,
dto: authTokens,
eventPattern: RmqApiEventPatterns.SIGNED_UP_USER,
};
this.client.emit(RmqApiEventPatterns.EVENT, payload);
Hope it's clear enough how the mechanism work.
Here a diagram:
I have a repository:
export class MyRepository
extends
Repository<MyEntity>
{
constructor(
protected readonly _clientId: string
) {
super()
}
// ... methods
}
I need to pass the client id through which isnt known until request time. As such, the only way I know how to do it would be to create a factory which can create the repo at run time (it's in GRPC metadata).
#Injectable()
export class MyRepositoryFactory {
create(clientId: string) {
return new MyRepository(
clientId,
);
}
}
I register this as a provider and then in my controller I call:
const { clientId } = context;
const repository = this.myRepositoryFactory.create(clientId);
However I get the error
"Cannot read property 'findOne' of undefined"
when trying to do a basic typeorm call. I can see this is because instead the repository should be registered in the module imports like:
imports: [
TypeOrmModule.forFeature([ MyRepository, MyEntity ])
],
However this only works when injecting the repository directly, not in a factory. I have no idea how to either overcome this problem, or use a different way of creating the repository at run time with GRPC meta data passed through. Any help would be appreciated, thanks.
you cant call new and have the repository connected to TypeORM.
for that you need to make call to getCustomRepository(MyCustomRepository (or connectio.getCustomRepository) so it would be connected to the connectiong pool and everything TypeORM hides under the hood.
IMO creating a custom repository per request is not such a great idea, maybe you can create a Provider that has scope REQUEST (so the provider would be created per-request) allowing NestJS to inject the repository the 'standard' way, modify it's client attribute and then use it's methods that uses the internal repository methods.
something like
#Injectable({ scope: Scope.REQUEST })
export class RequestRepositoryHandler {
client_id: string
constructor(#InjectRepository(MyEntity) private repo: Repository<MyEntity>){}
// other methods
}
// usage
#Injectable()
export class SomeService {
constructor(private providerRepo: RequestRepositoryHandler){}
async method(client_id: string){
this.providerRepo.client_id = client_id; // the provider is per-request
// now do some work
}
}
I try to implement a request async Response Pattern using pubsub. I'm doing this by using spring Integration. I've defined a topic and two subscriptions on both ends (Event Sender for async response and Event consumer for request). But If consumer sends Response it is send to Sender and Consumer. But consumer Event ist empty. All good so far. My Question is how can i defined a Filter for messages in springs pubsub Integration. It is a Feature in Google pubsub.
I suggest you use Spring Filters to let you take action whether the message should be dropped or carried forward to a message channel. You can see more information.
You can see these examples.
#Filter(inputChannel = "inputChannel", outputChannel = "secretChannel")
boolean filter(Message<?> message) {
String msg = message.getPayload().toString();
return msg.contains("secret");
}
You can see these examples on how to implement the filtering in spring. You can see more examples.
package com.zj.node.contentcenter.controller.content;
import lombok.RequiredArgsConstructor;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
/**
* Producer
*
* #author 01
* #date 2019-08-03
**/
#RestController
#RequiredArgsConstructor
public class TestProducerController {
private final Source source;
#GetMapping("/stream-send-msg")
public String streamSendMsg(String flagHeader) {
source.output().send(
MessageBuilder.withPayload("Message Body")
// Setting the header for filtering messages
.setHeader("flag-header", flagHeader)
.build()
);
return "send message success!";
}
}
I have a listener which listens to events but I also want that listener to call on other services to do stuff as part of the event. ie create a db notification, send an sms etc.
When I create a constructor to inject the dependant Service the Listener stops picking up the events and when I remove the constructor with the Service, it starts working again.
How do I need to structure this for the listener to be able to call other services such as NotificationsService in example below?
client-updated.listener.ts
#Injectable()
export class ClientUpdatedListener {
constructor(
#Inject(NotificationsService) private notificationService) {
}
private readonly logger = new Logger(ClientUpdatedListener.name);
#OnEvent(eventType.CLIENT_UPDATED)
handleClientUpdatedEvent(event: ClientUpdatedEvent) {
this.logger.log('Processing event: ' + eventType.CLIENT_UPDATED );
console.log(event);
this.notificationService.emailClient(event.id);
}
The Notifications Service.
It's a shell at the moment but I expect to perform logic and possibly db calls from within it.
#Injectable()
export class NotificationsService {
constructor(
#Inject(TENANT_CONNECTION) private tenantDb,
) {}
emailClient(id: string) {
console.log(id);
}
}
The calling service code
const clientUpdatedEvent = new ClientUpdatedEvent();
clientUpdatedEvent.id = id;
this.eventEmitter.emit(eventType.CLIENT_UPDATED, clientUpdatedEvent);
You'll notice in the docs that there is an explicit warning that Event Listening Services can not be REQUEST scoped. If you need access to the request scoped service, you'll need to pass on the request information as a part of the event payload, and use the ModuleRef and ContextIdFactory to generate the context id for the current sub-tree and then pulling the NotificationsService from that sub-tree.
I am currently developing a prototype to communicate data between a chatbot and website elements. I am using the Azure Bot Services BotFrameworkAdapter and DirectLine in order to communicate between the two applications.
I am having a problem which I have narrowed down to the 'pollingInterval' property of the DirectLine object. The documentation says:
Clients that poll using HTTP GET should choose a polling interval that matches their intended use.
Service-to-service applications often use a polling interval of 5s or 10s.
Client-facing applications often use a polling interval of 1s, and issue a single additional request shortly after every message that the client sends (to rapidly retrieve a bot's response). This delay can be as short at 300ms but should be tuned based on the bot's speed and transit time. Polling should not be more frequent than once per second for any extended period of time.
To my understanding this is how the DirectLine object receives events from the bot, however I only need the event to trigger once, and not on the polling interval. It seems like there is no way to say "I am finished with this event" and move on. As soon as it is triggered once, it is continuously triggered which is causing issues with the functionality of the application.
I have this BotConnection object that is used to create a DirectLine instance and subscribe event handlers to the received events:
import { DirectLine } from 'botframework-directlinejs';
export default class BotConnection {
constructor() {
//Connect to directline object hosted by the bot
this.connection = new DirectLine({
domain: 'http://localhost:3001/directline',
pollingInterval: 1000,
webSocket: false
})
}
getConnection() {
return this.connection;
}
subscribeEventHandler(name, handle) {
console.log('subscribeEventHandler');
this.connection.activity$
.filter(activity => activity.type === "event" && activity.name === name)
.subscribe(activity => handle(activity));
}
}
I am implementing the botConnection class in my App file like so:
props.botConnection.subscribeEventHandler('changePage', () => console.log('test'));
My Bot file takes a message and sends an event to the page that should be handled once on the client application:
const { ActivityHandler } = require('botbuilder');
class MyBot extends ActivityHandler {
constructor() {
super();
this.onMessage(async (context, next) => {
//Testing directline back-channel functionality
if(context.activity.text === "Change page") {
await context.sendActivity({type: 'event', name: 'changePage', value: '/test'});
}
await next();
}
}
Any help with this issue would be fantastic. I am unsure if there is something supported by Azure or if there is some custom magic that I need to do.
Thanks