I am looking for an option to use nest as a back-end Gateway service -
The idea is to poll on DB changes ( and maybe later to move it to event driven ) - de facto no listener would be required here.
On change the Nest would update a 3rd pt API calls
What would be best practice here ?
Take a look here, I'm doing something similar to what you're after: https://github.com/nerdybeast/sith-api/blob/feature/redis-cache/src/modules/api/sobjects/trace-flag/TraceFlagPoller.ts
I created a class that "polls" a backend and emits an event when it detects a change in that backend. You could have other code that listens for this event which makes the call to your 3rd party api.
UPDATE:
As you stated, Nest does have a basic application context which skips the http service setup, here's how you can do that:
index.ts
import { NestFactory } from '#nestjs/core';
import { ApplicationModule } from './ApplicationModule';
import { DatabaseService } from './DatabaseService';
(async () => {
const app = await NestFactory.createApplicationContext(ApplicationModule);
const databaseService = app.get<DatabaseService>(DatabaseService);
await databaseService.poll();
})();
DatabaseService.ts
#Injectable()
export class DatabaseService {
private expectedResult: any;
public async poll() : Promise<void> {
const result = await getData();
if(result !== this.expectedResult) {
this.expectedResult = result;
await axios.post('https://some-url.com', result);
}
//Poll every 5 seconds or whatever
setTimeout(() => this.poll(), 5000);
}
}
This could be the solution if you had to poll the database instead of being able to subscribe to it. With this approach, when you start the app, it will poll forever, constantly updating your 3rd party api.
I would start the index.ts file with pm2 or forever so that you can have a graceful restart if your process crashes for some reason.
I would personally use typeORM subscribers like I have done many times for similar requirements. However I use an eventEmitter to not block the saving action. This is a snippet of what I usually do.
#Injectable()
export class EntityUpdate implements EntitySubscriberInterface {
constructor(
#InjectConnection() readonly connection: Connection,
#InjectEventManager() emitter: AppEvents<AbstractEntity>,
) {
connection.subscribers.push(this);
}
afterInsert(event: InsertEvent<AbstractEntity>): void {
this.emitter('entity', {
method: 'update',
entity,
});
}
}
Then I could listen to the event anywhere within my application and handle that status change of the entity
Related
I am using NestJS HttpModule to make GET request for one end point. The code is somewhat like this:
#Injectable
export class AnimalService {
constructor(private httpService: HttpService){}
getAnimalData(variant: string): Observable<AxiosResponse<Animal>> {
return this.httpService
.get(`http://animal.test/${variant}`)
.pipe(map((response) => response.data));
}
}
And now I want to create a method which calls multiple endpoints simultaneously.
getAllAnimalsData() {
// const variants = ['birds', 'cats', 'dogs'];
// call
// http://animal.test/birds
// http://animal.test/cats
// http://animal.test/dogs
// simultaneously
// and process the response data
}
How can I achieve this using NestJS HttpModule?
How can I process every result?
How can I handle if there is a partial error (e.g. 1 of 3 request has an error)?
If you want to make multiple http requests simultaneously you can use the RxJS forkJoin operator. I also added a catchError to each Observable so that any errors will be passed to the subscribe callback. The subscribe callback triggers when all Observables are finished.
getAllAnimalsData() {
const variants = ['birds', 'cats', 'dogs'];
forkJoin(
variants.map(v =>
getAnimalData(v).pipe(catchError(e => of(e)))
)
)
.subscribe(([r1, r2, r3]) => /* ... */)
}
Let's say I have a module that provides a connection to something.
export const CONN string = 'CONN';
#Global()
#Module({})
export class ConnModule implements OnApplicationShutdown {
constructor(#Inject(CONN) private readonly conn: Connection) {}
onApplicationShutdown() {
this.conn.close();
}
static forRoot(options): DynamicModule {
const connectionProvider = {
provide: CONN,
useFactory: async (): Promise<Connection> => {
const connection = new Connection(options);
await connection.establish();
return connection;
},
};
return {
module: ConnModule ,
providers: [connectionProvider],
exports: [connectionProvider],
};
}
}
This connection is used through the application:
export class AService {
constructor(
#Inject(CONN) protected readonly conn: Connection,
) {}
someMethod() {
this.conn.doSomething(); // this may get 401ed
}
export class BService {
constructor(
#Inject(CONN) protected readonly conn: Connection,
) {}
otherMethod() {
this.conn.doSomethingElse(); // this may get 401ed
}
This is a connection to 3rd party app and it may start returning 401 anytime needing us to reauthenticate / recreate a connection.
How can I create a new connection on error and replace the current provider for all the modules that are using it?
Something like:
OnConnectionUnauthorizedException ->
create a new Connection ->
replace the current stale connection object that CONN provides with a new one ->
retry the original action
Thank you!
This sounds concerning to me. As it is hard to communicate to the outside what the status of your application is. Also, what will happen to processing events/requests during the time of the connection re-establishing? You are going to handle a lot of accidental complexity. Consider health checks at least to expose your application's status.
Yet there is a solution that I can imagine: Instead of providing the connection, provide a service that holds the connection in a private field. Make a getConection method, possibly async to gap disconnected times, and return the connection. This way you don't have to replace the injected value.
I would furthermore really discourage replacing an injected value. As this is really hard to communicate to code that might be dependent on this value. ("e.g. hey here is a new instance).
export class ConnectionService {
connection: Conection | undefined;
// Pro Tip: make getConnection async and leverage e.g. Promise.race to implement
// a timeout so that if the connection is down this code e.g. delays 5 seconds and
// if the connection gets re-established, answers with the new connection or throws
// a "Couldnt reconnect" error
getConnection(){
return this.connection;
}
}
This solution should be your last resort and instead, you should challenge the source of the required reconnect.
When I create an observable and I am done with it I unsubscribe it directly
const data$ = this.httpClient.get('https://jsonplaceholder.typicode.com/todos/1').subscribe(res => {
console.log('live', res);
data$.unsubscribe(); // <---- works fine
});
But say if I create an Observable using of and try to do the same
const obs$ = of(1).subscribe(e => {
console.log('test', e)
obs$.unsubscribe(); // <--- Problem while creating Observable by of
});
Whats different between these 2 observables?
Your code should be import Subscription and unsubscribe in ngOnDestroy
import { Observable, Subscription, of } from "rxjs";
private subscription$: Subscription;
this.subscription$ = of(1).subscribe(e => {
console.log('test', e)
});
ngOnDestroy() {
this.subscription$.unsubscribe();
}
Update: What I understand is http request is an observable that potentialy have incoming value in the future and of simply create a list of observable
And from #cartant comment
of completes synchronously, so you are attempting to call
obs$.unsubscribe before the assignment to obs$ has been made.
If you have only one observable in your component that would be the easiest scenario for unsubscribing by the following way:
ngOnDestroy() {
this.subscription$.unsubscribe();
}
the more complex scenario is having multiple observables, at that moment you will not to push each observable while subscribing in subscriptions: Subscription[] and while destroying do unsubscribe all added subscriptions in the array ngOnDestroy(){
this.subscriptions.forEach(sub => sub.unsubscribe());
}
or you can use that npm package
For ensuring unsubscription in a component, I would recommend creating a Subject which emits in ngOnDestroy() like this:
destroy$ = new Subject<boolean>();
...
ngOnDestroy() {
this.destroy$.next(true);
}
And adding a takeUntil on each Observable in the component:
myObs$.pipe(
takeUntil(this.destroy$),
tap(stuff => doStuff())
).subscribe();
This way you avoid polluting things with loads of unnecessary Subscription variables.
I have multiple #WebsocketGateways in my project, one implementing the OnGatewayConnection life cycle hook. It seems like the life cycle hook is called once for each Gateway, even though only one is implementing them. Is this the default behavior, a bug or am I doing something wrong?
CommonGateway
#WebSocketGateway()
export class CommonGateway implements OnGatewayConnection, OnGatewayDisconnect {
#WebSocketServer() server;
users: number = 0;
handleConnection() {
this.users++;
console.log('USER CONNECTED: ', this.users);
}
handleDisconnect() {
this.users--;
console.log('USER Disconnected: ', this.users);
}
}
DatesGateway
import {
WebSocketGateway,
SubscribeMessage,
WsResponse,
} from '#nestjs/websockets';
import { CommonService } from 'src/common/common.service';
#WebSocketGateway()
export class DatesGateway {
constructor(private readonly commonService: CommonService) {}
#SubscribeMessage('dates-now')
onNow(client): Promise<WsResponse<Date>> {
return Promise.resolve(this.commonService.now).then(now => ({
event: 'dates-now',
data: now,
}));
}
}
Console screenshot
A small repo demonstrating the issue can be found here
thanks
Update
This was fixed in 6.10.13.
Kamil's guidance to leave the #WebSocketGateway decorator empty now applies in the case where you have multiple WebSocket gateways (and wish to avoid handleConnection being called multiple times).
Pre v6.10.13
I just encountered this issue after creating several gateways. I'm using version 6.10.1 of #nestjs/platform-ws and #nestjs/websockets.
With the solution below, a single WebSocket server will be created and handleConnection will be called once (as desired). The handleDisconnect event always (correctly) fired once. It is important to note that:
A namespace must be included when defining the gateway.
When using the platform-ws adapter the namespaces can be either unique or identical.
When using the platform-socket.io adapter the namespaces must be identical.
A port must be included when defining the gateway. The ports should be identical but different from the HTTP server port to avoid EADDRINUSE and UnhandledPromiseRejectionWarning: Error: The HTTP/S server is already being used by another WebSocket server errors.
If you create many gateways you may see the MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit warning. I still need to look into this.
app.gateway.ts
#WebSocketGateway(8080, {namespace: 'foo'})
export class AppGateway implements OnGatewayConnection, OnGatewayDisconnect {
constructor() {}
public handleConnection(): void {}
public handleDisconnect(): void {}
#SubscribeMessage('app:event')
public handleGet(): WsResponse<string> {
return {
event: 'app:event',
data: 'foobar'
};
}
}
user.gateway.ts
#WebSocketGateway(8080, {namespace: 'foo'})
export class UserGateway {
constructor() {}
#SubscribeMessage('user:event')
public handleGet(): WsResponse<string> {
return {
event: 'user:event',
data: 'foobar'
};
}
}
main.ts
import { NestFactory } from '#nestjs/core';
import { AppModule } from './app.module';
import { WsAdapter } from '#nestjs/platform-ws';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.useWebSocketAdapter(new WsAdapter(app));
await app.listen(80);
}
bootstrap();
More insight into this would be welcome.
I found a solution.
Obviously it is a bug (?) in NestJS that it initiates multiple namespaces for all of your Gateways if you dont define any namespace at all. Then for each is a connection event.
The solution is to define the same namespace on each of your WebSocketGateways
#WebSocketGateway({namespace: 'yournamespace'})
And connect to the server in the client like so:
io.connect('http://localhost:4200/yournamespace');
This is because you have declared the gateway multiple times in modules instead of declaring it as an export in the main module and importing it in another. I also had a similar problem and found either removing an exporting or setting your module as a global decorator module helped. Please research the global decorator for the module before proceeding to use it: https://docs.nestjs.com/modules
I have many async functions in my system, so I need to go "async all the way down", which is to the point where the http.Server and express.Application app are created.
(This is unavoidable in an async system - there will be many async routines which are needed in constructors, which cannot be done, and so we need to use async factory functions instead, which lead to async creep all the way down to the entry point.)
But I'm not sure of the Node/TypeScript syntax to use to bootstrap the app.
My main entry point is System.ts:
class default export System {
public constructor() {
// init Express.Application
// init http.Server
// init other parts of the system
}
public async start(): Promise<void> {
// start the system asynchronously
// start listening with http.Server
}
}
Then I have a bootstrapping module Main.ts:
import System from "./System"
const system = new System();
export default ???; // PROBLEM IS HERE
Which should be run:
node ./dist/Main.js
But I'm not sure what to use in the export line. I tried all these:
export default await system.start(); // doesn't compile (obviously)
export default system.start(); // doesn't seem right
export default system.start().then(); // this works *maybe*
The last line works based on a smoke test - but I'm not sure if that's the way to do it, and whether there's something down the line that may fail.
What is the canonical way to start an asynchronous node app?
UPDATE
Based on #JacobGillespie's answer, the Main.ts bootstrapping module is now:
import System from "./System"
new System().start().then();
//new System().start().catch(e => console.error(e)); // alternative
In my case, System.ts has handlers for errors and unhandled promises, and does logging (otherwise use the "alternative" line). So the bootstrapping module just bootstraps the system.
async / await here are operating on promises, so you essentially want to "start" the promise by calling .then or .catch.
My go-to snippet for this is creating an async run or main function, then attaching error handling to the process, something like this:
async function run() {
// run the app, you can await stuff in here
}
run().catch(err => {
console.error(err.stack)
process.exit(1)
})
In your case that would look like (Main.ts):
import System from "./System"
async function run() {
const system = new System()
await system.start()
}
run().catch(err => {
console.error(err.stack)
process.exit(1)
})
You don't need to export anything since this module file isn't being imported anywhere else (it's the entry file).
You can just call system.then() or system.catch(), but personally I like the async function run() pattern since you may need to coordinate more than one async thing in the future and this makes the code more explicit.
system.start().then() => {
value => export default value
}
In my opinion, a better way would be:
System.ts:
function System():Promise<string>{
//setup express and the server
return new Promise((res,rej) => {
//the server var is just the http server instance
server.listen(8000,() => resolve("server created"));
});
}
export {System}
And then in Main.ts:
import {System} from "yourpath"
And then:
System().then(() => {
//code runs when server is created
}).catch(err => console.error(err));