I've recently added Bull to my project to offload things like synchronizing documents to 3rd party services and everything's working well, except errors occurring while processing jobs don't end up in Sentry. They're only logged on the jobs themselves, but since we're running our application on multiple configurations, it means I have to constantly monitor all these instances for job processing errors.
I know I can add an error handler to a processor, but I have quite a few processors already, so I'd prefer another, more global, solution
Is there any way to make sure these errors are also sent to Sentry?
I wasn't able to find a way to do this globally, but I was able to create a base processor class that implemented the OnQueueFailed Event Listener and reported failures to sentry. I have all my processors inherit from it and it seems to work well.
Base Class:
import { OnQueueFailed } from '#nestjs/bull';
import { Logger } from '#nestjs/common';
import * as Sentry from '#sentry/node';
import { Job } from 'bull';
export abstract class BaseProcessor {
protected abstract logger: Logger;
#OnQueueFailed()
onError(job: Job<any>, error: any) {
Sentry.captureException(error);
this.logger.error(
`Failed job ${job.id} of type ${job.name}: ${error.message}`,
error.stack,
);
}
}
Processor:
import { InjectQueue, Process, Processor } from '#nestjs/bull';
import { Logger } from '#nestjs/common';
import { Job, Queue } from 'bull';
import { BaseProcessor } from 'src/common/BaseProcessor';
import { BULL_QUEUES } from 'src/common/queues';
#Processor(BULL_QUEUES.SOME_QUEUE_NAME)
export class SomeProcessor extends BaseProcessor {
protected readonly logger = new Logger(SomeProcessor.name);
constructor(
// dependencies
) {
super();
}
#Process()
async processTask(job: Job) {
// processor code here
}
}
Related
I'm trying to develop a healthcheck endpoint with NestJS (in which I have no experience). One of the dependencies I want to check is Twilio's SMS service. So far, the best URL I've found to gather this information is https://status.twilio.com/api/v2/status.json. The problem here is that I don't want to merely ping this address, but to gather it's JSON response and present some of the information it provides, namely these:
Is it possible, using (or not) the Terminus module? In the official docs I didn't find anything regarding this, only simpler examples using pingCheck / responseCheck: https://docs.nestjs.com/recipes/terminus
Yes, it is possible.
I have never used this, but HttpHealthIndicator has responseCheck method to check depends on the API response message. You can specify a callback function to analyze responses from the API. The callback function should return boolean represents the status of the API.
I couldn't find this in the documents, but you can see it here.
Although meanwhile the logic for this healthcheck has changed (and so this question became obsolete), this was the temporary solution I've found, before it happened (basically a regular endpoint using axios, as pointed out in one of the comments above):
Controller
import { Controller, Get } from '#nestjs/common';
import { TwilioStatusService } from './twilio-status.service';
#Controller('status')
export class TwilioStatusController {
constructor(private readonly twilioStatusService: TwilioStatusService) {}
#Get('twilio')
getTwilioStatus() {
const res = this.twilioStatusService.getTwilioStatus();
return res;
}
}
Service
import { HttpService } from '#nestjs/axios';
import { Injectable } from '#nestjs/common';
import { map } from 'rxjs/operators';
#Injectable()
export class TwilioStatusService {
constructor(private httpService: HttpService) {}
getTwilioStatus() {
return this.httpService
.get('https://status.twilio.com/api/v2/status.json')
.pipe(map((response) => response.data.status));
}
}
Of course this wasn't an optimal solution, since I had to do this endpoint + a separated one for checking MongoDB's availability (a regular NestJS healthcheck, using Terminus), the goal being an healthcheck that glued both endpoints together.
It is possible to merge in any property to the resulting object. You can see that in the TypeScript Interface
/**
* The result object of a health indicator
* #publicApi
*/
export declare type HealthIndicatorResult = {
/**
* The key of the health indicator which should be uniqe
*/
[key: string]: {
/**
* The status if the given health indicator was successful or not
*/
status: HealthIndicatorStatus;
/**
* Optional settings of the health indicator result
*/
[optionalKeys: string]: any;
};
};
And here is an example:
diagnostics/health/healthcheck.controller
import { Controller, Get } from '#nestjs/common'
import { ApiTags } from '#nestjs/swagger'
import { HttpService } from '#nestjs/axios'
import { HealthCheckService, HealthCheck, HealthIndicatorStatus, HealthCheckError } from '#nestjs/terminus'
#ApiTags('diagnostics')
#Controller('diagnostics/health')
export class HealthController {
constructor(
private health: HealthCheckService,
private httpService: HttpService,
) { }
#Get()
#HealthCheck()
check() {
return this.health.check([
() => this.httpService.get('http://localhost:9002/api/v1/diagnostics/health').toPromise().then(({ statusText, config: { url }, data }) => {
const status: HealthIndicatorStatus = statusText === 'OK' ? 'up' : 'down'
return { 'other-service': { status, url, data } }
}).catch(({ code, config: { url } }) => {
throw new HealthCheckError('Other service check failed', { 'other-service': { status: 'down', code, url } })
}),
])
}
}
diagnostics/diagnostics.module.ts
import { Module } from '#nestjs/common'
import { TerminusModule } from '#nestjs/terminus'
import { HttpModule } from '#nestjs/axios'
import { HealthController } from './health/health.controller'
#Module({
imports: [
HttpModule,
TerminusModule,
],
controllers: [HealthController],
})
export class DiagnosticsModule { }
I have a mistake that I can't understand what I'm doing wrong. I am following the instructions set out at https://docs.nestjs.com/techniques/mongodb. The difference is that I created interfaces to build different strategies for implementing a service (and for that I am making a wrapper in Model ).
import { Inject, Injectable } from "#nestjs/common";
import { Model } from "mongoose";
import { Documents } from "src/domain/schemas/documents.schema";
interface IDocumentRepository {
save();
}
interface IDocumentRepositoryFactory {
new(doc?: any): IDocumentRepository;
}
export function createIDocumentRepository(ctor: IDocumentRepositoryFactory, doc?: any): IDocumentRepository {
return new ctor(doc);
}
#Injectable()
export class DocumentRepository implements IDocumentRepository {
constructor(
private doc?: Documents,
#Inject()
private repository?: Model<Documents>
) {}
/* others fields and methods */
save() {
this.doc.save()
}
}
In others code points I call:
someMethod(repository: DocumentRepository) {
/* others codes */
const mdoc = new this.repository(mongoModelDoc); // <<<<---- error
mdoc.save();
/* others codes */
}
It is causing the following error:
This expression is not constructable. Type 'DocumentRepository' has no construct signatures.
return new this.repository(document)
What is wrong and how to resolve to meet this implementation?
I am working on an API with NestJS, and because I have DTO's I am using an AutoMapper (made by #nartc and/or nestjsx), I have tried to make my example as small as I could with the Foo example, because I use multiple files.
This is my module:
// foo.module.ts
import { Module } from "#nestjs/common";
import { MongooseModule } from "#nestjs/mongoose";
import { Foo, FooSchema } from "./foo.entity.ts";
import { FooController } from "./foo.controller.ts";
import { FooService } from "./foo.service.ts";
import { FooProfile } from "./foo.profile.ts";
#Module({
imports: [
MongooseModule.forFeature([
{
name: Foo.name,
schema: FooSchema,
collection: "foos",
}
])
// FooProfile <-- if I uncomment this, the program will give the error (shown at the bottom of this question)
],
controllers: [FooController],
providers: [FooProivder],
})
export class FooModule {}
This is my entity:
// foo.entity.ts
import { Schema, SchemaFactory, Prop } from "#nestjs/mongoose";
import { Document } from "mongoose";
#Schema()
export class Foo extends Document { // if I remove the `extends Document` it works just fine
#Prop({ required: true })
name: string;
#Prop()
age: number
}
export const FooSchema = SchemaFactory.createForClass(Foo);
This is my DTO:
// foo.dto.ts
export class FooDTO {
name: string;
}
This is my controller:
// foo.controller.ts
import { Controller, Get } from "#nestjs/common";
import { InjectMapper, AutoMapper } from "nestjsx-automapper";
import { Foo } from "./foo.entity";
import { FooService } from "./foo.service";
import { FooDTO } from "./dto/foo.dto";
#Controller("foos")
export class FooController {
constructor(
private readonly fooService: FooService
#InjectMapper() private readonly mapper: AutoMapper
) {}
#Get()
async findAll() {
const foos = await this.fooService.findAll();
const mappedFoos = this.mapper.mapArray(foos, Foo, FooDTO);
// ^^ this throws an error of the profile being undefined (duh)
return mappedFoos;
}
}
This is my profile:
// foo.profile.ts
import { Profile, ProfileBase, InjectMapper, AutoMapper } from "nestjsx-automapper";
import { Foo } from "./foo.entity";
import { FooDTO } from "./foo.dto";
#Profile()
export class FooProfile extends ProfileBase {
constructor(#InjectMapper() private readonly mapper: AutoMapper) {
// I've read somewhere that the `#InjectMapper() private readonly` part isn't needed,
// but if I exclude that, it doesn't get the mapper instance. (mapper will be undefined)
super();
this.mapper.createMap(Foo, FooDTO);
}
}
If I uncomment the line I highlighted in the module, it will result in the following error..
[Nest] 11360 - 2020-08-18 15:53:06 [ExceptionHandler] Cannot read property 'plugin' of undefined +1ms
TypeError: Cannot read property 'plugin' of undefined
at Foo.Document.$__setSchema ($MYPATH\node_modules\mongoose\lib\document.js:2883:10)
at new Document ($MYPATH\node_modules\mongoose\lib\document.js:82:10)
at new Foo($MYPATH\dist\foo\foo.entity.js:15:17)
I have also referred to this answer on stackoverflow, but that doesn't work for me either. I have also combined that with the documentation, but with no luck.. How would I get the AutoMapper to register my profiles?
Update
The error seems to originate from the foo entity, if I remove the extends Document and the Schema(), Prop({ ... }) from the class it works fine, it seems like I have to inject mongoose or something?
In your module, just import the path to the profile like below:
import 'relative/path/to/foo.profile';
By importing the path to file, TypeScript will include the file in the bundle and then the #Profile() decorator will be executed. When #Profile() is executed, AutoMapperModule keeps track of all the Profiles then when it's turn for NestJS to initialize AutoMapperModule (with withMapper() method), AutoMapperModule will automatically add the Profiles to the Mapper instance.
With that said, in your FooProfile's constructor, you'll get AutoMapper instance that this profile will be added to
#Profile()
export class FooProfile extends ProfileBase {
// this is the correct syntax. You would only need private/public access modifier
// if you're not going to use this.mapper outside of the constructor
// You DON'T need #InjectMapper() because that's Dependency Injection of NestJS.
// Profile isn't a part of NestJS's DI
constructor(mapper: AutoMapper) {
}
}
The above answer will solve your problems with AutoMapper. As far as your Mongoose problem, I would need a sample repro to tell for sure. And also, visit our Discord for this kind of question.
What worked for me.
1. Updated all the absolute paths for models, schemas, entities (is easy if you search for from '/src in your projects, and update all the routes to relative paths)
from:
import { User } from 'src/app/auth/models/user/user.entity';
to:
import { User } from './../../auth/models/user/user.entity';
2. mongoose imports:
from:
import mongoose from 'mongoose';
to:
import * as mongoose from 'mongoose';
3. Remove validation pipe if you don't use it. For some reason (I think i don't use them on the controller, I didn't investigate, I've removed from one controller the Validation Pipe) so If you have this try it:
from:
#Controller('someroute')
#UsePipes(new ValidationPipe())
export class SomeController {}
to:
#Controller('someroute')
export class SomeController {}
I hope my solution worked for you ^_^
I need to access a service (provided by Nest TypeOrmModule) inside the intercept function (important note: not as constructor parameter!!!) because it depends of the passed options (entity in this case).
The service injection token is provided by the getRepositoryToken function.
export class PaginationInterceptor {
constructor(private readonly entity: Function) {}
intercept(context: ExecutionContext, call$: Observable<any>): Observable<any> {
// Here I want to inject the TypeORM repository.
// I know how to get the injection token, but not HOW to
// get the "injector" itself.
const repository = getRepositoryToken(this.entity);
// stuff...
return call$;
}
}
Is any concept of "service container" in Nest? This github issue didn't help me.
Example usage (controller action):
#Get()
#UseInterceptors(new PaginationInterceptor(Customer))
async getAll() {
// stuff...
}
Regarding dependency injection (if you really want/need it), I guess using a mixin class can do the trick. See the v4 documentation (Advanced > Mixin classes).
import { NestInterceptor, ExecutionContext, mixin, Inject } from '#nestjs/common';
import { getRepositoryToken } from '#nestjs/typeorm';
import { Observable } from 'rxjs';
import { Repository } from 'typeorm';
export function mixinPaginationInterceptor<T extends new (...args: any[]) => any>(entityClass: T) {
// declare the class here as we can't give it "as-is" to `mixin` because of the decorator in its constructor
class PaginationInterceptor implements NestInterceptor {
constructor(#Inject(getRepositoryToken(entityClass)) private readonly repository: Repository<T>) {}
intercept(context: ExecutionContext, $call: Observable<any>) {
// do your things with `this.repository`
return $call;
}
}
return mixin(PaginationInterceptor);
}
Disclaimer: this is valid TypeScript code but I didn't had the chance to test it in a real project, so it might need a bit of rework. The idea is to use it like this:
#UseInterceptors(mixinPaginationInterceptor(YourEntityClass))
Let me know if you have any question about the code. But I think the doc about mixin is pretty good!
OR You can also use getRepository from typeorm (passing the entity class). This is not DI, thus, it will oblige you to spyOn the getRepository function in order to do proper testing.
Regarding the container, I'm almost sure that the only way to access it is using the Execution Context, as pointed by Kim.
the Spring Batch docs say of the Map-backed job repository:
Note that the in-memory repository is volatile and so does not allow restart between JVM instances. It also cannot guarantee that two job instances with the same parameters are launched simultaneously, and is not suitable for use in a multi-threaded Job, or a locally partitioned Step. So use the database version of the repository wherever you need those features.
I would like to use a Map job repository, and I do not care about restarting, prevention of concurrent job executions, etc. but I do care about being able to use multi-threading and local partitioning.
My batch application has some partitioned steps, and at first glance it seems to run just fine with a Map-backed job repository.
What is the reason it said to be not possible with MapJobRepositoryFactoryBean? Looking at the implementation of Map DAOs, they are using ConcurrentHashMap. Is this not thread-safe ?
I would advise you to follow the documentation, rather than relying on implementation details. Even if the maps are individually thread-safe, there might be race conditions in changes than involve more than one of these maps.
You can use an in-memory database very easily. Example
#Grapes([
#Grab('org.springframework:spring-jdbc:4.0.5.RELEASE'),
#Grab('com.h2database:h2:1.3.175'),
#Grab('org.springframework.batch:spring-batch-core:3.0.6.RELEASE'),
// must be passed with -cp, for whatever reason the GroovyClassLoader
// is not used for com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver
//#Grab('org.codehaus.jettison:jettison:1.2'),
])
import org.h2.jdbcx.JdbcDataSource
import org.springframework.batch.core.Job
import org.springframework.batch.core.JobParameters
import org.springframework.batch.core.Step
import org.springframework.batch.core.StepContribution
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory
import org.springframework.batch.core.launch.JobLauncher
import org.springframework.batch.core.scope.context.ChunkContext
import org.springframework.batch.core.step.tasklet.Tasklet
import org.springframework.batch.repeat.RepeatStatus
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.context.annotation.AnnotationConfigApplicationContext
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.core.io.ResourceLoader
import org.springframework.jdbc.datasource.init.DatabasePopulatorUtils
import org.springframework.jdbc.datasource.init.ResourceDatabasePopulator
import javax.annotation.PostConstruct
import javax.sql.DataSource
#Configuration
#EnableBatchProcessing
class AppConfig {
#Autowired
private JobBuilderFactory jobs
#Autowired
private StepBuilderFactory steps
#Bean
public Job job() {
return jobs.get("myJob").start(step1()).build()
}
#Bean
Step step1() {
this.steps.get('step1')
.tasklet(new MyTasklet())
.build()
}
#Bean
DataSource dataSource() {
new JdbcDataSource().with {
url = 'jdbc:h2:mem:temp_db;DB_CLOSE_DELAY=-1'
user = 'sa'
password = 'sa'
it
}
}
#Bean
BatchSchemaPopulator batchSchemaPopulator() {
new BatchSchemaPopulator()
}
}
class BatchSchemaPopulator {
#Autowired
ResourceLoader resourceLoader
#Autowired
DataSource dataSource
#PostConstruct
void init() {
def populator = new ResourceDatabasePopulator()
populator.addScript(
resourceLoader.getResource(
'classpath:/org/springframework/batch/core/schema-h2.sql'))
DatabasePopulatorUtils.execute populator, dataSource
}
}
class MyTasklet implements Tasklet {
#Override
RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
println 'TEST!'
}
}
def ctx = new AnnotationConfigApplicationContext(AppConfig)
def launcher = ctx.getBean(JobLauncher)
def jobExecution = launcher.run(ctx.getBean(Job), new JobParameters([:]))
println "Status is: ${jobExecution.status}"