I'm testing a lambda using the serverless framework with the sls offline command, this lambda should connect to my local dynamoDB (initialized with a docker-compose image), and put a new data in Dynamo using aws-sdk, but I can never get the return of the put().promise() function, if I use the get function I don't get any return either .I checked and the data is being entered into dynamodb. Follow the code below
import ILocationData, { CreateLocationDTO } from '#domain/location/data/ILocationData';
import { LocationEntity } from '#domain/location/entities/LocationEntity';
import { uuid } from 'uuidv4';
import DynamoDBClient from './DynamoDBClient';
export default class LocationProvider extends DynamoDBClient implements ILocationData {
private tableName = 'Locations';
public async createLocation(data: CreateLocationDTO): Promise<LocationEntity> {
const toCreateLocation: LocationEntity = {
...data,
locationId: uuid(),
hasOffers: false,
};
try {
const location = await this.client
.put({
TableName: this.tableName,
Item: toCreateLocation,
ReturnValues: 'ALL_OLD',
})
.promise();
console.log(location);
return location.Attributes as LocationEntity;
} catch (err) {
console.log(err);
return {} as LocationEntity;
}
}
}
DynamoDBClient.ts -> Class file
import * as AWS from 'aws-sdk';
import { DocumentClient } from 'aws-sdk/clients/dynamodb';
abstract class DynamoDBClient {
public client: DocumentClient;
private config = {};
constructor() {
if (process.env.IS_OFFLINE) {
this.config = {
region: process.env.DYNAMO_DB_REGION,
accessKeyId: 'xxxx',
secretAccessKey: 'xxxx',
endpoint: process.env.DYNAMO_DB_ENDPOINT,
};
}
this.client = new AWS.DynamoDB.DocumentClient(this.config);
}
}
export default DynamoDBClient;
I assume locationId is your partition key and you assign it to uuid() which will be always unique so you will never update any existing items with your put operation. Put operation returns anything only if there is already existing item with the same partition key which will be overwritten by newly provided item.
Related
I have this code to change the default message from typeorm when a value in a unique column already exists. It just creates a custom message when we get an error 23505.
if (error.code === '23505') {
// message = This COLUMN VALUE already exists.
const message = error.detail.replace(
/^Key \((.*)\)=\((.*)\) (.*)/,
'The $1 $2 already exists.',
);
throw new BadRequestException(message);
}
throw new InternalServerErrorException();
I will have to use it in other services, so I would like to abstract that code.
I think I could just create a helper and then I import and call it wherever I need it. But I don’t know if there is a better solution to use it globally with a filter or an interceptor, so I don’t have to even import and call it in different services.
Is this possible? how can that be done?
If it is not possible, what do you think the best solution would be?
Here all the service code:
#Injectable()
export class MerchantsService {
constructor(
#InjectRepository(Merchant)
private merchantRepository: Repository<Merchant>,
) {}
public async create(createMerchantDto: CreateMerchantDto) {
try {
const user = this.merchantRepository.create({
...createMerchantDto,
documentType: DocumentType.NIT,
isActive: false,
});
await this.merchantRepository.save(user);
const { password, ...merchantData } = createMerchantDto;
return {
...merchantData,
};
} catch (error) {
if (error.code === '23505') {
// message = This COLUMN VALUE already exists.
const message = error.detail.replace(
/^Key \((.*)\)=\((.*)\) (.*)/,
'The $1 $2 already exists.',
);
throw new BadRequestException(message);
}
throw new InternalServerErrorException();
}
}
public async findOneByEmail(email: string): Promise<Merchant | null> {
return this.merchantRepository.findOneBy({ email });
}
}
I created an exception filter for typeORM errors.
This was the result:
import {
ArgumentsHost,
Catch,
ExceptionFilter,
HttpStatus,
InternalServerErrorException,
} from '#nestjs/common';
import { Response } from 'express';
import { QueryFailedError, TypeORMError } from 'typeorm';
type ExceptionResponse = {
statusCode: number;
message: string;
};
#Catch(TypeORMError, QueryFailedError)
export class TypeORMExceptionFilter implements ExceptionFilter {
private defaultExceptionResponse: ExceptionResponse =
new InternalServerErrorException().getResponse() as ExceptionResponse;
private exceptionResponse: ExceptionResponse = this.defaultExceptionResponse;
catch(exception: TypeORMError | QueryFailedError, host: ArgumentsHost) {
const ctx = host.switchToHttp();
const response = ctx.getResponse<Response>();
exception instanceof QueryFailedError &&
this.setQueryFailedErrorResponse(exception);
response
.status(this.exceptionResponse.statusCode)
.json(this.exceptionResponse);
}
private setQueryFailedErrorResponse(exception: QueryFailedError): void {
const error = exception.driverError;
if (error.code === '23505') {
const message = error.detail.replace(
/^Key \((.*)\)=\((.*)\) (.*)/,
'The $1 $2 already exists.',
);
this.exceptionResponse = {
statusCode: HttpStatus.BAD_REQUEST,
message,
};
}
// Other error codes can be handled here
}
// Add more methods here to set a different response for any other typeORM error, if needed.
// All typeORM erros: https://github.com/typeorm/typeorm/tree/master/src/error
}
I set it globally:
import { TypeORMExceptionFilter } from './common';
async function bootstrap() {
//...Other code
app.useGlobalFilters(new TypeORMExceptionFilter());
//...Other code
await app.listen(3000);
}
bootstrap();
And now I don't have to add any code when doing changes in the database:
#Injectable()
export class MerchantsService {
constructor(
#InjectRepository(Merchant)
private merchantRepository: Repository<Merchant>,
) {}
public async create(createMerchantDto: CreateMerchantDto) {
const user = this.merchantRepository.create({
...createMerchantDto,
documentType: DocumentType.NIT,
isActive: false,
});
await this.merchantRepository.save(user);
const { password, ...merchantData } = createMerchantDto;
return {
...merchantData,
};
}
}
Notice that now I don't use try catch because nest is handling the exceptions. When the repository save() method returns an error (actually it is a rejected promise), it is caught in the filter.
I am using AWS CDK to create a userpool and userpool client. I would like to be able to access the userpool id and userpool client id from a lambda once they have been created. I pass these two values to the lambda via environmental variables. Here is my code:
import { Construct } from 'constructs';
import {
IResource,
LambdaIntegration,
MockIntegration,
PassthroughBehavior,
RestApi,
} from 'aws-cdk-lib/aws-apigateway';
import {
NodejsFunction,
NodejsFunctionProps,
} from 'aws-cdk-lib/aws-lambda-nodejs';
import { Runtime } from 'aws-cdk-lib/aws-lambda';
import * as amplify from 'aws-cdk-lib/aws-amplify';
import {
aws_s3,
aws_ec2,
aws_rds,
aws_cognito,
aws_amplify,
Duration,
CfnOutput,
} from 'aws-cdk-lib';
export class FrontendService extends Construct {
constructor(scope: Construct, id: string) {
super(scope, id);
const userPool = new aws_cognito.UserPool(this, 'userpool', {
userPoolName: 'frontend-userpool',
selfSignUpEnabled: true,
signInAliases: {
email: true,
},
autoVerify: { email: true },
});
const userPoolClient = new aws_cognito.UserPoolClient(
this,
'frontend-app-client',
{
userPool,
generateSecret: false,
}
);
const bucket = new aws_s3.Bucket(this, 'FrontendStore');
const nodeJsFunctionProps: NodejsFunctionProps = {
environment: {
BUCKET: bucket.bucketName,
DB_NAME: 'hospoFEDB',
AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
USER_POOL_ID: userPool.userPoolId,
USER_POOL_CLIENT_ID: userPoolClient.userPoolClientId,
},
runtime: Runtime.NODEJS_14_X,
};
const registerLambda = new NodejsFunction(this, 'registerFunction', {
entry: 'dist/lambda/register.js',
memorySize: 1024,
...nodeJsFunctionProps,
});
const registerIntegration = new LambdaIntegration(registerLambda);
const api = new RestApi(this, 'frontend-api', {
restApiName: 'Frontend Service',
description: 'This service serves the frontend.',
});
const registerResource = api.root.addResource('register');
registerResource.addMethod('POST', registerIntegration);
}
}
Here is my lambda function and how I intend to use the USER_POOL_ID and USER_POOL_CLIENT_ID env variables:
import {
CognitoUserPool,
} from 'amazon-cognito-identity-js';
export const handler = async (event: any, context: any) => {
try {
console.log(process.env.USER_POOL_ID);
console.log(process.env.USER_POOL_CLIENT_ID);
const userPool = new CognitoUserPool({
UserPoolId: process.env.USER_POOL_ID as string,
ClientId: process.env.USER_POOL_CLIENT_ID as string,
});
return {
statusCode: 200,
};
} catch (error) {
if (error instanceof Error) {
const body = error.stack || (JSON.stringify(error, null, 2) as any);
return {
statusCode: 400,
headers: {},
body: JSON.stringify(body),
};
}
return {
statusCode: 400,
};
}
};
The idea with this setup is that I would create a cognito user pool and client then be able to pass those id's directly down. Currently if I run this locally via sam local start-api it generates the following USER_POOL_ID : Frontenduserpool87772999. If I try and use this id in the new CognitoUserPool({... part of my lambda function I get the following error:
Error: Invalid UserPoolId format.
If I deploy the app however and execute the lambda function from the deployed environment with the exact same code I get a USER_POOL_ID that looks more like: us-east-1_HAjkUj9hP. This works fine and I do not get the error above.
Should I assume that I can not create a user pool locally and will always have to point to the deployed user pool?
Should I assume that I can not create a user pool locally and will always have to point to the deployed user pool
Yes. See the docs: start-api creates an emulated local API endpoint and Lambda for local testing. It does not deploy or emulate other resources.
You can reference previously deployed AWS resources by passing a JSON file with the deployed physical values using the --env-vars flag.
I have install Hbase client and PostgreSql client install but how to connect two databases in a single node application
import { error } from "console";
const hbase = require('hbase');
export class Db {
private conn = hbase();
private config = { host: '0.0.0.0', port: 8080 };
public client = new hbase.Client(this.config);
constructor() {
this.conn = new hbase.Client(this.config);
}
public conection() {
this.conn.table('messages').exists((error: string, succuss: string) => {
if (!succuss) {
this.createTable('messages', 'message_data');
}
});
}
private createTable(TblName: string, CF: string) {
this.conn.table(TblName).create(CF, function (error: string, success: string) {
console.log(success);
return success
});
}
}
I would suggest creating two different classes for Hbase and PostgreSql. ANd use them in your application whenever needed.
Another thing also use dependency injection in the constructor instead of defining configs in class. That way you can inject any DB configuration in an instance.
Here's code example
Create Class to manage HBaseDB connection
import { error } from "console";
const hbase = require('hbase');
export class HBaseDB {
//Inject this config object in your class constructor
//private config = { host: '0.0.0.0', port: 8080 };
//Here we have injected config object
constructor(config) {
this.conn = new hbase.Client(config);
}
public conection() {
this.conn.table('messages').exists((error: string, succuss: string) => {
if (!succuss) {
this.createTable('messages', 'message_data');
}
});
}
private createTable(TblName: string, CF: string) {
this.conn.table(TblName).create(CF, function (error: string, success: string) {
console.log(success);
return success
});
}
}
Create Class to manage PostgreSQL connection
const pg = require('pg');
export class PostgresqlDB {
constructor(config) {
//config contains database,username,password etc... configs
this.pool = new pg.Pool({ config })
this.conn = undefined
}
public async conection() {
//If connection is already connected, no need to connect again
//This will save time
if (this.conn === undefined)
this.conn = await this.pool.connect()
}
public async query(query) {
await this.conection()
const response = await this.conn.query(query)
return response
}
}
Now you can use them in code like
const pgDB = require('./PostgresqlDB')
const hbaseDB = require('./HBaseDB')
const hbaseDBConn = new hbaseDB({ host: '0.0.0.0', port: 8080 })
const pgDBConn = new pgDB({ database: 'test', user:'test',password:'test'})
Note: Above code is for understanding purposes only, You may need to add validations and correct some syntax for actual use
index.ts is the entry point of this NodeJS program.
This is the code in index.ts:
import JobWorker from "./worker";
import { SwitchPlan } from "./jobs";
const worker = new JobWorker();
worker.addJob(SwitchPlan);
This is worker.ts:
import { CronJob } from "cron";
import mongoose from "mongoose";
import Config from "./config";
import logger from "./logger";
export default class JobWorker {
private jobs: CronJob[];
private config: {
NAME: string;
MONGO_URL: string;
};
constructor() {
this.config = Config;
this.connectDB();
this.jobs = [];
}
public async connectDB(): Promise<void> {
try {
await mongoose.connect(this.config.MONGO_URL,
{ useUnifiedTopology: true, useNewUrlParser: true, useCreateIndex: true },
);
logger.info("\nMONGODB has been connected\n");
} catch(err) {
logger.error("ERROR occurred while connecting to the database");
}
}
addJob(cronJob: CronJob) {
this.jobs.push(cronJob);
}
}
This is jobs.ts:
import moment from "moment";
import {
DatabaseOperations, Vehicle,
Plan1Doc, Plan1, VehicleDoc
} from "common-lib";
import logger from "../logger";
import { CronJob } from "cron";
const vehicleOps = new DatabaseOperations(Vehicle);
const SwitchPlan = new CronJob("25 * * * * *", async (): Promise<void> => {
const date: Date = moment(new Date()).startOf("date").toDate();
const expiringVehicles: VehicleDoc[] = vehicleOps.getAllDocuments(
{ "inspection.startTime": {
"$gte": date, "$lte": moment(date).startOf("date").add(1, "day").toDate()
}
},
{}, { pageNo: 0, limit: 0 }
).then((result: any) => {
logger.info("dsada");
}).catch((err: any) => {
logger.info("ssd");
});
});
SwitchPlan.start();
export { SwitchPlan };
I have omitted parts of code which are irrelevant to this problem. I ran this code through a debugger and there's no issue with the config. MonggoDB connected is getting printed at the start of the program. However the then block after getAllDocuments in jobs.ts is never reached and it always goes in the error block with the message, Operation vehicleinventories.find() buffering timed out after 10000ms. The getAllDocuments uses MongoDB's find() method and is working correctly because I am using this method in other projects where I have no such issues.
So far I have tried, deleting Mongoose from node_modules and reinstalling, tried connecting to MongoDB running on localhost, but the issue remains unsolved.
EDIT: DatabaseOperations class:
import { Model, Schema } from "mongoose";
class DatabaseOperations {
private dbModel: Model<any>;
constructor(dbModel: Model<any>) {
this.dbModel = dbModel;
}
getAllDocuments(
query: any,
projections: any,
options: { pageNo: number; limit: number },
sort?: any
): any {
const offset = options.limit * options.pageNo;
return this.dbModel
.find(query, projections)
.skip(offset)
.limit(options.limit)
.sort(sort ? sort : { createdAt: -1 })
.lean();
}
}
in your jobs.ts file you have the following line
SwitchToTier1Plan.start();
This line is called the moment you required the class file, hence before mongoose is connected, and all the models defined. Could this be the issue?
Another thing I noted is u are using mongoose.connect which may be wrong since mongoose.connect creates a global connection.
which means each new Worker you will be attempting to override the mongoose property with previous connection
Though i'm not sure what the implication is, but it could be because your .find could be using the old connection.
Since you are writing class, I would recommend using mongoose.createConnection which creates a new connection for each class initiation.
I am trying to mock AWS SSM using aws-sdk-mock with the code below but not working. Does not throw error, fetch the values from Actual store when getParametersByPath is called.
I had a look at the aws-sdk-mock documentation but does not seem to have an example for mocking ssm, is it supported or not.
AWSMock.mock('SSM', 'getParametersByPath', (params, callback) => {
callback(null, mockResponse);
});
I ran across this when trying to do a similar operation: When trying to mock SSM functionality the resources were still attempting to make requests to AWS and were not using the mock functionality.
Example:
import { mock } from 'aws-sdk-mock';
import { SSM } from 'aws-sdk';
import { GetParameterRequest, GetParameterResult } from 'aws-sdk/clients/ssm';
import 'mocha'
...
const ssm: SSM = new SSM();
mock('SSM', 'getParameter', async (request: GetParameterRequest) => {
return { Parameter: { Value: 'value' } } as GetParameterResult;
})
const request: GetParameterRequest = { Name: 'parameter', WithDecryption: true};
const result: GetParameterResult = await ssm.getParameter(request).promise();
expect(result.Parameter.Value).to.equal('value');
...
The error occurred when making the call to getParameter.
Turns out that the reason for our error was that we were instantiating the integration prior to declaring our mock. So the fix was to switch the order of execution and declare the mock before instantiating the integration.
Example:
import { mock } from 'aws-sdk-mock';
import { SSM } from 'aws-sdk';
import { GetParameterRequest, GetParameterResult } from 'aws-sdk/clients/ssm';
import 'mocha'
...
mock('SSM', 'getParameter', async (request: GetParameterRequest) => {
return { Parameter: { Value: 'value' } } as GetParameterResult;
});
// -> Note the following line was moved below the mock declaration.
const ssm: SSM = new SSM();
const request: GetParameterRequest = { Name: 'parameter', WithDecryption: true};
const result: GetParameterResult = await ssm.getParameter(request).promise();
expect(result.Parameter.Value).to.equal('value');
...