pact.io: Select specific endpoints for provider test - node.js

We are running a microservice architecture and want to set up contract testing in our project. Our consumers do not know which request is handled by which microservice. We want our microservices to select the interactions from the pacts that they should participate in.
Example:
Consumer A writes a test which is testing POST /users.
Consumer A writes a second test for POST /users with different parameters.
Consumer A writes a test for GET /users/$userId.
Consumer A writes a test for GET /articles/$articleId.
Microservice A handles all POST /users requests.
Microservice B handles all GET /users/$userId requests.
Microservice C handles all GET /articles/$articleId requests.
All of the consumer tests only have a single request in their interactions.
We want to put provider tests next to the Microservices. Each microservice should only test the endpoints that it is capable of handling. In this scenario, Microservice A would test all of the POST /users contracts. Microservice B would select the GET /users/$userId contracts and so on.
Is there a way to do so with pactflow.io and nodejs bindings for pact?
Edit: Added the architecture diagram:

No, there is no such in-built feature in Pact that supports that use case.
We've discussed the possibility of publishing expectations for messages this way, but not HTTP (because this is a bit more unusual, unlike message queues like Kafka where there is usually more indirection).
Are you using some form of dynamic API gateway or something?
One challenge you'll face is reverse engineering out the requests themselves in a reliable manner.
Ideas
The only suggestion I have would be to have a proxy on the provider side test that was aware of the different endpoints, and would redirect the requests to the correct provider. But then state handling gets difficult.
You can of course also manually fetch the pacts, and split them, but you'll lose a lot of value that pact has.
I'm not sure if the consumers not knowing about the providers is more of a philosophical thing, a practical thing or otherwise, but obviously the simplest solution is probably making the consumers aware of their providers.
Raise a Feature Request
Perhaps stating your use case more clearly and requesting a feature at https://pact.canny.io/ might be worthwhile, to see how relevant your use case is to the broader community and if it would be worth implementing.

We have found a solution to our particular problem:
The consumer does not know which service acts as a provider, but it knows the (HTTP method, URL) tuple that it calls.
The microservice knows every (HTTP method, URL) that it is responsible for.
We define a provider as the tuple (HTTP method, URL). As a result, a consumer contains many tests for many providers and a microservice also contains many tests for many providers.
Something like this in node.js for the consumer:
const consumer = "MyConsumer";
const providerGetArticles = new Pact({ consumer, provider: 'GET-articles', ... });
const providerGetArticlesArticleId = new Pact({ consumer, provider: 'GET-articles-:articleId', ... });
const providerPostUsers = new Pact({ consumer, provider: 'POST-users', ... });
const providerGetUsers = new Pact({ consumer, provider: 'GET-users', ... });
const providerGetUsersUserId = new Pact({ consumer, provider: 'GET-users-:userId', ... });
providerGetArticles.setup().then(() => {
providerGetArticles.addInteraction(
{
withRequest: { method: 'GET', path: '/articles' },
...
providerGetArticlesArticleId.setup().then(() => {
providerGetArticlesArticleId.addInteraction(
{
withRequest: { method: 'GET', path: '/articles/12345' },
...
providerPostUsers.setup().then(() => {
providerPostUsers.addInteraction(
{
withRequest: { method: 'POST', path: '/users' },
...
And like this for a microservice that handles GET /articles and GET /articles/:articleId
new Verifier({ provider: 'GET-articles', ... }).verifyProvider()...
new Verifier({ provider: 'GET-articles-:articleId', ... }).verifyProvider()...
We can now start the single microservice in isolation and run the provider tests.

Related

Logging Middleware Microservice

I am required to save logs into a MySQL database of each request and response made to the backend. The issue is that we are migrating to microservices architecture. The backend was made with NodeJS and Express, and it has a middleware that does this task. Currently, it has this middleware attached to each microservice.
I would like to isolate this middleware as its own microservice. The issue is that I don't know how to redirect the traffic this way. This is how I would like to manage it:
I would like to do it this way, because we can make changes or add features to the middleware without having to implement it in each microservice. This is the middleware's code:
const connection = require("../database/db");
const viewLog = (req, res, next) => {
const oldWrite = res.write,
oldEnd = res.end,
chunks = [],
now = new Date();
res.write = function (chunk) {
chunks.push(chunk);
oldWrite.apply(res, arguments);
};
res.end = function (chunk, error) {
if (chunk) chunks.push(chunk);
const bodyRes = Buffer.concat(chunks).toString("utf8");
connection.query("CALL HospitalGatifu.insertLog(?,?,?,?,?)", [
`[${req.method}] - ${req.url}`,
`${JSON.stringify(req.body) || "{}"}`,
bodyRes,
res.statusCode === 400 ? 1 : 0,
now,
]);
oldEnd.apply(res, arguments);
};
next();
};
module.exports = viewLog;
I think there might be a way to manage this with Nginx which is the reverse proxy that we are using. I would like to get an approach of how to change the logs middleware.
Perhaps you might want to take a look at the sidecar pattern which is used in microservice architectures for common tasks (like logging).
In short, a sidecar runs in a container besides your microservice container. One task of the sidecar could be intercepting network traffic and logging requests and responses (and a lot of other possible tasks). The major advantage of this pattern is that you don't need to change any code in your microservices and you don't have to manage traffic redirection yourself. The latter will be handled by the sidecar itself.
The disadvantage is that you are required to run your microservices containerized and use some kind of container orchestration solution. I assume this being the case since you are moving towards a microservices based application.
One question about the log service in between of the webapp and the NGNIX server. What if the logging services goes down for some reason, is it acceptable for the entire application to go down?
Let me give you not exactly what you requested but something to think about.
I can think on 3 solutions for the issue of logging in microservices, each 1 have its own advantages and disadvantages:
Create a shared library that handles the logs, I think its the best choice in must cases. An article I wrote about shared libraries
You can create API gateway, it is great solution for shared logic to all the requests. So it will probably be more work but then can be used for other shared logic. Further read (not written by me :) )
A third option (which I personally don't like) is create a log microservice that listens to LogEvent or something like that. Then from your MSs publish this event whenever needed.

How to implement rabbitMQ into node.js microservice app right way?

Greetings Stackoverflow.
I've been using stackoverflow for years to find answers, and this is my first attempts to make a question myself. So feel free to tell me if I'm doing it wrong way.
Currently I'm developing a data analytical system based on microservice architecture.
It is assumed that this system will consist of a dozen self-sufficient microservices communicating with each other by RabbitMQ. Each of them is encapsulated in a docker-container and the whole system is powered by docker-swarm in the production.
In particular each microservice is a node.js application and related database, connected with some ORM interface. Its task is to manage and serve data in a CRUD manner, and to provide results of some prepared queries based on the contained data. Nothing extraordinary.
To provide microservice-microservice communication I assume to use amqplib. But the way to use it is uncertain yet.
My current question is how to make use of amqplib in a OOP manner to link inter microservice communication network with application's object-related functionality? By OOP manner, I mean the possibility to replace amqplib (and RabbitMQ itself) in the long run without the need to make changes to the data-related logic.
What I really searching for is the example of currently working microservice application utilizing AMQP. I'd pretty much appreciate that if somebody could give a link to it.
And the second part of my question.
Does it make sense to build microservice application based on event-driven principals, and just pass messages from RabbitMQ to the application's main event queue? So that each procedure would be called the same way, despite the fact that it is an internal or external event.
As for the abstract example of single microservice:
Let's say I have an event service and a listener connected to this service:
class UserManager {
constructor(eventService) {
this.eventService = eventService;
this.eventServce.on("users.user.create-request", (payload) => {
User.create(payload); // User interface is omitted in this example
}
}
}
const eventService = new EventEmmiter();
const userManager = new UserManager(eventService);
On the other hand I've got RabbitMQ connection, that is waiting for messages:
const amqp = require('amqplib');
amqp.connect('amqp-service-in-docker').then(connection => {
connection.createChannel().then(channel => {
// Here we use topic type of exchange to be able to filter only related messages
channel.assertExchange('some-exchange', 'topic');
channel.assertQueue('').then(queue => {
// And here we are waiting only the related messages
channel.bind(queue.queue, 'some-exchange', 'users.*');
channel.consume(queue.queue, message => {
// And here is the crucial part
}
}
}
}
What I'm currently think off is to just parse and forward this message to eventService and use it's routing key as the name of the event, like this:
channel.consume(query.query, message => {
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
But how about RPC's? Should I make another exchange or even a channel for them with another approach, something like:
// In RPC channel
channel.consume(query.query, message => {
eventService.once('users.user.create-response', response => {
const recipient = message.properites.replyTo;
const correlationId = msg.properties.correlationId;
// Send response to specified recipient
channel.sendToQueue(
recipient,
Buffer.from(JSON.stringify(resonse)),
{
correlationId: correlationId
}
);
channel.ack(message);
});
// Same thing
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
And then my User class should fire 'users.user.create-response' event every time it creates a new user. Isn't this a crutch?

Microservices Create Entity Implementation

This is a follow-up question to my issue outlined here.
The Gateway serves as an entry point to the application, to which every request from the client is made. The gateway then allocates the request to the responsible microservices and also handles authentication.
In this case the gateway listens for HTTP POST /bok and notifies the Microservice A to create a book. Thus Microservice A ist responsible for managing and storing everything about the book entity.
The following pseudo-code is a simplified implementation of this architecture:
Queue Communication
Gateway
router.post('/book', (req, res) => {
queue.publish('CreateBook', req.body);
queue.consume('BookCreated', (book) => {
const user = getUserFromOtherMicroService(book.userId);
res.json({ book, user });
});
});
Microservcie A
queue.consume('CreateBook', (payload) => {
const book = createBook(payload);
eventStore.insert('BookCreated', book);
const createdBook = updateProjectionDatabase(book);
queue.publish('BookCreated', createdBook);
})
But I am not quite sure about this because of the following reasons:
The listener for consuming BookCreated in the Gateway will be recreated every time a user requests to create a new book
What if 2 users simultaneously create a book and the wrong book will be returned?
I don't know how to fetch additional data (e.g. getUserFromOtherMicroService)
That's why I though about implementing this architecture:
Direct and Queue Communication
Gateway
router.post('/book', async (req, res) => {
const book = await makeHttpRequest('microservice-a/create-book', req.body);
const user = await makeHttpRequest('microservice-b/getUser', book.userId);
res.json({ book, user });
});
Microservice A
router.post('/create-book', (req, res) => {
const book = createBook(req.body);
eventStore.insert('BookCreated', book);
const createdBook = updateProjectionDatabase(book);
queue.publish('BookCreated', createdBook);
res.json(createdBook);
})
But I am also not really sure about this implementation because:
Don't I violate CQRS when I return the book after creation? (because I should only return OK or ERROR)
Isn't it inefficient to make another HTTP request in a microservices system?
Based on the comments above .
Approach 1
In this case your api gateway will be used to drop the message in the queue. This approach is more appropriate if your process is going to take long time and you have a queue workers sitting behind to pick up the messages and process. But your client side has to poll to get the results. Say you are looking for airline ticket . You drop the message. Your get an ID to poll. Your client will keep on polling until the results are available.
But in this case you will have a challenge , as you drop the message how you are going to generate the ID that client would poll ? Do you assign the ID to message at Gateway and drop in the queue and return the same ID for the client to poll to get result ? again this approach is good for web/worker kind of scenario.
Approach 2
Since Your API gateway is custom application that would handle the authentication and redirect the request to respective service. Your Microsvc A would create book and publish the event and your Microservice B and C would be using it . Your Gateway will wait for the Microservice A to return response with ID (or event metadata of newly created object) of book that is created so you don't poll for it later and client has it. If you want you can have additional information from other microservices you can fetch at this time and can send aggregated response.
For any data that is available in Microservice A,B,C you will be getting via Gateway. Make sure your gateway is highly available.
Hope that helps . Let me know if you have any questions !

Unique configuration per vhost for Micro

I have a few Zeit micro services. This setup is a RESTful API for multiple frontends/domains/clients
I need to, in my configs that are spread throughout the apps, differentiate between these clients. I can, in my handlers, setup a process.env.CLIENT_ID for example that I can use in my config handler to know which config to load. However this would mean launching a new http/micro process for each requesting domain (or whatever method I use - info such as client id will prob come in a header) in order to maintain the process.env.CLIENT_ID throughout the request and not have it overwritten by another simultaneous request from another client.
So I have to have each microservice check the client ID, determine if it has already launched a process for that client and use that else launch a new one.
This seems messy but not sure how else to handle things. Passing the client id around with code calls (i.e. getConfg(client, key) is not practical in my situation and I would like to avoid that.
Options:
Pass client id around everywhere
Launch new process per host
?
Is there a better way or have I made a mistake in my assumptions?
If the process per client approach is the better way I am wondering if there is an existing solution to manage this? Ive looked at http proxy, micro cluster etc but none seem to provide a solution to this issue.
Well I found this nice tool https://github.com/othiym23/node-continuation-local-storage
// Micro handler
const { createNamespace } = require('continuation-local-storage')
let namespace = createNamespace('foo')
const handler = async (req, res) => {
const clientId = // some header thing or host
namespace.run(function() {
namespace.set('clientId', clientId)
someCode()
})
})
// Some other file
const { getNamespace } = require('continuation-local-storage')
const someCode = () => {
const namespace = getNamespace('foo')
console.log(namespace.get('clientId'))
}

Architecture for microservices

I've recently started to work with node.js and I have to build an architecture that should use multiple express.js services. Some of these services will have to be located on one server, anothers - on other server machines. I want to build a base service (like API Gateway), but I don't know what the proper way to communicate between this Gateway and microservices, or between two microservices.
Currently I'm working with a solution based on this:
# inside Gateway server I call another service:
http.get('http://127.0.0.1:5001/users', (service_res) ->
data = ''
service_res.on 'data', (chunk) ->
data += chunk
service_res.on 'end', ->
# some logic on data
).end()
I have a strong feeling that this approach is not right. What the proper way to build communication logic between API Gateway and microservices?
The logic you have is not incorrect but what would probably be better is to build a layer of abstraction on top of making requests to an another service eg. the API gateway to another microservice. Lets just call that microservice B for this instance (API gateway to make a request to B).
B in this case should provide its own client on how another service should interact with it, whether its through HTTP or WebSockets, the protocol is up to B because B understands how one should communicate with it. The argument for the client and the service being implemented together is that these two components should have a higher level of cohesion since technically they are bound by a contract eg. if a requests needs to be made to a service, it needs to adhere to the contract that the service requires.
In simple pseudocode with Express:
// implemented elsewhere, ideally next to the service that it communicates with
function BServiceClient() {
// ...
}
// the API gateway's calling code
app.get('...', function(request, response, next) {
// create an instance of the service client
var bServiceClient = new BServiceClient();
// retrieving the users from an abstracted endpoint
bServiceClient.GetUsers();
// do some processing and then render a response or call next
});
In order for it to be more testable, you might have to write your own wrapper around the app to do the proper dependency injection for injecting the client to make the routes more testable. Otherwise, you might be able to create another function that can inject the client and create the client at the handler level that calls the newly created function. The newly created function could then be tested. However, I prefer the former approach of using the wrapper. Hope this helps!
What i would do is,
Create separate modules for each microservice. Depending on what microservice you want to run, just have a route for that in express.
Inject the modules you want into an instance of express().
Example + shameless plug - https://github.com/swarajgiri/express-bootstrap/blob/master/core/index.js
Disclaimer - The above solution is a highly opinionated way of solving your problem.

Resources