Using load testing app in docker - node.js

I have this simple node socker seerver as follows:
var ws = require("nodejs-websocket")
var connectionCount = 0;
console.info("Node websocket started # 8002");
var server = ws.createServer(function (conn) {;
console.log("New connection", ++connectionCount);
conn.on("close", function (code, reason) {
console.log("Connection closed")
});
}).listen(8002);
Now I want to hit this server from machines. So to mimic these machines, I am using docker. I want to create around 10 different docker containers which will hit my server.
I want to hit the server from this docker container by using the load testing tool called thor (https://github.com/observing/thor), which can be run as easily as
thor --amount 1000 --messages 100 ws://localhost:8002
So I want to created 10 different docker container and each container should use this tool called thor and hit my server with
thor --amount 1000 --messages 100 ws://localhost:8002
How can I implement such dockor containers.
PS: I am a novice here.

I believe that it should be possible.
There are images available in the docker hub for node of varying size. Choose the appropriate image.
Here are the pseudo instructions to create an image that you needed.
Get the node image
Install thor from git(which you already have the details)
Run the container with your command(Hoping that your websocket app might already be running)
You can do the above in two ways either doing it manually or using Dockerfile.
I believe that you wanted to run in multiple containers, Dockerfile would be good option.
If you can use docker-compose, since multiple containers, it would even better approach.
Hope this is helpful.

Related

How to implement XRay in NodeJS project?

I've a nodejs project with Docker and ECS in AWS and i need to implement XRay to get the traces but I couldn't get it to work yet
I installed 'aws-xray-sdk' (npm install aws-xray-sdk), then I added
const AWSXRay = require('aws-xray-sdk');
in app.js
Then, before the routes I added
app.use(AWSXRay.express.openSegment('Example'));
and after the routes:
app.use(AWSXRay.express.closeSegment());
I hit some endpoints but I can't see any trace or data in xray, maybe do I need to setup something in AWS ? I have a default group in xray.
Thanks!
It sounds like you do not have the XRay Daemon running in your ECS environment. This daemon must be used in conjunction with the SDKs to send the trace data to AWS XRay service from the SDKs. The daemon listens for the trace data traffic on UDP port 2000. Read more about the daemon here:
https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html
See how to run the XRay Daemon on ECS via Docker here:
https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html
You would either need to look at X-Ray SDK, Agent or Open Telemetry SDK, Collector (AWS Distro for Open Telemetry)

How to setup automatic shutdown for Google Compute Instance?

I'm running a NodeJS app inside a docker container inside a container-optimized-OS GCE instance.
I need this instance to shutdown an self-delete upon its task completion. Only the NodeJS app is aware of the task completion.
I used to achieve this behavior by setting up this as a startup-script:
node ./dist/app.js
echo "node script execution finished. Deleting this instance"
export NAME=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/name -H 'Metadata-Flavor: Google')
export ZONE=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google')
gcloud compute instance-groups managed delete-instances my-group --instances=$NAME --zone=$ZONE
I've also used similar setups with additional logic based on the NodeJS app exit code.
How do I do it now?
There are two problems:
I don't know how to pass NodeJS exit event (preferably with exit code) up to the startup-script. How do I do that?
Container-optimized-OS GCE instance lacks gcloud. Is there different way of shutting down an instance?
Google Cloud's Healthcheck seems too troublesome and not universal. My app is not a web-server, I prefer not to install express or something else just for sake of handling health checks.
Right now my startup-script ends with docker run ... command. Maybe I should write the shutdown command after that and somehow make docker exit on NodeJS exit?
If you think the Healthcheck is the way to go, what would be the lightest setup for a health check given that my app is not a web-server?
Try to have your app trigger a Cloud Function when the app finishes the job
Cloud function can then have script to delete your VM. See sample script below
https://medium.com/google-cloud/start-stop-compute-engine-instance-from-cloud-function-bf9ae5199609

Chrome run fails in Azure Functions: An attempt was made to access a socket in a way forbidden by its access permissions

I wrote a web bot that uses Selenium framework to crawl. Installed ChromeDriver 72.0.3626.69 and also downloaded Chromium 72.0.3626.121. The app initializes ChromeDriver with this included Chromium binary (and NOT a locally installed Chrome binary). All this perfectly works on my machine locally.
I've been attempting now to port the app to Azure Functions. I wrote a function, tested it, and it works fine locally. But once I publish it to Azure Functions it fails due to about 182 errors of type:
An attempt was made to access a socket in a way forbidden by its
access permissions
I know this happens due to exceeding the TCP connection limits of Azure sandbox, but the only attempt here was to create an instance of ChromeDriver (not even navigate anywhere yet!)
Here is a screenshot of Azure Function call log.
That error appears about 182 times in a row, and that's basically just an attempt to create a browser instance (or ChromeDriver instance, to be precise - can't be sure if that's Chromium or ChromeDriver causing the issue).
The question: Have anyone experienced issues with ChromeDriver/Chromium creating so many (obviously excessive) connections when launching? And what might help to avoid this.
If that's of any help, this is basically a piece of code that crashes on the last line:
ChromeOptions options = new ChromeOptions();
options.BinaryLocation = this.chromePath;
options.AddArgument("no-sandbox");
options.AddArgument("disable-infobars");
options.AddArgument("--disable-extensions");
if (this.headlessMode)
{
options.AddArgument("headless");
}
options.AddUserProfilePreference("profile.default_content_setting_values.images", 2);
Log.LogInformation("Chrome options compiled. Creating ChromeDriverService...");
var driverService = ChromeDriverService.CreateDefaultService(this.driverPath);
driver = new ChromeDriver(driverService, options, timeout);
I believe you are running this function in a Windows Function App which is subject to quite a few limitations as described in this wiki.
But when running on Linux, functions are basically run in a docker container, removing most of these restrictions that windows has. I believe what you are trying should be possible there.
You could either just deploy your function to a Linux Function App or even build a container and use that directly as well.

Random connection errors to MS SQL from nodeJS app

We have an AWS server running some nodeJS services. The services connecting to MS sql are randomly crashing with message "Failed to connect to databaseserver:1433 - Could not connect (sequence)".
We are running on:
App server:
Linux Ubuntu 14.4
AWS m5
NodeJS: 8.11.2
Services are using package mssql latest version (4.3.0). This includes tedious 2.7.1.
DB server:
Windows server 2012.
sql server 2012
throughput: about 300 rpm, error also happens when throughput is lower (about 20 rpm).
App is running in a cluster through PM2 (runs 4 times). We see the error happening on all 4 at the same time, but sometimes also on 1 or 2 instances.
What we tried:
Upgrading to alpha version of mssql with tedious 3.0.1. Did not make a difference
Upgrading from Amazon M4 machine to M5 machine with enhanced networking
Changing the pool settings in the app. We tried setting min connections to 0 or low/high value. Max also to low/high value but no avail.
Duplicate server to new machine.
Setting idleTimeoutMillis to 1 second
Pinging DB server to see if there is a connection problem, but we see no weird pings when the error happens.
Connection on app startup:
App.sqlConnection = new App.SQL.ConnectionPool(config, function(err) {
if(err){
Log.error(err);
process.exit(1);
}
App.sqlConnection.on('error', err => {
Log.error(`There was a connection err : ${err}`);
process.exit(1);
});
});
request;
var request = new App.SQL.Request(App.sqlConnection);
request.query(sQuery, function(err,results)
{
});
Errors are catched by the "on error" handler.
The error happens randomly across services. Some have more instances of the error then others.
We are running out of options. Any idea if we can see more detailed errors?
I have a couple suggestions.
First, how sure are you that these errors are actually a problem? If your code simply retries, instead of exiting, are the connections stable afterwards, or can a connection drop in the middle of a query?
(Connections dropping in the middle of queries are obviously not good, but random failures on connection, that can be fixed by retries, are the best kind of problem to have IMHO.)
Ignoring the potential in-code fix, I'm wondering when you say you "duplicated server to new machine" - did you launch a new AMI using latest Windows Server 2012, or did you image and clone? If your database server is a couple years old, you might actually be running outdated network drivers in your instance, which could give you some hiccups.
If you wanted to explore that, you could attempt rebuilding the entire database server from scratch on a newly launched AMI. Alternately you can upgrade PV driver, network adapter, and EC2Config on your existing instance, you can find the instructions at the following links:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Upgrading_PV_drivers.html#aws-pv-upgrade
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/sriov-networking.html#enable-enhanced-networking
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_Install.html

How can file writes with NodeJS on Docker be inconsistent?

A while back I've created some feed fetching & processing scripts with NodeJS for an application I'm working on. I've been running this script on my local machine (OSX) manually for a while and momentarily I'm working on having a job do this. I figured I'd go with a Docker droplet ($5/mo) on Digital ocean. I've created a Dockerfile with NodeJS (9), cron and everything I think I need. Everything works fine on my local machine when I build and run the docker container.
However, when I deploy it to Digital Ocean there seems to be different behaviour from running it locally. Exactly the thing I wanted to prevent using Docker. I have a main shell script that calls 8 different node scripts sequentially. On Digital Ocean it seems that some NodeJS-scripts exit prematurely. So they haven't finished yet but the mail shell script continues with the next NodeJS script. An example:
execute() {
const outputPath = path.join(this._feedDir, this._outputFile);
return createDirIfNotExists(this._feedDir, this._log)
.then(() => this._doRequests())
.then(requestData => this._writeFile(outputPath, JSON.stringify({requests: requestData.map(data => JSON.parse(data))}), this._log))
}
When running the code:
Script does create the dir
Script does do all the requests to external sources (and accumulates the data)
Script sometimes write the data to file, and sometimes not. Without any error code.
Main (shell) script always picks up after that.
Any thoughts?

Resources