I am trying to create a clustered Cronjob using
cron-cluster and redis
on Node.js. We are running multiple servers with different roles (production, staging, test) and the same codebase. They are all connected to the same Redis instance.
How do I make cron-cluster run only once on each role?
Currently, it picks only one server across the whole fleet (production, staging, test) and runs everything there.
I had the same issue with cron-cluster library.
To solve it, you should pass a unique key, while initializing an instance. For example:
const ClusterCronJob = require('cron-cluster')(redisClient, { key: leaderKey }).CronJob;
Where { key: leaderKey } can be taken as follow:
const leaderKey = process.env.NODE_ENV;
Related
I am writing a ec2 scheduler logic to start and stop ec2 instances.
The lambda works for stopping instances. However the start function is not initiating ec2 start.
The logic is to filter based on tags and status of ec2 and start or stop based on current status.
Below is the code snippet to start EC2 instances. But this isn't starting the instances.
The filtering happens correctly and pushes the instances to "stopParams" object.
The same code works if I change the logic to ec2.stopInsatnces by filtering the running state instances. The role has permissions to start and stop .
Any ideas why its not triggering start ?
if (instances.length > 0){
var stopParams = { InstanceIds: instances };
ec2.startInstances(stopParams, function(err,data) {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
}
context.done(err,data);
});
Finally got this working. There were no issues with the nodejs lambda code. Even though was able to stop instances but start instances were not invoking the start method. Found that all volumes are encrypted.
To start an instance using API call the lambda role used by lambda should have permission to kms key which is used for encrypting the volume. After adding the lambda role arn in the principal section of kms key policy permission the lambda was able to start instances. But key permission is not necessary for stopping the instance. Hope this helps
I'm currently playing with Hazelcast Cloud. My use case requires me to upload 50mb of jar file dependencies to Hazelcast Cloud servers. I found out that the upload seems to give up after about a minute or so. I get an upload rate of about 1mb a second, it drops after a while and then stops. I have repeated it a few times and the same thing happens.
Here is the config code I'm using:
Clientconfig config = new ClientConfig();
ClientUserCodeDeploymentConfig clientUserCodeDeploymentConfig =
new ClientUserCodeDeploymentConfig();
// added many jars here...
clientUserCodeDeploymentConfig.addJar("jar dependancy path..");
clientUserCodeDeploymentConfig.addJar("jar dependancy path..");
clientUserCodeDeploymentConfig.addJar("jar dependancy path..");
clientUserCodeDeploymentConfig.setEnabled(true);
config.setUserCodeDeploymentConfig(clientUserCodeDeploymentConfig);
ClientNetworkConfig networkConfig = new ClientNetworkConfig();
networkConfig.setConnectionTimeout(9999999); // i.e. don't timeout
networkConfig.setConnectionAttemptPeriod(9999999); // i.e. don't timeout
config.setNetworkConfig(networkConfig);
Any idea what's the cause, maybe there's a limit on the free cloud cluster?
I'd suggest using the smaller jar because this feature, the client user code upload, was designed for a bit different use cases:
You have objects that run on the cluster via the clients such as Runnable, Callable and Entry Processors.
You have new or amended user domain objects (in-memory format of the IMap set to Object) which need to be deployed into the cluster.
Please see more info here.
I am using Serilog to write to a file and try to get more information about an error that is occurring in my production cluster...
In my local dev cluster the log files are created fine but they are not created in the VM's on my production cluster. I think this may be security related
Has anyone ever had this?
My production cluster has 5 nodes with a Windows 2016 VM on each
Even more strange is that this works on a single node cluster in Azure
public static ILogger ConfigureLogging(string appName, string appVersion)
{
AppDomain.CurrentDomain.ProcessExit += (sender, args) => Log.CloseAndFlush();
var configPackage = FabricRuntime.GetActivationContext().GetConfigurationPackageObject("Config");
var environmentName = configPackage.GetSetting("appSettings", "Inspired.TradingPlatform:EnvironmentName");
var loggerConfiguration = new LoggerConfiguration()
.WriteTo.File(#"D:\SvcFab\applog-" + appName + ".txt", shared: true, rollingInterval: RollingInterval.Day)
.Enrich.WithProperty("AppName", appName)
.Enrich.WithProperty("AppVersion", appVersion)
.Enrich.WithProperty("EnvName", environmentName);
var log = loggerConfiguration.CreateLogger();
log.Information("Starting {AppName} v{AppVersion} application", appName, appVersion);
return Log.Logger = log;
}
Paul
I wouldn't recommend logging into local files in Service Fabric, since your node may be moved to another VM any time and you won't have access to these files. Consider using another sinks which write to external system (database, message bus or logging system like loggly)
It is likely a permission issue. Your service might be trying to log to a folder where it does not have permission.
By default, your services will run under same user as the Fabric.exe process, that run as NetworkService, you can find more information about this on this link.
I would not recommend this approach, because many reasons, a few of them are:
Your services might be moved around the cluster so your files will be incomplete
You have to log on multiple machines to find the logs
The node might be gone with files (Scale up + Down, Failure, Disk error)
Multiple instances on same node trying to access the same file
and so on...
On Service Fabric, the recommended way is to use EventSource(or ETW) + EventFlow + Application Insights. They run smoothly together and bring you many features.
If you want to use stay on Serilog, I would recommend you use Serilog + Application Insights instead, it will give you move flexibility on your monitoring. Take a look at the Application Insights sink for serilog here.
This was actually user error! I was connecting to a different cluster of VMs than the one my service fabric was connected to! Whoops!
I have this simple node socker seerver as follows:
var ws = require("nodejs-websocket")
var connectionCount = 0;
console.info("Node websocket started # 8002");
var server = ws.createServer(function (conn) {;
console.log("New connection", ++connectionCount);
conn.on("close", function (code, reason) {
console.log("Connection closed")
});
}).listen(8002);
Now I want to hit this server from machines. So to mimic these machines, I am using docker. I want to create around 10 different docker containers which will hit my server.
I want to hit the server from this docker container by using the load testing tool called thor (https://github.com/observing/thor), which can be run as easily as
thor --amount 1000 --messages 100 ws://localhost:8002
So I want to created 10 different docker container and each container should use this tool called thor and hit my server with
thor --amount 1000 --messages 100 ws://localhost:8002
How can I implement such dockor containers.
PS: I am a novice here.
I believe that it should be possible.
There are images available in the docker hub for node of varying size. Choose the appropriate image.
Here are the pseudo instructions to create an image that you needed.
Get the node image
Install thor from git(which you already have the details)
Run the container with your command(Hoping that your websocket app might already be running)
You can do the above in two ways either doing it manually or using Dockerfile.
I believe that you wanted to run in multiple containers, Dockerfile would be good option.
If you can use docker-compose, since multiple containers, it would even better approach.
Hope this is helpful.
I'm trying to write a test to test a method that connects to mongo, but I don't actually want to have to have mongo running and actually make a connection to it to have my tests pass successfully.
Here's my current test which is successful when my mongo daemon is running.
describe('with a valid mongo string parameter', function() {
it('should return a rejected promise', function(done) {
var con = mongoFactory.getConnection('mongodb://localhost:27017');
expect(con).to.be.fulfilled;
done();
});
});
mongoFactory.getConnection code:
getConnection: function getConnection(connectionString) {
// do stuff here
// Initialize connection once
MongoClient.connect(connectionString, function(err, database) {
if (err) {
def.reject(err);
}
def.resolve(database);
});
return def.promise;
}
There are a couple of SO answers related to unit testing code that uses MongoDB as a data store:
Mocking database in node.js?
Mock/Test Mongodb Database Node.js
Embedded MongoDB when running integration tests
Similar: Unit testing classes that have online functionality
I'll make an attempt at consolidating these solutions.
Preamble
First and foremost, you should want MongoDB to be running while performing your tests. MongoDB's query language is complex, so running legitimate queries against a stable MongoDB instance is required to ensure your queries are running as planned and that your application is responding properly to the results. With this in mind, however, you should never run your tests against a production system, but instead a peripheral system to your integration environment. This can be on the same machine as your CI software, or simply relatively close to it (in terms of process, not necessarily network or geographically speaking).
This ENV could be low-footprint and completely run in memory (resource 1) (resource 2), but would not necessarily require the same performance characteristics as your production ENV. (If you want to performance test, this should be handled in a separate environment from your CI anyway.)
Setup
Install a mongod service specifically for CI. If repl sets and/or sharding are of concern (e.g. write concern, no use of $isolated, etc.), it is possible to mimic a clustered environment by running multiple mongod instances (1 config, 2x2 data for shard+repl) and a mongos instance on the same machine with either some init.d scripts/tweaks or something like docker.
Use environment-specific configurations within your application (either embedded via .json files, or in some place like /etc, /home/user/.your-app or similar). Your application can load these based on a node environment variable like NODE_ENV=int. Within these configurations your db connection strings will differ. If you're not using env-specific configs, start doing this as a means to abstract the application runtime settings (i.e. "local", "dev", "int", "pre", "prod", etc.). I can provide a sample upon request.
Include test-oriented fixtures with your application/testing suite. As mentioned in one of the linked questions, MongoDB's Node.js driver supports some helper libraries: mongodb-fixtures and node-database-cleaner. Fixtures provide a working and consistent data set for testing: think of them as a bootstrap.
Builds/Tests
Clean the associated database using something like node-database-cleaner.
Populate your fixtures into the now empty database with the help of mongodb-fixtures.
Perform your build and test.
Repeat.
On the other hand...
If you still decide that not running MongoDB is the correct approach (and you wouldn't be the only one), then abstracting your data store calls from the driver with an ORM is your best bet (for the entire application, not just testing). For example, something like model claims to be database agnostic, although I've never used it. Utilizing this approach, you would still require fixtures and env configurations, however you would not be required to install MongoDB. The caveat here is that you're at the mercy of the ORM you choose.
You could try tingodb.
TingoDB is an embedded JavaScript in-process filesystem or in-memory database upwards compatible with MongoDB at the API level.