Call nodejs kafka script inside Angular 8 application - node.js

How to call nodejs kafka consumer script from Angular component.
I am able to run the same script in terminal using "npm filename.js" but it is throwing an error while calling same from Angular component.

An angular component runs on the client, the kafka script runs on the server. You'll have to create a proxy endpoint to call the script. Depending on which server side Web API Framework you're using this would be done different ways.
To help you understand this further, if you were using express.js, you'd need to have an endpoint to trigger the call (it should be a POST endpoint):
// POST method route
app.post('/triggerKafka', function (req, res) {
const exec = require('child_process').exec, child;
const testscript = exec('bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic yourTopic --from-beginning');
testscript.stdout.on('data', function(data){
console.log(data);
// sendBackInfo();
});
testscript.stderr.on('data', function(data){
console.log(data);
// triggerErrorStuff();
});
})
Now, I should say that this is probably not the way you want to do this. For one thing, you don't want to give people outside your control the ability to start and stop Kafka listening.
You should try to set up your Kafka consumer on the startup of your web application (server side); and provide a mechanism for connecting to it in a similar fashion as above -- proxying the calls through an API endpoint that you set up.
To start Kafka when your web application starts; you need to put the correct information in your node.js web application server startup. In your package.json if you have a "scripts" section you can add the following:
"scripts" : {
"start" : "<whateveryouhavenow>.sh && npm filename.js"
}
The important part here is the && npmfile.js; that says to run your command as well as the command that preceded it. There are more options here, but I provided just one.
Depending on what server side framework you're running and what task-running tool you're using, it would be different (whether it's npm, gulp, grunt, or something else).

Related

why is so differult between the app generated by fastify-cli and the example in the documentation?

where is the start.js in the project which is generated by fastify-cli ?
i think its big different between the getting start example and the app generated by fastify-cli?
should i write the start function like this in the project created by fastify-cli?
const start = async () => {
try {
await sequelize.sync({})
app.log.info('database sync correctly')
await app.listen(PORT, '0.0.0.0')
app.swagger()
} catch (err) {
app.log.error(err)
process.exit(1)
}
}
start()
there are just a app.js in the project generated by fastify-cli.what a different!
where is the start.js in the project which is generated by fastify-cli ?
There is not, it is replaced by the CLI utility fastify your-file.js in the package.json (like mocha, jest etc.. does to run tests)
Usually the starter file is always the same, so it has been integrated in the cli and you can use the args to set the PORT or to reload the server automatically when you edit one file.
i think its big different between the getting start example and the app generated by fastify-cli?
The docs teach all you need to know about the framework, the plugins and utility around it want to ease the developer experience.. like manage a mongodb-connectio: it is one line with the official plugin.
should i write the start function like this in the project created by fastify-cli?
If you use fastify my-file.js you don't need it.
After some experience you will understand when you need the fastify-cli or not.
I think the cli is useful in the most use cases and it suggests good ways to implement configurations loading and encapsulation.
You won't need it for special use cases that need to run some async operation before the server is created

Set up a server from within an electron app

I haven't had any success looking for this because I mostly find misleading questions, about people wanting to use data from a server inside of their electron app. That's not my case.
I have a regular app, which uses a server on the internet, just like any other, but we want to make it available for schools without internet (without any or without reliable internet), so what I'm trying to do is to create a version of my server which runs from an electron exe and serves files for the students conected to the wifi (but no the internet) to access. After the process is done "offline", I will sync the data from the electron app itself.
I tried to run a server from express but I didn't have any progress so far. What I tried was to put the exact same code from my node server in my main.js file and had no luck.
I know that's not what electron is supposed to do, if you're positively sure there is no way to do that, please tell me so I can search for another alternative.
A simple approach is to create a cluster where the master process is the Electron Main and the worker process is the server.
Example:
Change the main on package.json to start.js
On start.js write:
const cluster = require('cluster');
if (cluster.isMaster) {
require('./main.js'); // your electron main file
cluster.fork();
} else {
require('./server.js'); // your server code
}

private chat using socket.io in MEAN app

i am trying to include chat features on my MEAN app, all i have completed till now is a medium where all connected users can communicate.not a separate group. i follow some of the tutorials but they do by trick like sending some key words in front of message(whistle as they say).
as far as i know every connected users are provided a seperate socket ID through which communication is carried but i failed getting that id.
module.exports = function(socket){
//console.log(socket);
var users =[];
socket.on('username',function(data){
users.push({id:socket.id,username:data.message});
socket.emit('username',users)
})
console.log('connected');
socket.on('typing',function(data){
//socket.emit('typing',{message:"helo angular"});
socket.broadcast.emit('typing',{message:data.message});
});
It shows me socket is not defined, any one has better idea how to perform private message using socket.io and node.js
can anyone enlighten me about this.
Make all users join their own unique group.
socket.join(socket.id);
Then you can send a message to one user by doing this
io.sockets.in(whichever_user.id).emit('msg', "Hello there user lets have a private chat!");
I would recommend using the Express framework of Node, and implenting Socket.io into it.
Here are some links telling you how to set up Express and Socket.io:
https://www.youtube.com/watch?v=WH5qsGnFkBM&lc=z135dbryaqbfitiee22pd33ouozhwft3q
---Youtube comment tells you how to set up express generator.
Adrian GÄ…siewicz:
For those who want to have similar directory structure as Bucky:
Install Express generator "sudo npm install -g express-generator"
Go to directory where you want to create project and type "express myapp --ejs" 3. Select newly created directory --> "cd myapp"
Install all node_modules --> "npm install"
Run server --> "npm start"
Open browser and type "http://localhost:3000/" Have fun ;)
Using socket.io in Express 4 and express-generator's /bin/www
---This tells you how to set up socket in Node.js Express project.

Node JS message queue on Heroku

I need to move my Node JS server running on Heroku to a message queue architecture. Currently, the server receives a HTTP request, does some processing, and responds. The problem is that the processing takes some time, especially when there are lots of requests. This lengthy processing time causes the server to timeout, overload, and crash! My reading tells me a need a background worker to do the processing.
I have zero experience with message queues and background workers and I'm looking for a very simple example to get started. Can anyone suggest a simple, understandable module or example to get started?
I found some examples but they are complex and I'm getting lost! I want a barebones example I can build from.
Let's see how to do this with RabbitMQ.
First, you will need a RabbitMQ server to work with in your development environment.
If you don't already have it (check "sudo service rabbitmq-server status") you can install (on ubuntu or similar) as follows:
sudo su -c "echo 'deb http://www.rabbitmq.com/debian/ testing main' >> /etc/apt/sources.list"
wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update
sudo apt-get install rabbitmq-server
rm rabbitmq-signing-key-public.asc
Then, get the server running with:
sudo service rabbitmq-server start
You also need to configure a RabbitMQ service for your Heroku deployment. Let's use CloudAMPQ for this example. You can add its Free Plan to your Heroku app with:
heroku addons:create cloudamqp:lemur
That will create a new CLOUDAMQP_URL environment variable in your Heroku app.
Next, you're going to need a suitable RabbitMQ client for your node.js app.
There are a few of them out there, but for this example, let's use ampqlib:
npm install ampqlib --save
That should add something like the following line in your package.json dependencies:
"amqplib": "^0.4.1",
Next thing is to add a background "worker" dyno to your Heroku app.
I assume that currently you only have a single Web dyno in your Procfile.
So, you need to add another line for instantiating a worker, such as:
worker: node myworker.js
Finally, you need to write the code that will enable your Web dyno to interact with your worker dyno via RabbitMQ.
For the sake of this example, I will assume that your Web dyno will be "publishing" messages to a RabbitMQ message queue, and your worker dyno will be "consuming" these messages.
So, let's start with writing code for publishing to a message queue. This code needs to run somewhere in your Web dyno:
// Define ampq_url to point to CLOUDAMPQ_URL on Heroku, or local RabbitMQ server in dev environment
var ampq_url = process.env.CLOUDAMQP_URL || "amqp://localhost";
var ampq_open = require('amqplib');
var publisherChnl;
function createPublisherChannel() {
// Create an AMPQ "connection"
ampq_open.connect(ampq_url)
.then(function(conn) {
// You need to create at least one AMPQ "channel" on your connection
var ok = conn.createChannel();
ok = ok.then(function(ch){
publisherChnl = ch;
// Now create a queue for the actual messages to be sent to the worker dyno
publisherChnl.assertQueue('my-worker-q');
})
})
}
function publishMsg() {
// Send the worker a message
publisherChnl.sendToQueue('my-worker-q', new Buffer('Hello world from Web dyno'));
}
You will need to call createPublisherChannel() during the initialisation of your Web dyno. Then, call publishMsg() whenever you want to send a message to the queue.
Finally, let's write the code for consuming the above message in the worker dyno. So, for example, add something like the following in myworker.js:
// Just like in Web dyno...
var amqp_url = process.env.CLOUDAMQP_URL || "amqp://localhost";
var open_ampq = require('amqplib').connect(amqp_url);
var consumerChnl;
// Creates an AMPQ channel for consuming messages on 'my-worker-q'
function createConsumerChannel() {
open_ampq
.then(function(conn) {
conn.createChannel()
.then(function(ch) {
ch.assertQueue('my-worker-q');
consumerChnl = ch;
});
});
}
function startConsuming() {
consumerChnl.consume('my-worker-q', function(msg){
if (msg !== null) {
console.log(msg.content.toString());
// Tell RabbitMQ server we have consumed the message
consumerChnl.ack(msg);
}
})
}
createConsumerChnl().then(startConsuming);
Finally, test with "heroku local". You should see that you now have 2 processes running in your app, "Web" and "worker". Whenever you call publishMsg() in your Web dyno, you should hopefully see the wroker dyno spit out the message contents to your console. To see what's happening in your RabbitMQ queues, you can use:
sudo rabbitmqctl list_queues
I found a very simple example (followed by deeper examples) here: https://www.rabbitmq.com/tutorials/tutorial-one-javascript.html

How to get grunt.js to start an express app for testing

My current situation is that I use grunt to make a production version of my express app (minimize and merge all the js/css, copy all the files in place) and then I have to run a script which sets an environment variables (my app only serves the test harness when running in TEST mode), creates an empty Mongo test database and then calls npm start on the application directory, and then I manually have to run the tests from either Chrome or Phantom, what I want to do is have grunt set the environment variable and run the server, run the tests, and then stop the server (in the future if all is successful it would be nice to have it deploy as well). However when I try to run the app in grunt, it gets stopped as soon as it is completed.
How do I have grunt start the app, wait until it is started and then run tests?
If you check grunt-express which is a plugin for express web-server tasks via grunt.
express-keepalive
Note that this server only runs as long as grunt is running. Once
grunt's tasks have completed, the web server stops. This behavior can
be changed by appending a express-keepalive task at the end of your
task list like so
grunt.registerTask('myServer', ['express', 'express-keepalive']);
Now when you run grunt myServer, your express server will be kept alive
until you manually terminate it.
Such feature can also be enabled ad-hoc by running the task like grunt express express-keepalive.
This design gives you the flexibility to use grunt-express in
conjunction with another task that is run immediately afterwards, like
the grunt-contrib-qunit plugin qunit task. If we force express task to
be always async, such use case can no longer happen.
From the guide grunt-contrib-qunit package is used to run QUnit unit tests in a headless PhantomJS instance. Also mind the last line, if you force express to be always async it would be of no use.
npm link for grunt-express.
I'm not sure if I understand your problem correctly, but probably this helps:
Do you know about Grunt's async function? For some time I used the following approach to start (and stop) my Express app. I used it with watch, so it automatically restarted on save. In this case you have to set watch's nospawn option to true.
var server = null;
grunt.registerTask('server', 'Start server', function() {
var done = this.async();
if (server !== null) {
server.close();
clearCache();
}
var app = require('./my-express-app.js');
server = http.createServer(app).listen(app.get('port'), function(){
done();
});
});
function clearCache() {
for (key in require.cache) {
if (key.indexOf(__dirname + '/node_modules/') == -1) {
delete require.cache[key];
}
}
}
It is ugly because of this require-cache-hack. However, it works.

Resources