Multithreaded Item Price Checking server in scala - multithreading

I am trying to implement a multithreaded item price checking server, where I have two classes : A Server and A Client. My Server should also have access to a file where it will hold items with respective prices, and my client should be able to ask for an item "ex. Coffee" and get its price from server "ex. $2.00". Now I don't know how to make a file and extract required information, so need help.
Here is my Server Code:
import java.io.{DataInputStream, DataOutputStream}
import java.net.ServerSocket
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
object server extends App {
// Register service on port 9000
val server = new ServerSocket(9000)
while(true) {
// Register service on port 1234
val socket = server.accept()
Future {
//store local socket references for processing
val client = socket
try {
// Get a communication stream associated with the socket
val is = new DataInputStream(client.getInputStream())
// Get a communication stream associated with the socket
val os = new DataOutputStream(client.getOutputStream())
// Read from input stream
var line: String = is.readLine()
// sending string
os.writeBytes("$2.00\n")
println(line)
} catch {
case e: Exception => e.printStackTrace
} finally {
// Close the connection, but not the server socket
client.close()
}
}
}
}
Here is my Client Code:
import java.io.{DataInputStream, DataOutputStream}
import java.net.Socket
object client extends App {
var client: Option[Socket] = None
try {
//Open your connection to a server, at port 9000
client = Some(new Socket("127.0.0.1", 9000))
// Get an input file handle from the socket
val is = new DataInputStream(client.get.getInputStream)
val os = new DataOutputStream(client.get.getOutputStream)
// write to server a string
//os.writeBytes("Hello from client\n")
os.writeBytes("Coffee\n")
//read from server a string
var line: String = is.readLine()
//printing output
println(line)
}
catch {
case e: Exception => e.printStackTrace()
}
finally {
client foreach (_.close())
}
}

Related

Ktor - Keep POST alive until receiving websocket communication

I am building an API with Kotlin and Ktor that should be able to receive a normal POST request.
Upon receiving it, he should keep it alive and establish a series of asynchronous communications with other systems using websocket.
Only at the end of these communications and receiving certain information will it be able to respond to the POST request.
Needless to say, the request must be kept alive.
I'm not sure how to make this possible.
I have investigated using coroutines and threads but my inexperience prevents me from understanding what would be the best solution.
By default sequential code inside a coroutine is executed synchronously so you can just put your code for communication via Websockets inside a route's handler and in the end send a response. Here is an example:
import io.ktor.client.*
import io.ktor.client.engine.okhttp.*
import io.ktor.client.plugins.websocket.*
import io.ktor.client.plugins.websocket.WebSockets
import io.ktor.server.application.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import io.ktor.server.response.*
import io.ktor.server.routing.*
import io.ktor.server.websocket.*
import io.ktor.websocket.*
fun main() {
val client = HttpClient(OkHttp) {
install(WebSockets)
}
embeddedServer(Netty, port = 12345) {
routing {
get("/") {
client.webSocket("ws://127.0.0.1:5050/ws") {
outgoing.send(Frame.Text("Hello"))
val frame = incoming.receive()
println((frame as Frame.Text).readText())
println("Websockets is done")
}
call.respondText { "Done" }
}
}
}.start(wait = false)
embeddedServer(Netty, port = 5050) {
install(io.ktor.server.websocket.WebSockets)
routing {
webSocket("/ws") {
outgoing.send(Frame.Text("Hello from server"))
}
}
}.start()
}

Communication between child processes in Node.js

I'm trying to make a program in node.js that creates two processes using the fork() method of childproccess. The processes are as follows:
Father.js
Son1.js
Son2.js
I want to transfer data between two child processes directly, not between father and children. I show you a graph of what I'm trying to do.
communication between child proccess
I tried with the following code, but it did not work for me.
In the father.js code, I 'm creating the childs processes as follows:
const cp = require("child_process");
var son1 = cp.fork("${__dirname}/son1.js");
var son2 = cp.fork("${__dirname}/son2.js");
console.log("father sending message to son1..");
son1.send({msg:'Hi son1',br:son2});
console.log("father sending message to son2..");
son2.send({msg:'Hi son1',br:son1});
The Son1.js's code:
var brother=null;
process.on('message', function(json)
{
console.log('message father in son1.js;', json.msg);
brother=json.br;
brother.send("hello I'm son1.js");
});
And the Son2.js 's code:
var brother=null;
process.on('message', function(json)
{
console.log('message father in son2.js;', json.msg);
brother=json.br;
brother.send("hello I'm son2.js");
});
How can I send and receive messages from son1.js to son2.js and vice versa without sending messages to father.js?
Here's what you can do:
Create an IPC server on the parent linked to a socket file.
Open a connection to the server (still within the parent), creating a socket pair.
Send the server's socket to one child, send the client's socket to the other child.
parent.js:
const net = require('net');
const cp = require('child_process');
let u_proc_1 = cp.fork(__dirname+'/child.js', ['#1']);
let u_proc_2 = cp.fork(__dirname+'/child.js', ['#2']);
// create IPC server on parent
let d_server = net.createServer((d_socket_2) => {
// send server socket to #2
u_proc_2.send('socket', d_socket_2);
});
// create socket file
let p_socket = __dirname+'/sibling.sock';
// bind server to socket file
d_server.listen(p_socket);
// create client socket; this also triggers creation of a server socket
let d_socket_1 = net.connect(p_socket, () => {
// have #1 send to #2
u_proc_1.send('hey!');
// have #2 send to #1
u_proc_2.send('hello');
});
// send client socket to #1
u_proc_1.send('socket', d_socket_1);
child.js:
const name = process.argv[2];
let d_socket_sibling;
process.on('message', (s_action, d_socket_msg) => {
// parent is sending a socket
if('socket' === s_action) {
console.log(name+' now has a socket');
// save socket to variable for later use
d_socket_sibling = d_socket_msg;
// receive data from sibling
d_socket_sibling.on('data', (s_data) => {
console.log(name+' received: '+s_data);
});
}
// otherwise, parent wants me to send message to sibling
else {
console.log(name+' is sending: '+s_action);
// send data to sibling
d_socket_sibling.write(s_action);
}
});
Output:
#2 now has a socket
#1 now has a socket
#2 is sending: hello
#1 is sending: hey!
#1 received: hello
#2 received: hey!
The result is a direct, two-way communication channel between the two child processes, existentially mediated by the parent. Keep in mind that you will have to clean up the socket file somehow (e.g., deleting it on startup or before closing the parent)
You would have to open up another communication channel, such as a local socket (tcp, udp, or unix) or a third-party service (e.g. redis).
For send data to child process can you use this
const fork = require('child_process').fork('./son1.js')
fork.send('message')
son1.js
process.on('message', message => {
console.log(message)
process.exit()
})

Network request in threads is crashing my application

Currently I have a code that is crashing (SEGFAULT) on me.
I am trying to compare a big amount of images that are in my drive to their counter parts in a server.
To speed up the process I get the image from the server and compare the images on a different thread.
From what I already tried and debugged, the issue is in getting the image from the server (that is why the other calls are commented out).
Also if I run without the QtConcurrent::run it does not crash, but if I put semaphore concurrentComparisons with only one resorce, it will crash.
Finally I also get the following errors
QObject::connect: Cannot connect (null)::configurationAdded(QNetworkConfiguration) to QNetworkConfigurationManager::configurationAdded(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationRemoved(QNetworkConfiguration) to QNetworkConfigurationManager::configurationRemoved(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::configurationChanged(QNetworkConfiguration) to QNetworkConfigurationManager::configurationChanged(QNetworkConfiguration)
QObject::connect: Cannot connect (null)::onlineStateChanged(bool) to QNetworkConfigurationManager::onlineStateChanged(bool)
QObject::connect: Cannot connect (null)::configurationUpdateComplete() to QNetworkConfigurationManager::updateCompleted()
Any help would be very appreciated.....
Relevant code:
QSemaphore FileComparisonInfo::concurrentComparisons(1);
QtConcurrent::run( [this, localPath, imageURL]()
{
ImageComparer cmp;
FileComparisonInfo::concurrentComparisons.acquire();
//cmp.setImageLeftPath(localPath);
cmp.setImageRightPath(imageURL);
//cmp.createDifferenceImage();
FileComparisonInfo::concurrentComparisons.release();
});
void ImageComparer::setImageRightPath(QString path)
{
this->rightImagePath = path;
this->imageRight = getImage(path);
}
QImage* ImageComparer::getImage(QString path)
{
QUrl url(path);
QFile file(path);
if(file.exists())
{
return new QImage(path);
}
else if(url.isValid())
{
return getImageFromURL(path);
}
}
QImage* ImageComparer::getImageFromURL(QString url)
{
QNetworkAccessManager * tempNAM = new QNetworkAccessManager();
QNetworkReply *imageConnection = tempNAM->get( QNetworkRequest( QUrl( url ) ));
QEventLoop loop;
connect(imageConnection, SIGNAL(finished()), &loop, SLOT(quit()));
loop.exec();
QImage * downloadedImage;
if(imageConnection->error() != QNetworkReply::NoError)
{
qDebug() << imageConnection->errorString();
downloadedImage = new QImage();
}
else
{
QByteArray data = imageConnection->readAll();
downloadedImage = new QImage(QImage::fromData(data));
}
tempNAM->deleteLater();
imageConnection->deleteLater();
return downloadedImage;
}
Unfortunately this had nothing to do with the code.
One of the images was corrupted and was segfaulting in the comparison.

Scala client server multithreaded using socket

I cant get my head around this one here
I am a beginner to Scala just few weeks old and have tried but failed
I have read and tried about Actors, Futures,...etc didnt work for me
Could you supply a code of a server client example (or at least the server side)
Suppose to open connection using a socket that receives a string (i.e. file path) from several clients and process each one in a thread
import java.net.{Socket, ServerSocket}
import java.util.concurrent.{Executors, ExecutorService}
import java.util.Date
import java.io._
import scala.io._
import java.nio._
import java.util._
import scala.util.control.Breaks
import java.security.MessageDigest
import java.security.DigestInputStream
import scala.util.Sorting
class NetworkService(port: Int, poolSize: Int) extends Runnable {
val serverSocket = new ServerSocket(port)
val pool: ExecutorService = Executors.newFixedThreadPool(poolSize)
def run() {
try {
var i = 0
while (true) {
// This will block until a connection comes in.
val socket = serverSocket.accept()
val in = new BufferedReader(new InputStreamReader(socket.getInputStream)).readLine
/*var f = new FileSplit(in) //FileSplit is another class that i would like each
// client's sent string to be passed as an instance of
f.move*/
pool.execute(new Handler(socket))
}
} finally {
pool.shutdown()
}
}
}
class Handler(socket: Socket) extends Runnable {
def message = (Thread.currentThread.getName() + "\n").getBytes
def run() {
socket.getOutputStream.write(message)
socket.getOutputStream.close()
}
}
object MyServer {
def main(args: Array[String]) {
(new NetworkService(2030, 2)).run
}
}
You have several options available. You could do same old java style app, basically just using java standard libraries and scala syntax.
Maybe this helps: Scala equivalent of python echo server/client example?
You would just need to write logic that handles each socket (the one you get from accept()) in a new thread.
However I would not recommend using plain old java approach directly. There are great libraries out there that can handle that for you. For example Akka:
http://doc.akka.io/docs/akka/2.3.3/scala/io-tcp.html
I would also urge you to read about futures as they are super useful to do stuff async.

How to calculate node.js socket buffer to avoid allocating memory and never using it?

I'm using node.js as a server between pairs of clients, to handle my online game.
Clients send short messages between hem [one message should not exceed 200bytes].
Currently I expect single client to send [on average] 1 message per second [keeping in mind it can be 5 seconds of nothing and 5 messages one after another].
I've downloaded a sample server using 'net' module and rewritten it to handle the messages the way I need them to be handled.
Basically, for every connected socket, it creates a Buffer with size of 1024*8.
Currently I'm testing my game with some bots, which simply connect, wait 3 seconds and disconnect. They only send 1 message. Nothing else happening.
function sendMessage(socket, message) {
socket.write(message);
}
server.on('connection', function(socket) {
socket.setNoDelay(true);
socket.connection_id = require('crypto').createHash('sha1').update( 'krystian' + Date.now() + Math.random() ).digest('hex') ; // unique sha1 hash generation
socket.channel = '';
socket.matchInProgress = false
socket.resultAnnounced = false;
socket.buffer = new Buffer(cfg.buffer_size);
socket.buffer.len = 0; // due to Buffer's nature we have to keep track of buffer contents ourself
_log('New client: ' + socket.remoteAddress +':'+ socket.remotePort);
socket.on('data', function(data_raw) { // data_raw is an instance of Buffer as well
if (data_raw.length > (cfg.buffer_size - socket.buffer.len)) {
_log("Message doesn't fit the buffer. Adjust the buffer size in configuration");
socket.buffer.len = 0; // trimming buffer
return false;
}
socket.buffer.len += data_raw.copy(socket.buffer, socket.buffer.len); // keeping track of how much data we have in buffer
var str, start, end
, conn_id = socket.connection_id;
str = socket.buffer.slice(0,socket.buffer.len).toString();
if ( (start = str.indexOf("<somthing>")) != -1 && (end = str.indexOf("</something>")) != -1) {
try {
if (!<some check to see if the message format is right>) {
sendMessage(socket, "<error message to the client>");
return;
}
<storing info on the socket>
} catch(err) {
sendMessage(socket, "<error message to the client>");
return;
}
socket.channel = <channel>;
str = str.substr(end + 11);
socket.buffer.len = socket.buffer.write(str, 0);
sockets[socket.channel] = sockets[socket.channel] || {}; // hashmap of sockets subscribed to the same channel
sockets[socket.channel][conn_id] = socket;
waiting[socket.channel] = waiting[socket.channel] || {};
waiting[socket.channel][conn_id] = socket;
sendMessage(socket, "<info message to the client>");
for (var prop in waiting[socket.channel]) {
if (waiting[socket.channel].hasOwnProperty(prop) && waiting[socket.channel][prop].connection_id != socket.connection_id) {
<here I'll try to advertise this client among other clients>
sendMessage(waiting[socket.channel][prop], "<info to other clients about new client>");
}
}
}
var time_to_exit = true;
do{ // this is for a case when several messages arrived in buffer
if ( (start = str.indexOf("<some other format>")) != -1 && (end = str.indexOf("</some other format>")) != -1 ) {
var json = str.substr( start+19, end-(start+19) );
var jsono;
try {
jsono = JSON.parse(json);
} catch(err) {
sendMessage(socket, "<parse error>");
return;
}
if (<message indicates two clients are going to play together>) {
if (waiting[socket.channel][jsono.other_client_id] && waiting[socket.channel][socket.connection_id]) {
delete waiting[socket.channel][jsono.other_client_id];
delete waiting[socket.channel][socket.connection_id];
var opponentSocket = sockets[socket.channel][jsono.other_client_id];
sendMessage(opponentSocket, "<start game with the other socket>");
opponentSocket.opponentConnectionId = socket.connection_id;
sendMessage(socket, "<start game with the other socket>");
socket.opponentConnectionId = jsono.other_client_id;
}
} else if (<check if clients play together>) {
var opponentSocket = sockets[socket.channel][socket.opponentConnectionId];
if (<some generic action between clients, just pass the message>) {
sendMessage(sockets[socket.channel][socket.opponentConnectionId], json);
} else if (<match is over>) {
if (<match still in progress>) {
<send some messages indicating who won, who lost>
} else {
<log an error>
}
delete sockets[socket.channel][opponentSocket.connection_id];
delete sockets[socket.channel][socket.connection_id];
}
}
str = str.substr(end + 20); // cut the message and remove the precedant part of the buffer since it can't be processed
socket.buffer.len = socket.buffer.write(str, 0);
time_to_exit = false;
} else { time_to_exit = true; } // if no json data found in buffer - then it is time to exit this loop
} while ( !time_to_exit );
}); // end of socket.on 'data'
socket.on('close', function(){ // we need to cut out closed socket from array of client socket connections
if (!socket.channel || !sockets[socket.channel]) return;
if (waiting[socket.channel] && waiting[socket.channel][socket.connection_id]) {
delete waiting[socket.channel][socket.connection_id];
}
var opponentSocket = sockets[socket.channel][socket.opponentConnectionId];
if (opponentSocket) {
sendMessage(opponentSocket, "<the other client has disconnected>");
delete sockets[socket.channel][socket.opponentConnectionId];
}
delete sockets[socket.channel][socket.connection_id];
_log(socket.connection_id + " has been disconnected from channel " + socket.channel);
}); // end of socket.on 'close'
}); // end of server.on 'connection'
server.on('listening', function(){ console.log('Listening on ' + server.address().address +':'+ server.address().port); });
server.listen(cfg.port);
I've pasted the above code [very stripped version of the original] to give you and idea about how simple the server is.
I've got an array of sockets, who joined the game and array of sockets on the waiting list, waiting for another client to play with.
Nothing else is going on.
Still the script is memory hungry - 5 hours of connecting and disconnecting gave me this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31461 ec2-user 20 0 995m 91m 7188 S 0.7 15.4 1:29.07 node
I think this is way too much.
I'm using nodetime.com free service at the moment to monitor the script, but none of the metrics would suggest the script gained so much memory (it starts with just 10-12MB).
I believe this is due to the buffers, and because they allocate too much memory.
I'm only wondering, if my assumptions regarding buffer size are correct.
Should I adjust the buffer to reflect the amount of data I expect from the client?
If I expect the client to send 5 messages with a very short time between them, 200 bytes max each, should I assume that 1024*3 would be enough?
Or should I adjust buffer size according to the message size I expect, so if I'm sure the message will never go above 300 bytes, I should be fine with buffer size of 512?
Thanks,
Krystian
EDIT:
Node version:
$ node -v
v0.10.5
$ npm -v
1.2.19
EDIT2:
I've tested the script with 400 connections connecting and disconnecting and memory usage dropped significantly to around 60MB. After changing the test setup back to 4 connections it went up again.
The kernel has a socket receive buffer which is at least 8k., which takes care of multiple incoming messages on the socket. You don't need to buffer messages you've already read, so your application buffer doesn't need to be any bigger than the largest expected message.

Resources