How do I go about connecting to a server and check if there is a file to be downloaded using Java ME?
Should I use POST method, post all my required data to server and see what the server response back? And also what are the headers do I have to post together with my data?
Or do I have to use a Get method?
You don't need it to do by hand. Just use
HttpConnection connection = Connector.open(url);
DataInputStream stream = connection.openDataInputStream();
while(stream.read() != -1) {
//your stuff if anything there
}
Related
Introduction
Say that on the same local network we have two Node JS servers set up with Express: Server A for API and Server F for form.
Server A is an API server where it takes the request and saves it to MongoDB database (files are stored as Buffer and their details as other fields)
Server F serves up a form, handles the form post and sends the form's data to Server A.
What is the most efficient way to send files between two NodeJS servers where the receiving server is Express API? Where does the file size matter?
1. HTTP Way
If the files I'm sending are PDF files (that won't exceed 50mb) is it efficient to send the whole contents as a string over HTTP?
Algorithm is as follows:
Server F handles the file request using https://www.npmjs.com/package/multer and saves the file
then Server F reads this file and makes an HTTP request via https://github.com/request/request along with some details on the file
Server A receives this request and turns the file contents from string to Buffer and saves a record in MongoDB along with the file details.
In this algorithm, both Server A (when storing into MongoDB) and Server F (when it was sending it over to Server A) have read the file into the memory, and the request between the two servers was about the same size as the file. (Are 50Mb requests alright?)
However, one thing to consider is that -with this method- I would be using the ExpressJS style of API for the whole process and it would be consistent with the rest of the app where the /list, /details requests are also defined in the routes. I like consistency.
2. Socket.IO Way
In contrast to this algorithm, I've explored https://github.com/nkzawa/socket.io-stream way which broke away from the consistency of the HTTP API on Server A (as the handler for socket.io events are defined not in the routes but the file that has var server = http.createServer(app);).
Server F handles the form data as such in routes/some_route.js:
router.post('/', multer({dest: './uploads/'}).single('file'), function (req, res) {
var api_request = {};
api_request.name = req.body.name;
//add other fields to api_request ...
var has_file = req.hasOwnProperty('file');
var io = require('socket.io-client');
var transaction_sent = false;
var socket = io.connect('http://localhost:3000');
socket.on('connect', function () {
console.log("socket connected to 3000");
if (transaction_sent === false) {
var ss = require('socket.io-stream');
var stream = ss.createStream();
ss(socket).emit('transaction new', stream, api_request);
if (has_file) {
var fs = require('fs');
var filename = req.file.destination + req.file.filename;
console.log('sending with file: ', filename);
fs.createReadStream(filename).pipe(stream);
}
if (!has_file) {
console.log('sending without file.');
}
transaction_sent = true;
//get the response via socket
socket.on('transaction new sent', function (data) {
console.log('response from 3000:', data);
//there might be a better way to close socket. But this works.
socket.close();
console.log('Closed socket to 3000');
});
}
});
});
I said I'd be dealing with PDF files that are < 50Mb. However, if I use this program to send larger files in the future, is socket.io a better way to handle 1GB files as it's using stream?
This method does send the file and the details across but I'm new to this library and don't know if it should be used for this purpose or if there is a better way of utilizing it.
Final thoughts
What alternative methods should I explore?
Should I send the file over SCP and make an HTTP request with file details including where I've sent it- thus, separating the protocols of files and API requests?
Should I always use streams because they don't store the whole file into memory? (that's how they work, right?)
This https://github.com/liamks/Delivery.js ?
References:
File/Data transfer between two node.js servers this got me to try socket-stream way.
transfer files between two node.js servers over http for HTTP way
There are plenty of ways to achieve this , but not so much to do it right !
socket io and wesockets are efficient when you use them with a browser , but since you don't , there is no need for it.
The first method you can try is to use the builtin Net module of nodejs, basically it will make a tcp connection between the servers and pass the data.
you should also keep in mind that you need to send chunks of data not the entire file , the socket.write method of the net module seems to be a good fit for your case check it : https://nodejs.org/api/net.html
But depending on the size of your files and concurrency , memory consumption can be quite large.
if you are running linux on both servers you could even send the files at ground zero with a simple linux command called scp
nohup scp -rpC /var/www/httpdocs/* remote_user#remote_domain.com:/var/www/httpdocs &
You can even do this with windows to linux or the other way.
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
the client scp for windows is pscp.exe
Hope this helps !
I am working with a nodejs express server which uses socket.io to communicate an iOS client, and am having a little trouble trying to test how many clients can connect and exchange data at any one time.
My goal is to be able to run a script which connects to socket.io with thousands of different sessions, as well as send and receive data to understand our system's scale. Currently we are using a single dyno on Heroku but will likely be considering other options on AWS soon.
I have found code which should do what I am trying to do for earlier versions of socket.io, such as this, but have had issues since it seems v1.x has a very different handshake protocol. I tried out using the socket.io-client package, but trying to connect multiple times only simulates use of one session, I need to simulate many in independent users.
I have been picking apart the socket.io-client code, but have only gotten so far as creating a connection - I am stuck on the sending data part. If anyone has any knowledge or could point to some written resources on how data is sent between a client and a socket.io server, it would help me out a lot.
Here's what I have so far:
var needle = require('needle'),
WebSocket = require('ws'),
BASE_URL = 'url-to-socket-host:5002';
var connectionNo = 0;
needle.get('http://' + BASE_URL + '/socket.io/?EIO=3&transport=polling&t=1416506501335-0', function (err, resp) {
// parse the sid
var resp = JSON.parse(resp.body.toString().substring(5, resp.body.toString().length));
// use the sid to connect using websockets
var url = 'ws://' + BASE_URL + '/socket.io/?EIO=3&transport=websocket&sid=' + resp.sid;
console.log(connectionNo + ' with sid: ' + resp.sid);
var socket = new WebSocket(url, void(0), {
agent: false
});
socket.on('open', function () {
console.log('Websocket connected: ' + connectionNo);
// I don't understand how to send data to the server here,
// from looking at the source code it should use some kind
// of binary encoding, any ideas?
socket.on('message', function (msg) {
console.log(msg);
});
});
});
I will continue deconstructing the socket.io-client code but if anyone has any clues or recourses that may help, let me know. Thanks.
I ended up setting for using the socket.io-client npm package which has the ability to connect to a new session on every connection. I found an example benchmark in this issue.
There is not so much need for me to manually connect to socket.io using pure websockets and HTTP, but thanks to Yannik for pointing out the parser in use. The spec of the inner workings of v1.x can be found here.
Thanks!
The problem my reside in the fact that you are not using socket.io in your client code. You have imported ('ws') which is another module whose docs are here: https://www.npmjs.org/package/ws.
You probably want to ws.send('something');. When you receive a message in ws, it also comes with an object with a property indicating whether it is binary data or not. If it is, you will need to concatenate the chunks incrementally. There is a canonical way to do this which you can find via google. But it looks a little like this:
var message;
socketConnection.on('data', function(chunk){ message += chunk});
I'm searching a solution to store a WebM file into Redis.
Let's explain the situation:
The NodeJS server receive a WebM file from a client, and save it into server file system.
Then it have to save this file in redis, because I don't want to manage redis and file system too. In this way I can delete the video just with redis command.
I think to read file with fs.readFile() and then save it into a Buffer, but I don't know witch encode format to use, and I don't know how to refer this process to give back the WebM video to a client when it make a request.
Is this a good way to proceed? Any suggestion?
PS: I use formidable to upload file.
EDIT: I found a way to proceed, but theres another problem:
var file = fs.readFileSync("./video.webm");
client.set("video1", file1, function(){
client.get("video1", function(err, data) {
var buffer = new Buffer(data, 'binary');
// file ≠ buffer
});
});
Is this an encode problem? Like unicode/UTF8/ASCII?
Maybe node and redis use different encode?
Solution found!
The problem became when you create the client object.
Usually this is what is done
var client = redis.createClient();
And return_buffers param will be set as false.
In this way
var client = redis.createClient(6379, '127.0.0.1', {
return_buffers: true,
auth_pass: null
});
everything gone right! ;)
this is the issue page they help me
I don't know much about NodeJS and WebM files.
Redis stores C char type on 8 bit String, so it should be binary friendly. Check the js code and configuration to ensure your js redis client sends / receives data as bytearray and not as UTF-8 string, probably there is a bad conversion in JS of data.
I need to make download script by NodeJs,which limit connections number and speed for each session.
I can read file and write is to response by NodeJs, maybe I can limit speed by sleep package of npm, but how should I support more than one connection to download file in NodeJs?
for e.g when user download it by IDM he/she can see 7 or 8 connection during download.
In apache it can do it by mode_limitipconn and bw_mod
I mean something like RapidShare which limit speed and connections number for each category.
What you want to do is limit access to a resource to only one IP address at a time.
This is fairly easy to do with Node since all requests are served from a single process.
You can enter IP addresses into an object when the request is made and then check that object when new requests come in for duplicates.
If you're using Express and you have your routes in modules, you can put the IP connection object in the top level of the route module.
var connectedIPs = {};
exports.myDownloadRoute = function(request, response) {
var IP = request.connection.remoteAddress;
if(connectedIPs.IP) {
response.redirect("http://mysite.com/download_rules.html");
return;
}
connectedIPs.IP = true;
// pseudo send a file call, replace this with your code
send_a_file(function(err) {
// done sending or error, remove from connectedIPs
delete connectedIPs.IP;
});
}
I'm new to Node.js, and I've been playing with the "chat" example provided with the Socket.io install package. Is shows in a few lines of code how you can push some data to several clients (browsers) in a push-fashion (no pulling).
Here is the code on the server side : http://pastie.org/1537175
I get how you can send a message to a client with client.broadcast(msg), but I don't get how you can do it outside of the
io.on('connection', function(client){
... }
loop
I would like to invoke a client.broadcast(msg) when someone hits a particular url (like '/test.html'), see line #32. The device asking for the '/test.html' is not a typical "ajax-enabled" browser, but a mere text-based browser, so I cannot initialize an asynchronous request with the server. Any idea?
Thank you.
you can use .broadcast on your io object
case '/test.html':
io.broadcast('test'); // This is where I would like to invoke a client.broadcast(msg);
break;