I have the following code that works fine for sending a file by FTP, but it blocks my UI.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ftpUrl + filename);
request.UsePassive = false;
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential(ftpUser, ftpPass);
request.Timeout = 10000; //10 second timeout
byte[] fileContents = File.ReadAllBytes(fullPath);
request.ContentLength = fileContents.Length;
//Stream requestStream = await request.GetRequestStreamAsync();
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
I want to switch the Stream to the commented line so I call asynchronously and don't block my UI, and it works fine, except for the timeout, which according to the docs is for synchronous use only.
The question is how to make a timeout work on an async call?
From document FtpWebRequest.Timeout Property ,Timeout is the number of milliseconds that a synchronous request made with the GetResponse method waits for a response and that the GetRequestStream method waits for a stream. So there is no more api to use it asynchronously .
Maybe this can be a good way to realize it.Putting FtpWebRequest code into the Task to have a try.
// Start a new task (this launches a new thread)
Task.Factory.StartNew (() => {
// Do some work on a background thread, allowing the UI to remain responsive
DoSomething();
// When the background work is done, continue with this code block
}).ContinueWith (task => {
DoSomethingOnTheUIThread();
// the following forces the code in the ContinueWith block to be run on the
// calling thread, often the Main/UI thread.
}, TaskScheduler.FromCurrentSynchronizationContext ());
Related
I am developing an application targetting mobile devices, so I have to consider bad network connectivity. In one use case, I need to reduce the timeout for a request, because if no network is available, that's okay, and I'd fall back to default data immediately, without having the user wait for the HTTP response.
I found that HttpMixin.MakeWebRequest() has a timeout parameter (with default=null) but DownloadUrl() never makes use of it, so the forementioned function always waits for up to 15 seconds:
request.Timeout(timeout ?? TimeSpan.FromSeconds(15),
BlobCache.TaskpoolScheduler).Retry(retries);
So actually I do not have the option to use a different timeout, or am I missing something?
Thanks for considering a helpful response.
So after looking at the signature for DownloadUrl in
HttpMixin.cs
I saw what you are talking about and am not sure why it is there but, it looks like the timeout is related to building the request and not a timeout for the request itself.
That being said, in order to set a timeout with a download, you have a couple options that should work.
Via TPL aka Async Await
var timeout = 1000;
var task = BlobCache.LocalMachine.DownloadUrl("http://stackoverflow.com").FirstAsync().ToTask();
if (await Task.WhenAny(task, Task.Delay(timeout)) == task) {
// task completed within timeout
//Do Stuff with your byte data here
//var result = task.Result;
} else {
// timeout logic
}
Via Rx Observables
var obs = BlobCache.LocalMachine
.DownloadUrl("http://stackoverflow.com")
.Timeout(TimeSpan.FromSeconds(5))
.Retry(retryCount: 2);
var result = obs.Subscribe((byteData) =>
{
//Do Stuff with your byte data here
Debug.WriteLine("Byte Data Length " + byteData.Length);
}, (ex) => {
Debug.WriteLine("Handle your exceptions here." + ex.Message);
});
I have a stream process like this:
Incomming file via HTTP (original stream)
-> Check if zipfile
- Yes -> push through an unzip2-stream
- No -> push to S3
When the unzip2-stream finds zip-entries, these are pushed through the same chain of streams, i.e.
Incomming file entry from zip file ("child" stream)
-> Check if zipfile
- Yes -> push through an unzip2-stream
- No -> push to S3
Thanks to https://stackoverflow.com/users/3580261/eljefedelrodeodeljefe I managed to solve the main problem after this conversation:
How to redirect a stream to other stream depending on data in first chunk?
The problem with creating new "child" streams for every zip entry is that these will have no connection to the original stream, so I cannot get a unified onFinish for all the streams.
I don't want to send a 202 of to the sender before I have processed (unzipped and sent to S3) every file. How can I accomplish this?
I'm thinking that I might need some kind of control object which awaits onFinish for all child streams and forces the process to dwell in the original onFinish event until all files are processed. Would this be overkill? Is there a simpler solution?
I ended up making a separate counter for the streams. There is probably a better solution, but this works.
I send the counter object as an argument to the first call to my saveFile() function. The counter is passed along to the unzip stream so it can be passed to saveFile for every file entry.
Just before a stream is started (i.e. piped) I call streamCounter.streamStarted().
In the last onFinish in the pipe chain I call streamCounter.streamFinished()
In the event of a stream going bad I call streamCounter.streamFailed()
Just before I send the 202 in the form post route I wait for streamCounter.streamPromise to resolve.
I'm not very proud of the setInterval solution. It'd probably be better with some kind of event emitting.
module.exports.streamCounter = function() {
let streamCount = 0;
let isStarted = false;
let errors = [];
this.streamStarted = function(options) {
isStarted = true;
streamCount += 1;
log.debug(`Stream started for ${options.filename}. New streamCount: ${streamCount}`);
};
this.streamFinished = function(options) {
streamCount -= 1;
log.debug(`Finished stream for ${options.filename}. New streamCount: ${streamCount}`);
};
this.streamFailed = function(err) {
streamCount -= 1;
errors.push(err);
log.debug(`Failed stream because (${err.message}). New streamCount: ${streamCount}`);
};
this.streamPromise = new Promise(function(resolve, reject) {
let interval = setInterval(function() {
if(isStarted && streamCount === 0) {
clearInterval(interval);
if(errors.length === 0) {
log.debug('StreamCounter back on 0. Resolving streamPromise');
resolve();
} else {
log.debug('StreamCounter back on 0. Errors encountered.. Rejecting streamPromise');
reject(errors[errors.length-1]);
}
}
}, 100);
});
};
At first I tried this concept with a promise array and waited for Promise.all() before sending status 202. But Promise.all() only works with static arrays as far as I can tell. My "streamCount" is changing during the streaming so I needed a more dynamic "Promise.all".
I have a function which calls Concurrency::create_task to perform some work in the background. Inside that task, there is a need to call a connectAsync method on the StreamSocket class in order to connect a socket to a device. Once the device is connected, I need to grab some references to things inside the connected socket (like input and output streams).
Since it is an asynchronous method and will return an IAsyncAction, I need to create another task on the connectAsync function that I can wait on. This works without waiting, but complications arise when I try to wait() on this inner task in order to error check.
Concurrency::create_task( Windows::Devices::Bluetooth::Rfcomm::RfcommDeviceService::FromIdAsync( device_->Id ) )
.then( [ this ]( Windows::Devices::Bluetooth::Rfcomm::RfcommDeviceService ^device_service_ )
{
_device_service = device_service_;
_stream_socket = ref new Windows::Networking::Sockets::StreamSocket();
// Connect the socket
auto inner_task = Concurrency::create_task( _stream_socket->ConnectAsync(
_device_service->ConnectionHostName,
_device_service->ConnectionServiceName,
Windows::Networking::Sockets::SocketProtectionLevel::BluetoothEncryptionAllowNullAuthentication ) )
.then( [ this ]()
{
//grab references to streams, other things.
} ).wait(); //throws exception here, but task executes
Basically, I have figured out that the same thread (presumably the UI) that creates the initial task to connect, also executes that task AND the inner task. Whenever I attempt to call .wait() on the inner task from the outer one, I immediately get an exception. However, the inner task will then finish and connect successfully to the device.
Why are my async chains executing on the UI thread? How can i properly wait on these tasks?
In general you should avoid .wait() and just continue the asynchronous chain. If you need to block for some reason, the only fool-proof mechanism would be to explicitly run your code from a background thread (eg, the WinRT thread pool).
You could try using the .then() overload that takes a task_options and pass concurrency::task_options(concurrency::task_continuation_context::use_arbitrary()), but that doesn't guarantee the continuation will run on another thread; it just says that it's OK if it does so -- see documentation here.
You could set an event and have the main thread wait for it. I have done this with some IO async operations. Here is a basic example of using the thread pool, using an event to wait on the work:
TEST_METHOD(ThreadpoolEventTestCppCx)
{
Microsoft::WRL::Wrappers::Event m_logFileCreatedEvent;
m_logFileCreatedEvent.Attach(CreateEventEx(nullptr, nullptr, CREATE_EVENT_MANUAL_RESET, WRITE_OWNER | EVENT_ALL_ACCESS));
long x = 10000000;
auto workItem = ref new WorkItemHandler(
[&m_logFileCreatedEvent, &x](Windows::Foundation::IAsyncAction^ workItem)
{
while (x--);
SetEvent(m_logFileCreatedEvent.Get());
});
auto asyncAction = ThreadPool::RunAsync(workItem);
WaitForSingleObjectEx(m_logFileCreatedEvent.Get(), INFINITE, FALSE);
long i = x;
}
Here is a similar example except it includes a bit of Windows Runtime async IO:
TEST_METHOD(AsyncOnThreadPoolUsingEvent)
{
std::shared_ptr<Concurrency::event> _completed = std::make_shared<Concurrency::event>();
int i;
auto workItem = ref new WorkItemHandler(
[_completed, &i](Windows::Foundation::IAsyncAction^ workItem)
{
Windows::Storage::StorageFolder^ _picturesLibrary = Windows::Storage::KnownFolders::PicturesLibrary;
Concurrency::task<Windows::Storage::StorageFile^> _getFileObjectTask(_picturesLibrary->GetFileAsync(L"art.bmp"));
auto _task2 = _getFileObjectTask.then([_completed, &i](Windows::Storage::StorageFile^ file)
{
i = 90210;
_completed->set();
});
});
auto asyncAction = ThreadPool::RunAsync(workItem);
_completed->wait();
int j = i;
}
I tried using an event to wait on Windows Runtime Async work, but it blocked. That's why I had to use the threadpool.
Node.js is famous for concurrency, however, I'm confused by how to make it work concurrently. I started two requests from Chrome one by one very quickly, and I Expected the outputs in console should be:
"get a new request"
immediately after my second request, "get a new request" should be printed
after several seconds, "end the new request"
after several seconds, "end the new request"
However, what I saw is:
"get a new request"
after several seconds, "end the new request"
"get a new request"
after several seconds, end the new request
That means the second request is NOT handled until the first one is done. Below is my sample code, anything I missed?
var http = require("http");
var url = require("url");
function start(route) {
http.createServer(function(request, response) {
console.log('get a new request');
// a time consuming loop
for (var i=0; i<10000000000; ++i) {
}
route(url.parse(request.url).pathname);
response.writeHead(200, {"Content-Type": "text/plain"});
response.end();
console.log('end the new request');
}).listen(5858);
}
function saySomething(something) {
console.log(something);
}
exports.start = start;
exports.saySomething = saySomething;
You don't have to do anything.
It's based on non blocking I/O. Put simply, there is an event loop. A certain set of sync code are run, once done, the next iteration is run that picks up the next set of sync code to run. Anytime an async op is run (db fetch, setTimeout, reading a file, etc) the next tick of the event loop is run. This way there is never any code just waiting.
It's not threaded. In your example, the for loop is in one continuous chunk of code, so js will run the entire for loop before it can handle another http request.
Try putting a setTimeout around the for loop so that node can switch to the next event loop and in your case handle a web request.
node can't handle these:
for (var i=0; i<10000000000; ++i) {}
concurrently. But it handles IO concurrently
You might want to look at Clusters:
http://nodejs.org/api/cluster.html#cluster_how_it_works
http://rowanmanning.com/posts/node-cluster-and-express/
T̶h̶i̶s̶ ̶i̶s̶ ̶t̶h̶e̶ ̶e̶x̶p̶e̶c̶t̶e̶d̶ ̶b̶e̶h̶a̶v̶i̶o̶r̶,̶ ̶w̶e̶ ̶c̶a̶l̶l̶ ̶t̶h̶i̶s̶ ̶̶b̶l̶o̶c̶k̶i̶n̶g̶̶.̶ ̶T̶h̶e̶ ̶s̶o̶l̶u̶t̶i̶o̶n̶ ̶f̶o̶r̶ ̶h̶a̶n̶d̶l̶i̶n̶g̶ ̶c̶o̶n̶c̶u̶r̶r̶e̶n̶t̶ ̶r̶e̶q̶u̶e̶s̶t̶ ̶i̶s̶ ̶m̶a̶k̶i̶n̶g̶ ̶t̶h̶e̶ ̶c̶o̶d̶e̶ ̶̶n̶o̶n̶-̶b̶l̶o̶c̶k̶i̶n̶g̶̶.̶ ̶A̶s̶ ̶s̶o̶o̶n̶ ̶a̶s̶ ̶y̶o̶u̶ ̶c̶a̶l̶l̶e̶d̶ ̶̶r̶e̶s̶p̶o̶n̶s̶e̶.̶w̶r̶i̶t̶e̶H̶e̶a̶d̶̶ ̶t̶h̶e̶ ̶c̶o̶d̶e̶ ̶b̶e̶g̶a̶n̶ ̶b̶l̶o̶c̶k̶i̶n̶g̶ ̶w̶a̶i̶t̶i̶n̶g̶ ̶f̶o̶r̶ ̶̶r̶e̶s̶p̶o̶n̶s̶e̶.̶e̶n̶d̶̶.
EDIT 7/8/14:
Had to deal with this problem recently and found out you can use threads for this:
https://www.npmjs.org/package/webworker-threads
Webworker-threads provides an asynchronous API for CPU-bound tasks that's missing in Node.js:
var Worker = require('webworker-threads').Worker;
require('http').createServer(function (req,res) {
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
this.onmessage = function (event) {
postMessage(fibo(event.data));
}
});
fibo.onmessage = function (event) {
res.end('fib(40) = ' + event.data);
};
fibo.postMessage(40);
}).listen(port);
And it won't block the event loop because for each request, the fibo worker will run in parallel in a separate background thread.
I have to write a small app which downloads a few thousand files. Some of these files contain reference to other files that must be downloaded as part of the same process. The following code downloads the initial list of files, but I would like to download the others files as part of the same loop. What is happening here is that the loop completes before the first request come back. Any idea how to achieve this?
var countdownLatch = new CountdownEvent(Urls.Count);
string url;
while (Urls.TryDequeue(out url))
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.BeginGetResponse(
new AsyncCallback(ar =>
{
using (HttpWebResponse response = (ar.AsyncState as HttpWebRequest).EndGetResponse(ar) as HttpWebResponse)
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
string myFile = sr.ReadToEnd();
// TODO: Look for a reference to another file. If found, queue a new Url.
}
}
}), webRequest);
}
ce.Wait();
One solution which comes to mind is to keep track of the number of pending requests and only finish the loop once no requests are pending and the Url queue is empty:
string url;
int requestCounter = 0;
int temp;
AutoResetEvent requestFinished = new AutoResetEvent(false);
while (Interlocked.Exchange(requestCounter, temp) > 0 || Urls.TryDequeue(out url))
{
if (url != null)
{
Interlocked.Increment(requestCounter);
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.BeginGetResponse(
new AsyncCallback(ar =>
{
try {
using (HttpWebResponse response = (ar.AsyncState as HttpWebRequest).EndGetResponse(ar) as HttpWebResponse)
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
string myFile = sr.ReadToEnd();
// TODO: Look for a reference to another file. If found, queue a new Url.
}
}
}
finally {
Interlocked.Decrement(requestCounter);
requestFinished.Set();
}
}), webRequest);
}
else
{
// no url but requests are still pending
requestFinished.WaitOne();
}
}
You are tryihg to write a webcrawler. In order to write a good webcrawler, you first need to define some parameters...
1) How many request do you want to download simultaneously? In other words, how much throughput do you want? This will determine things like how many requests you want outstanding, what the threadpool size should be etc.
2) You will have to have a queue of URLs. This queue is populated by each request that completes. You now need to decide what the growth strategy of the queue is. For eg, you cannot have an unbounded queue, as you can pump workitems into the queue faster than you can download from the network.
Given this, you can design a system as follows:
Have max N worker threads that actually download from the web. They take one time from the queue, and download the data. They parse the data and populate your URL queue.
If there are more than 'M' URLs in the queue, then the queue blocks and does not allow anymore URLs to be queued. Now, here you can do one of two things. You can either cause the thread that is enqueuing to block, or you can just discard the workitem being enqueued. Once another workitem completes on another thread, and a URL is dequeued, the blocked thread will now be able to enqueue succesfully.
With a system like this, you can ensure that you will not run out of system resources while downloading the data.
Implementation:
Note that if you are using async, then you are using an extra I/O thread to do the download. THis is fine, as long as you are mindful of this fact. You can do a pure async implementation, where you can have 'N' BeginGetResponse() outstanding, and for each one that completes, you start another one. THis way you will always have 'N' requests outstanding.