I've got a web role on Azure that one of its jobs is to upload a picture, format it and then upload to a BLOB.
I do this with a temp directory on the web role - so there is a temp file there which I delete after it is uploaded to a BLOB.
Sometime the uploading is interrupted or the web role has some problems and the temp image file stays on the web role.
I want to create a worker role that once in X hours will clean that folder. It is possible that I will have 100 web roles (each one in its own isolated environment) and only 2 worker roles - So they have to somehow get to the web roles, one by one, and delete those files.
So my question is - Is this even possible?! if so, how?
Thanks!
Should you create a worker role it would run on a separate VM, not the same as your web role and that would defeat the whole idea since you can't get to another VM without a carefully crafted interface and such interface would definitely be an overkill for this task.
What you really want is just a separate thread (System.Threading.Thread) that you start from within the web role entry point and that continuously monitors the temporary folder for leftover files. That will be cheap and working.
Related
I present my problem if anyone can help will be appreciated:
I have a web site (implemented in WebForms in Azure) where the user can make configurations that will result into a XML with all the data of the configuration.
In the other hand, I have a Worker Role which is working and need this XML configuration.
The question is: how can I send this XML file generated by the user to the Worker Role?
I have looked for something similar to a REST API as interol communication seems not to be the correct path to follow.
Save the XML to the blob storage
Use Azure Queue - web app will add the "job" message pointing to XML (XML url + maybe some other data) and worker roles periodically check for "jobs" in this queue and processing them
It is recommended practice in Azure, and it works very well (I am using it on some projects with very high load - hundreds of thousands "jobs" per day. Azure queues are very reliable and fast, those performance issues you have read about were certainly in some "user code".
Fairly new to Azure and the whole worker role concept, previously if I wanted some back end work done I would just create a windows forms application and have it as a scheduled task.
With my new site I have created a windows form application which I have running every hour which reads in XML feeds and does all the processing an inserts the information into sql azure.
There is also image links that I want to store in azure blob storage and possibly resize them , which I had trouble doing from my vb.net application.
My question is should I move all the processing from my windows form application to the worker role or should I set up a worker role to just process the image to blob storage?
How much compute time does a worker role use? I have seen examples of where there is a sleep timer but is it possible to run it every hour on the hour?
You can easily set up a timer to trigger every hour on the hour. Where you run your code this depends on your application architecture. If you have a web role, you can place this in your web role instead of a dedicated worker role unless you really need the extra processing power of a separate instance and are willing to pay for it. Also, the number of instances of each role (web/worker) will add complications to the solution.
A detailed outline of your architecture would provide a better frame of reference for the answer you seek.
Worker role is designed for doing all sort of background activities. so you can move all of processing logic from windows application to workerrole. It is simply a class library with an entry point class.
You can not schedule any work automatically in worker role. You will have to right a logic for your self. worker role is executing a piece of code in an infinite loop. You will be charged for the no. of hours you have used in worker role (compute), it does not matter your worker role is idle or processing something. plus if you are using blob and queue, you will also be charged for accessing them and for storing data in them.
I am reaching a conflict in my azure project,
I'm trying to seclude task :
on every 16th of the month of .xml file form individual web site.
In my project I have one web role and one worker role,
I trying to seclude on every 16th of the month that the web role will create a message(connect to the site and download the .xml file), insert it to the queue storage, the worker role gets the message process it and delete the message.
Suggestion anyone??
Have you taken a look at this post - http://blog.smarx.com/posts/building-a-task-scheduler-in-windows-azure? It discusses one approach for building a task scheduler, which seems pretty much like what you're looking to do.
In your worker role you can start a timer or use this: http://quartznet.sourceforge.net/
An Azure web role is a windows server, and as such has a task scheduler built in, just like any other windows server. You can use a startup task to add a scheduled task to the task scheduler on a web or worker role which will then be fired on the 16th of every month.
However, keep in mind that if you have multiple web roles / worker roles, each one will have this scheduled task set up. So you'll need to have some way of knowing if another role has taken care of the work yet or not, or the scheduled task will run once for each web role.
I am considering moving my web application to Windows Azure for scalability purposes but I am wondering how best to partition my application.
I expect my scenario is typical and is as follows: my application allows users to upload raw data, this is processed and a report is generated. The user can then review their raw data and view their report.
So far I’m thinking a web role and a worker role. However, I understand that a VHD can be mounted to a single instance with read/write access so really both my web role and worker role need access to a common file store. So perhaps I need a web role and two separate worker roles, one worker role for the processing and the other for reading and writing to a file store. Is this a good approach?
I am having difficulty picturing the plumbing between the roles and concerned of the overhead caused by the communication between this partitioning so would welcome any input here.
Adding to Stuart's excellent answer: Blobs can store anything, with sizes up to 200GB. If you needed / wanted to persist an entire directory structure that's durable, you can mount a VHD with just a few lines of code. It's an NTFS volume that your app can interact with, just like any other drive.
In your case, a vhd doesn't fit well, because your web app would have to mount a vhd and be the sole writer to it. And if you have more than one web role instance (which you would if you wanted the SLA and wanted to scale), you could only have one writer. In this case, individual blobs fit MUCH better.
As Stuart stated, this is a very normal and common pattern. And again, with only a few lines of code, you can call upon the storage sdk to copy a file from blob storage to your instance's local disk. Then you can process the file using regular File IO operations. When your report is complete, another few lines of code lets you copy your report into a new blob (most likely in a well-known container that the web role knows to look in).
You can take this a step further and insert rows into an Azure table that are partitioned by customer, with row key identifying the individual uploaded file, and a 3rd field representing the URI to the completed report. This makes it trivial for the web app to display a customer's completed reports.
Blob storage is the easiest place to store files which lots of roles and role instances can then access - with none of them requiring special access.
The normal pattern suggested seems to be:
allow the raw files to be uploaded using instances of a web role
these web role instances return the HTTP call without doing processing - they store the raw files in blob storage, and add a "do this work message" to a queue.
the worker role instances pick up the message from the queue, read the raw blob, do the work, store the report result, then delete the message from the queue
all the web roles can then access the report when the user asks for it
That's the "normal pattern suggested" and you can see it implemented in things like the photo upload/thumbnail generation apps from the very first Azure PDC - its also used in this training course - follow through to the second page.
Of course, in practice you may need to build on this pattern depending on the size and type of data you are processing.
We have a WebRole which deals with request coming in off a WCF service, this validates then puts the messages into an Azure queue in Cloud storage. In the same application we have a WorkerRole which reads the information from the queue and makes a call to our persistence layer which processes the request and returns the result. We were wondering why the worker roles didn't pick up any of our configuration settings and hence was not providing the Trace information we were looking for. We realised that the worker role was likely looking for an app.config and couldn't find it.
Is there a way to point the worker role to the web config, or to be able to load our Enterprise Library settings into the ServiceConfiguration.cscfg file which in either case would mean both could read from a common place?
Many thanks in advance
Kindo
As far as I'm aware there is no way for your worker role to get access to the web config of a web role out of the box.
If you move your configuration items to the ServiceConfiguration.csfg file and both the worker and web role are in the same cloud project, the settings will be in the same file. But because the web role and the worker role are different projects within that cloud project, their settings are in different sections of that .csfg file. If you want the settings to be the same for both of them, you will have to duplicate the settings.
Putting your settings in this file gives you the advantage that you can change the settings while the roles are running and have the roles respond however you like e.g. you might want certain settings to restart the roles, for others you may just want to update a static variable. In order to update a web.config or app.config you need to redeploy that role.
You do need to be aware though that the ServiceConfiguration file is not a replacement for a webconfig. If you're using tools that look for their settings in a web or app config, unless they're particularly smart and aware of the Azure environment, they won't go looking for settings in the ServiceConfiguration file.
I know you didn't ask this question, but if you're expecting your worker role to be providing an almost synchronous response to your web role by using a request queue and a response queue, you'll probably find that this won't scale very well. If you want to do synchronous calls to a worker role in Azure you're best off using WCF (this isn't highlighted anywhere in the guides). As you said that all of this is just a WCF service anyway, the answer to your problem may be to just do away with the worker role.
You cannot share web.config's OR .cscfg's across Roles in Azure, as there is no guarantee a role is in the same host, cluster, or even datacenter as another role.
If you are simply trying to share items like connection Strings, app-specific variables, etc., I would simply create your own "config manager" class that would obtain some XML and parse it into a collection of settings. This way, you could store that config on Azure Blob Storage and changes would be as simply as updating that blob and signaling your apps to reload. (very easy using the Service Management API).