I need to implement scheduled tasks, so that every X time the job will start running and will start an .exe file.
I did this those tasks in Azure very easily, but can't find something appropriate in Amazon Web Services.
Can you tell me if there is something similar in AWS for Azure WebJobs?
The most similar piece of AWS services that fits your needs is AWS Lambda. But as your comment states you do not want to code.
When comparing AWS to other cloud services it pops out that AWS focus on a very primitive services that can be connect and build complex systems. This is an advantage as one can tailor the cloud to its needs. However it can be more complex to setup when compared to a PaaS.
Related
My website has a free "live test" tool. In order to reduce serverless costs and avoid abuse, I'd like to run it on one separate machine. If users spam the machine, it'll become slower, but it won't ruin me.
Is there a tool in Azure or AWS that would allow this? To have an Azure function 24/7 running on a low-cost server -- without any scaling and so on.
Is there a tool in Azure or AWS that would allow this? To have an
Azure function 24/7 running on a low-cost server -- without any
scaling and so on.
Yes, both providers have a pay per consumption model that you could use Azure Functions / AWS Lambda. However, you pay based on resource consumption * time spent to process each request. It means you can easily suffer Denial of Wallet
As an alternative, you can spin up a very cheap Virtual Machine and route requests to it.
Please help me understand why we say azure functions is a serverless compute service. It does require cloud to host it and run. Cloud is also a server still why we are saying it is serverless?
Serverless computing does not mean that servers are out of the picture. Servers are very much required, just like they have been for all these years, or else, where will your code run. The reason why the phrase was coined is that as a developer, you do not need to worry about what server your code runs on. In fact, you do not know which server it eventually runs on. Once your code is deployed, Azure assigns the responsibility of executing the code to the next available server. What Azure ensures, and what is ultimately important for you, is that your code will execute whenever required.
Ref: Serverless Computing with Azure Functions
Hope it makes sense :)
To get a better idea this is how we evolved. Cloud providers are making sure we should only worry about the business logic but nothing else.
IaaS (Infrastructure as a service)
You get a running VM somewhere in the data centre but you are required to maintain everything. From Deployment to patching your VMs or anything running on the VM.
PaaS(Platform as a service)
You are not longer required to maintain platForm but you are still responsible to manage your server in terms of load balancing etc.
FaaS(Function as a service)
Servers are abstracted from you . You are only required to maintain your code without worrying about what's under the hood or how to load balance your servers. It's then cloud provider responsibility to package your code and run it for you. But servers are still there.
Going by the official documentation of Azure Serverless computing service, Azure Functions can be defined as;
Azure Functions is a serverless compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure. Use Azure Functions to run a script or piece of code in response to a variety of events.
Azure Functions is an event driven, compute-on-demand experience that
extends the existing Azure application platform with capabilities to
implement code triggered by events occurring in virtually any Azure or
3rd party service as well as on-premises systems. Azure Functions
allows developers to take action by connecting to data sources or
messaging solutions, thus making it easy to process and react to
events. Azure Functions scale based on demand and you pay only for the
resources you consume.
Here the serverless compute service is like a metaphor which implies, the end user doesn't need to manage the servers or infrastructure to run the applications over the Azure and can spend time to focus on managing and improving the business logic.
Few more points to consider,
Serverless in Azure builds on an open-source foundation, the core of
which is Azure Functions, an event-driven compute experience and open
source project. Community contributions include support for new
languages, integrations and deployment targets.
Azure Functions can be used on-premises, in hybrid environments such as Azure Stack, on IoT Edge devices and deployed on top of orchestrators such as Kubernetes – as well as in other clouds.
They enable faster time to market with lower infrastructure and operating costs.
There are heaps of definition of serverless which you can easily google. But I will share my understanding anyways.
1. It does require cloud to host it and run.
You are correct with this. But anything on Cloud requires Cloud, doesn't it? Azure being one of the cloud providers consists of hundreds of services to cater to different needs people are after from using Cloud.
2. Cloud is also a server still why we are saying is serverless
This is not quite right. Cloud is different from a server. Server is a physical box sitting somewhere. With hundreds of thousands of servers all over the world, Cloud hosts all sorts of different services on these servers.
The reason we say Functions are serverless is that the infrastructure of hosting a Function is abstracted away from devs. It is still deployed to some servers but Azure is responsible for all the resource managing, configuration, load balancing, scaling and networking etc. It allows developers to focus primarily on their code, not having to worry about servers.
We currently have a window service which send some notification emails to users after doing some processing on database(SQL database). Runs once in day.
We want to move this on azure cloud. One alternate is to put it on Azure VM as is. but I am finding some other best possible solution for that.
I study about recurring and on demand Web jobs but I am not sure is this is best solution.
Also is there any possibility to update configuration of service code in App.config without re-deploy the code of service on cloud. I means we can manage configuration from Azure portal.
Thanks in advance.
Update 11/4/2016
Since this was written, there are 2 additional features available in Azure that are both excellent choices depending on what functionality you need:
Azure Functions (which was based on the WebJobs described below): Serverless code that can be trigger/invoked in various ways, and has scaling support.
Azure Service Fabric: Microservice platform, with support for actor model, stateful and stateless services.
You've got 3 basic options:
Windows service running on VM
WebJob
Cloud service
There's a lot of information out there on the tradeoffs between these choices, but here's a brief summary.
VM - Advantages: you can move your service basically as it is without having to change much or any of your code. They also have the easiest connectivity with other resources in Azure (blob storage, virtual networks, etc). The disadvantage is you're giving up all the of PaaS advantages and are still stuck managing your own VM infrastructure
WebJob - Advantages: Multiple invocation options (queues, blobs, manually, queue receive loops, continuous while-loop style, etc), scheduled (would cover your case). Easy to deploy (can go with website, as a console app, automatically through Kudu), has some built in logging in Azure portal - and yes, to answer your question, you can alter the configuration in the portal itself for connection strings and app settings.
Disadvantages - you'll need to update code, you don't have access to underlying resources (if you need that), and more of something to keep in mind than a disadvantage - it uses the same resources as the webapp it's deployed with.
Web Jobs are the newest of the options, but at the same time appear to have active development going on to increase the functionality and usefulness.
Cloud Service - like a managed VM, has some deployment options, access to underlying VM if needed. Would require some code changes from your existing service.
There's nothing you've mentioned in your use case that makes me think a Web Job shouldn't be first thing you try.
(Edit: Troy Hunt has a great and relatively recent blog post illustrating most of the points I've mentioned about Web Jobs above: http://www.troyhunt.com/2015/01/azure-webjobs-are-awesome-and-you.html)
Is there any technology like CloudFormation for aws that would work on any IaaS based cloud to do the same thing? I mean you write it once and then it runs on any IaaS based cloud platform like azure, aws, openstack, and so on?
I think what you are looking for is exactly OpenStack Heat Project http://wiki.openstack.org/Heat The project is under active development.
Your question is making some assumptions that isn't quite correct - there's a lot of different types of clouds - Google AppEngine, OpenShift, Heroku and Azure (for example) work at a very different layer than the cloud services you get from Amazon AWS, CloudStack, OpenStack, etc.
The first is more of an platform as a service (PaaS) play, the later are more infrastructure as a service (IaaS). That said, and given that you're asking about Amazon's CloudFormation, it's worth mentioning that there's an equivalent of that technology for OpenStack clouds called HEAT (although I'm not aware of one that works with CloudStack).
To answer you're higher level of question of "is there a way to write it once and run it on any cloud?", today the answer is a qualified "no". There's lots of APIs and helper libraries you can use to bring several cloud provider options more into compatibility, and some companies are making all their pennies of exactly this game (enStratus, for example). But none of them go so far as to allow you to "run it anywhere" - especially when you lump IaaS style clouds in with PaaS style clouds.
What you are describing is a Platform-as-a-Service (PaaS) that works on multiple clouds. There is a similar question here: Cloud Mangement for Amazon IaaS
The answers also link to a comparison of PaaS options.
Check out enstratus. It's a UI for interfacing with multiple Cloud APIs. So you can store and shift images between different clouds then spin them up and down. Might be what you want.
I've got my head around creating cloud instances in AWS, Azure and Rackspace. However, I need to turn my instances off at the end of the day and on in the morning as this will half my hosting cost (they are for development).
I've looked at a few management services but they blew my brains out. Is there a simple way to do this?
Azure
REST:
You can do this to Azure deployments by using the Windows Azure Service Management REST API. Because it is REST you can use most programming languages to access it.
You could have an application running on your local machine that schedules calls to these services to delete at a certain time at the end of office hours and then create your service again in the morning.
PowerShell:
Or you can manage your deployments in the same way but instead of using REST you can use Azure PowerShell cmdlets. I have done this way myself and it works nicely.
To help you get started there is a nice tutorial on how to do use PowerShell to deploy Azure applications.
also if you didn't already know I should also mention there is a 3month free trial with Azure if you are simply looking for cutting costs whilst developing.
Approach
You could always roll your own solution, insofar most cloud providers offer a respective API to start/stop instances on demand (or even on schedule), which is what those management services are actually using as well of course - the AmazonEC2 Java interface offers all relevant methods for example (amongst many others), specifically:
StartInstances()
StopInstances()
RebootInstances()
Via Scripting (EC2)
The most simple approach for this regarding Amazon EC2 would be to craft yourself some Python scripts by means of the excellent boto (An integrated interface to current and future infrastructural services offered by Amazon Web Services), which exposes all EC2 methods mentioned above; you could then start those scripts on demand or via your operating system scheduler.
Via Continuous Integration / Automation (EC2)
Another option would be to facilitate a continuous integration server as an automation engine (a sometimes overlooked aspect of these systems), in case you happen to run one anyway; it would allow you to both start/stop instances on demand or scheduled similar to cron.
We do exactly this by means of the Bamboo AWS Plugin (it's Open Source and the code is available on Bitbucket), see my answer to How to start and stop an Amazon EC2 instance programmatically in java for more details on this approach. While Atlassian Bamboo is a commercial offering, there should be something similar available for popular Open Source CI solutions like e.g. Jenkins as well.
NOTE: As for June of 2013, IaaS Instances can be placed in a "stopped (deallocated)" state. In this state you are only billed for storage of any disks associated with the VM. The original answer below describes a VM instance that is in a "stopped" but not deallocated state. The deallocated state is currently the default for VM stop actions taken via the Azure management portal.
The only way to accomplish this in Widows Azure today is to delete the deployment.
If you stop the service, you are still billed (like renting office space, you pay for it even if you aren't in it), and you can't set the instance count to zero. An option may use is to just reduce the instance count to absolute minimum (1) an then scale it back up during needed hours. But the cost benefits of this will depend on the size of your instances.
Old thread I know, but Microsoft introduced 'Runbooks' for Azure in 2014 that you can use for automation, including scheduled startups and shutdowns. As mentioned above, be sure you are in stopped (deallocated) state, as opposed to just stopped, in order to prevent charges.
More info:
Script to stop your VMs
Azure automation, official MS docs.
Yes Automation Runbook are there by which we can schedule the job. I created the script for stopping (De-allocated) Azure VM.
https://gallery.technet.microsoft.com/Deallocate-all-VM-under-79049c69
Please read about how to use runbook http://azure.microsoft.com/blog/2014/06/19/azure-automation-runbook-management/
Dellocation and stop are different, since stop vm will also incur cost.
The best article on automation + switching on/off VMs I have found so far. [05 February 2015]. http://clemmblog.azurewebsites.net/using-azure-automation-start-und-stop-virtual-machines-schedule/
Recommended solution for AWS:
The AWS Data Pipeline is uniquely suited to this task. Data Pipeline
uses AWS technologies and can be configured to run AWS CLI commands on
a set schedule with no external dependencies. Data Pipeline can write
logs to S3 and runs in the context of an IAM role, which eliminates
key management requirements. Data Pipeline is also cost effective; for
example, the Data Pipeline free tier can be used to stop and start
instances once per day.
https://aws.amazon.com/premiumsupport/knowledge-center/stop-start-ec2-instances/
Refer to this article, there some options to turn your instances on/off inside AWS.
AWS Datapipeline
AWS Lambda scheduled events
Scheduled Cron on EC2 instance
Scheduled Scaling of Auto Scaling Group
So in your case I'd recommend the followings:
For AWS:
Through Shell Command like AWS CLI commands: See Turn on/off
Cloud instances using AWS Pipeline. this method will initiate a
separate EC2 instance to be started and terminated for each AWS API
call that running times affect to your Bill.
Through programming languages like Node.js / Python: See Turn
on/off Cloud instances using AWS Lambda. The task running twice a
day for typically less than 3 seconds with memory consumption up to
128MB typically costs less than $0.0004 USD/month
For Azure and Rackspace (or other platforms you may have):
Use the above tools to provide a Respective API to start/stop instances on demand.
You may also consider to set scripts-per-boot which runs each time your instance is started.
AWS
AWS SDK is your best bet but I am using TotalCloud.io to start and stop instances under the free tier. Very customizable.
Easy to setup.