A number of years ago I implemented an asynchronous peer-to-peer Message-Oriented-Middle-ware that was very friendly to use in Excel VBA, and I find myself again needing to do lots of calculations which could be trivially distributed, if I had the mechanism.
I could re-implement the MOM layer, but I'd prefer to use a third party product if one existed.
My requirements are these:
to be able to send messages easily from Excel VBA and VB6,
to have a resource discovery mechanism to find the calculation services,
to have asynchronous message sends (I don't want to lock up Excel while the calculation is being done),
to provide a queuing mechanism so I can have multiple servers doing the work easily,
to have low admin for setup
Can anyone suggest anything?
Many thx
-- DM
The next release of Microsoft Windows Server 2008 HPC edition includes something called "Excel Runner" which is specifically designed to deal with the kind of problem you are describing.
RESTFul Web Services are easy to use from Excel VBA. You can use a reference to MSXML2 or WinHttp - Sounds like you want to use in Async mode so as not to block the Excel UI.
Start with something simple:
Simple discovery (scan your subnetfor worker nodes).
Divide your task up.
Share out tasks.
Wait or poll for results.
Update cells with new results.
If any sub task takes too long round robin again.
Nice to have visual indication on cells pending results.
These links may help you.
Peer-to-Peer Programming with WCF and .NET Framework 3.5
Peer-to-Peer Programming
MPAPI - Parallel and Distributed Applications Framework
Related
I am writing payroll management web application in nodejs for my organisation. In many cases application shall involve cpu intensive mathematical calculation for calculating the figures and that too with many users trying to do this simulatenously.
If i plainly write the logic (setting aside the fact that i already did my best from algorithm and data structure point of view to contain the complexity) it will run synchronously blocking the event loop and make request, response slow.
How to resolve this scenario? What are the possible options to do this asynchronously? I also want to mention that this calculation stuff can be let to run in the background and later i can choose to tell user via notification about the status. I have searched for the solution all over this places and i found some solutions but only in theory & i haven't tested them all by implementing. Mentioning below:
Clustering the node server
Use worker threads
Use an alternate server and do some load balancing.
Use a message queue and couple it with worker thread to do backgound tasks.
Can someone suggest me some tried and battle tested advice on this scenario? and also some tutorial links associated with that.
You might wanna try web workers,easy to use and documented.
https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
We have an oracle forms application, and one of the many thoughts (considered converting to non-oracle-form technology) was to use JHeadStart (oracle product) that converts the oracle forms to ADF application. But we would like to not use ADF, so is there any way that we can remove the dependency on ADF?
If anyone feels this is not the question to ask, instead of giving me -ve marks please comment me and I will remove this question.
Thanks.
As always, it depends on what you want to achieve. I don't know JHeadStart, but to me, it sounds like a tool converting a legacy application to a framework that might be considered legacy soon. There are a few supporters of ADF, so I believe it's a good thing if you're ready to live with the compromises a full-stack framework brings. But in general, ADF is not popular among JSF developers (mostly because of those compromises, which often turn out to be too restrictive). Even more generally speaking, JSF is not popular among UI developers. That, in turn, is a bit unfair, but I observe a huge movement to pure JavaScript UI frameworks.
This indicates that using a tool like JHeadStart isn't the most future-proof approach. It's (probably) good to survive the next month, but in the long run, it'll probably backfire.
Let's have a look at the question from another angle. Why do you want to get rid of Oracle forms? Most likely, it's because of recruiting problems, but it might also have something to do with architecture. Oracle Forms supports a programming style integrating the database layer tightly with the UI layer. That's a very efficient way to write small applications, but it scales badly if your application grows.
So I'd recommend spending some extra money and time to re-implement your application from scratch. Automated tools tend to generate code that's hard to maintain. Re-designing your application from scratch gives you the opportunity to build an application that lasts a decade.
Oh, and I don't think it's possible to use JHeadStart without introducing ADF. Simply because JHeadStart has been designed with ADF in mind.
I have a Node.js app with a small set of users that is currently architected with a single web process. I'm thinking about adding an after save trigger that will get called when a record is added to one of my tables. When that after save trigger is executed, I want to perform a large number of IO operations to external APIs. The number of IO operations depends on the number of elements in an array column on the record. Thus, I could be performing a large number of asynchronous operations after each record is saved in this particular table.
I thought about moving this work to a background job as suggested in Worker Dynos, Background Jobs and Queueing. The article gives as a rule of thumb that tasks that take longer than 500 ms be moved to background job. However, after working through the example using RabbitMQ (Asynchronous Web-Worker Model Using RabbitMQ in Node), I'm not convinced that it's worth the time to set everything up.
So, my questions are:
For an app with a limited amount of concurrent users, is it ok to leave a long-running function in a web process?
If I eventually decide to send this work to a background job it doesn't seem like it would be that hard to change my after save trigger. Am I missing something?
Is there a way to do this that is easier than implementing a message queue?
For an app with a limited amount of concurrent users, is it ok to leave a long-running function in a web process?
this is more a question of preference, than anything.
in general i say no - it's not ok... but that's based on experience in building rabbitmq services that run in heroku workers, and not seeing this as a difficult thing to do.
with a little practice, you may find that this is the simpler solution, as I have (it allows simpler code, and more robust code, as it splits the web away from the background processor - allowing each to run without knowing about each other directly)
If I eventually decide to send this work to a background job it doesn't seem like it would be that hard to change my after save trigger. Am I missing something?
are you missing something? not really
as long as you write your current in-the-web-process code in a well structured and modular fashion, moving it to a background process is not usually a big deal
most of the panic that people get from having to move code into the background, comes from having their code tightly coupled to the HTTP request / response process (i know from personal experience how painful it can be)
Is there a way to do this that is easier than implementing a message queue?
there are many options for distributed computing and background processing. i personally like RabbitMQ and the messaging patterns that it uses.
i would suggest giving it a try and seeing if it's something that can work well for you.
other options include redis with pub/sub libraries on top of it, using direct HTTP API calls to another web server, or just using a timer in your background process to check database tables on a given frequency and having the code run based on the data it finds.
p.s. you may find my RabbitMQ For Developers course of interest, if you are wanting to dig deeper into RMQ w/ node: http://rabbitmq4devs.com
I have a telephony scenario in which the following happens:
Customer calls a Voice Gateway
TCL script runs and a code is taken from customer
Authentication is done through a RADIUS server
Customer will hear correct voice menu
The problem is that RADIUS server must connect to a SQL Database and check the credentials. I have currently designed the solution using cisco secure ACS and through managed stored procedures on MS SQL server.
My question is: Is the VoiceXML a better tool to do this job and because some extenstions and wrappers of VoiceXML exists in .net, does it fit in this simple scenario??
Sincerely speaking, I am a little confisued with the technology and looking for a good tutorial on its features as well.
Thanks
In a strict sense, only step 4 is implemented by VoiceXML. Other aspects are handled by the platform or external code. VoiceXML is the standards mechanism for implementing step 4, but if all you are going to do is limited audio output and simple input, it may be overkill depending on the solutions available to you.
The following is just an example of a way to solve your problem and is fairly fictitious given I don't know anything about your environment nor constraints.
Given most VoiceXML platforms, upon receiving of a call your VoiceXML application will be executed. If this is a servlet/ASP based solution, you can perform steps 2 & 3 then generate/return the VoiceXML to play the menu, gather the input and move to the next step. If this is a static VoiceXML 2.1 solution, you can use a Data element call to make an HTTP request to a system that can perform these actions. The system will need to return XML that the Javascript/ECMAScript in VoiceXML application can parse and provide the correct audio output and input processing.
Since you are asking about VoiceXML, I'm assuming your challenge is the telephony aspect of the problem. Unless you have a system already available, choosing and activating a premise or hosted solution is far more complicated than the call flow code involved. Depending on your requirements, there are solutions as low as a single line, analog modem that supports audio output and DTMF input to massively scaled on premise and hosted solutions to handle 10,000s of concurrent calls that implement VoiceXML as well as a wide range of other call flow technologies.
VoiceXML would work fine in this scenario. There is a an open source project called VoiceModel that uses ASP.NET MVC to generate the VoiceXML and therefore integrates nicely with the .NET stack. There are a lot of examples in the project with discussions on how to use the examples in this blog. The examples use Voxeo Prophecy as the VoiceXML platform which has a SIP interface that will connect with a Voice Gateway. You can download two ports for free to try it out.
While multithreading is faster in some cases, sometimes we just want to spawn multiple worker processes to do work. This has the benefits of not crashing the main app if one of the worker crashes, and that the user doesn't need to worry a lot about inter-locking stuffs.
COM+'s Application Pooling seems like a good way to achieve this on Windows. The downside is that we need to write a COM+ wrapper for the worker process.
However, when I search for Application Pooling on Google, it seems like most of its usages are related to IIS. Don't other applications (such as scientific/graphics) find it useful to spawn multiple worker processes?
So there are several questions:
Why isn't COM+ more popular in areas other than IIS? If I write a non-IIS application and want to use process management on Windows, should I go with COM+ or are there better alternatives out there?
What would be the cross platform way to do it? Are there libraries out there that give me a "process pool" (worker processes will intelligently pick up work, can be managed, etc.)
I can't offer any answers to the COM aspect of your question, but it's worth noting there's another world (besides HPC MPI) where multi-processing (rather than the more common multi-threading approach) is apparently alive, well and thriving: Python.
Why ? Python's GIL ("global interpreter lock") cripples most attempts to multithread python code so badly that multiprocessing is the generally recommended approach to parallelising Python on SMP. The standard library includes process pools; there are various other options too.
Python certainly ought to satisfy any multi-platform requirement!
You might want to investigate how the apache web server manages process pools. From version 2.0 it runs natively on windows and one of the multi-processing models it supports are process pools. A part of apache is also APR (apache portable runtime), which handles platform-specific issues.
No one can answer why something is not popular because may be no body is looking for what you are looking for. After .NET came in picture, people shifted from COM to Managed Environment, before .NET, COM and ATL and relative other technologies were quite painful to implement and they would crash and were also quite difficult to debug.
That is the reason, managed environment came in existence.
However, .NET 4 onwards, parallel libraries give much more power to user for parallel programming and also you can spawn and control other proceeses.
For multiplatform, you can look for zvrba's answer.
Yes, other applications--especially science applications--find it useful to spawn multiple processes. Since few super-computers run Microsoft Windows, scientists generally avoid using anything that ties them to a Microsoft platform. Nothing related to COM will help scientists leverage their enormous existing code base written in Fortran.
People who choose to run IIS have generally already drunk the Microsoft Koolaid, so they have fewer inhibitions to tying themselves to Microsoft's proprietary platforms, which is why COM-specific terminology will get lots of hits related to IIS.
One of the open standards for doing what you want is the Message Passing Interface. Several implementations exist and some of them run on supercomputers using Fortran. Some of them run on cheaper computers using sexier languages.
See http://en.wikipedia.org/wiki/Message_Passing_Interface
There hasn't been a mob rushing through the doors of COM application pooling primarily because of two factors:
COM is a pain in the ass to deal with compared to just about anything else
Threading can be a headache, but it's a lot easier and more convenient to manage than inter-process communication
COM application pooling was essentially created for IIS. It has one very specific benefit over normal multithreading: the multiple processes are fully isolated from each other. This is important for data security and for app stability when dealing with third party plugins of questionable stability.
Scientific computing generally doesn't need strong data security isolation between operations, and I would venture to guess that scientific computing doesn't rely much on third party plugins of questionable stability. When doing big math operations, you're either using a sexy numerics library that had better be rock solid to be taken seriously, or you're using your own code, in which case crashes should be fixed and repeat offenders should be spanked.
Oh, and all crashes except stack overflow can be trapped and dealt with within a multithreaded app, especially if it's your own code.
In short, COM app pooling is overkill for just about anything other than IIS.
Google's webbrowser chrome is a multi-process architecture software. It is open source, so you can check out its code and see how to manage processes.