For fun and maybe for profit, i want to implement the following:
scheduled or manually triggered process that logs into all my bank accounts
process knows bank site structure, and goes through "cashback/partner deals" pages
all deal information is collected in one place and in one format
when i'm going to buy something, i can quickly see if any of my cards has a special offer for that place. so that i can pay with the card that offers the best deal. ideally, this should happen on my android phone. that doesn't imply a standalone app though, e.g. you can search inside an email or googledoc or anything.
any ideas on implementation? don't limit yourself. suggest crazy things, as long as they work.
PS i did look for an existing website that offers something similar, but they all seem to focus on cards themselves rather than on specific deals for stores.
Depending on the bank website, your code may need to be able to execute JavaScript to interact with it. Take a look at CasperJS for the web scraping part.
I will soon be developing an application to log and priorities worker shifts. It only needs to be small, and simple, but I am wondering what framework to use. All that needs to happen is boxes with names are in 3 lists, and the user can switch these around at will. It must log the times, which I will need access to in real time.
Im new to application development of this sort, and would like any suggestions. I have time to learn new technologies / languages.
Portability / device independence would also be useful. Should I consider a Web Application in Javascript? or something more like Python.
Thanks.
Even if your application is going to be simple that does not mean that whole system will be simple too. I can imagine that your app will serve only as a front end to something much bigger. If that is a case and you really have freedom to choose what language you will develop your app with consider choosing something that you will feel comfortable to work with. But before you will make your decision I would go to whoever gave you that task and try to get as much information about it as you can because expected features can help in choosing technology.
First of all, it seems that it is up to you to decide if it should be web or desktop app. In my opinion it is completely wrong situation. You should get clear requirements on what kind of application customer expects and in what environment it should work. And I would not move a finger towards a code before somebody gave me that information. You write that portability and device independence would be useful but is it actual requirement or you just think it would be nice feature to deliver?
EDITED (to answer a question in comment)
Probably there is as much possible solutions as people that would answer you so in the end you will have to make your own choices.
One way of doing it (considering that you want to learn something new and have a challenge :) would be implement WCF service that would act as a data provider from your database (so every GetUsers(), GetVacationDays() methods would be in it) and it would take care of some business logic (for example CalculateMaxValidWorkingTime() or whatever). That service would be a real power horse of your system. Since you don't have clear requirement whether it should be desktop or web app you could satisfy both possibilities by creating thin clients that would communicate with your service and they would be just a pretty front ends. And if you keep in mind that you can consume webservices practically with everything from C++ to .NET (C#/VB) to Javascript to Python to PHP after you done with service you would not be constrained with one particular technology/language.
Regarding databases I won't advice because personally I hate dealing with them and it always was somebody else's task to provide me with pretty API :)
I'm sure many of you are familiar with the IBM i5 series emulator (looks like this poop)
My company uses this religiously and there is no Biz logic in it so anytime somone in our finance dpt makes a human error it accepts it and adds it to the database. Not to mention its ugly, hard to use, not intuitive, etc....
I would like to create a frontend for this interface so that we can control the logic before its submitted to the system (we dont control the system itself) so in effect I need to make my own emulator app.
However I cant seem to find any information on how to interface with the i series, namely login, send commands, and view or gather data from the screens it would normally send back.
Any suggestions?
The problem is not the iSeries but the software package your company is running on it.
There ARE advantages to use green screens: it's fast and it's almost unbeatable at data entry, provided you get used to it.
But to answer your question, the iSeries is a J2EE enabled machine: a HTTP server comes installed and depending of the version of the iSeries, WebSphere might be already installed, or are entitled to install it. Then you can use JT400, which is the java toolkit for the os400 containing the jdbc drivers to connect the database and the necessary classes for calling programs.
If you prefer php, there is a flavor of the Zend framework made to work on the iSeries but I never tried it.
I'd recommend that you take a look at both the Attachmate Verastream Host Integrator (VHI) and IBM's Host Access Transformation Services (HATS) products. They effectively just screen scrape the green screen terminals to allow you to pull and push data and provide macro recording and editing tools to automate the process. App integration can be achieved via web services or html/jsp/servlet programming (plus .Net for VHI and EJB's for HATS). They do come with enterprise pricing however which may be an obstacle for some. They do have free trial offerings for evaluation purposes to help determine if they are an appropriate solution to your problem.
What software packages are they using? Most programs that I use in the 5250 emulator has some business logic that error checks the data before adding it to the database. Can you get us some more information so we can direct you in a better direction.
There are vendors that sell products that screen-scrape the 5250 data stream and produces a web front-end. Or you can write your own front-end in the language of your choice and just do SQL calls to the database.
THere's got to be some source code. Start by looking at the menu and menu option your users are accessing and figure that's running behind them.
Use command STRPDM to look for source code - look in different libraries (they are like folders)
You might have source code in a "member" called something like xxxMNUSRC xxxRPGSRC (rpg program source) or xxxCLSRC (cl programs), xxxDDSSRC (display/screen source, physical/logical file source)
Objects a "compiled" objects such as files (tables), screens, priter files (reports)
Stay away from Qxxx and #xxx libraries - those are system libraries.
http://systeminetwork.com/ is a good resource for iSeries related questions.
How can I communicate in a very simple and effective way the path the user takes through my application?
I'm currently working in a Ruby and Rails environment, so I tend to visualize my application in terms of RESTful URIs. So for example, if I want my users to sign up, I could match a new route called /users/new. The thing is, I would like to see beyond that specific action, and visualize how many pages or forms does it take to create an account and some business logic associated with the process in general. In other words, I'd like to see a mix of a workflow diagram and some implementation details (at an interface level).
I was thinking in showing mockup pictures wrapped in boxes, and relate them through arrows with their corresponding GET, POST, PUT, DELETE methods and URIs attached to them. I think it is a good idea, but I haven't seen examples yet that inspire me.
In your experience, what helps you see the big picture? Balsamiq mockups allow to define links and navigate through the app, but it doesn't help to conceptualize.
Have you thought of using a mind-map? You could try the free FreeMind
If you stick with UML, you could consider an Activity diagram.
I think you're on the right path. Showing different screens with possible combinations of users' transactions between them is a good technique. Basically you would be able to show user's flow through your application and stress out decisions a user will make on the way.
The good example for it was presented here http://vimeo.com/43869717
This technique called Storyboarding. You should be able to find some examples. But the one I mentioned above is one of the best Storyboarding techniques. I use it all the time to show the big picture and present application workflow from user perspective to my team.
Is there a way of offering the flexibility of Excel/Access development that end users love while instilling centralised IT management so data and logic is secure, backed up, version controlled etc. The common options are to re-write in C#/ASP.Net/Java/Python/Your Choice, but that takes away control from the users. Is there a better way, and what do you do at your site?
There is a universal issue of users creating fantastically useful Excel/Access mini-apps that the IT department would like to bring under control. Users love the flexibility that Excel affords, especially on the fly changes, graphing and data import/export. In Access we have brilliant QBE. The downside is that after a short while there are legions of out of control spreadsheets/mdbs which are mission critical, with lots poorly understood business logic, and brittle code, they're a pain to support especially as staff move on.
This puts the IT dept in an awkward spot, they'd like to support these apps, but don't know enough about them. This is made more difficult as they are typically insecure with zero documentation.
Having been of both sides of the fence I would go after the root cause of the problem. Why do uses make their own little apps? Because it is too hard/expensive/time consuming/never turns out right when they go through the “proper” channels.
The other thing is they tend to know the business very well so whilst their coding might not be very good their knowledge of what needs doing is very good.
So what can we do to combat this problem? I personally think their should be a small team of people within IT whose job (or one of their jobs) is to develop these small applications. They should work very closely with the end users and not be locked in the ivory tower of IT.
In my current role I’m on the non-IT side of the fence, I have a few quite major applications that needed to be developed so I asked for an install of visual studio and some space on an SQL server. I had my request denied. So I just asked for SQL server space, again request denied (each request taking about a week to go through) So in the end I’m “stuck” in access.
Now these are very nice access apps with version control, comments in the (shock!) and all the other nice things but at the end of the day I was trying to do things the “right” way and ended up being forced down the access route. So when my apps try to get scaled up and I’m quoting a long time for a rewrite who is to blame?
Have you considered looking at SharePoint for department-level applications? Many professional developers will balk at the idea of using Sharepoint for "application development," but it truthfully can be a great way for "power users" to start putting their data and tools in a managed framework.
With SharePoint, you can manage the overall structure of the site and then set up users with elevated permissions within their respective departments. There are some great 3rd-party tools to help with keeping an eye on what's going on in your SharePoint site.
SharePoint is not a silver bullet by any means, but it is great for many multi-user applicatinos that need to keep up with a list of data.
(The following is not really related to my above answer, but your question really hit home and I thought I'd share my similar experiences and insights.)
Our company will be going through a similar process in the near future. I'm on the "end user" side of things and can sympathize with a lot of what Kevin Ross said. Sometimes Access and Excel are simply the best tools available for me to get the job done.
Here's an example: I was asked several years ago to come up with a system for creating Purchase Orders to a vendor in China for product for which there is a 3 month lead time. Our ERP software had a few features for procurement, but nothing that even came close to the complexity of the situation we were facing. Years later, after going through several iterations of the application in Excel (VLOOKUP was a lifesaver), Access ("So that is why people using relational databases. Awesome!), and back in Excel ("let's not make this so complicated"), I still find that these Micorosft Office apps are the best tools to get the job done.
What's the cost to not use these tools to get the job done?
Contract work to our ERP vendor to add a special feature for this ordering process: are you kidding me? We'd likely pay tens of thousands of dollars for an unflexible monolithic application with horrendous user experience...and we would still end up back in Excel.
Buy third party software designed for this exact process: I've seen an on-site demo of software that does exactly what I want for our procurement process. It starts at $100,000. There are probably other tools that we can get for a few thousand dollars, but at that price point, I've already emulated most of their features in my own application.
Try to finish the job "by hand." : Ha! I'm a programmer at heart, which means I'm lazy. If it takes a solid week of sitting at a desk to work up a purchase order (it actually did take this long), you can bet I'm going to work up a solution so that it only takes me a few hours (and now it does). Perhaps the guy after me will go back to doing most of it by hand, but I'll use the tools in my toolbox to save myself time and stress.
It's so hard to find the perfect application to allow for maximum creativity on the user end but still allow IT to "manage" it. Once you think you've found a solution for one thing, you realize it doesn't do something else. Can I write I printable report in this solution like I used to do in Access? Can I write complicated Excel formulas that tie multiple data sources together from different sheets ("You want me to learn what? No, I've never heard of a "SQuirreL query" before. VLOOKUP is just fine thankyouvermuch)? Can I e-mail the results to the people in my department? Can it automatically pull data from our back-end database like I do in Excel and Access? Can I write my own code, VBA or otherwise, to make my job easier? The list goes on.
In the end, the best advice I can give to any IT manager in your situation is to respect the other workers at your company. Let them know their work is important (even if it's only useful to them and the guy at the next desk over). Let them know you are not trying to make their job harder. Don't assume they are morons for creating mission-critical applications in office productivity software; they are just trying to get the job done with the tools at hand and are usually quite capable and intelligent people. Invite them to explore different solutions with you instead of just removing the tools they currently have in their toolbox and then replacing them with ones they don't know how to use.
At the end of the day, if you have users who are smart enough to shoot themselves in the foot by creating complicated apps in Excel and Access, they are probably smart enough to learn to use the appropriate tools to accomplish the same tasks. Invest the time and energy to involve them in the process and you will have a solution that works for everyone at the end.
You could try a hybrid approach: Allow your users to use Excel/Access to home-brew their own, specialized tools, but take the mission-critical stuff and put it under IT control. There are a few strategies that could help you with this:
Make sure that your IT department is firm on VBA. Not the "yeah-everybody-can-write-a-few-lines-of-basic" type of knowledge, but in-depth training, just like you would if it were a less simple programming language. Although "real programmers" will tell you otherwise, it is possible to write large, stable applications in VBA.
If you currently have the data in Access databases, move away from that and migrate it to an SQL Server. This allows you to do centralized backup and management, while still giving your power users the flexibility to "link" these SQL Server tables to their Access frontend.
Commonly used business logic should be under control of your IT department. This can be done either with VBA, by creating an Access library that is linked by your users, or in any of the .net languages, using COM interop. The latter sounds more complicated than it is, and it will increase the satisfaction of your IT department, since developing in .net is just much more rewarding than VBA (version control possible, etc.).
I would second one of Kevin Ross's main points:
I personally think their should be a
small team of people within IT whose
job (or one of their jobs) is to
develop these small applications. They
should work very closely with the end
users and not be locked in the ivory
tower of IT.
I think any IT department that has a lot of users using Access/Excel should have at least one properly trained and experienced specialist in developing apps on those platforms. That person would be the go-between to make sure that:
IT's priorities and policies get properly implemented in the home-grown apps.
the end users get expert help in converting their home-grown efforts into something more stable and well-designed.
I would second Tony's point that whoever works with the end users in revising these apps to meet IT standards should work side-by-side with the users. The Access/Excel specialist should be an advocate for the end users, but also for the IT policies that have to be followed.
I also think that an IT department could have a specialist or two on staff, but should also have a full-time professional Access and/or Excel developer as a consultant, since the on-staff people could probably handle day-to-day issues and management of the apps, while the professional consultant could be called in for planning and architecture and for the implementation of more complex feature sets.
But all of that would depend on the size of the organization and the number of apps involved. I don't know that it would be desirable to have someone on salary who is nothing but an Access/Excel specialist, precisely because of the problem you get with all salaried employees compared to consultants -- the employees don't see as wide a variety of situations as an active consultant with the same specialization is likely to see and thus the consultant is going to have broader experience.
Of course, I recognize that many companies do not like to outsource anything, or not something that important. I think that's unwise, but then again, I'm the person that gets hired by the people who decide to do it!
If it's mission critical, and it's in Access or Excel, is built poorly, and no one understands it, it is probably time to rebuild it properly.
When the 'users' are in control it usual means one particular person is in control of the architecture, design, coding and documentation... except they normally omit the documentation step. Source control and bug reporting, the touchstone of software development, is usually absent. Few instances of code reuse, due to the nature of Office apps (code modules usually embedded into documents) and VBA (little OOP, most VBA coders don't use Implements, etc). All this means that the resulting applications are not subject to get proper scrutiny and quality can suffer, meaning there are likely to be maintenace issues, escpecially when that one user leaves. I know because I used to be that person ;)
So in order to satisfy the IT department, the proper process needs to be applied. That one 'power' user can continue to own the design and coding but will get peer review, perhaps the serivces of a technical author and a dedicated tester, be required to use source control, perhaps consider integrating with enterprise systems, etc.
There is no getting around the use of Excel/Access. It's what's available, and still very powerful and flexible. The best thing to do is offer some guidelines as to how files should look and be set up. If everyone is using similar standards then the files will live longer and more productive lives, beyond the creator's tenure at the company.
You've got some excellent answers regarding dealing with the folks and the business side of things. So my response will be more technical.
If you are going to redesign the app have the developers work in the same offices as the users. Given the users updates every day or two. If the users have any minor suggestions give those to the users within a day or two. Ultra Frequent Application Deployment
Give the power users an Access MDB/ACCDB linked to the tables with a bunch of starter queries. Let them create the queries they need to export the data to Excel for their own purposes and distribution to clients.