I've been asked to figure out a way to do this, so please fill me in on whether this is even possible or if it shouldn't be done.
The goal is to automate data entry into VMS (we use E-Term32 for connecting to VMS). Things that have been discussed for this purpose: Excel spreadsheets, Dynamic Data Exchange, the macro tools available in E-Term32 (Emulation Command Language - ECL), OLE Automation, etc.
The envisioned process would go like:
Receive Excel file (or other data format like a text file)
Connect to VMS
Run Macro
Macro navigates the menu system and uses data from Excel file to enter data
I know there are "better" or easier ways to do this like building an application to enter the data, but my supervisors are concerned about circumventing the business logic built into the "Blue Screen" menu/applications. They are also sticklers on building new applications for stuff like this anyways.
How is the data stored on OpenVMS, may we assume in native RMS (indexed) files? or some database (RDB, Oracle, Adabas,...) ?
Whichever, it is sure to be perfectly possible to write directly in the datastore through some ODBC or JDBC method. Freeware, or commercial (ConnX, Attunity,...). Just google: OpenVMS ODBC
Once you find a method for direct data access one should indeed be concerned about the business logic. Field formatting, values ranges, foreign keys,...
Thus access can only be granted to (software managed by) trusted team players.
But that can be perfectly manageable and you may find the new method can be made more robust than those green-screen apps.
If direct data access is no negotiable, then there are still many options.
Screen-scrapers have been build, you should not attempt to write from scratch.
Check out commercial terminal-centric modernization tools like: http://www.rocketsoftware.com/products/rocket-legasuite-gui
presentation: http://de.openvms.org/Spring2009/03-Dutch_Police_FINAL.pdf
(I am not associated with the fine folks # Rocket, it is just one example I am aware of. There are surely more (commercial) options.
Now about those business rules. How clean is this implement?
Strict form/function seperation? Probably not, otherwise you would not be asking.
There are several RPC, GSOAP, methods available, free and for fee, that allow one to call OpenVMS service routines, passing in external data. That's probably the best path forward.
The company I work for, Attunity, sells such 'legacy adapter' tools in addition to ODBC/JDBC data access to files directly.
For example, using Attunity Connect software you can connect a row in a table, to the call of a subroutine. The basic plan would be to just use an SQL INSERT statement on Linux or Windows to an ODBC datasource which is connected to an OpenVMS target. The connect software will then call an OpenVMS subroutine in a shareable library to process the row, using or at least re-using the existing business logic for validation, normalization or de-normalization as the case might be. And yes, they can also expose a SELECT or MODIFY for lookups that are more complex than can be described in SQL.
Everybody happy! You can use modern tools, they can use the old code and checks.
And ofcourse another time-honored method is to just have an FTP drop point for data to be entered. Some OpenVMS job scans an agreed upon directory for 'action' files, and runs an OpenVMS program to process the data in the fashion similar to the terminal UI app. Cobol, Basic, re-using as much of the existing terminal code and logic as possible
Good luck!
Hein
Related
I am trying to figure out a way to create a secure data entry process using Microsoft Office (cannot use Azure by the way). The obvious way is to use Microsoft Forms, but the form would be too large as there are over 150 data points that need to be entered.
I have instead created an Excel file that has the ability to load responses to a SharePoint list. This works as long as the list is public and viewers have contribute privileges. The downside to this is that the data is not secure and anyone can view the list. The data has phone numbers, ssn, passports, etc. I tried to make a work around for this by creating a flow that captured any new entries and moved them to my personal private SharePoint list, then deleted the original. This works great, but after testing with my colleague it seems that anyone can join the first team and make an identical flow to that team to capture data themselves.
Is there a way to accept data from anyone internally while also preventing them from accessing the data? I am capable of using SharePoint, Teams, Power Automate, and Access to accomplish this. I am less familiar with access but have used it before. I'm just not sure how secure it is.
I'll be happy to provide further information as needed. Thanks
You could consider Access forms, and then have the tables reside on SharePoint.
You can then say base the form on a query, and in that query you could provide (have) the user name included as part of the query where clause (thus a user removing or changing the forms filter will only provide a view of their own data.). However, such uses if they fired up Access (or even Excel) could wind up with seeing all rows in the table and not what the Access query restricts the data to (the one user).
Unfortantly with Access desktop? And like most desktop software? Well the goal and design and history is that of allowing ease of data editing and viewing. So, just like when you walk into a bank, and wait in line? Well, when you get to the teller, they will fire up a access form based like system and pull up your account information. And any and all tellers can see/view and pull up that data because that is what the software supposed to do, and do well. (get at any customer data).
Now, flip this problem, and create a user web banking system, or even a instant teller. Now the problem is backwards. The software's goal is not ease of editing data in a table, but ONLY allowing the user to see THEIR data. So a very different kind of problem.
It is very (beyond very) to note that web development tools do not out of the blue or automatics restrict data to JUST the one user. It is YOU the developer that makes that choice. So, if we could press a button and have a access form converted to the web? Well, then that access form would do its job in life. That job is to allow you to edit all that data in a table. Not just YOUR data.
For example, have you ever written a Access application that JUST displays each users own data? I am betting not! It is hard to do. (because the roots of Access is ease of editing data). As noted, I cannot stress that web development tools do the SAME thing. They can let you edit data. The editing of data is thus a software issue and development issue. The data system or Access or even SQL server? They just hold data. It is YOUR software that has to decide that ONLY data for the one user is to be displayed.
Now so very often web systems ONLY show YOUR data. But that VERY common case is due to the nature of web systems, and that your not a bank teller, or a company employee, but that of a consumer of that system. And each consumer now is 100% opposite of what most software systems do naturally (that is get data from a table).
So, I can't stress this concept enough. That concept is that web software and systems do NOT make this restricting of data choice for you. It is your developer tools that MUST have this ability and ALSO the choice of how you build such systems.
Web systems can do this data restricting data better for several reasons.
One big reason? Well, most of the time you adopt some kind of security and logon system.
Thus, parts of the web site now are restricted based on your logon system. (hey, maybe you use Facebook logons). So web systems have a very good and wide and broad system of supporting some kind of logon system. For desktop software? not so much.
But EVEN with that very robust and type of security system and logon system?
You the developer STILL have to make the choice of what data they see. Maybe it is traveling salesmen on the road. And thus every user of that web system still has to be able to edit, view and add customers to that system. This common case is not really different then a typical Access form to edit data.
So only SOME types of web applications need that restricting of data to JUST the one logged on user. But, because web systems have that "better" logon system? Then you the developer can thus now use that logon to restrict data with greater ease then what desktop software dev tools often offer (say like access).
So, a lot of this will come down to the web tools used. If you pulling data with Excel, or Access? Then it considerable more difficult to pull JUST data that belongs to a given user. And your software will require designs from day one with these restrictions in mind.
Again, at the risk of repeating myself?
the tools don't' make the data restrictions for you. If you need secure data for ONLY the given user, then you simply have to adopt a set of developer tools that allow this concept in mind. And that tends to mean the tools you use do NOT by default allow users to directly edit or link to some big table of data that contains all users data.
This is also why say a program written in VB6, or now vb.net can be more secure then Access. (because those tools by default don't provide forms that are designed from day one to edit all rows of data). So, the UI does not present default forms and things that gives users the ability to edit all rows. As a result, then the software can now control what records the users sees, and since they don't' have some default form that allows viewing of all rows.
It thus becomes more practical to write software that only dishes out their own data. But again, and I can't stress this:
The choice of what data to display is a choice of the software developer - not the database nor the web development tools.
It thus stands to reason then you have to chose the appropriate tools that will allow you to write software that restricts the data in a manor you require.
General questions on SO in regards to what flavor of ice cream or what dev tools to use tends to be frowned upon, and is quite much against the general rules for solving software and code issues you have and post on SO.
But, if you come from say Access with VBA skills? Then I would suggest adopting the free verison of Visual Studio and go with vb.net + asp.net web forms. Web forms are close to end of life, but they are by far and away the best choice, the least effort (coming from Access) and will give you a robust security model in which you can restrict data on a per user basis.
I would not duck tape and try to cobble together such a system with office tools, since they are not really designed for that type of data restricting you need. So, go with some good quality web tools like Visual Studio (free version is fine) and use SQL server (free version again) is really the way to go here. If you need a truck in place of a car, then get the right kind of truck, and don't try and use a car for that transport of goods, or in this case build a secure web database system.
Looking for an IT data mapping solution that integrates with SharePoint 2013.
Need to be able to capture information about IT systems, how they relate, how the data flows through each system. Must be able to dynamically generate diagrams on the fly.
As far as I am aware, there is nothing like this that Integrates to SharePoint. You would have to create something bespoke.
Indeed, I would not recommend that approach as I think that the level of complexity and size of data is likely to quickly outstrip anything that would be especially performant in SP. SP is already a complex set of data and adding further complexity is not wise in my experience.
SP lists are good for simple data. I've been starting to create a service catalogue for ourselves in SP but it will be a very simple list & even as I've been thinking about it, I've quickly realised that it might not be suitable for long. Also, I'm not planning to hold much in the way of data interfaces.
Instead, I would recommend a solution that is more focussed with a database and a web front end that could easily be integrated to the SP site or simply linked.
These kind of systems tend to get very complex as you begin to discover links between systems and unless your organisation is relatively small and simple, I think you will regret it.
If your system architecture is at all complex, you would be better off with a fully-fledged Enterprise Architecture tool. That would allow you to begin to correctly map the logic between systems and even to map systems to infrastructure.
In short, to use SP, keep it very simple (simplistic) and forget the diagrams for now as you would have to create a bespoke interface anyway. Consider a bespoke database with web front end that can be linked to your AD for authentication and authorisation the same as SP and simply linked to from SP. Or for a more complex architecture, investigate the use of an Enterprise Architecture tool.
UPDATE:
OK, so you need something simple to draw out the data. That is a bit easier. I'm still not aware of a dedicated web part but to be honest, I've not looked.
My approach would probably be to use a JavaScript "Graph" tool - note the phrase "graph" not to be confused with "chart". A chart is a pictorial view of (often numeric) data. A graph is a set of nodes and "edges" (I probably haven't explained that very well). Graphviz is the most famous of such tools but that is a command line tool. However, there are plenty of JavaScript tools that might be used.
SharePoint is delightfully easy to integrate with JavaScript (apart from the bizarre field names and hopelessly overcomplex data structures!) & even with my limited skills, I was able to knock up a Google map "view" that uses the data from a normal SharePoint list. With a bit of help from a JavaScript library of course.
You simply hide the normal View Web Part. The data is available to the web page still and you can feed it straight into a JavaScript library that will do the graphing for you.
Here are some free and none-free libraries in case you chose to go this route:
Free:
SigmaJS
VisJS
Arbor
Springy
Not Free
GoJs - Not used it but this looks fab. Interactive Diagrams for the
Web - unlimited trial but watermarked, USD thousands
jGraph USD 10's thousands
ZoomCharts - USD hundreds
Worth also noting that, in theory, you could do this with Visio. However, there are not many examples out there because it is so complex.
I work in a corporate office environment which routinely uses an IBM Mainframe and AS\400 as the main source of the Information System.
However, the company is growing faster than the information technology group can support.
Therefore, it is necessary to create ancillary applications to accommodate the ever changing business needs and new ventures.
I’ve introduced a web based application using PHP and MySql database (as we have routinely received bulk Mainframe downloads and imported into Microsoft Access Databases before upsizing the databases to a SQL server due to their ever increasing size).
It was been mandated that all new application be similar to look and functionality as the Mainframe. It must be a text based user interface where it’s not necessary to use the mouse, only function keys to navigate menus and pressing the enter key to add new data (records) or to execute a search.
What modern day programming language will satisfy this Text-based user interface mandate? Preferably something that can interact with MySql to fetch/store data and look similar to the mainframe and possibly support XML formatting (font colors, input fields, etc).
Thanks in advanced!
You can build this in pretty much any language. Almost all programming languages have support for these things. Choose a modern language that your development team already knows well and use that. Ideally, it would be a high level language (don't use C) to speed up development time.
its been a while since i have used it but ncurses might be a good solution
Java, Delphi, Visual Studio, etc, etc, etc. As #Oleksi said; pretty much any modern fat-client GUI language that has a database library that can handle MySQL will do.
If you're not sure which to use, you should either hire someone with experience or try a few small prototypes.
I'm looking for a way of programmatically exporting Facebook insights data for my pages, in a way that I can automate it. Specifically, I'd like to create a scheduled task that runs daily, and that can save a CSV or Excel file of a page's insights data using a Facebook API. I would then have an ETL job that puts that data into a database.
I checked out the oData service for Excel, which appears to be broken. Does anyone know of a way to programmatically automate the export of insights data for Facebook pages?
It's possible and not too complicated once you know how to access the insights.
Here is how I proceed:
Login the user with the offline_access and read_insights.
read_insights allows me to access the insights for all the pages and applications the user is admin of.
offline_access gives me a permanent token that I can use to update the insights without having to wait for the user to login.
Retrieve the list of pages and applications the user is admin of, and store those in database.
When I want to get the insights for a page or application, I don't query FQL, I query the Graph API: First I calculate how many queries to graph.facebook.com/[object_id]/insights are necessary, according to the date range chosen. Then I generate a query to use with the Batch API (http://developers.facebook.com/docs/reference/api/batch/). That allows me to get all the data for all the available insights, for all the days in the date range, in only one query.
I parse the rather huge json object obtained (which weight a few Mb, be aware of that) and store everything in database.
Now that you have all the insights parsed and stored in database, you're just a few SQL queries away from manipulating the data the way you want, like displaying charts, or exporting in CSV or Excel format.
I have the code already made (and published as a temporarily free tool on www.social-insights.net), so exporting to excel would be quite fast and easy.
Let me know if I can help you with that.
It can be done before the week-end.
You would need to write something that uses the Insights part of the Facebook Graph API. I haven't seen something already written for this.
Check out http://megalytic.com. This is a service that exports FB Insights (along with Google Analytics, Twitter, and some others) to Excel.
A new tool is available: the Analytics Edge add-ins now have a Facebook connector that makes downloads a snap.
http://www.analyticsedge.com/facebook-connector/
There are a number of ways that you could do this. I would suggest your choice depends on two factors:
What is your level of coding skill?
How much data are you looking to move?
I can't answer 1 for you, but in your case you aren't moving that much data (in relative terms). I will still share three options of many.
HARD CODE IT
This would require a script that accesses Facebook's GraphAPI
AND a computer/server to process that request automatically.
I primarily use AWS and would suggest that you could launch an EC2
and have it scheduled to launch your script at X times. I haven't used AWS Pipeline, but I do know that it is designed in a way that you can have it run a script automatically as well... supposedly with a little less server know-how
USE THIRD PARTY ADD-ON
There are a lot of people who have similar data needs. It has led to a number of easy-to-use tools. I use Supermetrics Free to run occasional audits and make sure that our tools are running properly. Supermetrics is fast and has a really easy interface to access Facebooks API's and several others. I believe that you can also schedule refreshes and updates with it.
USE THIRD PARTY FULL-SERVICE ETL
There are also several services or freelancers that can set this up for you at little to no work on your own. Depending on where you want the data. Stitch is a service I have worked with on FB-ads. There might be better services, but it has fulfilled our needs for now.
MY SUGGESTION
You would probably be best served by using a third-party add-on like Supermetrics. It's fast and easy to use. The other methods might be more worth looking into if you had a lot more data to move, or needed it to be refreshed more often than daily.
I'm sure many of you are familiar with the IBM i5 series emulator (looks like this poop)
My company uses this religiously and there is no Biz logic in it so anytime somone in our finance dpt makes a human error it accepts it and adds it to the database. Not to mention its ugly, hard to use, not intuitive, etc....
I would like to create a frontend for this interface so that we can control the logic before its submitted to the system (we dont control the system itself) so in effect I need to make my own emulator app.
However I cant seem to find any information on how to interface with the i series, namely login, send commands, and view or gather data from the screens it would normally send back.
Any suggestions?
The problem is not the iSeries but the software package your company is running on it.
There ARE advantages to use green screens: it's fast and it's almost unbeatable at data entry, provided you get used to it.
But to answer your question, the iSeries is a J2EE enabled machine: a HTTP server comes installed and depending of the version of the iSeries, WebSphere might be already installed, or are entitled to install it. Then you can use JT400, which is the java toolkit for the os400 containing the jdbc drivers to connect the database and the necessary classes for calling programs.
If you prefer php, there is a flavor of the Zend framework made to work on the iSeries but I never tried it.
I'd recommend that you take a look at both the Attachmate Verastream Host Integrator (VHI) and IBM's Host Access Transformation Services (HATS) products. They effectively just screen scrape the green screen terminals to allow you to pull and push data and provide macro recording and editing tools to automate the process. App integration can be achieved via web services or html/jsp/servlet programming (plus .Net for VHI and EJB's for HATS). They do come with enterprise pricing however which may be an obstacle for some. They do have free trial offerings for evaluation purposes to help determine if they are an appropriate solution to your problem.
What software packages are they using? Most programs that I use in the 5250 emulator has some business logic that error checks the data before adding it to the database. Can you get us some more information so we can direct you in a better direction.
There are vendors that sell products that screen-scrape the 5250 data stream and produces a web front-end. Or you can write your own front-end in the language of your choice and just do SQL calls to the database.
THere's got to be some source code. Start by looking at the menu and menu option your users are accessing and figure that's running behind them.
Use command STRPDM to look for source code - look in different libraries (they are like folders)
You might have source code in a "member" called something like xxxMNUSRC xxxRPGSRC (rpg program source) or xxxCLSRC (cl programs), xxxDDSSRC (display/screen source, physical/logical file source)
Objects a "compiled" objects such as files (tables), screens, priter files (reports)
Stay away from Qxxx and #xxx libraries - those are system libraries.
http://systeminetwork.com/ is a good resource for iSeries related questions.