Short version: (after finding out the answer)
I have an Excel VBA application with an MS Access database (.accdb) on a SharePoint library.
The behavior that was examined (and apparently documented - see answer):
It takes a long time to perform the ADODB Open and Close methods (~15 seconds).
If multiple users are connected at the same time to the database, only the changes made by the user which closed the database connection LAST are saved. Changing cursor types, cursor locations or lock types didn't help. No error shown.
Why does this happen?
Original Question:
First question here. Hope this isn't too wordy:
I've built an Excel application using VBA to communicate with an MS Access database (.accdb) that should have support for concurrent users accessing it. It is meant to be placed on a Sharepoint site as an accessible file (not integrated into it in any other way). When I was testing the Excel file and the database on my home network it worked like a charm, transactions and all. However, once I migrated it to Sharepoint, I've noticed some extreme differences from the way it acted on my personal network:
The ADODB {.open} and {.close} methods take at least 15 seconds each (making Excel freeze until done). Due to this, I've decided to open and close connections only once throughout the lifetime of the application, and restore the connection if it is broken along the way. I'm aware of the fact that this is highly not recommended, but can't afford having my users wait so long. This hasn't caused any problems that I'm aware of, perhaps apart from the one I'm about to explain.
The problem: Changes aren't saved to the actual database unless all active user connections to the database are closed, even if the only active thing is the connection. Everything passes without errors for each user when attempting to update, and each user can access his/her changes, I suppose until all connections are terminated. I tried all possible cursor types and lock types, nothing seemed to work. It is as if a local copy of the database is stored on the user's computer (hence the long wait while opening and closing the connection), and updates are stored on the temporary version, not the actual one.
I tried all possible combinations for cursor types, cursor locations, lock types and what not (found out along the way that dynamic cursors aren't supported in my case - I wonder if that's the answer).
Due to this I have no other choice but to make the program accessible to only one user at a time, or changes seem to get lost along the way, making the program highly unreliable.
I read something about having to "flush the buffer" or "refresh the cursor". Is this even possible/necessary? Or the case? If I'm using a keyset cursor, shouldn't my edited records be shown to all other users? (not talking about new ones)
For what it's worth, I map the path to the sharepoint folder before accessing it.
Have any of you experienced something like this? Or have any suggestions?
If you need samples of my code I'll post it soon. Thanks so much!
I found the solution to my problem:
Although you can save an Access database file to OneDrive or a SharePoint document library, we recommend that you avoid opening an Access database from these locations. The file may be downloaded locally for editing and then uploaded again once you save your changes to SharePoint. If more than one person opens the Access database from SharePoint, multiple copies of the database may get created and some unexpected behaviors may occur. This recommendation applies to all types of Access files including a single database, a split database, and the .accdb, .accdc, .accde, and .accdr file formats. For more information on deploying Access, see Deploy an Access application.
Source: Ways to share an Access desktop database
Related
I have no knowledge on computer programming and I need a bit of help.
I'm using automate.io (a drag and drop integration software) to take a new row in excel and insert it into salesforce. That bit works all ok.
What I worry about is my excel document it is connected to an SQL server and auto refreshes every minute. The problem is that I have to have the Excel document open at all times for this to auto refresh to take place.
To combat this I used task scheduler to open the document at 7am even when there is no one logged in.
My question is,
will this work and is it reliable?
Will it work?
Only your testing can answer that.
Watch out for false positives, e.g. new record in database not picked up or not refreshed, and therefore not input to SalesForce in a timely manner.
Is it reliable?
Here are some ways to achieve what you want, in approximate descending order of reliability:
Get a third party to integrate the two databases directly (SalesForce and your SQL server), with updates triggered by any change in the data in your SQL server. There is a whole sub-industry of SalesForce integration businesses and individuals who would consider taking this on in return for money.
Get a standalone script (not Excel) running on a server near your database to monitor your DB for changes, and push new records to SalesForce via a direct API.
Get a standalone script (not Excel) running on a server near your database to monitor your DB for changes, and push new records to text files (not Excel) which are subsequently loaded into SalesForce.
Get Excel to refresh your DB for changes regularly via a data link (i.e. what you outlined), but have it new records to text files (not Excel) which are subsequently loaded into SalesForce.
Get Excel to refresh your DB for changes regularly via a data link (i.e. what you outlined), and have it push new records to SalesForce via third-party software as a substitute for actual integration.
You will notice your proposed solution is at or near the end. The list may inspire you to determine what tweaks you might make to the process to move up the list a little ways, without changing the approach completely (unless you want to and can justify it).
Any tweak that removes a link in the dependency chain helps reliability. Shorter chains are generally more reliable. Right now your chain sounds something like: Database > Server?/internet?/network? > Excel data link > Excel file > Task scheduler > internet > automate.io > API > Force.com > SalesForce.
Transaction volume, mission criticality, and other subjective criteria will help guide you as to what is most appropriate in your situation.
This question may be opinion based, but please bear with me.
Context: Company has asked me to develop an Excel add-in, that all employees will use. I use a shared network folder to store the add-in. All users just creates an reference to that add-in and loads it from network.
Problem: Company would like to see usage statistics of the add-in. Basically, lets say the add-in has 10 specific functions, which are invoked by a press of a button. Each time a button is pressed, this information should be stored.
Question: Where to store this information?
I added a new function and included in all methods that would just write the number of each function use of a session in the add-in sheet, and on the workbook_close method, print everything to a .txt file and save it to the network drive and run reports based on that.
Now the problem is that everyone has access to the network drive, can accidentally delete the log files or just plainly change them for whatever reason. I can overcome the changing issue by encrypting the log files, so users won't know what's inside them, but then they can just delete them by thinking that it's trash.
It all comes to this: where to store user usage data, which would be accessible to me or my manager, to use that data for reports, without anyone else being able to interfere with the data?
By interfering I don't mean that anybody would do that of spite, but an accidental deletion, or just a cleanup of the network folders.
My idea: run a script to monitor the log folder, and when a new file appears, move it to a safe directory and remove it from the original folder. But it won't work if I turn off my computer and seems to me that I'm missing something and over-complicating this issue.
Thank you for your time.
Let Company decide.
Present the issue to your client, along with all the options and risks and get their feedback. In this way:
you will get information if they have a strong preference
make the client aware of any risks
limit your liability in case something "bad" happens
Note:
This is not the proper forum for this question. Expect it to be Closed soon.
I have been reading up a lot about CouchDB (and PouchDB) and am still unsure what the best option would be for a project of mine.
I do have a possible way to solve the project in my head based on what I have read so far, but I am unsure about things like performance and would love to get some insights. Or perhaps there's a better place to ask this question? Please let me know if that's the case! (Already tried their IRC channel and the mailing list, but no answers there as of yet)
So the project is basically an 'offline-first' mobile application. The users are device installers. They get assigned a few locations and devices to install every day. They need to walk around buildings and update the data (eg. device X has been installed at location Y; Or property A of device B on location C has been changed to D, etc...)
Some more info about the basic data.
There are users, they are the device installers. They need to log into the app.
There are locations, all the places that the device installers need to visit.
There are devices, all the different devices that can be installed by the users.
There are todos, basically a planned installation for a specific user at a specific location for specific devices.
Of course I have tried to simplify the data, but this should contain the gist.
Now, these are important characteristics of the application:
Users, locations and devices can be changed by an administrator (back-end software).
Todos can be planned by an administrator (back-end software).
App user (device installer) only sees his/her own todos/planning for today + 1 week ahead.
Multiple app users (device installers) might be assigned to the same location and/or todos, because for a big building there might be multiple installers at work.
Automatic synchronization between the data in each app in use and the global database.
Secure, it should only be possible for user X to request his/her own todos/planning.
Taking into account these characteristics I currently have the following in mind:
One global 'master' database containing all users, locations, devices, todos.
Filtered replication/sync using a selector object which for every user replicates only the data that may be accessible for this specific user.
Ionic application using PouchDB which does full/normal replication/sync with his/her own user database.
Am I correct in assuming the following?
The user of the application using PouchDB will have full read access on his own user database which has been filtered server-side?
For updating data I can make use of validate_doc_update to check whether the user may or may not modify something?
Any changes done on the PouchDB database will be replicated to the 'user' database?
These changes will then also be replicated from the 'user' database to the global 'master' database?
Any changes done on the global 'master' database will be replicated to the 'user' database, but only if required (only if there have been new/changed(/deleted) documents for this user)?
These changes will then also be replicated from the 'user' database to the PouchDB database for the mobile app?
If all this holds true, then it might be a good fit for this project. At least I think so? (Correct me if I'm wrong!) But I did read some 'performance' problem regarding filtered replication. Suppose there are hundreds of users (device installers) (there aren't this many right now, but there might be in the future). Then would it be a problem to have this filtered replication running for hundreds of 'user' databases? I did read about CouchDB 2.0 and 2.1 having a selector object to do filtered replication instead of the usual JS MapReduce which is supposed to be up to 10x faster. But my question is still: does this work well, even for hundreds (or even thousands) of 'filtered' databases? I don't know enough about the underlying algorithms and limitations but I am wondering whether any change to the global 'master' database does or does not require expensive calculations to run to decide which 'filtered' databases to replicate to. And if it does... does it matter in practice?
Please, any advice would be welcome. I did also consider using other databases. My first approach would actually have been to use a relational database. But one of the required characteristics of this app must be the real-time synchronization. In the past I have been able to handle this myself using revision fields in a RDBMS and with a lot of code, but I would really prefer something as elegant as CouchDB/PouchDB for the synchronization. This is really an area that would save me a lot of headache. Keeping this in mind, what are my options? Am I going in a possible right path or could performance become an issue down the road?
Also note that I have also thought about having separate databases for each user ('one database per user'), but I think it might not be the best fit for this project because some todos might be assigned to multiple users and when one user updates something for a todo, it must be updated for the other user as well.
Hopefully some CouchDB experts can shed some light on my questions. Much appreciated!
I understand there might be some debate but I am only interested in the facts and expertise of others.
I have an Excel workbook I need to share with my client but make sure it is not opened outside of their office network.
How to protect my Excel workbook like this?
In short, exactly what you request isn't (easily) possible.
Theoretically you could set the workbook to draw its data from a network resource you control, and set that resource to only allow read access from specific network addresses such as the client network. However, there are a number of hoops to jump through first, such as setting up the database server and creating the data source, testing the worksheet, and then setting up the network permissions to allow the client's addresses (and of course discovering their IP addresses). That's actually a lot more work than it even sounds.
Even all of that work wouldn't prevent the user from copying that data into another workbook (either via copy/paste or manually retyping it), or using a VPN to connect to their work-network and reading the file just fine from Tahiti or Moscow.
Sadly, the only foolproof solution is not sharing the data at all, and that doesn't let the customer see the data at all.
In IT we frequently try to reframe the problem not in terms of "how can we make this solution" but instead as "what is the problem we're trying to solve?" In this case, it sounds like the problem you're trying to solve is keeping sensitive information from falling into the wrong hands. The only solution to THAT problem that has ever been found is not to trust it to the wrong people. If you can't trust your client with this data, I'd suggest either the client isn't trustworthy, or there's something wrong with the data you're looking to share (as in, why do they need data this sensitive?).
I'd suggest re-evaluating the problem you're trying to solve. Maybe send them a screenshot of the document from a resource you control and can delete the image from after they've viewed it. They'd still be able to print or save the image, but if they're THAT persistent in getting the data, you're not going to stop them if they can see it at all, and they're not terribly trustworthy.
I have created a process where a (registered) user can upload (after client-side and server-side validation) a (zipped with a very uncommon extension) access database to my server through an asp.net webform, that will sit in a nice secure location until a scheduled SSIS package comes along at night, to flow relevant data from the access db to the sql server.
After that, my access db is deleted. There will be no other execution of that db. Access is not installed on the server.
I've done research, of course, but am I introducing a vulnerability (script inside the access db for instance?) that SSIS might trigger?
Thank you in advance.
SSIS likely uses ODBC or OLEDB to get to the data in the Access/Jet/ACE database, so there is nothing there to execute any code -- ODBC and OLEDB know nothing about anything but data and all the dangerous functions that could be executed in SQL statements are blocked.
So, without Access installed, no, there's no real danger here. If you're concerned that there is, you could process the file with DAO before you open it and delete everything in the QueryDefs collection and in the Modules document collection. Or, you could use a buffer database where you import nothing but the data tables, and then pass that to SSIS.
But I don't really think SSIS is looking at anything but the data tables to begin with.
BTW, there has never once been any virus or exploit that has been propagated via Access, so the concerns over Access vulnerabilities are vastly overblown (with massive inconveniance to end users as a result, with blocked macros, sandbox mode and from A2007 on, the need to define Trusted Locations).
I'd agree with David (although using the word "never" is always dangerous!). One extra thing you might consider is having the person who zips up the database prior to uploading apply encryption to the zip file with a password specific to each user.
The actual encryption doesn't necessarily have to be particularly strong, even, although it would help if your communications were being hacked: the point is that it serves to identify the person who originated the Access upload. If the SSIS process tries to open it with each known password in turn and fails with all of them then the package can be considered to have been introduced into the system by some unauthorised person and therefore be suspect.
This is less to prevent malicious code and more to prevent malicious data being entered into your system.
hth
Mike