I have an excel file about 340 mb which contains more than 2000 worksheets and dozens of long VBA program. The file is getting so large that it takes around 10-15 mins to open the file and often get into "NO RESPONSE", "NOT ENOUGH USEABLE RESOURCES" when I save or debug the file.
I searched online people suggest migrating to Acesss. However, I have never used Access before. SO I wonder
1) How to migrate the excel file to access?
2) Will the VBA program be carried to the new access file
3) Do i need to modify the excel VBA code to fit Access?
2) Can Access handle a 300-500mb file?
thank you
500MB Excel file does not mean 500MB Access file. Moving from Excel to Access is a very good thought. However, you need to understand the basics first. Excel it completely different from MS Access access except when Access looks similar to excel in a datasheet form.
Access is a "Relational"-Database + graphical front-end software where you can present your data in a form or through components. In a relational database you identify entities and define relationships between them. By doing so you eliminate [Data inconsistency] throughout your application. A proper modelling might reduce your data from 500MB to just 50MB.
To answer your questions:
1> How to migrate the excel file to access?
First of all migration in your case needs a fresh new MS Application. Start modelling your business first. Read about Rational-Databases, Read about MS Access tables, relationships, queries. Think if Access is a suitable platform for your business. Create the application in MS Access and then you can come back to us and ask about migration.
OR:
You can use the current Excel sheet as an external data-source and link them as linked table within the access application. This is very very not recommended because you are effectively not benefiting anything better than your current situation. (Except queries to find a data)
2> Will the VBA program be carried to the new access file
Do you mean VBA forms? no it won't. Access has its own Forms and controls which looks better than Excel userforms. You have to redesign that part completely in Access.
3> Do i need to modify the excel VBA code to fit Access?
In most cases YES unless its a generic function. Most Excel VBA codes are referring to a cell or worksheet which Access does not have! In other words, it depends how big and complex your codes are. Any specific code has to be adjusted to MS Access platform but the adjustments are very minor not a major task. Again it depends on your code complexity.
4> Can Access handle a 300-500mb file?
Yes it can! newest versions can reach up to 2GB but i personally would not stay with access when i have to work with such amount of data. I would look for splitting/upgrading the database to a proper dedicated database engine such as MSSQL or free MySQL servers.
Some advise from me: 500MB of Excel sheet is potentially dangerous you should seek an alternative very soon. If you are going to fiddle on you own please always "backup" because Access throws random errors which are very hard to understand. Find an experienced IT person to help you before you delete/update/wipe off your data. Good luck
Interesting
I faced much of the same problem years ago on a POS system that recorded the transaction history in excel via some VBA. I moved to Access and found (Access) get a little corruptible the larger the file gets. I had to build in safeguards to restore from a backup should the file quit on me. I eventually moved to Visual FoxPro as my VBA could just about be translated straight across. Works to this day as a matter of fact.
How can I develop an Excel plug-in to edit external data in an Excel data table?
Excel can make connections to external data sources but as far as I am can see they are one-direction read-only data tables. What I am trying to do is something like TFS plug-in for excel. I am sure there are many more ones like that.
For those who do not know that plug-in:
When installed, TFS Excel plug-in takes place as a new menu in Excel. Through that menu you can open a connection to a TFS server and bring your (work item) records into Excel as an Excel table. You can add new rows or edit the data in the table. Some cells has drop down lists attached to them but only valid options are shown in the list and that is different for each record. You can edit rows in the table and you can bulk push those records back to server.
I don't know if it makes a difference but the connection and update operations on my datasource will be through web services.
I guess this would require some serious development but I am lost between web pages about external data ranges (which are only for reading). Can someone please direct me to some further reading on the topic?
External Data Ranges will not help you so you can stop reading web pages about them. You're correct that they're read only. You could use them for the read part of your operation, but you'll be doing so much coding around the write part, you might as well just control everything. You just won't get enough benefit from External Data Ranges to warrant using them at all for this type of situation. In my opinion, of course.
If you were reading and writing to a database, you would likely use ActiveX Data Objects (ADO). You would read in a recordset, monitor it's changes, then write back to the database using UPDATE, DELETE, and INSERT statements as necessary.
If you'll be interacting with the database through an API, as you seem to indicate, you will probably use Microsoft XML library, specifically the MSXML2.XMLHTTP object. You can use GET, POST, PUT, DELETE, and anything else you can do through HTTP.
If you've never used XMLHTTP before, you'll have a little learning to do. But it's not particularly difficult and there's a ton of info available. The hard part, in my opinion, is tracking the changes made to the Excel sheet. If you allow the user to use Excel's native editing features, it can be difficult to keep track of the changes made. If you go to a total lockdown situation where the user has to, say, use your menu item to delete a record, then you have to ask why you're using Excel (there still may be good reasons, but familiarity with Excel's interface won't be one of them because you'll be replacing it with yours).
Maybe you already have a strategy for this. But if not, search for "detect deleted row with worksheet change event" to get a feel for some of the challenges you'll face. If you have a way forward, then go read up on XMLHTTP and you should be all set.
My client wants to store basic relational data in Access. So far, so good. However, ideally, he'd like for me to create an Excel spreadsheet that would allow users to create and modify data types without having to work with Access software or know about databases. To be more specific, he wants a single master spreadsheet that would let people manage data for several different "projects." Each project would have basic attributes and other related data such as employees working on it, numbered to do items with associated data, etc. I've worked with databases before and it's a neat, textbook example of a relational database. I have a model for the data already, and making an Access form to fill it in would be straightforward.
However, here's the thing: he wants creating new attributes and tables completely intuitive within the Excel spreasheet--as easy as clicking an "add student" button or even add a new category of data. For instance, in the future, he may add a list of contractors working on the project, and it would be nice to be able to have a button that would allow you to essentially create that new table. There won't be a great amount of data, though, and I'm not sure if referential integrity and normalization is crucial. For instance, the list of contractors he creates wouldn't need to be perfectly linked up so that each company only appears once in the database.
So, what should I do? Can I accomplish this within Excel spreadsheets using macros? Can you make buttons in Excel that would say "create a new table," which would (run a VBScript to) create a new database table to be associated with each project, and then allow you to format it? Should I not bother with Excel at all and basically write a Visual Basic program? I'm familiar with general programming and databases, but I am fairly new to Excel, Access, and Visual Basic. If you could point me in the right direction--to tutorials, examples, advice, general concepts, etc--it would be much appreciated.
Excel is essentially for analyzing data, while Access is essentially for storing and processing relational data. Now, having said that, what you are trying to do is probably possible but it is really not taking advantage of the features the software where optimized for.
Furthermore, adding "tables to be associated with each project" does not seem as the "relational way of doing it", like a complex solution for a simple problem.
Perhaps you should consider some alternatives:
If the amount of data is small and not very complex, would there really be any need for Access or could you just as well use Excel for data storage and data manipulation?
Depending on how the data is structured, perhaps you can create a view or stored procedure in Access and used it as an linked table in Excel?
Perhaps you can develop the set of forms you need in Access and turn it to an stand-alone application (no need for Access installed on the client's computers)
For a person entering data, Access is built so they don't need to understand relational data at all. If you let them enter data in Excel, you will have to excessively code it to give you the same control in Access or you run the risk of letting them free form data to the point you won't be able to import it back into Access.
Unless there are very complex calculations and a need for the user to 'tweak' the report layout, give them a data entry form.
Beware of the "I'm just so use to doing it in Excel I don't want to relearn it in Access" notion. The data entry can be made very intuitive and may save them time in the long-run.
Seems like there is an owner/manager who understands Excel and wants the ability to update it without you if needed.
I run a sports program where i have a master roll of who is in which class in excel. I want to link this to a database in access that stores the other information about each athlete, e.g. address, parents name, school, medical details. I want to be able to add names to class in the excel speadsheet and have this automatically generate a record for that person in access. There also needs to be some failsafe for athletes that are in multiple classes. I was also doing class roles as pivot tables out of the access database so i need to code for classes and also have this allow for athletes in multiple classes/disciplines.
It is easy enough to update an Access table from Excel via ADO, after that it is very much about your tables and indexes. If you are not familiar with relational databases, you might like to read http://r937.com/relational.html. That being said, it would be a lot easier to work in Access and output to Excel when necessary.
I agree I think this is a classic case of trying to get excel to do something its not best for. If you try to create some kind of hybrid system with excel pushing data into access then it will end in tears at some point.
The best thing in this case would be to port the whole thing to some kind of database. If the number of uses and the usage falls into the range for access/jet then that would be a great choice. If more users/higher usage is going to be needed then maybe look to SQL express to hold the data and access as a front end.
There was a thread a few days ago about someone being sick when maintaining an access DB, he wanted to rewrite it in .net. The point of that thread boiled down to using the correct tool for the correct job. No one can blankly say “Access sucks, everything should be in SQL server/.net” because if used in the correct way and for the correct projects access is a great tool.
So to bring it back to this thread it looks like you have “outgrown” excel and should be looking at some kind of database with access being a strong candidate
If you want to display the data in Excel (so you can do sorts, filters, etc.) then you could store the data in Access as has been suggested, then instead of exporting a report every time you want to use it, link your Excel file to Access using a Database Query.
In Excel 2003 go to Data->Import External Data->New Database Query and create a new data source to your Access mdb.
That way your data is stored in a much better way, whilst still having the Excel viewability that everyone(?) loves.
I am building a small application for a friend and they'd like to be able to use Excel as the front end. (the UI will basically be userforms in Excel). They have a bunch of data in Excel that they would like to be able to query but I do not want to use excel as a database as I don't think it is fit for that purpose and am considering using Access. [BTW, I know Access has its shortcomings but there is zero budget available and Access already on friend's PC]
To summarise, I am considering dumping a bunch of data into Access and then using Excel as a front end to query the database and display results in a userform style environment.
Questions:
How easy is it to link to Access from Excel using ADO / DAO? Is it quite limited in terms of functionality or can I get creative?
Do I pay a performance penalty (vs.using forms in Access as the UI)?
Assuming that the database will always be updated using ADO / DAO commands from within Excel VBA, does that mean I can have multiple Excel users using that one single Access database and not run into any concurrency issues etc.?
Any other things I should be aware of?
I have strong Excel VBA skills and think I can overcome Access VBA quite quickly but never really done Excel / Access link before. I could shoehorn the data into Excel and use as a quasi-database but that just seems more pain than it is worth (and not a robust long term solution)
Any advice appreciated.
Alex
I'm sure you'll get a ton of "don't do this" answers, and I must say, there is good reason. This isn't an ideal solution....
That being said, I've gone down this road (and similar ones) before, mostly because the job specified it as a hard requirement and I couldn't talk around it.
Here are a few things to consider with this:
How easy is it to link to Access from Excel using ADO / DAO? Is it quite limited in terms of functionality or can I get creative?
It's fairly straitforward. You're more limited than you would be doing things using other tools, since VBA and Excel forms is a bit more limiting than most full programming languages, but there isn't anything that will be a show stopper. It works - sometimes its a bit ugly, but it does work. In my last company, I often had to do this - and occasionally was pulling data from Access and Oracle via VBA in Excel.
Do I pay a performance penalty (vs.using forms in Access as the UI)?
My experience is that there is definitely a perf. penalty in doing this. I never cared (in my use case, things were small enough that it was reasonable), but going Excel<->Access is a lot slower than just working in Access directly. Part of it depends on what you want to do....
In my case, the thing that seemed to be the absolute slowest (and most painful) was trying to fill in Excel spreadsheets based on Access data. This wasn't fun, and was often very slow. If you have to go down this road, make sure to do everything with Excel hidden/invisible, or the redrawing will absolutely kill you.
Assuming that the database will always be updated using ADO / DAO commands from within Excel VBA, does that mean I can have multiple Excel users using that one single Access database and not run into any concurrency issues etc.?
You're pretty much using Excel as a client - the same way you would use a WinForms application or any other tool. The ADO/DAO clients for Access are pretty good, so you probably won't run into any concurrency issues.
That being said, Access does NOT scale well. This works great if you have 2 or 3 (or even 10) users. If you are going to have 100, you'll probably run into problems. Also, I tended to find that Access needed regular maintenance in order to not have corruption issues. Regular backups of the Access DB are a must. Compacting the access database on a regular basis will help prevent database corruption, in my experience.
Any other things I should be aware of?
You're doing this the hard way. Using Excel to hit Access is going to be a lot more work than just using Access directly.
I'd recommend looking into the Access VBA API - most of it is the same as Excel, so you'll have a small learning curve. The parts that are different just make this easier. You'll also have all of the advantages of Access reporting and Forms, which are much more data-oriented than the ones in Excel. The reporting can be great for things like this, and having the Macros and Reports will make life easier in the long run. If the user's going to be using forms to manage everything, doing the forms in Access will be very, very similar to doing them in Excel, and will look nearly identical, but will make everything faster and smoother.
I do this all the time. If you're using ADO, you're not really using Access, but Jet, the underlying database. That means anybody with Excel can use the app - Access not required. Oh I should mention, the place I work bought a bunch of Office Small Business licenses - no Access. Prior to working here, I would have assumed that anyone who had Excel would also have Access. Not so.
I create one class for every table in Access. I very rarely run queries through ADO, instead I keep that logic in the class modules. I read in with a SELECT statement and write out with and UPDATE or INSERT using the Execute method of the ADODB.Connection object.
See http://www.dailydoseofexcel.com/archives/2008/12/21/vba-framework-ii/
if you want to see how I set up my code.
To answer your questions: It will be a small learning curve for you if you already know Excel VBA, but there will be some learning to do; you will pay a performance penalty over doing it all in Access, but it's not that bad and only you can decide if it's worth it; and you can have multiple people accessing the database.
Just skip the excel part - the excel user forms are just a poor man's version of the way more robust Access forms. Also Access VBA is identical to Excel VBA - you just have to learn Access' object model. With a simple application you won't need to write much VBA anyways because in Access you can wire things together quite easily.
If the end user has Access, it might be easier to develop the whole thing in Access. Access has some WYSIWYG form design tools built-in.
Unless there is a strong advantage to running your user form in Excel then I would go with a 100% Access solution that would export the reports and data to Excel on an ad-hoc basis.
From what you describe, Access seems the stronger contender as it is built for working with data:
you would have a lot more tools at your disposal to solve any data problems than have to go around the limitations of Excel and shoehorn it into becoming Access...
As for your questions:
Very easy. There have been some other questions on SO on that subject.
See for instance this one and that one.
Don't know, but I would guess that there could be a small penalty.
The biggest difficulty I see is trying to get all the functionalities that Access gives you and re-creating some of these in Excel.
Yes, you can have multiple Excel users and a single Access database.
Here again, using Access as a front-end and keeping the data in a linked Access database on your network would make more sense and it's easy as pie, there's even a wizard in Access to help you do that: it's just 1 click away.
Really, as most other people have said, take a tiny bit of time to get acquainted with Access, it will save you a lot of time and trouble.
You may know Excel better but if you've gone 80% of the way already if you know VBA and are familiar with the Office object model.
Other advantages of doing it in Access: the Access 2007 runtime is free, meaning that if you were to deploy to app to 1 or 30 PC it would cost you the same: nothing.
You only need one full version of Access for your development work (the Runtime doesn't have the designers).
It really depends on the application. For a normal project, I would recommend using only Access, but sometimes, the needs are specific and an Excel spreadsheet might be more appropriate.
For instance, in a project I had to develop for a former employer, the need was to give access to different persons on forms(pre-filled with some data, different for each person) and have them complete them, then re-import the data.
Since the form was using heavy number crunching, it made more sense to build it in Excel.
The Excel workbooks for the different persons were built from a template using VBA, then saved in a proper location, with the access rights on the folder.
All workbooks were attached as External tables to the workbooks, using named ranges. I could then query the workbooks from the Access Application. All administrative stuff was made from the db, but the end users only had access to their respective workbook.
Developping an Excel/Access application this way was a pleasant experience and the UI was more user-friendly than it would have been using Access.
I have to say that in this case, it would have taken a lot more time doing it in Access than it took using Excel. Also, the Application Object Model seems better though in Excel than in Access.
If you plan to use Excel as a front-end, do not forget to lock all the cells, but the editable ones and don't be affraid to use masked rows and columnns (to construct output tables for the access database, to perform intermediate calculations, etc).
You should also turn off autocalculation while importing data.
It's quite easy and efficient to use Excel as a reporting tool for Access data.
A quick "non programming" approach is to set a List or a Pivot Table, linked to your External Data source. But that's out of scope for Stackoverflow.
A programmatic approach can be very simple:
strProv = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & SourceFile & ";"
Set cnn = New ADODB.Connection
cnn.Open strProv
Set rst = New ADODB.Recordset
rst.Open strSql, cnn
myDestRange.CopyFromRecordset rst
That's it !
Given the ease of use of Access, I don't see a compelling reason to use Excel at all other than to export data for number crunching. Access is designed to easily build data forms and, in my opinion, will be orders of magnitude easier and less time-consuming than using Excel. A few hours to learn the Access object model will pay for itself many times over in terms of time and effort.
I did it in one project of mine. I used MDB to store the data about bills and used Excel to render them, giving the user the possibility to adapt it.
In this case the best solution is:
Not to use any ADO/DAO in Excel. I implemented everything as public functions in MDB modules and called them directly from Excel. You can return even complex data objects, like arrays of strings etc by calling MDB functions with necessary arguments. This is similar to client/server architecture of modern web applications: you web application just does the rendering and user interaction, database and middle tier is then on the server side.
Use Excel forms for user interaction and for data visualisation.
I usually have a very last sheet with some names regions for settings: the path to MDB files, some settings (current user, password if needed etc.) -- so you can easily adapt your Excel implementation to different location of you "back-end" data.
To connect Excel to Access using VBA is very useful I use it in my profession everyday. The connection string I use is according to the program found in the link below. The program can be automated to do multiple connections or tasks in on shot but the basic connection code looks the same. Good luck!
http://vbaexcel.eu/vba-macro-code/database-connection-retrieve-data-from-database-querying-data-into-excel-using-vba-dao
It Depends how much functionality you are expecting by Excel<->Acess solution. In many cases where you don't have budget to get a complete application solution, these little utilities does work. If the Scope of project is limited then I would go for this solution, because excel does give you flexibility to design spreadsheets as in accordance to your needs and then you may use those predesigned sheets for users to use. Designing a spreadsheet like form in Access is more time consuming and difficult and does requires some ActiveX. It object might not only handling data but presenting in spreadsheet like formates then this solution should works with limited scope.
You could try something like XLLoop. This lets you implement excel functions (UDFs) on an external server (server implementations in many different languages are provided).
For example you could use a MySQL database and Apache web server and then write the functions in PHP to serve up the data to your users.
BTW, I work on the project so let me know if you have any questions.