I have no previous experience in Access, VBA coding or in Excel macros prior to teaching myself the past month via these forums. Thank you forums and contributors. I have enjoyed my Access learnings so far, the challenge that it has provided and appreciate any help that I can get. As such, the code and methods that I have used to this point may well be convoluted and confusing. I will do my best to provide relevant details and accurate terminology.
I work in a lab and I am creating an Access Form for semi-automated reporting. Samples are received from clients and are logged into the Excel Table R&D Log. The worksheet is InProcess. Samples are sorted based on the site in which they originate and given a one or two letter site code (G, D, WH, etc.) and an ID "yy-000" in separate Excel columns (i.e. D 18-096). Samples may be submitted for multiple analyses (Metals, Water, Soil, etc.) and may even have multiple rows of reporting if multiple analytes are identified in the sample. There are several other columns, such as receipt date, reporting date, units, etc. Once samples are reported, I manually copy and paste them into the Archived worksheet, and delete the record and blank row from the InProcess worksheet. Since one sample may have multiple analyses and even more potential results, each record would be reported on a new Excel row (with the same D 18-096 ID number). Thus, there is not a single unique identifier or primary key for each sample in the current format. R&D Log is updated manually by lab technicians and the worksheet InProcess is a linked table in an Access Database.
The Access Database is using two combo boxes on a Form frmInProcess to filter a Query qryInProcess of the linked table. The combo boxes are filtering the report destination (one client may receive multiple site codes) and the analysis (reports are separated based on type of analysis). The Query is also filtering out blank results and blank dates, so only completed samples will appear on the filtered Form. I have generated VBA code to this point that will export the Form to a .pdf, save the file with unique filename, and open outlook to mail out the report. I have also managed to export the filtered Form frmInProcess to an Excel file Access Test (not the linked file).
What I would like to do now is to automate the transfer of completed test results from the Excel worksheet R&D Log: InProcess to R&D Log: Archived and delete the record from the InProcess worksheet. I am not sure if I can export the filtered Form into a linked Excel table, or if I must use a separate Excel file (or if it even matters for simplicity of code?). I would now like to read the exported filtered Form in Excel Access Test, lookup matching rows in R&D Log based on several criteria (site, ID, Analysis, Analyte, Report Date) and automate the transfer of records between R&D Log worksheets. End result being that Access generates reports for completed tests, and the records are removed from InProcess testing and transferred to Archived testing in Excel. I am guessing that I may need to close the Access application and perform this in Excel. Hope this is easy enough to follow.
Thank you.
In my experience, importing an Excel document into a temporary NEW (or totally empty) Access table is usually the easiest way to go. Then you do not have to worry about cell references like you do in Excel VBA. Even if the Excel document has old data in it with just a few new changes each time, importing it into a temporary Access table could be the simplest way to go, because then you can compare the data in this table with the data in another, permanent Access table and update the latter based on the former.
As far as the original Excel file, if you need to delete rows there, it might be quicker to export a new Excel file with just the data the old one is supposed to end up with, and then use VBA to delete (or - safer! - rename) the old file.
So the development process goes something like this:
Save import steps by first importing an Excel file via Access' ribbon options "External Data" (tab) ->"Excel" and when you finish, be sure to check the "Save import steps" box and note the name you give the "saved import" because you will need that in your VBA code.
In Access, write a function for deleting the table. The VBA code is:
Const cTable = "MyExcelTempTable"
If TableExists(cTable) Then
DoCmd.DeleteObject acTable, cTable
End If
Now you can test your delete function on the data you imported.
Write VBA code to import the same spreadsheet to create the same table:
Const cSavedImport = "Import-MyExcelTempTable"
' Import the Excel file
DoCmd.RunSavedImportExport cSavedImport
Write more VBA function(s) to check the imported table for bad data and then to copy it into the permanent table. You might be updating existing records or adding new ones. Either way, you could use Access queries or SQL to do this and run them from VBA.
Write a VBA function to rename the old Excel file. (You could use an InputBox if the Excel file name is different each time. I do this for importing Excel files, and I set a default value so I do not have to type as much.)
Write a VBA function to export the new version of the Excel file.
Make yourself a button on a form that, when clicked, runs a VBA function. Inside that function, run Steps 2 through 6, above.
I am not sure my answer exactly matches what you are trying to do, but hopefully you get enough of a picture of the workflow to figure out the details of what you need.
Related
We're looking at allowing our customers to download an Excel file from our web application which contains a raw export of their data along with some basic charts and pivot tables based on that data.
The basic way, we want to make this work is that we have a fixed Excel file which contains all the reporting elements in one worksheet and have room for the underlying data in another worksheet. When the user requests their Excel report, we programmatically fill out the data worksheet with their results and send them the final Excel file.
Everything seemed a bit to easy when doing the prototyping with a fixed set of data. The dataset we worked with was added to the Excel Data Model and we then set up the charts and other reporting elements. However, when using that file as the template for the generated Excel file in our application we are finding that the definition of the data model still remains - meaning, that we built the "protype" with a table definition of $A$1:$T$5879
but when generating the report, that definition isn't changed to contain whatever size the added dataset might have.
We're using EPPlus to work with the generation of our Excel sheets and have so far been unable to find any sort of solution to this kind of problem. This might very much be due to us being quite Excel novices. The goal is to have the user experience, that the charts and pivot tables contained in the Excel sheet reflects the total dataset contained in the Excel file without them having to do anything.
Ok, I've actually found a solution for it.
The solution was right infront of us.
We define the dataset as a named set - this is done under under the "Formulas" and inside the "Name Manager". We have a range which defines our dataset - the "Refers To" field when defining a range can take a formula. So intead of giving it a fixed size, we use this: =OFFSET(Data!$A$1;0;0;COUNTA(Data!$A:$A);COUNTA(Data!$1:$1))
This counts the amount of rows and columns, with reference to A1 in our Data worksheet. All our pivots are set to reload on startup and that seems to work.
I have a master spreadsheet that we use for tracking a certain activity. The spreadsheet contains various lines with item descriptions, and open columns where end users are to fill in dates.
An updated master spreadsheet goes out to various end users to update date values and return it to the coordinator. The coordinator then takes the various feedback sheets, and updates the master accordingly.
Up to now, this has been quite a labor-intensive task as end-users change filtering settings before sending it back, and there are 1000+ lines, so it is hard to see what the end-users updated.
Does someone know of an effective method to do a task like this, using Excel directly or VBA?
I would recommend creating a server version of the master spreadsheet so that end-users could check out that document,fill in their dates and avoid the redundancy of having someone retype everything. Sharepoint's ideal for this kind of thing.
If you want to keep it to Excel alone, though, perhaps you could save all the modified user-input-provided workbooks to a designated folder. Then write a macro to loop through each of these files in the folder. Supposing that the cells the users fill/inputting data into were previously empty, check to see if they're now not empty and copy those changes to the master file.
In my Access 2013 Desktop database, I want to validate an Excel form through VBA before importing its rows to a table.
The problem is that the imported Excel table may contain in the same column values that are numeric, string, null, etc. depending on the row due to mistakes from the user, so these values have to be initially of VBA type Variant before validation.
If the row is invalid (e.g. one value is not of numeric type while it should be, one mandatory field is null, etc.), I want to be able to store its values and let the user correct it later on. I am therefore looking for a way to build a table where I could store these rows, but where the fields are kept intact, so with the possible mixed data types.
This seems not to be possible, as Access tables need apparently to have fields of a defined type, so nothing equivalent to Variant.
How could I achieve this? What would be the simplest and most efficient way to store rows of Variant values while keeping the structure of the columns? The only way I could think of so far is to concatenate these values for each row in a string using some defined separator, and store this in a String column in an Access table, but maybe there is an easier way to do?
Thanks a lot for your help.
EDIT: Reading your answers I realized that I forgot to specify that the people filling and sending the Excel forms (who typically do not have Access on their computers) are not the same users than the ones who have the Access database and need to import the data. In this configuration it is impossible to use directly Access forms to input data, which would of course simplify tremendously the task. While we try to enforce rules for people who fill the Excel form at the first place there will be mistakes and the Access application needs to cope with them. And the importing process for the users manipulating Access should be as simple as possible (these users are not experts in IT).
I though about your problem for a while and made a few notes as I thought below.
I would strongly advise that unless the excel workbook is providing other functionality that you do not use excel, but create an simple Access database with a form they can use to interact with "their" data. They do not necessarily need an Access license to do this. Access 2013 even has new web features that can be used if you have sharepoint to collect data via a Access created web frontend.
If you are to collect data in Excel, then you must use excel validation and VBA code to validate the data as much as possible before transferring to access.
There are complexities that may or may not be an issue for you, things like:
Can users edit/created/delete the same rows concurrently - what happens if they do! You may need to "lock" rows when they are downloaded from Access. But what happens if tow users add the same record, or ones deletes a row before another user commits it back again.
Can a user open multiple excel files and edit the data they store without committing the data to Access?
Can multiple users login to the same Workbook with uncommitted changes and edit the data.
Using access will simplify your code as it will be able to prevent erroneous data being entered and remove the need to deal with the above issues and others
In summary you are using excel as a front end "form" to data stored in Access.
Each excel file can have data that has yet to be up
I would suggest that the primary key of a row is the "path and name" of the excel file that was used to create it & a unique numeric identifier. The unique identifier is a counter that is maintained for each excel file. The "path and filename" could be replaced by a unique identifier created for each file.
Many users enter data using multiple instances of Excel into a form which results in one or more rows being updated/inserted/deleted in an Excel table stored in each spreadsheet.
I would expect that whenever Excel is opened the user enters a username and excel will grab "their data" from Access. Alternatively a workbook might be set up for each department or "case type" and only interact with data that matches this "custom criteria set up in the workbook". Excel would not necessarily need to store data itself longterm. The workbook might always save data back to the database when it is saved.
You say Excel VBA performs limited validation on the data (but no complete validation). It should be used as much as possible and arguably should be able to do exactly the same validation that access can do. At the very least it should enforce datatypes etc (eg using the standard data validation rules on the excel data menu) or perhaps use VBA/controls to get any access data it needs to validate the data entered.
After updating the data the user can "commit" (ie save) the data to the access database. Before closing the workbook you might want to commit the data.
(An issue is whether a user might open many workbooks perhaps on many machines without committing the data.)
An Access "staging" table can be created with all columns having the datatype "shorttext" and not-required.
The process that loads data from an excel table into Access, will uploaded all excel rows into this staging table. It will then validate each row in the staging table and process all valid data into the main table(s) that have data types, relationships constraints etc.. Any valid rows in the staging table are flagged as "VALIDATED & TRANSFERRED", Invalid rows are flagged as INVALID. The "VALIDATED AND TRANSFERRED ROWS" are subsequently ignored, but kept so you can check what processing has happened, perhaps only whilst testing.
The data in excel is then updated with the Valid/Invalid status from Access and suitable messages given to the excel user. The user may correct and then re-commit the data.
Each excel file has a status of "changed/Unchanged" to indicate whether a user has changed data in the file.
When a user opens an excel workbook and status is unchanged it will refresh it's data. If it's changed a refresh probably can't be done.
In order for the data in Access to be updated/deleted/inserted with the changes made in Excel, there will need to be a unique identifier for each row that cannot be changed by the users in excel. This is likely to be the Path&Filename or the UserName logged into Excel and a numeric counter (which is maintained for each file or user). (This assumes that the user will have to commit changes in one excel file before they work on another.)
Anyway, without knowing more it's difficult to fully design what you will need, but I hope these thoughts help you
Harvey
My Access programming is a little rusty, & I've never worked with Excel files all that much.
I have a requirement to bring data from Excel spreadsheets into Access 2007. These spreadsheets have a fixed (predictable) format, but it includes a "header area" where I need to read single data items from specific cells, followed by a mass of tabular data (~500 rows in the one sample I've seen so far). I will be processing all of this into a set of tables that are normalized quite differently from the flat layout of the spreadsheet.
I know how to open an ADO recordset on the tabular data, and it should work fairly well for my purposes. I also figure that I can reference the Excel object model and open the sheets through Automation to get the "header area" data items.
My question is this: since I have to (I think) use the Automation approach for the "header area", am I better off just leaving it open in this mode to move on to the tabular data (with cell/range navigation), or closing that mode & going over to ADO? I suspect it's the latter--and I'd be more comfortable with it--but I don't want to do the wrong thing just because it's more familiar.
Edit
It seems I wasn't clear that I need to build this capability into the "application", as something that a user can repeat down the line. I'm assured that I can trust the format of the spreadsheet (though I'll include error trapping for graceful failure if that turns out to be false). These spreadsheets are "official design documents" for hardware, and my app needs to handle bringing in new &/or updated ones to track the things that are described in the tabular data in ways that the flat Excel format diesn't allow for.
Of those two options, I would choose the second simply because I find it more convenient to work with an ADO recordset. It should be fairly simple if you can assign a named range to your spreadsheet's tabular data.
Edit: If your spreadsheet includes field names, the recordset approach would be less prone to break due to spreadsheet changes such as one or more new columns inserted before or between the existing columns or a re-ordering of the existing columns.
But actually, I think the TransferSpreadsheet Method might be more convenient. You can specify the spreadsheet range as a named range or by cell address as in this example from the linked page:
DoCmd.TransferSpreadsheet acImport, 3, _
"Employees","C:\Lotus\Newemps.wk3", True, "A1:G12"
Also, you can choose between importing the spreadsheet range directly into an Access table, or linking to the range as a "virtual" table ... whichever best meets your application's needs.
Edit2: Creating a link (acLink instead of acImport) with TransferSpreadsheet would allow you to execute SQL statements against the link table:
INSERT INTO DestinationTable (field1, field2, field3)
SELECT foo, bar, bat FROM LinkedTable;
If the header information is really complicated, this can simplify your coding work:
In the official design Excel file, create a hidden tab.
In that tab, make a 1-row table connecting to all the header elements you're interested in. (i.e. set row 1 column 1 to "Document#" and row 2 column 1 to Sheet1:A1)
Then you can re-use the same VBA procedure to import both your tabular data and your header data.
I would do it all via Automation. Why have two separate processes where one will do? After you've read the header information reading the tabular information will be quite easy.
I inherited an application back in mid-2000 that was built to import Excel spreadsheets that were basically reporting output from MYOB (an accounting program). What had been done was to simply create a template table that had all the columns necessary to accomodate the report, using text data type for all columns. Then the non-data rows were filtered out and processed into the eventual destination table.
It's not elegant, and doesn't require a lot of programming, though the implementation I inherited used a dedicated temp table for each report layout that was being imported. You could easily replace all of those with a single table with 100 text columns of 255 (or memo fields, for that matter, if that was a requirement), and just re-use it.
I'm not sure if I'd recommend it or not, but it really is quite easy without requiring much in the way of code.
This question is long winded because I have been updating the question over a very long time trying to get SSIS to properly export Excel data. I managed to solve this issue, although not correctly. Aside from someone providing a correct answer, the solution listed in this question is not terrible.
The only answer I found was to create a single row named range wide enough for my columns. In the named range put sample data and hide it. SSIS appends the data and reads metadata from the single row (that is close enough for it to drop stuff in it). The data takes the format of the hidden single row. This allows headers, etc.
WOW what a pain in the butt. It will take over 450 days of exports to recover the time lost. However, I still love SSIS and will continue to use it because it is still way better than Filemaker LOL. My next attempt will be doing the same thing in the report server.
Original question notes:
If you are in Sql Server Integrations Services designer and want to export data to an Excel file starting on something other than the first line, lets say the forth line, how do you specify this?
I tried going in to the Excel Destination of the Data Flow, changed the AccessMode to OpenRowSet from Variable, then set the variable to "YPlatters$A4:I20000" This fails saying it cannot find the sheet. The sheet is called YPlatters.
I thought you could specify (Sheet$)(Starting Cell):(Ending Cell)?
Update
Apparently in Excel you can select a set of cells and name them with the name box. This allows you to select the name instead of the sheet without the $ dollar sign. Oddly enough, whatever the range you specify, it appends the data to the next row after the range. Oddly, as you add data, it increases the named selection's row count.
Another odd thing is the data takes the format of the last line of the range specified. My header rows are bold. If I specify a range that ends with the header row, the data appends to the row below, and makes all the entries bold. if you specify one row lower, it puts a blank line between the header row and the data, but the data is not bold.
Another update
No matter what I try, SSIS samples the "first row" of the file and sets the metadata according to what it finds. However, if you have sample data that has a value of zero but is formatted as the first row, it treats that column as text and inserts numeric values with a single quote in front ('123.34). I also tried headers that do not reflect the data types of the columns. I tried changing the metadata of the Excel destination, but it always changes it back when I run the project, then fails saying it will truncate data. If I tell it to ignore errors, it imports everything except that column.
Several days of several hours a piece later...
Another update
I tried every combination. A mostly working example is to create the named range starting with the column headers. Format your column headers as you want the data to look as the data takes on this format. In my example, these exist from A4 to E4, which is my defined range. SSIS appends to the row after the defined range, so defining A4 to E68 appends the rows starting at A69. You define the Connection as having the first row contains the field names. It takes on the metadata of the header row, oddly, not the second row, and it guesses at the data type, not the formatted data type of the column, i.e., headers are text, so all my metadata is text. If your headers are bold, so is all of your data.
I even tried making a sample data row without success... I don't think anyone actually uses Excel with the default MS SSIS export.
If you could define the "insert range" (A5 to E5) with no header row and format those columns (currency, not bold, etc.) without it skipping a row in Excel, this would be very helpful. From what I gather, noone uses SSIS to export Excel without a third party connection manager.
Any ideas on how to set this up properly so that data is formatted correctly, i.e., the metadata read from Excel is proper to the real data, and formatting inherits from the first row of data, not the headers in Excel?
One last update (July 17, 2009)
I got this to work very well. One thing I added to Excel was the IMEX=1 in the Excel connection string: "Excel 8.0;HDR=Yes;IMEX=1". This forces Excel (I think) to look at all rows to see what kind of data is in it. Generally, this does not drop information, say for instance if you have a zip code then about 9 rows down you have a zip+4, Excel without this blanks that field entirely without error. With IMEX=1, it recognizes that Zip is actually a character field instead of numeric.
And of course, one more update (August 27, 2009)
The IMEX=1 will succeed importing data with missing contents in the first 8 rows, but it will fail exporting data where no data exists. So, have it on your import connection string, but not your export Excel connection string.
I have to say, after so much fiddling, it works pretty well.
P.S. If you are using a x64 bit version, make sure you call the DTExec from C:\Program Files\Microsoft SQL Server\90\DTS.x86\Binn. It will load the 32 bit Excel driver and work fine.
Would it be easier to create the Excel Workbook in a script task, then just pick it up later in the flow?
The engine part of SSIS is good but the integration with Excel is awful
"Using SSIS in conjunction with Excel is like having hot tar funnelled up your iHole in a road cone"
Dr. Zim, I believe you were the one that originally brought up this question. I totally feel your pain. I love SSIS overall, but I absolutely hate the limited tools that come standard for Excel. All I want to do is Bold the Heading or Row1 record in Excel, and not bold the following records. I have not found a great way to do that; granted I am approaching this with no script tasks or custom extensions, but you would think something this simple would be a standard option. Looks like I may be forced to research and program up something fancy for a task that should be so fundamental. I've already spent a rediculous amount of time on this myself. Does anyone know if you can use Excel XML with Excel versions: 2000/XP/2003? Thanks.
This is an old thread but what about using a flat file connection and writing the data out as a formatted html document. Set the mime type in the page header to "application/excel". When you send the document as an attachment and the recipient opens the attachment, it will open a browser session but should pop Excel up over the top of it with the data formatted according to the style (CSS) specified in the page.
Can you have SSIS write the data to an Excel sheet starting at A1, then create another sheet, formatted as you like, that refers to the other sheet at A1, but displays it as A4? That is, on the "pretty" sheet, A4 would refer to A1 on the SSIS sheet.
This would allow SSIS to do what it's good for (manipulate table-based data), but allow the Excel to be formatted or manipulated however you'd like.
When excel is the destination in SSIS, or the target export type in SSRS, you do not have much control over formatting and specifying how you want the final file to be. I have written a custom excel rendering engine for SSRS once, as my client was so strict about the format of final Excel report generated. I used 'Excel xml' to get the job done inside my custom renderer. May be you can use XML output and convert it to Excel XML using XSLT.
I understand you would rather not use a script component so perhaps you could create your own custom task using the code that a script contains so that others can use this in the future. Check here for an example.
If this seems feasible the solution I used was CarlosAg Excel Xml Writer Library. With this you can create code which is similar to using the Interop library but produces excel in xml format. This avoids using the Interop object which can sometimes lead to excel processes hanging around.
Instead of using a roundabout way to do this exercise of trying to write data to particular cell(s), format the cell(s), style them which is indeed a very tedius effort considering the support SSIS has for EXCEL, we could go the "template" way to do this.
assume we need to write data in the so & so cell with all the custom formating thats done on it. Have all the formatting in a sheet, say "SheetActual", Whereas the cells that will hold the data will actually have Lookups/ refrences/ Formulaes to refer to the original data that SSIS exports in a hidden sheet say "SheetMasterHidden" of the same Excel connection. This "SheetMasterHidden" will essentially hold the master data in default format that SSIS writes data to the excel. This way you need not worry about formatting the data runtime.
Formatting the Excel is a one time work "IF" the formatting dont change very often. If the format changes and the format is decided runtime this solution maynot go very well.
The answer is in the question. Over time, it became a progress status. However, there is SSRS that will create Excel files if you create TABLE presentations. It works pretty well too.