I have a system that is using PQ to import various CSV files. In those CSV files each row has a StaffID that uniquely identifies each member of staff. I have another sheet which has all current Staff details so I can match the import data to the existing staff by linking on the StaffID field.
If a new member of staff joins the company, I would only get notification as an unknown StaffID would appear in the CSV files I am importing. Is the correct approach here to either a) write a PQ to find StaffIDs in the new CSV files that don't match the master list in Excel or b) run the import into Excel and process those 'orphan' rows in Excel ie StaffID's in the new import that don't have a matching StaffID in the Master List Excel sheet?
I'm looking for any advice on 'best practice' here. I could solve this using only Excel / VBA but I am new to PQ so I and trying to find more about how it can be used and in what scenarios.
Thanks in advance for any feedback.
Expected result: Advice on a 'best practice' or sensible approach to use Excel and PQ to manage the handling of 'orphaned' or data without a matching PK.
The best approach to finding deltas like this is a simple outer join and then remove the matching rows. It will highlight rows which exist on the left side but don't exist on the right and vice versa. You can then load that to a table either in Excel or PowerBI with an alert.
Related
I have 1000 excel workbooks and I have to summarize data in one excel workbook. Each workbook consists of data of one property (id of property, region, market value etc.) In the summary workbook I want to insert in a column the id of property and automatically search across the file of workbooks and insert the value for its region, market value etc.
Thanks
I am going to first assume that there is some structure to the workbooks you are discussing, for example all market value workbooks are the same but only the data is different. it would also be great if they were in different folders and if they were named in a consistent way.
This is only an example with links to different content that will help solve your problem. Your question is too vague and does not provide me with specific enough information to provide exact solution.
Combining records.
https://www.youtube.com/watch?v=rSQwZ1d3b1g&list=PLrRPvpgDmw0m3ohSvgwoHvd0KO8QsQdiK&index=5&t=0s
Merge Records.
https://www.youtube.com/watch?v=8F7v6YvnsiY&list=PLrRPvpgDmw0m3ohSvgwoHvd0KO8QsQdiK&index=6
Pivot Table Slicer for summary reports.
https://www.youtube.com/watch?v=zgt7SdrYJqg
Your best bet would be to use the power query feature to combine data from workbooks inside a folder
From DATA TAB -> Select GET DATA drop down from Get & Transform ribbon -> From File -> From Folder.
You should be able to combine all the workbooks in folder If your problem is simple.
If each workbook has multiple pages of info and the pages differ then you will need to create a custom function in power query to handle this issue.
If workbooks are all in one folder then you will need to run power query multiple times per condition and use a filter with power query assuming that the files have some kind of organized naming convention.
Once you have combined files and cleaned up the data you should be able to combine the records.
You will do this using the power query feature merge feature.
From DATA TAB -> Select GET DATA drop down from Get & Transform ribbon -> Combine Queries -> Merge.
Once records are all merged you should be able to have your final table and use a pivot table to give you the option to select ID with filters or pivot table slicers so that you can have the summary report by ID going across all the workbooks like you wanted.
If anything is unclear please provide more info so i can help you specifically
I have no previous experience in Access, VBA coding or in Excel macros prior to teaching myself the past month via these forums. Thank you forums and contributors. I have enjoyed my Access learnings so far, the challenge that it has provided and appreciate any help that I can get. As such, the code and methods that I have used to this point may well be convoluted and confusing. I will do my best to provide relevant details and accurate terminology.
I work in a lab and I am creating an Access Form for semi-automated reporting. Samples are received from clients and are logged into the Excel Table R&D Log. The worksheet is InProcess. Samples are sorted based on the site in which they originate and given a one or two letter site code (G, D, WH, etc.) and an ID "yy-000" in separate Excel columns (i.e. D 18-096). Samples may be submitted for multiple analyses (Metals, Water, Soil, etc.) and may even have multiple rows of reporting if multiple analytes are identified in the sample. There are several other columns, such as receipt date, reporting date, units, etc. Once samples are reported, I manually copy and paste them into the Archived worksheet, and delete the record and blank row from the InProcess worksheet. Since one sample may have multiple analyses and even more potential results, each record would be reported on a new Excel row (with the same D 18-096 ID number). Thus, there is not a single unique identifier or primary key for each sample in the current format. R&D Log is updated manually by lab technicians and the worksheet InProcess is a linked table in an Access Database.
The Access Database is using two combo boxes on a Form frmInProcess to filter a Query qryInProcess of the linked table. The combo boxes are filtering the report destination (one client may receive multiple site codes) and the analysis (reports are separated based on type of analysis). The Query is also filtering out blank results and blank dates, so only completed samples will appear on the filtered Form. I have generated VBA code to this point that will export the Form to a .pdf, save the file with unique filename, and open outlook to mail out the report. I have also managed to export the filtered Form frmInProcess to an Excel file Access Test (not the linked file).
What I would like to do now is to automate the transfer of completed test results from the Excel worksheet R&D Log: InProcess to R&D Log: Archived and delete the record from the InProcess worksheet. I am not sure if I can export the filtered Form into a linked Excel table, or if I must use a separate Excel file (or if it even matters for simplicity of code?). I would now like to read the exported filtered Form in Excel Access Test, lookup matching rows in R&D Log based on several criteria (site, ID, Analysis, Analyte, Report Date) and automate the transfer of records between R&D Log worksheets. End result being that Access generates reports for completed tests, and the records are removed from InProcess testing and transferred to Archived testing in Excel. I am guessing that I may need to close the Access application and perform this in Excel. Hope this is easy enough to follow.
Thank you.
In my experience, importing an Excel document into a temporary NEW (or totally empty) Access table is usually the easiest way to go. Then you do not have to worry about cell references like you do in Excel VBA. Even if the Excel document has old data in it with just a few new changes each time, importing it into a temporary Access table could be the simplest way to go, because then you can compare the data in this table with the data in another, permanent Access table and update the latter based on the former.
As far as the original Excel file, if you need to delete rows there, it might be quicker to export a new Excel file with just the data the old one is supposed to end up with, and then use VBA to delete (or - safer! - rename) the old file.
So the development process goes something like this:
Save import steps by first importing an Excel file via Access' ribbon options "External Data" (tab) ->"Excel" and when you finish, be sure to check the "Save import steps" box and note the name you give the "saved import" because you will need that in your VBA code.
In Access, write a function for deleting the table. The VBA code is:
Const cTable = "MyExcelTempTable"
If TableExists(cTable) Then
DoCmd.DeleteObject acTable, cTable
End If
Now you can test your delete function on the data you imported.
Write VBA code to import the same spreadsheet to create the same table:
Const cSavedImport = "Import-MyExcelTempTable"
' Import the Excel file
DoCmd.RunSavedImportExport cSavedImport
Write more VBA function(s) to check the imported table for bad data and then to copy it into the permanent table. You might be updating existing records or adding new ones. Either way, you could use Access queries or SQL to do this and run them from VBA.
Write a VBA function to rename the old Excel file. (You could use an InputBox if the Excel file name is different each time. I do this for importing Excel files, and I set a default value so I do not have to type as much.)
Write a VBA function to export the new version of the Excel file.
Make yourself a button on a form that, when clicked, runs a VBA function. Inside that function, run Steps 2 through 6, above.
I am not sure my answer exactly matches what you are trying to do, but hopefully you get enough of a picture of the workflow to figure out the details of what you need.
I have written some pretty lengthy VBA code in excel for the comparison of 2 worksheets. My code does the following:
Lets you import 2 sheets for comparison
arranges the columns
removes departments which require different comparisons into a new worksheet
In sheet 1 checks if the id's appear more than once then checks, which row of data to use for comparison based on the latest update, and deletes the old rows
compares the sheets based on the header and then the cell contents as header names are different, for different values it then highlights them red
finally giving me a breakdown per column per department of differences and any id's that are missing
I have now found that my data set is becoming to big and looking to use MS Access, is it possible to copy my VBA code over to access? What do you guys suggest for this?
Any advice would be helpful.
From the nature of your question it sounds like you may not have used a database before. If you were using access, you would need to totally re-write the code using SQL statements. eg An Aggregating SQL SELECT statement to find the most recently updated update and ignore the rest.
You can use conditional formatting in an access form, but it's no better than using it in excel. How many rows does your data have? Will it fit in an excel sheet?
You might use access to pre-process the data to remove the unwanted rows that you use in excel. OR use power query or sql directly from excel to remove them.
You have a way to go.
Harvy
We need to upload a small amount of additional records to a table from an Excel sheet. Is there a way to use the Access Import function to add the additional data to the table (truncate it). The table was created by uploading the same Excel sheet. But now, when records are added, we need to add them to the table. The tables are linked to SQL but I do not want to use an SSIS because there are only a few records and there must be a way to use Access functions. Suggestions please.
It may be easiest to link the excel sheet and run an append query to add data from Excel to existing table. Once linked, this can be done in the query design window.
You did not specify versions of Excel or Access.
I did this with a test 2003 Excel sheet with cells containing 1000+ characters. An import in Access 2003 detects the data type as a memo field, which is correct, when there are that many characters, so it should work for you. It may be your Excel data has other ingredients causing an import issue. How is the excel data derived?
Have you tried importing to Access? It should work fine. If your ultimate target is another database why use Access as an intermediary?
I agree a linked table seems like a really simple method to update a table if you are using Access, but that is your choice.
I want to import data from Excel into corresponding tables based on different column data's on based on ID's like customer data on based on CustomerID present in Customer table.
Means we have to extract data from the table and Excel source on basis of ID's.
Could you please help me out on this?
Use the SQL Server Data Import Wizard - see an article on it here.
(source: databasedesign-resource.com)
This wizard allows you to define your Excel file to import, it allows you to define the target where to put the data, it allows you to define mappings between columns in Excel and columns in your SQL table, and much more.
Update: based on your comment to the other answer, if you need to import the Excel sheet and match it up to some pre-existing lookup data, then you should definitely look at the SQL Server Integration Services (SSIS) which are there exactly for this kind of import/lookup scenario.
Your question's gamma is a bit all over the place so not entirely sure what you are asking about but here goes.
You can save you excel spreadsheet as a CSV file and then import that into your database. There a number of tutorials on this if you search google. Try searching "import CSV into database".