I have to import the data from excel to SQL database. Before inserting data I have to apply some business rules. So what is best approach.
1. Should I read all the excel and put it in collection and the apply business rules then
Or
1. I read excel, get the record , apply the business rule and insert the record in database
I would put the data without applying any rules into the database. Afterwards you can easily apply the business rules using SQL.
Although this approach assumes:
Your data is well-enough formatted. Maybe one cell is not equivalent to one field? Or maybe the Excel file was edited by several people following different editing conventions. From my experience it's much easier to do this data cleaning before the import to SQL.
You are able to apply your business logic using SQL. After all you need to be able to realize it and if you are a better VBA programmer than SQL or PL/SQL programmer, then you should probably apply the business rule before inserting everything into the database.
Related
Building a BI system (dashboards) in Excel using imported tables (from excel files). We're using Excel 2016 query, data model, measures using DAX expressions, resulting in more pivot tables (some of which are reloaded into data model), etc.
My question: is there "best practice" on 1) naming these data elements and 2) documenting these bits to have a more complete system documentation.
Background: I'm the senior "hacker" munging these things together. But I need to move this towards being sustainable. I did some prototyping work and when I went back a week later it was challenging to reconstruct my thoughts and relationships...
I've seen folks refer to use of PowerBI flow diagrams to support documentation; but it seems to be more of the "icing on the cake" than the "cake" itself.
So what "bread and butter" documentation approaches have you, more experienced developers, taken to ensure that your systems are clearly documented so that others can pick up where you left off???
For naming, I follow the Kimball Group's advice for data warehouses/marts, e.g.
https://www.kimballgroup.com/2014/07/design-tip-168-whats-name/
I rename many/most Query steps to reference the column or table name, e.g. Added Custom => Added Customer Name, Append Queries => Append Customers. The idea is to be able to pick the right step first time when coming back for maintenance.
You can select all the Queries in the Query Editor window and copy their code, then paste it into Word etc as the starting point for your documentation. You can also screen-shot the Query Editor's Query Dependancies pop-up.
For the Power Pivot logic, try this solution:
https://powerpivotpro.com/2014/03/automatically-create-data-dictionary-for-your-power-pivot-model/
I have to work with data CSV file. They look like this
sample
It represents products with options/cars etc. at the web-store.
It has a lot of columns with duplicated values and in my work in often need to copy some part of this data to another sheet, deduplicate it, edit and then paste it back by matching it for one of the columns that were untouched. More this purpose I'm using Ablebits Excel suit.
Is it possible by any excel function to automate this process or maybe there is some other software that could handle this? Something not so complicated as relational databases like Access, but something close to spreadsheet editor with relationships
I already tried Power Query in Excel and Power Bi, but they seem to be more analytics tools and not the data edit
2nd edition:
Data has a layer structure with duplicates.
Title1|Part number 1|Car1
Title1|Part number 1|Car2
Title2|Part number 2|Option1
Title2|Part number 3|Option2
I want to have opportunity to:
Edit values that duplicate without using "Replace All" or at least have more flexible "Find&Replace".
Extract columns with deduplicating them and saving a reference to the place they were taken. So if you edit some data there it was changed in the 1st place. For example, I have titles(a lot of titles) but need to edit it. Instead of copying it with some id to reference it I want to open it like they appears in filters, edit it, confirm and get it edited in all column
I would use Power Query (aka Get & Transform on the Data ribbon in Excel 2016). The only limitation I see with what you want to do is that Power Query will deliver a new Excel Table with the output of a Query - it can't update existing cells.
If you can get past that, Power Query is very flexible, easy to learn (WYSIWYG query editor), scales well and is integrated with other Microsoft products (as well as Power BI, there is integration with SQL Server Analysis Services in preview and hopefully SQL Server Integration Services one day).
In powerpivot, Related(Othertable[field]) retrieves the associated column from a related table.
I would like to import ALL such columns, doing the equivalent of a join.
Is it possible to do this ?
nicolas,
the smartest thing to do from my perspective is to merge your queries into one so that you can keep your original tables.
I would suggest using new PowerQuery Merge funcionality, which is very easy and works reliably (and also supports loading data directly into your PowerPivot data model).
Or you can write you custom Query in PowerPivot - if you use MSSQL (or any other) database as your source, you can actually use JOIN directly in the PowerPivot window with Table Import Wizard that makes things a bit easier.
So the answer is: keep your original data tables intact, and create a new one that will be merging them together just for the purpose of your desired report.
Hope this helps.
We need to upload a small amount of additional records to a table from an Excel sheet. Is there a way to use the Access Import function to add the additional data to the table (truncate it). The table was created by uploading the same Excel sheet. But now, when records are added, we need to add them to the table. The tables are linked to SQL but I do not want to use an SSIS because there are only a few records and there must be a way to use Access functions. Suggestions please.
It may be easiest to link the excel sheet and run an append query to add data from Excel to existing table. Once linked, this can be done in the query design window.
You did not specify versions of Excel or Access.
I did this with a test 2003 Excel sheet with cells containing 1000+ characters. An import in Access 2003 detects the data type as a memo field, which is correct, when there are that many characters, so it should work for you. It may be your Excel data has other ingredients causing an import issue. How is the excel data derived?
Have you tried importing to Access? It should work fine. If your ultimate target is another database why use Access as an intermediary?
I agree a linked table seems like a really simple method to update a table if you are using Access, but that is your choice.
I want to import data from Excel into corresponding tables based on different column data's on based on ID's like customer data on based on CustomerID present in Customer table.
Means we have to extract data from the table and Excel source on basis of ID's.
Could you please help me out on this?
Use the SQL Server Data Import Wizard - see an article on it here.
(source: databasedesign-resource.com)
This wizard allows you to define your Excel file to import, it allows you to define the target where to put the data, it allows you to define mappings between columns in Excel and columns in your SQL table, and much more.
Update: based on your comment to the other answer, if you need to import the Excel sheet and match it up to some pre-existing lookup data, then you should definitely look at the SQL Server Integration Services (SSIS) which are there exactly for this kind of import/lookup scenario.
Your question's gamma is a bit all over the place so not entirely sure what you are asking about but here goes.
You can save you excel spreadsheet as a CSV file and then import that into your database. There a number of tutorials on this if you search google. Try searching "import CSV into database".