I have to import excel to sql 2005 with a win7 os x86 it does not support DTS,SSIS,anything Import, Export is also not present in sql 2005 and there are millions of records that too in a very different rows like say after 50 records the cell breaks into 1 or 2 gives the subheadline and again too many records so there are millions of records how should i do it?
Consider posting a sample of the data for a better answer.
Simply, you should force the data into normalised tables, and then use your favourite tool to push them into SQL Server. If you have no additional tools, then you can use the Bulk copy (bcp) utilities to move large volumes of delimited or fixed format data into tables - fast.
You can, either:
Export the workbooks into text files and process them with scripts into the form you require, or
Use VBA or alternative automation techniques in Excel to generate the normalised tables to import into the database
You can learn about the process in Database normalization.
Related
I have around half a dozen Powerpivot files containing data extracted from an SQL database. Each file has around 30 million lines of data in an identical format. I do not have access to the underlying database but each powerpivot file contains the data.
I would like to connect to all of these files in one new workbook using Powerquery so that I can append them, add them to the data model and work with them in Excel.
I have found various solutions on how to get the data into CSV format using DAX studio but I would prefer to avoid this as it seems an unwieldy solution to export hundreds of millions of lines of data to CSV and then import it back to Powerquery when I already have the formatted data in Powerpivot.
I also don't have any experience of using SQL so would prefer to avoid that route.
I've tried creating a linkback as described here https://www.sqlbi.com/articles/linkback-tables-in-powerpivot-for-excel-2013/ but when I connect to this it only returns 1,048,576 lines of data (i.e. what Excel is limited to).
Is there an option for Powerquery to use Powerpivot data in a separate workbook as a source or another straightforward solution?
Thanks
You can either materialise the data (which you've tried, using Linkback tables), or you can copy the queries. There's no other way to reference powerpivot model data.
I have an excel sheet with a single workbook with data in it. The data is around 1000 rows and 50 columns. I need to import these data to an Oracle DB every week. Here comes the problem, the columns in the sheet belongs to different tables with some columns go in multiple tables. I use SQL Developer V.18.1.0.095. Thanks in advance for the help.
Note: I created a temp table and copied all data to it, then wrote the query to push each column to its respective tables. But, I feel its complex and think it won't work. Is there any better way.
PL/SQL Developer has special tool for tasks like this, calls ODBC Importer (Menu 'Tools'-> ODBC Importer).
For use it you have to set Excel File in USER / System DSN field and your domain user and password, and push Connect after.
After connection developer will ask you path of excel file, and after you can create table in heiborhood tab for your dataset.
Or, you can use sql loader. Ask google how to. It's easy.
I have Excel file which has about a million records in one Sheet. When I open in Excel it can't load properly. How do I open this record? Can other software to open this data properly?
10 Lakh records means 1 million records, to open such a file you need a really good processor and good RAM, I use a Octa Core machine with 32GB RAM which has no issues opening such large files
Excel has a limit of round about 1 million rows.You Cannot load more than a million rows in Excel.
If you wish to perform this kind of analysis in Excel, then you can use Power Query (Free add-in for Excel 2010 & 2013, but included in Excel 2016 as Get & Transform).
Power Query is a Microsoft Development and is part of Excel from 2016 onwards..
With Power Query, it loads millions of rows into an internal database engine. and you perform analysis generally by pivotting. You will not be able to see the data in a worksheet because the limit of 1048K still exists, but any aggregation analysis is possible.
Background:
I currently run reporting on a monthly basis from a source csv file that's roughly 50 columns by 15k rows. I have an existing system where I am importing the data into sql, using multiple stored procedures to handle the data transformations, then using excel connections to view the reports in excel after the data transformations. These transformations are relatively complex and comprise of ~4 stored procedures at ~5 pages and around 200 lines of code each.
Problem:
The amount of code and tables in sql to handle the transformations is becoming overwhelming. QA is a pain in the ass to track trough all the tables and stored procedures to find out where the problem lies. This whole process including extensive QA is taking me 3 days to complete, where ideally I'd like to to take half a day total. I can run through all the stored procedures and excel connections/formatting in a few hours, but currently it's more efficient to run QA after every single step.
Potential Solutions:
Would integrating SSIS help the automation and QA process?
I am new to SSIS, so how does data transformations work with SSIS?
Do I just link a stored proc as a step in the SSIS flow?
Note: I should specify that the results need to be displayed in Excel on a heavily formatted worksheet. Currently, there is a feeder sheet in excel that fetches data from SQL views, and the report page has formula links to that feeder sheet.
I appreciate all the help in advance.
I've done something similar and partially converted a SQL stored procedure (SP) solution to SSIS.
You can call existing SPs using the SSIS Execute SQL Task, so I would start with an SSIS "Master" package that just executes all your SPs. Right out of the box that gives you control over dependancies, parallel execution, restartability & logging.
Then I would incrementally chip away at replacing the SPs with SSIS Data Flow Tasks - this opens up the full range of SSIS transformation capabilities and is almost always a lot faster to build and run than SPs.
I would replace the Excel layer with an Report Services Report, but this would probably be a lower priority.
I have loads of excel files with huge amount of numbers and graphs and formulas. What I would like to do is to store everything in database. It could be oracle, ms sql or my sql. Main goal is to have all excels in one place and create graphs, tables, comparison and so on, in one place. Is there online some product/software (free or not) for this kind of system? I could edit it, but don't want to make it from scratch.