I'm trying to extract the data from another excel file, but I tried to make it dinamyc.
This is the context so I'm extractig data with a survey for different cities and the questions are the same but the data is different in each city, so I'm trying to create visualisations for each city but just replacing the data in the file, so this data is exported as a file we can call it "results-city.xlsx" and my goal is just placing this document with another with the same name and columns but obviusly different responses in each column so I'm trying to use power query and the advanced editor and this is my formula but is not getting success also the path will be dynamic that's the reason I included folder in the formula. Help please to achive this
let
Source = Excel.Workbook(File.Contents("C:\Users\iotal\OneDrive\Desktop\stack\folder\results-city.xlsx"), null, true),
Sheet1_Sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Sheet1_Sheet,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}})
in
#"Changed Type"
Update 2:
Example of data:
File with data city 1
So here is an example with the data from two cities File with data city 2, when I tried to use the formula the power query don't import the data to excel.
My desired output is have the data imported in a sheet in excel that doesn't mind which of the two is in the folder but can be updated.
This is my desired output with images:
Image 1 is the first data that I will import
And the second image is when I replace the data for city2 should look like this just replacing the file for another one with the same name
And what is not working is the formula that doesn't import to a sheet the data as a table
When I replace the file from city1 for the one from city2 I got this error:
In powerquery, something like this will combine all tabs in all xlsx files in a specified hardcoded directory
let Source = Folder.Files("C:\subdirectory\directory"),
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".xlsx")),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Name", "Content"}),
#"Added Custom" = Table.AddColumn(#"Removed Other Columns", "GetFileData", each Excel.Workbook([Content],true)),
#"Expanded GetFileData" = Table.ExpandTableColumn(#"Added Custom", "GetFileData", {"Data", "Hidden", "Item", "Kind", "Name"}, {"Data", "Hidden", "Item", "Kind", "Sheet"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded GetFileData",{"Content", "Hidden", "Item", "Kind"}),
List = List.Union(List.Transform(#"Removed Columns"[Data], each Table.ColumnNames(_))),
#"Expanded Data" = Table.ExpandTableColumn(#"Removed Columns", "Data", List,List)
in #"Expanded Data"
Alternatively, give a cell a range name, like path in excel and put your filepath in that cell, like C:\temp\a.xlsx
Then in powerquery, use that range name in place of hardcoding the file name like
let location= Excel.CurrentWorkbook(){[Name="path"]}[Content]{0}[Column1],
Source = Excel.Workbook(File.Contents(location), null, true),
...
this second alternative assumes that the tab name is constant. Otherwise I recommend the first method
You can combine the two alternatives if you want
To make the folder location dynamic with respect to a cell with named range "file_path" (B2, below/ here), modify an ordinary power query data import for one of the files/locations (here, "C:\temp\Folder1\File1.txt") as follows:
ordinary power query:
let
Source = Csv.Document(File.Contents("C:\temp\Folder1\File1.txt"),[Delimiter=" ", Columns=3, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
in
#"Promoted Headers"
updated power query:
let
MyFolder = Excel.CurrentWorkbook(){[Name="file_path"]}[Content][Column1]{0},
Source = Csv.Document(File.Contents(MyFolder),[Delimiter=" ", Columns=3, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
in
#"Promoted Headers"
i.e. file name XYZ in [Name ="XYZ"] replaced with variable MyFolder, defined as follows:
MyFolder = Excel.CurrentWorkbook(){[Name="file_path"]}[Content][Column1]
(optional)
Include VB code to refresh query every time the drop down list changes in the cell with named range ("file_path") (i.e. cell B2 above gif) - as follows:
(you don't have to do this - you could just as easily make the query refreshable periodically via the properties UI associated with the query, for instance)..
Private Sub Worksheet_Change(ByVal Target As Range)
If Not Intersect(Target, Range("file_path")) Is Nothing Then
Application.ScreenUpdating = False
Calculate
ActiveWorkbook.Connections("Query - dynamic_file").Refresh
Application.ScreenUpdating = True
Calculate
End If
End Sub
Kudos to Cristiano Galvão (here) for more details RE: dynamic links per above.
ta
Related
I've recently run into an issue which hopefully is solvable.
Currently, I have power query pointing at a folder containing several CSV files. This is normally no issue, however, in this instance not all of the files have the same columns.
Is there a way to have power query return every unique column found in the folder populating empty data observations with null values?
Assume that my folder has csv files similar to the following (note that the rows are indexed using letters for easy reference):
I would like my final table to look something like:
This seems like it should be pretty simple, but I can't figure it out for the life of me! Any help would be greatly appreciated!
Assuming you're using Folder.Files, I think you can:
Grab the Content column of the table returned by Folder.Files -- which should give you a list of binary values.
Parse each item in the list as a CSV document using List.Transform and Csv.Document -- which should give you a list of tables.
Then merge your list of tables with Table.Combine -- which should give you one single table. Table.Combine should take care of the details (like aligning column names).
You've not provided any code in your question, so it's hard to give a relevant example, but I think the code below gives me your expected output.
I've turned the row indexes into an ID column, just to make the final table easier to verify/follow.
let
firstCsv =
"ID,one,two,three
A,1,4,7
B,2,5,8
C,3,6,9",
secondCsv =
"ID,one,two,three,four
D,1,6,11,16
E,2,7,12,17
F,3,8,13,18
G,4,9,14,19
H,5,10,15,20",
thirdCsv =
"ID,one,two,yes,no,maybe
I,1,1,1,1,1
J,2,2,2,2,2
K,3,3,3,3,3
L,4,4,4,4,4
M,5,5,5,5,5",
// For example's sake, let's suppose that the contrived table below was
// returned by calling Folder.Files
filesInFolder = Table.FromColumns({
List.Transform({firstCsv, secondCsv, thirdCsv}, Text.ToBinary),
List.Transform({"1".."3"}, each "CSV file " & _ & ".csv"),
List.Repeat({"someFolderPath"}, 3)
}, type table [Content = binary, Name = text, Folder = text]),
parsed = List.Transform(filesInFolder[Content], each
let
csv = Csv.Document(_, [Delimiter = ",", QuoteStyle = QuoteStyle.Csv]),
promoted = Table.PromoteHeaders(csv, [PromoteAllScalars = true])
in promoted
),
// The step below should match the expected output in your question.
combined = Table.Combine(parsed)
in
combined
Obviously, you'll need to adjust for your own folder path and actually call Folder.Files as you presumably already are in your own code.
I've always used something like this
//read all files in specified directory you fill in here
let Source = Folder.Files("C:\directory\subdirectory"),
//filter only csv files
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".csv")),
//Pull contents of each file into table with an index
#"Added Custom1" = Table.AddColumn(#"Filtered Rows", "Custom", each Table.AddIndexColumn(Csv.Document(File.Contents([Folder Path]&"\"&[Name]),[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),"Index")),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom1", "Custom", {"Column1", "Index"}, {"Column1", "Index"}),
#"Removed Other Columns" = Table.SelectColumns(#"Expanded Custom",{"Column1", "Index", "Name"}),
#"Pivoted Column" = Table.Pivot(#"Removed Other Columns", List.Distinct(#"Removed Other Columns"[Name]), "Name", "Column1"),
#"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in #"Removed Columns"
I have a folder with 3 different subfolders where I store some daily exports. I want to get only the latest updated file from each folder path. As I show in the pic.
I need to keep the binary and the attributes columns of the latest "date modified" file for each same path. The latest date modified is different for each different subfolder. My subfolders refere to each month of the year (their name are May, Jun, July and soon it will be incorporated August).
In this way, as I have 3 different folder paths values in my column Folder Path, I will only have 3 csv to expand.
I have tried to group by folder path and maximum Date modified, but I lost the other fields. If I add, maximun of attributes or all rows, does not solve me problem either.
I also tried this solution: https://community.powerbi.com/t5/Desktop/Keep-only-the-latest-date-for-duplicate-entries/td-p/638447
But I get stuck in an error: Function type value cannot be converted to Table type. Details: Value = [Function], Type = [Type].
M Query code:
let
Source = AzureStorage.DataLake(".../usersDailyData"),
#"Filtered Hidden Files1" = Table.SelectRows(#"Sorted Rows", each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transformar archivo", each #"Transformar archivo"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transformar archivo"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transformar archivo", Table.ColumnNames(#"Transformar archivo"(#"Archivo de ejemplo"))),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{ Columns name here })
in
#"Changed Type"
Any ideas to solve it?
Thank you very much.
I'd suggest adding a column that has the max datetime for each Folder Path, then comparing each rows datetime to that max, and filtering out non-matches. Along the lines of below. I think it goes after the Invoke Custom Function1 step , but you can move it where needed
...
#"Added Custom" = Table.AddColumn(#"Invoke Custom Function1","MaxDate",(i)=>List.Max(Table.SelectRows( #"Invoke Custom Function1", each [Folder Path]=i[Folder Path]) [Date modified]), type datetime ),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom", each if [Date modified]=[MaxDate] then "keep" else null),
#"Filtered Rows" = Table.SelectRows(#"Added Custom1", each ([Custom] = "keep")),
...
I am trying to use Power Query in Excel 2013 to import a folder full of 121 text files. Each text file has a column of numbers:
24
2.0000E+07
1.0000E+07
5.0000E+06
2.0000E+06
1.0000E+06
1.0000E+05
1.0000E+04
1.0000E+03
1.0000E+02
1.0000E+01
1.0000E+00
6.2500E-01
5.0000E-01
4.0000E-01
3.0000E-01
2.0000E-01
1.0000E-01
8.0000E-02
6.0000E-02
4.0000E-02
3.0000E-02
2.0000E-02
1.0000E-02
2.0000E-04
1.0000E-05
1.0516E-05
9.3907E-06
3.3497E-04
1.8445E-03
1.3411E-03
5.4756E-03
9.4254E-03
1.2390E-02
1.4350E-02
1.5677E-02
1.7293E-02
4.0507E-03
2.0602E-03
2.1823E-03
3.1392E-03
7.5455E-03
9.1609E-02
7.5750E-02
1.2536E-01
1.9400E-01
1.2207E-01
1.2811E-01
1.1341E-01
5.2564E-02
56
2.0000E+07
6.4300E+06
4.3000E+06
3.0000E+06
1.8500E+06
1.5000E+06
1.2000E+06
8.6100E+05
7.5000E+05
6.0000E+05
4.7000E+05
3.3000E+05
2.7000E+05
2.0000E+05
5.0000E+04
2.0000E+04
1.7000E+04
3.7400E+03
2.2500E+03
1.9200E+02
1.8800E+02
1.1800E+02
1.1600E+02
1.0500E+02
1.0100E+02
6.7500E+01
6.5000E+01
3.7100E+01
3.6000E+01
2.1800E+01
2.1200E+01
2.0500E+01
7.0000E+00
6.8800E+00
6.5000E+00
6.2500E+00
5.0000E+00
1.1300E+00
1.0800E+00
1.0100E+00
6.2500E-01
4.5000E-01
3.7500E-01
3.5000E-01
3.2500E-01
2.5000E-01
2.0000E-01
1.5000E-01
1.0000E-01
8.0000E-02
6.0000E-02
5.0000E-02
4.0000E-02
2.5300E-02
1.0000E-02
4.0000E-03
1.0000E-05
I want to use Power Query to import the entire folder into Excel, with the data in each text file having its own column, and the column header being the name of the text file.
Like this
The problem is that Power Query only seems to import the file names, but not the data within them.
So I get something like:
this
With no data underneath its respective column. What am I doing wrong? Would it have something to do with Power Query seeing the data as 'binary' instead of 'text'?
This should do what you want ... read in all .txt files in a directory, and then place the values from each into its own column where the column headers is the filename.
Obviously, change the path in the first step
Assumes a single column of data in each source file
let Source = Folder.Files("C:\directory\subdirectory\"),
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".txt")),
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each Table.AddIndexColumn(Csv.Document(File.Contents([Folder Path]&"\"&[Name]),[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),"Index",1)),
#"Expanded Custom.1" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column1", "Index"}, {"Column1", "Index"}),
#"Removed Other Columns" = Table.SelectColumns(#"Expanded Custom.1",{"Name", "Column1", "Index"}),
#"Pivoted Column" = Table.Pivot(#"Removed Other Columns", List.Distinct(#"Removed Other Columns"[Name]), "Name", "Column1"),
#"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in #"Removed Columns"
I have, in my excel workbook, a Table called ResultsTable, in that table there is a file path
C:\Users\XXXX\OneDrive - WORK\Digital
Soil\Data\Results
I have Query that should get all excel files from the folder and transform the data into something usefull looking like this:
let
Source = Folder.Files("ResultsTable"),
#"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File from Analyseresultater", each #"Transform File from Analyseresultater"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File from Analyseresultater"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File from Analyseresultater", Table.ColumnNames(#"Transform File from Analyseresultater"(#"Sample File"))),
#"Removed Other Columns" = Table.SelectColumns(#"Expanded Table Column1",{"Key", "Attribute", "Value"})
in
#"Removed Other Columns"
But I get the error
DataFormat.Error: The supplied folder path must be a valid absolute
path. Details:
ResultsTable
I hope someone can help me get through this error :)
EDIT: Added screenshot of how my sheet with tables are set up
You can fix the code like,
let
FilePath = Excel.CurrentWorkbook(){[Name="ResultsTable"]}[Content][Path to results]{0},
Source = Folder.Files(FilePath),
In the original code, Folder.Files() was receiving the literal text "ResultsTable", not the cell value in ResultsTable. You need to first pick the cell value with Excel.CurrentWorkbook(), and then pass it to Folder.Files().
I am trying to load (combine) multiple Excel files into Power BI (October 2019 version). Every file has only 1 sheet. Each sheet has 1 range, and each range has the same schema across all files. (The sheet names are different, though.) A sample sheet name is '200704'.
Here are my steps:
Get Data \ Folder \ Connect
specify the Folder path
Combine & Load
select one of the files as my sample file; click on the file name as
my Parameter1; click OK
After I click OK, the cursor spins for a bit, and then it stops. Nothing happens. So, I go to Edit Queries \ Edit Queries. There is a warning symbol on my data query that reads:
An error occurred in the 'Transform File' query. Expression.Error:
The key didn't match any rows in the table.
Details: Key = Item=200704 Kind=Sheet Table=[Table]
How do I resolve this error?
If it helps, Power BI generate 5 queries for me, and the structure is:
Transform File from data [2]
Helper Queries [3]
Parameter1 (Sample File)
Sample File
Transform File
Transform Sample File
Other Queries [1]
data
Interestingly, if it helps to diagnose the issue, if I set sample file = First file or if I set sample file to my first file manually, the following error is thrown in the dialog, but it doesn't show what query is in error when I try to view / edit the query.
Failed to save modifications to the server. Error returned: 'OLE DB or ODBC error: [Expression.Error] The key didn't match any rows in the table..'.
And, to be sure, when I attempt to load this file (or any file in the folder, for that matter) individually (via Excel connection), it loads successfully. So, something must be wrong with the M code in my Folder connection.
I figured out the cause of my problem and the solution. The issue is that the row in my template query was being referenced incorrectly (i.e., the primary key between the template query and the regular query is wrong, and it has hard-coding of sheet names). To fix that, I had to remove all other columns in the template query table except the Data column, as described here. (It's odd that no MS documentation on combining multiple Excel files discusses this very important step.)
For comparison, here is my former (incorrect) M code:
Transform Sample File:
let
Source = Excel.Workbook(Parameter1, null, true),
#"Sample_Sheet" = Source{[Item="sample",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(#"Sample_Sheet", [PromoteAllScalars=true])
in
#"Promoted Headers"
test:
let
Source = Folder.Files("C:\some folder path"),
#"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File", each #"Transform File"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File", Table.ColumnNames(#"Transform File"(#"Sample File"))),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{{"Source.Name", type text}, {"ID", type text}, {"Name", type text}})
in
#"Changed Type"
And here is my new (correct) code:
Transform Sample File:
let
Source = Excel.Workbook(Parameter1, null, true),
#"Removed Columns" = Table.RemoveColumns(Source,{"Name", "Item", "Kind", "Hidden"}),
Data = #"Removed Columns"{0}[Data],
#"Promoted Headers" = Table.PromoteHeaders(Data, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"ID", type text}, {"Name", type text}})
in
#"Changed Type"
test:
let
Source = Folder.Files("C:\some folder path"),
#"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File", each #"Transform File"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File", Table.ColumnNames(#"Transform File"(#"Sample File")))
in
#"Expanded Table Column1"
Notice the 'Removed Columns' step in the new template query. This is the "secret sauce" to the key problem. Also notice that I kept all default steps after my 'Data' step (i.e., 'Promoted Headers' and 'Changed Type') in my template query. This is because all of my sheets have the same schema. If this weren't true, then I would need to move those steps to the regular query.
I had exactly the same error simply because the PowerBI VNET Gateway could not authenticate to the source of the data to refresh dataset hosted in PowerBI premium capacity workspace. Totally unexpected and confusing, but once the correct credentials were set for the Gateway configuration - everything worked fine and the error had gone away.