Power BI - Call Azure API with nextLink (next page) - azure

Apologies, I'm new to Power BI. I'm using Power BI to call an Azure API that will list all the VMs in my subscription, however it will only show the first 50 before having a nextLink.
Here is the API I'm calling;
https://management.azure.com/subscriptions/< subscription >/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01
I've seen other pages and forums with a similar issue (such as Microsoft API), but not for Azure API. I messed about with their fix, but could not work out how to apply it to mine.
Their code;
let
GetUserInfo = (Path)=>
let
Source = Json.Document(Web.Contents(Path)),
LL= #Source[value],
result = try #LL & #GetUserInfo(Source[#"#odata.nextLink"]) otherwise #LL
in
result,
Fullset = GetUserInfo("https://graph.microsoft.com/beta/users?$select=manager&$expand=manager"),
#"Converted to Table" = Table.FromList(Fullset, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "displayName", "manager"}, {"Column1.id", "Column1.displayName", "Column1.manager"}),
#"Expanded Column1.manager" = Table.ExpandRecordColumn(#"Expanded Column1", "Column1.manager", {"id", "displayName"}, {"id", "displayName"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded Column1.manager",{{"Column1.displayName", "Employee Full Name"}, {"Column1.id", "Employee Id"}, {"id", "Manager Id"}, {"displayName", "Manager Full name"}})
in
#"Renamed Columns"
Compared to the start of mine once I've connected the source by the simple web link;
let
Source = Json.Document(Web.Contents("https://management.azure.com/subscriptions/< subscription >/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01")),
#"Converted to Table" = Record.ToTable(Source)
in
#"Converted to Table"
If I were to adjust it, I suspected it would look something like this;
let
GetUserInfo = (Path)=>
let
Source = Json.Document(Web.Contents(Path)),
LL= #Source[value],
result = try #LL & #GetUserInfo(Source[#"#odata.nextLink"]) otherwise #LL
in
result,
Fullset = GetUserInfo("https://management.azure.com/subscriptions/< subscription >/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01"),
#"Converted to Table" = Record.ToTable(Source)
in
#"Converted to Table"
However I am prompted with the following error once clicking OK;
Expression.Error: The name 'Source' wasn't recognized. Make sure it's spelled correctly.
Any help on this would be greatly appreciated.

For anyone interested, here is what I ended up doing thanks to this link:
https://datachant.com/2016/06/27/cursor-based-pagination-power-query/
let
iterations = 10,
url =
"https://management.azure.com/subscriptions/< subscription >/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01",
FnGetOnePage =
(url) as record =>
let
Source = Json.Document(Web.Contents(url)),
data = try Source[value] otherwise null,
next = try Source[nextLink] otherwise null,
res = [Data=data, Next=next]
in
res,
GeneratedList =
List.Generate(
()=>[i=0, res = FnGetOnePage(url)],
each [i]<iterations and [res][Data]<>null,
each [i=[i]+1, res = FnGetOnePage([res][Next])],
each [res][Data])
in
GeneratedList
1 whole day of Googling headache :S

Related

Excel Power Query to only show unhidden rows

i have an excel workbook with 3 worksheets. each of the worksheet contains hidden and unhidden rows. i’ve managed to combine all 3 worksheets into one single worksheet via power query. however, the worksheet generated shows all the hidden and unhidden rows. is there anyways i could just generate the worksheet with just the unhidden rows?
i’ve read about using a helping column and applying a filter on it but i am unsure of how to do that as well. any new suggestions/walkthrough is greatly appreciated
You can do this in Power Query.
You can unzip the XLSX file to XML tables, and examine the row attributes - then get the sheet data, and merge the hidden attribute with the row number.
Pass this function the file name (including path) and sheet name, and it will return the sheet contents, with an added "Row hidden" column:
//fnGetRowHiddenStatus
(MyFileName as text, MySheetName as text) =>
let
fnUnzip = (ZIPFile, Position, FileToExtract, DataSoFar) =>
let
MyBinaryFormat = try BinaryFormat.Record([DataToSkip=BinaryFormat.Binary(Position),
MiscHeader=BinaryFormat.Binary(18),
FileSize=BinaryFormat.ByteOrder(BinaryFormat.UnsignedInteger32, ByteOrder.LittleEndian),
UnCompressedFileSize=BinaryFormat.Binary(4),
FileNameLen=BinaryFormat.ByteOrder(BinaryFormat.UnsignedInteger16, ByteOrder.LittleEndian),
ExtrasLen=BinaryFormat.ByteOrder(BinaryFormat.UnsignedInteger16, ByteOrder.LittleEndian),
TheRest=BinaryFormat.Binary()]) otherwise null,
MyCompressedFileSize = try MyBinaryFormat(ZIPFile)[FileSize]+1 otherwise null,
MyFileNameLen = try MyBinaryFormat(ZIPFile)[FileNameLen] otherwise null,
MyExtrasLen = try MyBinaryFormat(ZIPFile)[ExtrasLen] otherwise null,
MyBinaryFormat2 = try BinaryFormat.Record([DataToSkip=BinaryFormat.Binary(Position), Header=BinaryFormat.Binary(30), Filename=BinaryFormat.Text(MyFileNameLen), Extras=BinaryFormat.Binary(MyExtrasLen), Data=BinaryFormat.Binary(MyCompressedFileSize), TheRest=BinaryFormat.Binary()]) otherwise null,
MyFileName = try MyBinaryFormat2(ZIPFile)[Filename] otherwise null,
GetDataToDecompress = try MyBinaryFormat2(ZIPFile)[Data] otherwise null,
DecompressData = try Binary.Decompress(GetDataToDecompress, Compression.Deflate) otherwise null,
NewPosition = try Position + 30 + MyFileNameLen + MyExtrasLen + MyCompressedFileSize - 1 otherwise null,
AsATable = Table.FromRecords({[Filename = MyFileName, Content=DecompressData]}),
#"Appended Query" = if DecompressData = null then DataSoFar else if (MyFileName = FileToExtract) then AsATable else
if (FileToExtract = "") and Position <> 0 then Table.Combine({DataSoFar, AsATable})
else AsATable
in
if (MyFileName = FileToExtract) or (#"Appended Query" = DataSoFar) then
#"Appended Query"
else
#fnUnzip(ZIPFile, NewPosition, FileToExtract, #"Appended Query"),
Unzipped = fnUnzip(File.Contents(MyFileName), 0, "", null),
WorkbookXML = Xml.Tables(Table.SelectRows(Unzipped, each Text.Contains([Filename],"xl/workbook"))[Content]{0}),
WorkbookData = Table.SelectRows(WorkbookXML, each [Name] = "sheets"){0}[Table]{0}[Table],
SheetIDs = Table.ExpandTableColumn(WorkbookData, "http://schemas.openxmlformats.org/officeDocument/2006/relationships", {"Attribute:id"}, {"Attribute:id"}),
SheetID = Text.Replace(Table.SelectRows(SheetIDs, each ([#"Attribute:name"] = MySheetName))[#"Attribute:id"]{0},"rId","sheet"),
SheetXML = Xml.Tables(Table.SelectRows(Unzipped, each Text.Contains([Filename], "worksheets/" & SheetID))[Content]{0}),
SheetData = Table.SelectRows(SheetXML, each [Name]="sheetData"){0}[Table]{0}[Table],
Renamed = Table.RenameColumns(SheetData,{{"Attribute:r", "Row"}}),
#"Row Data" = Table.SelectColumns(Renamed,{"Row", "Attribute:hidden"}, MissingField.Ignore),
#"Changed Type" = Table.TransformColumnTypes(#"Row Data",{{"Row", Int64.Type}}),
#"Row Status" = Table.AddColumn(#"Changed Type", "Hidden", each try if [#"Attribute:hidden"] = "1" then true else false otherwise false, type logical),
Workbook = Excel.Workbook(File.Contents(MyFileName)),
Worksheet = Table.SelectRows(Workbook, each ([Name] = MySheetName))[Data]{0},
#"Added Row" = Table.AddIndexColumn(Worksheet, "Row", 1, 1, Int64.Type),
#"Merged Row Status" = Table.NestedJoin(#"Added Row", {"Row"}, #"Row Status", {"Row"}, "Row Status", JoinKind.LeftOuter),
#"Expanded Row Status" = Table.ExpandTableColumn(#"Merged Row Status", "Row Status", {"Hidden"}, {"Row hidden"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Row Status",{"Row"})
in
#"Removed Columns"
Example:
let
Source = fnGetRowHiddenStatus("C:\Temp\rowhide.xlsx", "Sheet1")
in
Source

How can i get all the sub-children?

source link
Hello guys, so i have a function ("flecheD"),
(ColChild,ColParent,ParentActuel,source)=>
let
mylist=Table.Column(Table.SelectRows(source,each Record.Field(_,ColParent)=ParentActuel),ColChild),
resultat=Text.Combine(mylist)
in
Text.Trim(
if resultat ="" then "" else # resultat &"|" & # flecheD(ColChild,ColParent,resultat,source),"|")
which loops through 2 columns (Parent,Child) to get all children of the main parent (output->Children column). The problem is that when the function is confronted with several children, the resultat variable no longer has a single letter/child but several, which blocks the function from looking for the other sub-children.
In order to solve this, I tried to create a custom function "SubChilldren" with List.Generate()
(children as text, ColChild,ColParent,source)=>
let
i = 1,
length = Text.Length(children),
subchildren = List.Generate( ()=>#flecheD(ColChild,ColParent,Text.At(children,i-1),source), i<=length, i+1 )
in
Text.Combine(subchildren)
which when coupled with my initial function
(ColChild,ColParent,ParentActuel,source)=>
let
mylist=Table.Column(Table.SelectRows(source,each Record.Field(_,ColParent)=ParentActuel),ColChild),
resultat=Text.Combine(mylist)
in
Text.Trim(
if resultat ="" then "" else if Text.Length(resultat) = 1 then # resultat &"|" & # flecheD(ColChild,ColParent,resultat,source)
else #resultat &"|"& SubChildren(resultat,ColChild,ColParent,source),"|")
should normally get the sub-children of each children. However, it doesnt work . Could you please help me . Thx
I thought this was a fun way, but you could write a recursive function as well. I have it hard coded to 4 levels of children deep
(not sure how in your source data D child can have two parents, c and J, but whatever)
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Grouped Rows" = Table.Group(Source, {"Parent"}, {{"data", each List.RemoveNulls(_[Child]), type list}}),
Parent_List = List.Buffer(#"Grouped Rows"[Parent] ),
Child_List = List.Buffer(#"Grouped Rows"[data] ),
Process = (n as list) as list =>
let children = List.Transform(List.Transform(n, each Text.ToList(_)), each Text.Combine( List.Distinct(List.Combine(List.Transform(_, each try Child_List{List.PositionOf( Parent_List, _ )} otherwise null))))) in children,
Level1=Process(Source[Parent]),
Level2=Process(Level1),
Level3=Process(Level2),
Level4=Process(Level3),
Final=List.Transform(List.Positions(Level1),each Level1{_}&"|"&Level2{_}&"|"&Level3{_}&"|"&Level4{_}&"|"),
#"Replaced Value" = Table.ReplaceValue(Table.FromList(Final),"||","",Replacer.ReplaceText,{"Column1"}),
custom1 = Table.ToColumns(Source) & Table.ToColumns(#"Replaced Value"),
custom2 = Table.FromColumns(custom1,Table.ColumnNames(Source) & {"Children"})
in custom2
edited to be generic so it can take text as well as numerical inputs
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Parent", type text}, {"Child", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Parent"}, {{"data", each List.Transform(List.RemoveNulls(_[Child]), each Text.From(_)), type list}}),
Parent_List = List.Buffer(List.Transform(#"Grouped Rows"[Parent], each Text.From(_))),
Child_List = List.Buffer(#"Grouped Rows"[data]),
Process = (n as list) as list =>let children = List.Transform(List.Transform(n, each Text.Split(_,",") ) , each try Text.Combine(List.Distinct(List.Combine(List.Transform(_, each try Child_List{List.PositionOf( Parent_List, _ )} otherwise ""))),"," ) otherwise "") in children,
Level1=Process(#"Changed Type"[Parent]),
Level2=Process(Level1),
Level3=Process(Level2),
Level4=Process(Level3),
Final=List.Transform(List.Positions(Level1),each Level1{_}&"|"&Level2{_}&"|"&Level3{_}&"|"&Level4{_}&"|"),
#"Replaced Value" = Table.ReplaceValue(Table.FromColumns({Final}),"||","",Replacer.ReplaceText,{"Column1"}),
custom1 = Table.ToColumns(#"Changed Type") & Table.ToColumns(#"Replaced Value"),
custom2 = Table.FromColumns(custom1,Table.ColumnNames(#"Changed Type") & {"Children"})
in custom2

Excel power query gives DataSource.Error web.content failed to get content from api url (404):Not Found

My excel uses data from an api url, I run the query every 10 mins, but if the api has a lot of traffic and returns a 404 error, I get a pop up error "DataSource.Error web.content failed to get content from api url (404);Not Found" and the whole spread sheet isn't working until I click okay, what I need to do is to let the query to keep trying the api url until it gets data to return instead of throwing an error.
here's the query that I run:
let
Source = Json.Document(Web.Contents("api url")),
#"Converted to Table" = Record.ToTable(Source),
#"Removed Top Rows" = Table.Skip(#"Converted to Table",9),
#"Removed Bottom Rows" = Table.RemoveLastN(#"Removed Top Rows",6)
in
#"Removed Bottom Rows"
I tried to add MaunalStatusHandling but it didn't solve the problem as I kept getting error messages.
I hope that someone can show me a way to solve this
thanks in advance for your help
I figured out a way to solve the problem, I'll post it in case someone else has a similar problem
let
api_fnName = let
Value.WaitFor = (producer as function, interval as function, optional count as number) as any =>
let
list = List.Generate(
() => {0, null},
(state) => state{0} <> null and (count = null or state{0} < count),
(state) => if state{1} <> null
then {null, state{1}}
else {1 + state{0}, Function.InvokeAfter(() => producer(state{0}), interval(state{0}))},
(state) => state{1})
in
List.Last(list),
Web.ContentsCustomRetry = (url as text, optional options as record) => Value.WaitFor(
(i) =>
let
options2 = if options = null then [] else options,
options3 = if i=0 then options2 else options2 & [IsRetry=true],
result = Web.Contents(url, options3 & [ManualStatusHandling={500}]),
buffered = Binary.Buffer(result), /* avoid risk of double request */
status = if buffered = null then 0 else Value.Metadata(result)[Response.Status],
actualResult = if status = 500 then null else buffered
in
actualResult,
(i) => #duration(0, 0, 0, i*0.1))
in
Json.Document(Web.ContentsCustomRetry("api url")),
results = api_fnName[results],
results1 = results{0},
data = results1[data],
#"Converted to Table" = Record.ToTable(data),
#"Kept Range of Rows" = Table.Range(#"Converted to Table",4,1)
in
#"Kept Range of Rows"

How to get data from within (many) documents in SharePoint library with Power Query

Power BI junior here
How to look in each excel file from a SharePoint list and extract contents from predefined cells.
I am currently accessing a few intranet Sharepoint libraries containing .xlsx files and with the metadata of those files I am doing some reporting. For example, a library contains 10 excel files so I can graph who uploaded them, when they were uploaded, and wat category they were assigned to...
However, is there a way with Power Query to look into each and every of the files, take the value from, say cell A1 of the excel, and add it as a new column "CellA1Content"? I.e., make your own metadata from the content of the files and add them to the imported metadata table.
I've found some functions that I possibly might need:
File.Contents
Excel.CurrentWorkbook
However I am not well-versed enough in Power Query to put it all together, if it's even possible at all. I would have to do a foreach operation of some kind.
Edit: Solution
This worked. I selected the first non-hidden sheet in the excel and I also made the function so that I can pass the column and row number.
Main query:
let
Source = SharePoint.Contents("http://mysharepoint", [Implementation=null, ApiVersion=15]),
... ... ...
//Open each excel and get cell D5
#"AddedColumn1" = Table.AddColumn(#"Filtered Rows", "AddedColumn1", each GetCellContent([Content],4,5))
in
AddedColumn1
Blank query in Power BI, called GetCellContent:
let
Source = (binaryParameter,col,row) => let
Source = Excel.Workbook(binaryParameter, null, false),
UnhiddenSheets = Table.SelectRows(Source, each if [Hidden]=false and [Kind]="Sheet" then true else false),
Sheet = UnhiddenSheets{0}[Data],
Column = Table.SelectColumns(Sheet,{Text.Combine({"Column",Number.ToText(col)})}),
Cell = Record.Field(Column{row-1}, Text.Combine({"Column",Number.ToText(col)}) )
in
Cell
in
Source
You'll need a Function used in a column like this.
This is my local interpretation of your problem, without sharepoint. The same logic is shared though.
Main Query
let
Source = Folder.Contents("YourDirectory"),
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".xlsx")),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Content", "Name"}),
#"Added Custom" = Table.AddColumn(#"Removed Other Columns", "Row1Col1", each PullRow1Col1([Content]))
in
#"Added Custom"
PullRow1Col1:
let
Source = (binaryParameter) => let
Source = Excel.Workbook(binaryParameter, null, false),
Sheet1_sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
Column1 = Sheet1_sheet{0}[Column1]
in
Column1
in
Source

Update a line "Advanced editor" in power query with Macro

Quick question. Got a data link to a site that is already designed (without macro)in excel.
I want to just change a small line in the advanced editor in power query editor with macro.
The code:
let
Source = Web.Page(Web.Contents("http://xxx.xxxxxx.xx.xx/xxx/XXX/user_id=111")),
Data0 = Source{0}[Data],
#"Removed Columns1" = Table.RemoveColumns(Data0,{"name"}),
#"Removed Bottom Rows" = Table.RemoveLastN(#"Removed Columns1",1),
#"Changed Type" = Table.TransformColumnTypes(#"Removed Bottom Rows",{{"number", type number}})
in
#"Changed Type"
I want to update the user_id from 111 to 112 using macro(will make user input the date)
I got lots of tables like these and redesigning it will take more time that I can spare.
Best regards.
Niko
You could use code similar to this:
Sub UpdateQueries()
Dim oQ As WorkbookQuery
'Find queries using this table
For Each oQ In ActiveWorkbook.Queries
oQ.Formula = Replace(oQ.Formula,"user_id=111""","user_id=112""")
Next
End Sub

Resources