Difference between dates in Power Query - excel
In power query editor, I added a custom column with the following formula:
=if[Type7]="REST" then [Til7]-[Fra7] else 0
That one works fine.
[Type#] is either "REST" or "REGULAR".
[Fra#] and [Til#] is a time and date in the format dd.mm.yyyy hh:mm (the format is not a problem), I subtract the smaller one from the bigger one to get the amount of time in decimals ([Fra#] is always the smaller one).
However, when I try to chain several of them together, I get Error. I tried to do it like this:
=((if[Type7]="REST" then [Til7]-[Fra7] else 0)
+(if[Type6]="REST" then [Til6]-[Fra6] else 0)
+(if[Type5]="REST" then [Til5]-[Fra5] else 0)
+(if[Type4]="REST" then [Til4]-[Fra4] else 0)
+(if[Type3]="REST" then [Til3]-[Fra3] else 0)
+(if[Type2]="REST" then [Til2]-[Fra2] else 0)
+(if[Type1]="REST" then [Til1]-[Fra1] else 0)
+(if[Type]="REST" then [Til]-[Fra] else 0))*24
I tried removing the *24 bit, but that is not the problem either.
Why is it not working? Thanks in advance for any help :)
Edit:
Data screenshot
Comment to screenshot: I get Error as output in the rightmost column when I run the code.
let
Source = Excel.Workbook(File.Contents("C:\Users\xxxxxx\Downloads\export.xlsx"),
null, true),
#"ag-grid_Sheet" = Source{[Item="ag-grid",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(#"ag-grid_Sheet", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"X_CREATED", type datetime}, {"X_CREATED_BY", type text}, {"X_DEFAULT_TZ", type text}, {"LocationData", type text}, {"Shift", type text}, {"ScheduleStart", type any}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"ScheduleStart"}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Removed Columns", "Shift", Splitter.SplitTextByDelimiter("}", QuoteStyle.Csv), {"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Replaced Value" = Table.ReplaceValue(#"Split Column by Delimiter",",{hours:,paidHours:,signature:,overtimeType:,","",Replacer.ReplaceText,{"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","]","",Replacer.ReplaceText,{"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Replaced Value2" = Table.ReplaceValue(#"Replaced Value1",null,"",Replacer.ReplaceValue,{"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Replaced Value3" = Table.ReplaceValue(#"Replaced Value2",",comment:","",Replacer.ReplaceText,{"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Replaced Value4" = Table.ReplaceValue(#"Replaced Value3","[{hours:,paidHours:,signature:,overtimeType:,","",Replacer.ReplaceText,{"Shift.1", "Shift.2", "Shift.3", "Shift.4", "Shift.5", "Shift.6", "Shift.7", "Shift.8", "Shift.9"}),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Replaced Value4", "Shift.1", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.1.1", "Shift.1.2"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Shift.1.1", type text}, {"Shift.1.2", type text}}),
#"Split Column by Delimiter2" = Table.SplitColumn(#"Changed Type1", "Shift.2", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.2.1", "Shift.2.2"}),
#"Changed Type2" = Table.TransformColumnTypes(#"Split Column by Delimiter2",{{"Shift.2.1", type text}, {"Shift.2.2", type text}}),
#"Split Column by Delimiter3" = Table.SplitColumn(#"Changed Type2", "Shift.3", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.3.1", "Shift.3.2", "Shift.3.3", "Shift.3.4", "Shift.3.5"}),
#"Changed Type3" = Table.TransformColumnTypes(#"Split Column by Delimiter3",{{"Shift.3.1", type text}, {"Shift.3.2", type text}, {"Shift.3.3", type text}, {"Shift.3.4", type text}, {"Shift.3.5", type text}}),
#"Removed Columns1" = Table.RemoveColumns(#"Changed Type3",{"Shift.3.3", "Shift.3.4", "Shift.3.5"}),
#"Split Column by Delimiter4" = Table.SplitColumn(#"Removed Columns1", "Shift.4", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.4.1", "Shift.4.2"}),
#"Changed Type4" = Table.TransformColumnTypes(#"Split Column by Delimiter4",{{"Shift.4.1", type text}, {"Shift.4.2", type text}}),
#"Split Column by Delimiter5" = Table.SplitColumn(#"Changed Type4", "Shift.5", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.5.1", "Shift.5.2"}),
#"Changed Type5" = Table.TransformColumnTypes(#"Split Column by Delimiter5",{{"Shift.5.1", type text}, {"Shift.5.2", type text}}),
#"Split Column by Delimiter6" = Table.SplitColumn(#"Changed Type5", "Shift.6", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.6.1", "Shift.6.2"}),
#"Changed Type6" = Table.TransformColumnTypes(#"Split Column by Delimiter6",{{"Shift.6.1", type text}, {"Shift.6.2", type text}}),
#"Split Column by Delimiter7" = Table.SplitColumn(#"Changed Type6", "Shift.7", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.7.1", "Shift.7.2"}),
#"Changed Type7" = Table.TransformColumnTypes(#"Split Column by Delimiter7",{{"Shift.7.1", type text}, {"Shift.7.2", type text}}),
#"Split Column by Delimiter8" = Table.SplitColumn(#"Changed Type7", "Shift.8", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Shift.8.1", "Shift.8.2"}),
#"Changed Type8" = Table.TransformColumnTypes(#"Split Column by Delimiter8",{{"Shift.8.1", type text}, {"Shift.8.2", type text}}),
#"Split Column by Delimiter9" = Table.SplitColumn(#"Changed Type8", "Shift.1.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.1.2.1", "Shift.1.2.2"}),
#"Changed Type9" = Table.TransformColumnTypes(#"Split Column by Delimiter9",{{"Shift.1.2.1", type text}, {"Shift.1.2.2", type text}}),
#"Removed Columns2" = Table.RemoveColumns(#"Changed Type9",{"Shift.1.2.1"}),
#"Split Column by Delimiter10" = Table.SplitColumn(#"Removed Columns2", "Shift.2.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.2.2.1", "Shift.2.2.2"}),
#"Changed Type10" = Table.TransformColumnTypes(#"Split Column by Delimiter10",{{"Shift.2.2.1", type text}, {"Shift.2.2.2", type text}}),
#"Removed Columns3" = Table.RemoveColumns(#"Changed Type10",{"Shift.2.2.1"}),
#"Split Column by Delimiter11" = Table.SplitColumn(#"Removed Columns3", "Shift.3.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.3.2.1", "Shift.3.2.2"}),
#"Changed Type11" = Table.TransformColumnTypes(#"Split Column by Delimiter11",{{"Shift.3.2.1", type text}, {"Shift.3.2.2", type text}}),
#"Removed Columns4" = Table.RemoveColumns(#"Changed Type11",{"Shift.3.2.1"}),
#"Split Column by Delimiter12" = Table.SplitColumn(#"Removed Columns4", "Shift.4.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.4.2.1", "Shift.4.2.2"}),
#"Changed Type12" = Table.TransformColumnTypes(#"Split Column by Delimiter12",{{"Shift.4.2.1", type text}, {"Shift.4.2.2", type text}}),
#"Removed Columns5" = Table.RemoveColumns(#"Changed Type12",{"Shift.4.2.1"}),
#"Split Column by Delimiter13" = Table.SplitColumn(#"Removed Columns5", "Shift.5.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.5.2.1", "Shift.5.2.2"}),
#"Changed Type13" = Table.TransformColumnTypes(#"Split Column by Delimiter13",{{"Shift.5.2.1", type text}, {"Shift.5.2.2", type text}}),
#"Removed Columns6" = Table.RemoveColumns(#"Changed Type13",{"Shift.5.2.1"}),
#"Split Column by Delimiter14" = Table.SplitColumn(#"Removed Columns6", "Shift.6.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.6.2.1", "Shift.6.2.2"}),
#"Changed Type14" = Table.TransformColumnTypes(#"Split Column by Delimiter14",{{"Shift.6.2.1", type text}, {"Shift.6.2.2", type text}}),
#"Removed Columns7" = Table.RemoveColumns(#"Changed Type14",{"Shift.6.2.1"}),
#"Split Column by Delimiter15" = Table.SplitColumn(#"Removed Columns7", "Shift.7.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.7.2.1", "Shift.7.2.2"}),
#"Changed Type15" = Table.TransformColumnTypes(#"Split Column by Delimiter15",{{"Shift.7.2.1", type text}, {"Shift.7.2.2", type text}}),
#"Removed Columns8" = Table.RemoveColumns(#"Changed Type15",{"Shift.7.2.1"}),
#"Split Column by Delimiter16" = Table.SplitColumn(#"Removed Columns8", "Shift.8.2", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Shift.8.2.1", "Shift.8.2.2"}),
#"Changed Type16" = Table.TransformColumnTypes(#"Split Column by Delimiter16",{{"Shift.8.2.1", type text}, {"Shift.8.2.2", type text}}),
#"Removed Columns9" = Table.RemoveColumns(#"Changed Type16",{"Shift.8.2.1"}),
#"Replaced Value5" = Table.ReplaceValue(#"Removed Columns9",null,"",Replacer.ReplaceValue,{"Shift.1.1", "Shift.1.2.2", "Shift.2.1", "Shift.2.2.2", "Shift.3.1", "Shift.3.2.2", "Shift.4.1", "Shift.4.2.2", "Shift.5.1", "Shift.5.2.2", "Shift.6.1", "Shift.6.2.2", "Shift.7.1", "Shift.7.2.2", "Shift.8.1", "Shift.8.2.2", "Shift.9"}),
#"Merged Columns" = Table.CombineColumns(#"Replaced Value5",{"Shift.1.1", "Shift.1.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged"),
#"Merged Columns1" = Table.CombineColumns(#"Merged Columns",{"Shift.2.1", "Shift.2.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.1"),
#"Merged Columns2" = Table.CombineColumns(#"Merged Columns1",{"Shift.3.1", "Shift.3.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.2"),
#"Merged Columns3" = Table.CombineColumns(#"Merged Columns2",{"Shift.4.1", "Shift.4.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.3"),
#"Merged Columns4" = Table.CombineColumns(#"Merged Columns3",{"Shift.5.1", "Shift.5.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.4"),
#"Merged Columns5" = Table.CombineColumns(#"Merged Columns4",{"Shift.6.1", "Shift.6.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.5"),
#"Merged Columns6" = Table.CombineColumns(#"Merged Columns5",{"Shift.7.1", "Shift.7.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.6"),
#"Merged Columns7" = Table.CombineColumns(#"Merged Columns6",{"Shift.8.1", "Shift.8.2.2"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged.7"),
#"Removed Columns10" = Table.RemoveColumns(#"Merged Columns7",{"LocationData"}),
#"Replaced Value6" = Table.ReplaceValue(#"Removed Columns10","/KVAKT","",Replacer.ReplaceText,{"X_CREATED_BY"}),
#"Split Column by Delimiter17" = Table.SplitColumn(#"Replaced Value6", "X_CREATED_BY", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, false), {"X_CREATED_BY.1", "X_CREATED_BY.2"}),
#"Changed Type17" = Table.TransformColumnTypes(#"Split Column by Delimiter17",{{"X_CREATED_BY.1", type text}, {"X_CREATED_BY.2", type text}}),
#"Renamed Columns" = Table.RenameColumns(#"Changed Type17",{{"X_CREATED_BY.1", "Stilling"}, {"X_CREATED_BY.2", "Fartøy"}, {"X_CREATED", "Dato_ført"}}),
#"Removed Columns11" = Table.RemoveColumns(#"Renamed Columns",{"X_DEFAULT_TZ"}),
#"Changed Type18" = Table.TransformColumnTypes(#"Removed Columns11",{{"Merged", type text}}),
#"Split Column by Position" = Table.SplitColumn(#"Changed Type18", "Merged", Splitter.SplitTextByPositions({0, 25}, false), {"Merged.1.1", "Merged.2.1"}),
#"Changed Type19" = Table.TransformColumnTypes(#"Split Column by Position",{{"Merged.1.1", type text}, {"Merged.2.1", type text}}),
#"Replaced Value7" = Table.ReplaceValue(#"Changed Type19","T"," ",Replacer.ReplaceText,{"Merged.1.1"}),
#"Replaced Value8" = Table.ReplaceValue(#"Replaced Value7","L"," ",Replacer.ReplaceText,{"Merged.1.1"}),
#"Replaced Value9" = Table.ReplaceValue(#"Replaced Value8","from: ","",Replacer.ReplaceText,{"Merged.1.1"}),
#"Renamed Columns1" = Table.RenameColumns(#"Replaced Value9",{{"Merged.1.1", "Fra"}}),
#"Replaced Value10" = Table.ReplaceValue(#"Renamed Columns1",",to:L","",Replacer.ReplaceText,{"Merged.2.1"}),
#"Replaced Value11" = Table.ReplaceValue(#"Replaced Value10","T"," ",Replacer.ReplaceText,{"Merged.2.1"}),
#"Split Column by Delimiter18" = Table.SplitColumn(#"Replaced Value11", "Merged.2.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.2.1.1", "Merged.2.1.2"}),
#"Changed Type20" = Table.TransformColumnTypes(#"Split Column by Delimiter18",{{"Merged.2.1.1", type datetime}, {"Merged.2.1.2", type text}}),
#"Renamed Columns2" = Table.RenameColumns(#"Changed Type20",{{"Merged.2.1.1", "Til"}}),
#"Changed Type21" = Table.TransformColumnTypes(#"Renamed Columns2",{{"Fra", type datetime}}),
#"Replaced Value12" = Table.ReplaceValue(#"Changed Type21","type:","",Replacer.ReplaceText,{"Merged.2.1.2"}),
#"Replaced Value13" = Table.ReplaceValue(#"Replaced Value12","RES","REST",Replacer.ReplaceText,{"Merged.2.1.2"}),
#"Renamed Columns3" = Table.RenameColumns(#"Replaced Value13",{{"Merged.2.1.2", "Type"}}),
#"Replaced Value14" = Table.ReplaceValue(#"Renamed Columns3","from:L","",Replacer.ReplaceText,{"Merged.1"}),
#"Split Column by Position1" = Table.SplitColumn(#"Replaced Value14", "Merged.1", Splitter.SplitTextByPositions({0, 19}, false), {"Merged.1.1", "Merged.1.2"}),
#"Changed Type22" = Table.TransformColumnTypes(#"Split Column by Position1",{{"Merged.1.1", type datetime}, {"Merged.1.2", type text}}),
#"Renamed Columns4" = Table.RenameColumns(#"Changed Type22",{{"Merged.1.1", "Fra1"}, {"Merged.1.2", "Til1"}}),
#"Replaced Value15" = Table.ReplaceValue(#"Renamed Columns4",",to:L","",Replacer.ReplaceText,{"Til1"}),
#"Split Column by Delimiter19" = Table.SplitColumn(#"Replaced Value15", "Til1", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Til1.1", "Til1.2"}),
#"Changed Type23" = Table.TransformColumnTypes(#"Split Column by Delimiter19",{{"Til1.1", type text}, {"Til1.2", type text}}),
#"Replaced Value16" = Table.ReplaceValue(#"Changed Type23",",type","",Replacer.ReplaceText,{"Til1.1"}),
#"Replaced Value17" = Table.ReplaceValue(#"Replaced Value16","T"," ",Replacer.ReplaceText,{"Til1.1"}),
#"Changed Type24" = Table.TransformColumnTypes(#"Replaced Value17",{{"Til1.1", type datetime}}),
#"Renamed Columns5" = Table.RenameColumns(#"Changed Type24",{{"Til1.1", "Til1"}, {"Til1.2", "Type1"}}),
#"Replaced Value18" = Table.ReplaceValue(#"Renamed Columns5","from:L","",Replacer.ReplaceText,{"Merged.2"}),
#"Split Column by Delimiter20" = Table.SplitColumn(#"Replaced Value18", "Merged.2", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.2.1", "Merged.2.2"}),
#"Changed Type25" = Table.TransformColumnTypes(#"Split Column by Delimiter20",{{"Merged.2.1", type text}, {"Merged.2.2", type text}}),
#"Replaced Value19" = Table.ReplaceValue(#"Changed Type25",",type","",Replacer.ReplaceText,{"Merged.2.1"}),
#"Replaced Value20" = Table.ReplaceValue(#"Replaced Value19","T"," ",Replacer.ReplaceText,{"Merged.2.1"}),
#"Replaced Value21" = Table.ReplaceValue(#"Replaced Value20","L","",Replacer.ReplaceText,{"Merged.2.1"}),
#"Split Column by Delimiter21" = Table.SplitColumn(#"Replaced Value21", "Merged.2.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.2.1.1", "Merged.2.1.2"}),
#"Changed Type26" = Table.TransformColumnTypes(#"Split Column by Delimiter21",{{"Merged.2.1.1", type datetime}, {"Merged.2.1.2", type text}}),
#"Renamed Columns6" = Table.RenameColumns(#"Changed Type26",{{"Merged.2.1.1", "Fra2"}}),
#"Replaced Value22" = Table.ReplaceValue(#"Renamed Columns6","to:","",Replacer.ReplaceText,{"Merged.2.1.2"}),
#"Changed Type27" = Table.TransformColumnTypes(#"Replaced Value22",{{"Merged.2.1.2", type datetime}}),
#"Renamed Columns7" = Table.RenameColumns(#"Changed Type27",{{"Merged.2.1.2", "Til2"}, {"Merged.2.2", "Type2"}}),
#"Replaced Value23" = Table.ReplaceValue(#"Renamed Columns7","from:L","",Replacer.ReplaceText,{"Merged.3"}),
#"Split Column by Delimiter22" = Table.SplitColumn(#"Replaced Value23", "Merged.3", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.3.1", "Merged.3.2"}),
#"Changed Type28" = Table.TransformColumnTypes(#"Split Column by Delimiter22",{{"Merged.3.1", type text}, {"Merged.3.2", type text}}),
#"Replaced Value24" = Table.ReplaceValue(#"Changed Type28",",type","",Replacer.ReplaceText,{"Merged.3.1"}),
#"Replaced Value25" = Table.ReplaceValue(#"Replaced Value24","T"," ",Replacer.ReplaceText,{"Merged.3.1"}),
#"Replaced Value26" = Table.ReplaceValue(#"Replaced Value25","L","",Replacer.ReplaceText,{"Merged.3.1"}),
#"Split Column by Delimiter23" = Table.SplitColumn(#"Replaced Value26", "Merged.3.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.3.1.1", "Merged.3.1.2"}),
#"Changed Type29" = Table.TransformColumnTypes(#"Split Column by Delimiter23",{{"Merged.3.1.1", type datetime}, {"Merged.3.1.2", type text}}),
#"Replaced Value27" = Table.ReplaceValue(#"Changed Type29","to:","",Replacer.ReplaceText,{"Merged.3.1.2"}),
#"Changed Type30" = Table.TransformColumnTypes(#"Replaced Value27",{{"Merged.3.1.2", type datetime}}),
#"Renamed Columns8" = Table.RenameColumns(#"Changed Type30",{{"Merged.3.1.1", "Fra3"}, {"Merged.3.1.2", "Til3"}, {"Merged.3.2", "Type3"}}),
#"Replaced Value28" = Table.ReplaceValue(#"Renamed Columns8","from:L","",Replacer.ReplaceText,{"Merged.4"}),
#"Split Column by Delimiter24" = Table.SplitColumn(#"Replaced Value28", "Merged.4", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.4.1", "Merged.4.2"}),
#"Changed Type31" = Table.TransformColumnTypes(#"Split Column by Delimiter24",{{"Merged.4.1", type text}, {"Merged.4.2", type text}}),
#"Replaced Value29" = Table.ReplaceValue(#"Changed Type31",",type","",Replacer.ReplaceText,{"Merged.4.1"}),
#"Replaced Value30" = Table.ReplaceValue(#"Replaced Value29","L","",Replacer.ReplaceText,{"Merged.4.1"}),
#"Replaced Value31" = Table.ReplaceValue(#"Replaced Value30","T"," ",Replacer.ReplaceText,{"Merged.4.1"}),
#"Split Column by Delimiter25" = Table.SplitColumn(#"Replaced Value31", "Merged.4.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.4.1.1", "Merged.4.1.2"}),
#"Changed Type32" = Table.TransformColumnTypes(#"Split Column by Delimiter25",{{"Merged.4.1.1", type text}, {"Merged.4.1.2", type text}}),
#"Replaced Value32" = Table.ReplaceValue(#"Changed Type32","to:","",Replacer.ReplaceText,{"Merged.4.1.2"}),
#"Changed Type33" = Table.TransformColumnTypes(#"Replaced Value32",{{"Merged.4.1.1", type datetime}, {"Merged.4.1.2", type datetime}}),
#"Renamed Columns9" = Table.RenameColumns(#"Changed Type33",{{"Merged.4.1.1", "Fra4"}, {"Merged.4.1.2", "Til4"}, {"Merged.4.2", "Type4"}}),
#"Replaced Value33" = Table.ReplaceValue(#"Renamed Columns9","from:L","",Replacer.ReplaceText,{"Merged.5"}),
#"Split Column by Delimiter26" = Table.SplitColumn(#"Replaced Value33", "Merged.5", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.5.1", "Merged.5.2"}),
#"Changed Type34" = Table.TransformColumnTypes(#"Split Column by Delimiter26",{{"Merged.5.1", type text}, {"Merged.5.2", type text}}),
#"Replaced Value34" = Table.ReplaceValue(#"Changed Type34",",type","",Replacer.ReplaceText,{"Merged.5.1"}),
#"Replaced Value35" = Table.ReplaceValue(#"Replaced Value34","T"," ",Replacer.ReplaceText,{"Merged.5.1"}),
#"Replaced Value36" = Table.ReplaceValue(#"Replaced Value35","L","",Replacer.ReplaceText,{"Merged.5.1"}),
#"Split Column by Delimiter27" = Table.SplitColumn(#"Replaced Value36", "Merged.5.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.5.1.1", "Merged.5.1.2"}),
#"Changed Type35" = Table.TransformColumnTypes(#"Split Column by Delimiter27",{{"Merged.5.1.1", type datetime}, {"Merged.5.1.2", type text}}),
#"Replaced Value37" = Table.ReplaceValue(#"Changed Type35","to:","",Replacer.ReplaceText,{"Merged.5.1.2"}),
#"Changed Type36" = Table.TransformColumnTypes(#"Replaced Value37",{{"Merged.5.1.2", type datetime}}),
#"Renamed Columns10" = Table.RenameColumns(#"Changed Type36",{{"Merged.5.1.1", "Fra5"}, {"Merged.5.1.2", "Til5"}, {"Merged.5.2", "Type5"}}),
#"Replaced Value38" = Table.ReplaceValue(#"Renamed Columns10","from:L","",Replacer.ReplaceText,{"Merged.6"}),
#"Split Column by Delimiter28" = Table.SplitColumn(#"Replaced Value38", "Merged.6", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.6.1", "Merged.6.2"}),
#"Changed Type37" = Table.TransformColumnTypes(#"Split Column by Delimiter28",{{"Merged.6.1", type text}, {"Merged.6.2", type text}}),
#"Replaced Value39" = Table.ReplaceValue(#"Changed Type37",",type","",Replacer.ReplaceText,{"Merged.6.1"}),
#"Replaced Value40" = Table.ReplaceValue(#"Replaced Value39","T"," ",Replacer.ReplaceText,{"Merged.6.1"}),
#"Replaced Value41" = Table.ReplaceValue(#"Replaced Value40","L","",Replacer.ReplaceText,{"Merged.6.1"}),
#"Split Column by Delimiter29" = Table.SplitColumn(#"Replaced Value41", "Merged.6.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.6.1.1", "Merged.6.1.2"}),
#"Changed Type38" = Table.TransformColumnTypes(#"Split Column by Delimiter29",{{"Merged.6.1.1", type datetime}, {"Merged.6.1.2", type text}}),
#"Replaced Value42" = Table.ReplaceValue(#"Changed Type38","to:","",Replacer.ReplaceText,{"Merged.6.1.2"}),
#"Changed Type39" = Table.TransformColumnTypes(#"Replaced Value42",{{"Merged.6.1.2", type datetime}}),
#"Renamed Columns11" = Table.RenameColumns(#"Changed Type39",{{"Merged.6.1.1", "Fra6"}, {"Merged.6.1.2", "Til6"}, {"Merged.6.2", "Type6"}}),
#"Replaced Value43" = Table.ReplaceValue(#"Renamed Columns11","from:L","",Replacer.ReplaceText,{"Merged.7"}),
#"Split Column by Delimiter30" = Table.SplitColumn(#"Replaced Value43", "Merged.7", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, true), {"Merged.7.1", "Merged.7.2"}),
#"Changed Type40" = Table.TransformColumnTypes(#"Split Column by Delimiter30",{{"Merged.7.1", type text}, {"Merged.7.2", type text}}),
#"Replaced Value44" = Table.ReplaceValue(#"Changed Type40","T"," ",Replacer.ReplaceText,{"Merged.7.1"}),
#"Replaced Value45" = Table.ReplaceValue(#"Replaced Value44","L","",Replacer.ReplaceText,{"Merged.7.1"}),
#"Replaced Value46" = Table.ReplaceValue(#"Replaced Value45",",type","",Replacer.ReplaceText,{"Merged.7.1"}),
#"Split Column by Delimiter31" = Table.SplitColumn(#"Replaced Value46", "Merged.7.1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Merged.7.1.1", "Merged.7.1.2"}),
#"Changed Type41" = Table.TransformColumnTypes(#"Split Column by Delimiter31",{{"Merged.7.1.1", type datetime}, {"Merged.7.1.2", type text}}),
#"Replaced Value47" = Table.ReplaceValue(#"Changed Type41","to:","",Replacer.ReplaceText,{"Merged.7.1.2"}),
#"Changed Type42" = Table.TransformColumnTypes(#"Replaced Value47",{{"Merged.7.1.2", type datetime}}),
#"Renamed Columns12" = Table.RenameColumns(#"Changed Type42",{{"Merged.7.1.1", "Fra7"}, {"Merged.7.1.2", "Til7"}, {"Merged.7.2", "Type7"}}),
#"Removed Columns12" = Table.RemoveColumns(#"Renamed Columns12",{"Shift.9"}),
#"Replaced Value48" = Table.ReplaceValue(#"Removed Columns12","REST ","REST",Replacer.ReplaceText,{"Type", "Type1", "Type2", "Type3", "Type4", "Type5", "Type6", "Type7"}),
#"Replaced Value49" = Table.ReplaceValue(#"Replaced Value48"," REST","REST",Replacer.ReplaceText,{"Type", "Type1", "Type2", "Type3", "Type4", "Type5", "Type6", "Type7"}),
#"Replaced Value50" = Table.ReplaceValue(#"Replaced Value49","REGULAR ","REGULAR",Replacer.ReplaceText,{"Type", "Type1", "Type2", "Type3", "Type4", "Type5", "Type6", "Type7"}),
#"Replaced Value51" = Table.ReplaceValue(#"Replaced Value50"," REGULAR","REGULAR",Replacer.ReplaceText,{"Type", "Type1", "Type2", "Type3", "Type4", "Type5", "Type6", "Type7"}),
#"Added Custom" = Table.AddColumn(#"Replaced Value51", "Hours rest", each ((if[Type7]="REST" then [Til7]-[Fra7] else 0)
+(if[Type6]="REST" then [Til6]-[Fra6] else 0)
+(if[Type5]="REST" then [Til5]-[Fra5] else 0)
+(if[Type4]="REST" then [Til4]-[Fra4] else 0)
+(if[Type3]="REST" then [Til3]-[Fra3] else 0)
+(if[Type2]="REST" then [Til2]-[Fra2] else 0)
+(if[Type1]="REST" then [Til1]-[Fra1] else 0)
+(if[Type]="REST" then [Til]-[Fra] else 0))*24)
in
#"Added Custom"
You can't directly subtract dates
Either try
Duration.Days([Fra]-[Fra1])
or
Number.From ([Fra])-Number.From([Fra1])
See potential duration functions https://learn.microsoft.com/en-us/powerquery-m/duration-functions
also maybe use some error catching
=( try if [Type6]="REST" then Duration.Days([Til6]-[Fra6]) else 0 otherwise 0)
or
=( try if [Type6]="REST" then Number.From([Til6]) -Number.From([Fra6]) else 0 otherwise 0)
Related
Name not recognized error in List.Generate
I have a table with 10 columns, StepDet 1 - StepDet 10. In each one is an embedded table. I want to expand each table but not with all the columns. I only want to expand Step Name and Step 1 - 10.
I tried the below code, but I get an error that Expression.Error: The name 'prevTable' wasn't recognized. Make sure it's spelled correctly.
I would appreciate any help to move forward.
let
// Load the query
Source = Source_name,
// Generate the list of columns to expand
columnsToExpand = List.Generate(
() => [i = 1],
each [i] <= 10,
each [i = [i] + 1],
each "StepDet " & Text.From([i])
),
// Expand the columns
expandedTable = List.Last(List.Generate(
() => [i = 0, prevTable = Source],
each [i] <= List.Count(columnsToExpand)-1,
each [
i = [i] + 1,
prevTable = Table.ExpandTableColumn(
prevTable,
columnsToExpand{[i]},
{"Step Name", "ActionDet 1", "ActionDet 2", "ActionDet 3", "ActionDet 4", "ActionDet 5", "ActionDet 6", "ActionDet 7", "ActionDet 8", "ActionDet 9", "ActionDet 10"},
List.Transform(
{"Step Name", "ActionDet 1", "ActionDet 2", "ActionDet 3", "ActionDet 4", "ActionDet 5", "ActionDet 6", "ActionDet 7", "ActionDet 8", "ActionDet 9", "ActionDet 10"},
each columnsToExpand{[i]} & "." & _
)
)
],
each [prevTable]
))
in
expandedTable
Does something like this get you closer?
let Source = < >,
columnsToExpand = List.Transform({1 .. 10}, each "StepDet "&Text.From(_)) & {"Step Name"},
#"Unpivoted Columns" = Table.UnpivotOtherColumns(Source, List.Difference(Table.ColumnNames(Source),columnsToExpand), "Attribute", "Value"),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(#"Unpivoted Columns", "Value"), each if _ is table then Table.ColumnNames(_) else {}))),
#"Expanded" = Table.ExpandTableColumn(#"Unpivoted Columns", "Value",ColumnsToExpand ,ColumnsToExpand )
in #"Expanded"
only list-like objects are allowed to be passed to isin(), you passed a [NoneType]
Am getting this on callback, am using Plotly dash, The data frame is of object types, is this an issue with "value =" in dcc.dropdown?. But a similar dropdown works fine!!
dcc.Dropdown(
id="account_name_selector",
options=[{"label": i, "value": i} for i in df["Account_Name"].unique()],
multi=True,
value=[i for i in df["Account_Name"].unique()],
className="dcc_control",
),
#app.callback(
Output("main_graph", "figure"),
[
Input("account_name_selector", "value"),
Input("week_selector", "value"),
]
)
def main_graph_fig(account_name_value, week_value):
dff = df[df.Week.isin(week_value) & df.Account_Name.isin(account_name_value)]
Perform an operation on multiple tables in a list
Trying to teach myself Power Query and can't get my head around this particular process.
I understand there may be other, perhaps more efficient methods of attaining the desired result, but my purpose here is to understand the process of performing multiple operations on each table in a list of tables, where the number of entries in the list is unknown and/or large.
My original data has pairs of information [Credit, Name] in adjacent columns with an unknown number of pairs.
Since it is a table, the different column/pairs have different Names.
Credit|Name|Credit1|Name1|...
If I demote the headers and transpose the table, the column headers will wind up in Column 1, and I can strip off the differentiating digit.
Using Table.Split, I can then create a number of tables where each pair of columns has the Identical headers.
I can then combine these tables to create a single, two column table, where I can Group and aggregate to get the results.
My problem is that I have not been able to figure out how to do the:
Table.PromoteHeaders(Table.Transpose(table))
operation on each table.
This M-code produces the desired result for the four pairs of columns in the provided data, but is clearly not scalable since the number of tables needs to be known in advance.
let
//Create the table
Tbl1= Table.FromRecords({
[Credit = 1, Name = "Bob", Credit2 = 2, Name2 = "Jim", Credit3 = 1, Name3 = "George", Credit4 = 1.75, Name4="Phil"],
[Credit = 2, Name = "Phil", Credit2 = 4, Name2="George", Credit3 = 2.5, Name3 = "Stephen",Credit4 = 6, Name4="Bob"]
}),
//Demote headers and transpose
transpose1 = Table.Transpose( Table.DemoteHeaders(Tbl1)),
//Create matching names for what will eventually be the final Column Headers
#"Split Column by Character Transition" = Table.SplitColumn(transpose1, "Column1", Splitter.SplitTextByCharacterTransition((c) => not List.Contains({"0".."9"}, c), {"0".."9"}), {"Column1.1", "Column1.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Character Transition",{"Column1.2"}),
//Create multiple tables from above
multTables = Table.Split(#"Removed Columns",2),
/*transpose and promote the headers for each table
HOW can I do this in a single step when I don't know how many tables there might be???
*/
tbl0 = Table.PromoteHeaders(Table.Transpose(multTables{0}),[PromoteAllScalars=true]),
tbl1 = Table.PromoteHeaders(Table.Transpose(multTables{1}),[PromoteAllScalars=true]),
tbl2 = Table.PromoteHeaders(Table.Transpose(multTables{2}),[PromoteAllScalars=true]),
tbl3 = Table.PromoteHeaders(Table.Transpose(multTables{3}),[PromoteAllScalars=true]),
combTable = Table.Combine({tbl0,tbl1,tbl2,tbl3})
in
combTable
Original Table
Demoted headers / Transposed table
Desired Result
Any help would be appreciated.
You could also try replacing this part of your code:
tbl0 = Table.PromoteHeaders(Table.Transpose(multTables{0}),[PromoteAllScalars=true]),
tbl1 = Table.PromoteHeaders(Table.Transpose(multTables{1}),[PromoteAllScalars=true]),
tbl2 = Table.PromoteHeaders(Table.Transpose(multTables{2}),[PromoteAllScalars=true]),
tbl3 = Table.PromoteHeaders(Table.Transpose(multTables{3}),[PromoteAllScalars=true]),
combTable = Table.Combine({tbl0,tbl1,tbl2,tbl3})
in
combTable
with this:
Custom1 = List.Transform(multTables, each Table.PromoteHeaders( Table.Transpose(_),[PromoteAllScalars=true])),
#"Converted to Table" = Table.FromList(Custom1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandTableColumn(#"Converted to Table", "Column1", {"Credit", "Name"}, {"Credit", "Name"})
in
#"Expanded Column1"
A bit clunky but seems to work with any number of rows and any number of 2 column paired columns.
First, a bunch of indexes modified in different ways, then a filter into two table, unpivot and merge
let Tbl1= Table.FromRecords({
[Credit = 1, Name = "Bob", Credit2 = 2, Name2 = "Jim", Credit3 = 1, Name3 = "George", Credit4 = 1.75, Name4="Phil"],
[Credit = 2, Name = "Phil", Credit2 = 4, Name2="George", Credit3 = 2.5, Name3 = "Stephen",Credit4 = 6, Name4="Bob"],
[Credit = 3, Name = "Sam", Credit2 = 5, Name2="Allen", Credit3 = 3.5, Name3 = "Ralph",Credit4 = 7, Name4="Nance"]
}),
#"Transposed Table" = Table.Transpose(Tbl1),
#"Added Index" = Table.AddIndexColumn(#"Transposed Table", "Index", 0, .5),
#"Added Custom" = Table.AddColumn(#"Added Index", "Index2", each Number.RoundDown([Index])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index"}),
#"Added Index1" = Table.AddIndexColumn(#"Removed Columns", "Index", 0, 1),
#"Added Custom1" = Table.AddColumn(#"Added Index1", "Custom", each Number.Mod([Index],2)),
#"Removed Columns1" = Table.RemoveColumns(#"Added Custom1",{"Index"}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Removed Columns1", {"Index2", "Custom"}, "Attribute", "Value"),
#"Filtered Rows" = Table.SelectRows(#"Unpivoted Other Columns", each ([Custom] = 0)),
#"Filtered Rows2" = Table.SelectRows(#"Unpivoted Other Columns", each ([Custom] = 1)),
#"Merged Queries" = Table.NestedJoin(#"Filtered Rows2",{"Index2", "Attribute"},#"Filtered Rows",{"Index2", "Attribute"},"Filtered Rows",JoinKind.LeftOuter),
#"Expanded Filtered Rows" = Table.ExpandTableColumn(#"Merged Queries", "Filtered Rows", {"Value"}, {"Value.1"}),
#"Removed Columns2" = Table.RemoveColumns(#"Expanded Filtered Rows",{"Index2", "Custom", "Attribute"})
in #"Removed Columns2"
Another way of doing it is to create two tables by selecting and unpivoting groups of columns based on their column names, then combining them using a custom column referring to the row index in each table
let Tbl1= Table.FromRecords({
[Credit = 1, Name = "Bob", Credit2 = 2, Name2 = "Jim", Credit3 = 1, Name3 = "George", Credit4 = 1.75, Name4="Phil"],
[Credit = 2, Name = "Phil", Credit2 = 4, Name2="George", Credit3 = 2.5, Name3 = "Stephen",Credit4 = 6, Name4="Bob"],
[Credit = 3, Name = "Sam", Credit2 = 5, Name2="Allen", Credit3 = 3.5, Name3 = "Ralph",Credit4 = 7, Name4="Nance"]
}),
Credit = List.Select(Table.ColumnNames(Tbl1), each Text.Contains(_, "Credit")),
Name = List.Select(Table.ColumnNames(Tbl1), each Text.Contains(_, "Name")),
// create table of just Names with index
#"Removed Columns1" = Table.RemoveColumns(Tbl1,Credit),
#"Added Index" = Table.AddIndexColumn(#"Removed Columns1", "Index", 0, 1),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
// create table of just Credits with index
#"Removed Columns2" = Table.RemoveColumns(Tbl1,Name),
#"Added Index2" = Table.AddIndexColumn(#"Removed Columns2", "Index", 0, 1),
#"Unpivoted Other Columns2" = Table.UnpivotOtherColumns(#"Added Index2", {"Index"}, "Attribute", "Value"),
#"Added Index1" = Table.AddIndexColumn(#"Unpivoted Other Columns2", "Index.1", 0, 1),
//merge two table together and remove excess columns
#"Added Custom" = Table.AddColumn(#"Added Index1", "Custom", each #"Unpivoted Other Columns"{[Index.1]}[Value]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index", "Attribute", "Index.1"})
in #"Removed Columns"
Insert new column with list of values in PowerQuery/M
If I have the following source:
#"My Source" = Table.FromRecords({
[Name="Jared Smith", Age=24],
[Name = "Tom Brady", Age=44],
[Name="Hello Tom", Age = null],
[Name = "asdf", Age = "abc"]
}),
How would I add a new column from a list of values, for example:
Table.AddColumn(#"My Source", "New Col", {'x', 'y', 'z', null})
Now my table would have three columns. How could this be done?
Here's another way. It starts similarly to the approach used by Ron, by adding an index, but then instead of using merge it just uses the index as a reference to the appropriate list item.
let
Source1 = Table.FromRecords({
[Name="Jared Smith", Age=24],
[Name = "Tom Brady", Age=44],
[Name="Hello Tom", Age = null],
[Name = "asdf", Age = "abc"]
}),
#"Added Index" = Table.AddIndexColumn(Source1, "Index", 0, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each {"x", "y", "z", null}{[Index]}),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index"})
in
#"Removed Columns"
I'm a PQ beginner, so there may be more efficient methods, but here's one:
Add an Index column to each of the tables
Merge the two tables, using the Index column as the key
Delete the Index column
let
Source1 = Table.FromRecords({
[Name="Jared Smith", Age=24],
[Name = "Tom Brady", Age=44],
[Name="Hello Tom", Age = null],
[Name = "asdf", Age = "abc"]
}),
#"Added Index" = Table.AddIndexColumn(Source1, "Index", 0, 1),
Source2 = Table.FromRecords({
[New="x"],
[New = "y"],
[New = "z"],
[New = null]
}),
#"Added Index2" = Table.AddIndexColumn(Source2, "Index", 0, 1),
Merge = Table.Join(#"Added Index", "Index",#"Added Index2", "Index"),
#"Removed Columns" = Table.RemoveColumns(Merge,{"Index"})
in
#"Removed Columns"
Trying to convert a CSV file into a Parquet file using Spark
Here is the spark-shell script which I am using to convert csv data into parquet:
import org.apache.spark.sql.types._;
val sqlContext = new org.apache.spark.sql.SQLContext(sc);
val df = sqlContext.read.format("com.databricks.spark.csv").option("header","true").load("/uploads/01ff5191-27c4-42db-a8e0-0d6594de3a5d/Worker_Snapshot_50_100000.csv");
val schema= StructType(Array(
StructField("CF_EmplID", StringType,true),
StructField("Worker", StringType,true),
StructField("CF_Sup Org Level 1", StringType,true),
StructField("CF_Sup Org Level 2", StringType,true),
StructField("CF_Sup Org Level 3", StringType,true),
StructField("CF_Sup Org Level 4", StringType,true),
StructField("Business Process Transaction", StringType,true),
StructField("Record Date", StringType,true),
StructField("CF_Fiscal Period", StringType,true),
StructField("Business Process Type", StringType,true),
StructField("Business Process Reason", StringType,true),
StructField("Active Status", BooleanType,true),
StructField("Age Group", StringType,true),
StructField("Annual Base Pay", StringType,true),
StructField("Base Pay Segment", StringType,true),
StructField("Compa-Ratio", StringType,true),
StructField("Company", StringType,true),
StructField("Compensation Grade", BooleanType,true),
StructField("Contingent Worker Type", StringType,true),
StructField("Cost Center", StringType,true),
StructField("Current Rating", StringType,true),
StructField("Employee Type", StringType,true),
StructField("Ending Headcount", IntegerType,true),
StructField("Ethnicity", StringType,true),
StructField("Exempt", BooleanType,true),
StructField("FTE", StringType,true),
StructField("Gender", StringType,true),
StructField("Highest Degree", StringType,true),
StructField("Hire Count", IntegerType,true),
StructField("Hire Year Text", IntegerType,true),
StructField("Hiring Source", StringType,true),
StructField("Involuntary Termination", StringType,true),
StructField("Involuntary Termination Count", IntegerType,true),
StructField("Is Critical Job", BooleanType,true),
StructField("Is High Loss Impact Risk", BooleanType,true),
StructField("Is High Potential", BooleanType,true),
StructField("Is Manager", BooleanType,true),
StructField("Is Retention Risk", BooleanType,true),
StructField("Job Category", StringType,true),
StructField("Job Code", IntegerType,true),
StructField("Job Family", IntegerType,true),
StructField("Job Family Group", StringType,true),
StructField("Job Profile", StringType,true),
StructField("Length of Service in Years including Partial Year", StringType,true),
StructField("Location", StringType,true),
StructField("Location - Country", StringType,true),
StructField("Loss Impact", StringType,true),
StructField("Management Level", StringType,true),
StructField("Manager", StringType,true),
StructField("Manager Count", IntegerType,true)
));
val dataFrame = spark.createDataFrame(df.rdd, schema)
var newDf = dataFrame
for(col <- dataFrame.columns){
newDf = newDf.withColumnRenamed(col,col.replaceAll("\\s", "_"))
}
newDf.write.parquet("/output_dir/parquet")
Seems pretty straight forward so far, but I am running into this exception which seems to be about trying to parse non-int value into a int field.
Here is the exception I am getting:
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:573)
at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:573)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:315)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:258)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:256)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:261)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.String is not a valid external type for schema of int
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.evalIfFalseExpr22$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_9$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
... 20 more
Am I doing something wrong when applying the schema to the data frame? I tried using the "inferSchema" option in the "sqlContext.read.format", but that seems to be guessing the types incorrectly.
Instead of
val dataFrame = spark.createDataFrame(df.rdd, schema)
use:
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header","true")
.schema(schema)
.load(...);
Try comparing schema of df and custom schema , convert the datatype of the columns as it matches both schema column-data types.