I am accessing a SSAS DMV through Power Query in Excel via:
let
Source = AnalysisServices.Database(TabularServerName, TabularDBName,
[Query="select * from $SYSTEM.TMSCHEMA_EXPRESSIONS"])
in
Source
This works great in Power BI, but in Excel, the Expression column is limited to a max of 1024 characters. How do I get Power Query in Excel to give me the entire value? My largest values are around 15000 characters, so still within the stated limits of Power Query that I can find.
If I set up a table with a connection and query behind it, Excel can pull in the entire Expression column, but the downside is the server and database cannot be parameterized and have to be manually changed in the connection. Also I don't remember how to do this manually, so I always have to access the DMV from DAX Studio and export to Excel to set it up!
Update
I did some heavy transformations of this column. I parsed out a value, I used it to merge the file with itself and add a column that I then did a bunch of transformations on, and then used it to replace text within the original problem column. And something in that pulled in the whole value. I tried just doing small parts of this, like adding a column that referenced the problem column, or doing a replace in the problem column, and none of that worked.
So, no, not easy to duplicate or figure out which step fixed it, but for my purposes, I now have what I need.
I think it is related to the type of the column your are loading in Excel. I had the same issue and read your answer (with Table.ReplaceValue).
Your solution is hiding the initial point : The function used in the expression you shared for Table.ReplaceValue() is Replacer.ReplaceText that as the additional specificity to convert a field of type Any
to type Text.
I tried to juste change the type of my field that was truncated when loaded in Excel, from type Any to type Text. Result : the complete values were then loaded in my worksheet.
I had to change this query today, and after I changed it, the values were truncated again. I added a Replace Value step at the end of the query on the truncated column and that seemed to fix it.
#"Replaced Value" = Table.ReplaceValue(#"Last Step","in ","in ",Replacer.ReplaceText,{"Truncated Column Name"})
in
#"Replaced Value"
Related
I am retrieving data through Power Query from an Oracle DB live to an Excel workbook. In PQ, under the "Transform" tab, there is a function to change the data type of a column, that I use to get all the decimal numbers displayed. In the M-code the function is called TransformColumnTypes. However I have some strings in the data that I cannot change to decimal number and produce an error. Is there a way to exclude these? Because the function takes the whole column at the moment.
Before applying function
Function producing error
Code
I don't think so. If you have multiple types within a column, text is the only one that doesn't produce errors.
But if it is only the first row like in your image, promoting it to header before setting the column type will fix the issue.
I'm experimenting with PowerQuery and I got to a good point, but I'm stuck with something. I checked here and other places, but couldn't find anything that helped me solve the issue.
I have a Source in PQ I can use and transform. Once this Source (a .csv) is loaded in PQ editor, I would need to verify if a specific column value is in another table/list or not.
So I created a small table in another sheet and created another Source for it.
I am trying to create a new column and validate now with "each if" if the current main Source value in that column (that obviously varies, they are names) is part of the list/table that is the other Source. And in case it is, the added column value will be YES, otherwise NO.
A simple check that in Excel I would have done probably with vlookup.
I always referred here as list/table as i tried to have it created as List or table, without success.
Is anybody here able to help?
= Table.AddColumn(#"Promoted Headers", "Real_A", each if Table.ContainsAny(Source, EMEAL, {[Analyst]=[EMEA]}) then "EMEA" else "XX")
This is what I tried last. The "Source" is the .CSV I read already and [Analyst] is one column of it.
[EMEA] is the name of the column in EMEAL (list).
The error I have:
Expression.Error: We cannot convert the value "Pippo" to type Record.
Details:
Value=Pippo
Type=Type
"Pippo" is the value in [Analyst] column, but as well one of the value in the EMEAL list... so, also here quite confused where the issue really is.
thanks!
The equivalent to this kind of VLOOKUP is a Merge-opertion like described here: http://www.myonlinetraininghub.com/excel-power-query-vlookup
I'm trying to read an Excel sheet from an XLS or XLSX file in memory using Delphi 7. When possible I use automation to read the cells one by one, but when Excel is not installed, I revert to using the ADO/ODBC Jet driver.
I connect using either
Provider=Microsoft.Jet.OLEDB.4.0; Data Source=file.xls;Extended Properties="Excel 8.0;Persist Security Info=False;IMEX=1;HDR=No";
Provider=Microsoft.ACE.OLEDB.12.0; Data Source=file.xlsx;Extended Properties="Excel 12.0;Persist Security Info=False;IMEX=1;HDR=No";
My problem then is that when I use the following query:
SELECT * FROM [SheetName$]
the returned results do not contain the empty rows or empty columns, so if the sheet contains such rows or columns, the following cells are shifted and do not end up in their correct position. I need the sheet to be loaded "as is", ie know exactly from what cell position each value comes from.
I tried to read the cells one by one by issuing one query of the form
SELECT F1 FROM `SheetName$A1:A1`
but now the driver returns an error saying "There is data outside the selected region". btw I had to use backticks to enclose the name because using brackets like this [SheetName$A1:A1] gave out a syntax error message.
Is there a way to tell the driver to select the sheet as-is, whithout skipping blanks? Or maybe a way to know from which cell position each value is returned?
For internal policy reasons (I know they are bad but I do not decide these), it is not possible to use a third party library, I really need this to work from standard Delphi 7 components.
I assume that if your data is say in the range B2:D10 for example, you want to include the column A as an empty column? Maybe? Is that correct? If that's the case, then your data set, when you read the sheet (SELECT * FROM [SheetName$]) would also return 1 million rows by 16K columns!
Can you not execute a query like: SELECT * FROM [SheetName$B2:D10] and use the ADO GetRows function to get an array - which will give you the size of the data. Then you can index into the array to get what data you want?
OK, the correct answer is
Use a third party library no matter what your boss says. Do not even
try ODBC/ADO to load arbitrary Excel files, you will hit a wall sooner or later.
It may work for excel files that contain a single data table, but not when you want to cherry pick data in a sheet primarily made for human consumption (ie where a single column contains some cells with introductory text, some with numerical data, some with comments, etc...)
Using IMEX=1 ignores empty lines and empty columns
Using IMEX=0 sometimes no longer ignores empty lines, but now some of the first non empty cells are considered field names instead of data, although HDR=No. Would not work anyway since valules in a column are of mixed types.
Explicitly looping across cells and making a SELECT * FROM [SheetName$A1:A1] works until you reach an empty cell, then you get access violations (see below)
Access violation at address 1B30B3E3 in module 'msexcl40.dll'. Read of address 00000000
I'm too old to want to try and guess the appropriate value to use so it works until someone comes with yet another mix of data in a column. Sorry for having wasted everybody's time.
In my SSAS cube I have a dimension that includes 6 date fields. They are all defined in the same way, with a Key field that is a date type and a Name field that is char(10) in format yyyy-mm-dd.
When I include those fields in an Excel pivot table, they all work fine except one. That one field is displayed correctly, but doesn't behave correctly when filtered. In particular, specifying a between filter always returns zero rows. Same with a greater than filter. Begins with works fine.
Once again, this only happens for one of the six date fields. But all six date fields are configured identically as far as I can tell. What type of mistake could cause this?
EDIT
Using SQLServer Profiler I can see that the MDX generated from Excel are identical for the dates that work and the one that doesn't (except for the field names changing of course). If I restrict the pivot table to a single date and add a between filter then the MDX is:
SELECT NON EMPTY Hierarchize({DrilldownLevel({[Participation Program].[Participation Start Date].[All]},,,INCLUDE_CALC_MEMBERS)})
DIMENSION PROPERTIES PARENT_UNIQUE_NAME,HIERARCHY_UNIQUE_NAME ON COLUMNS
FROM (SELECT Filter([Participation Program].[Participation Start Date].[Participation Start Date].AllMembers,
([Participation Program].[Participation Start Date].CurrentMember.member_caption>="1985"
AND [Participation Program].[Participation Start Date].CurrentMember.member_caption<="1990")) ON COLUMNS
FROM [Compass3])
WHERE ([Child].[Child is Handicapped].&[T],[Measures].[Child Count]) CELL PROPERTIES VALUE, FORMAT_STRING, LANGUAGE, BACK_COLOR, FORE_COLOR, FONT_FLAGS
That returns correct results for [Participation Start Date], but if I do the same thing with [Participation Stop Date] it returns 0 results. So it is a problem on the SSAS side rather than Excel side. But I still can see no difference in the way the two dates are configured in the cube, and I am 100% certain that there is data that should match the specified date range.
I would change the Key for those attributes to a numeric representation of the date in YYYYMMDD format. I would achieve this via a SQL View where I would use the CONVERT function.
I would not use the date datatype at all in SSAS, as it's internal representation is obscure/uncertain.
I've come across a similar problem using epplus. If you haven't defined the date explicitly, Excel may not realize that the data is a date. This causes problems with sorts and filtering. (Note that the code below is C#, but the idea behind it with respect to the excel formulas/formats should be the same.)
When you generate the excel cell, explicitly define a date formula in the cell:
ws.Cells[rowCount, columnCount].Formula = "=DATE(" + myDate.ToString("yyyy,M,d") + ")";
Then, set the cell formatting to display the date the way you want:
ws.Cells[rowCounter, columnCount].Style.Numberformat.Format = "d-mmm-yyyy";
I am developing a SSIS package, trying to update an existing SQL table from a CSV flat file. All of the columns are successfully updating except for one column. If I ignore this column on truncate, my package completes successfully. So I know this is a truncate problem and not error.
This column is empty for almost every row. However, there are a few rows where this field is 200-300 characters. My data conversion task identified this field as a DT_WSTR, but from what I've read elsewhere maybe this should be DT_NTEXT. I've tried both and I even set the DT_WSTR to 500. But none of this fixed my problem. How can I fix? What data type should this column be in my SQL table?
Error: 0xC02020A1 at Data Flow Task 1, Source - Berkeley812_csv [1]: Data conversion failed. The data conversion for column "Reason for Delay in Transition" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
Error: 0xC020902A at Data Flow Task 1, Source - Berkeley812_csv [1]: The "output column "Reason for Delay in Transition" (110)" failed because truncation occurred, and the truncation row disposition on "output column "Reason for Delay in Transition" (110)" specifies failure on truncation. A truncation error occurred on the specified object of the specified component.
Error: 0xC0202092 at Data Flow Task 1, Source - Berkeley812_csv [1]: An error occurred while processing file "D:\ftproot\LocalUser\RyanDaulton\Documents\Berkeley Demographics\Berkeley812.csv" on data row 758.
One possible reason for this error is that your delimiter character (comma, semi-colon, pipe, whatever) actually appears in the data in one column. This can give very misleading error messages, often with the name of a totally different column.
One way to check this is to redirect the 'bad' rows to a separate file and then inspect them manually. Here's a brief explanation of how to do that:
http://redmondmag.com/articles/2010/04/12/log-error-rows-ssis.aspx
If that is indeed your problem, then the best solution is to fix the files at the source to quote the data values and/or use a different delimeter that isn't in the data.
I've had this issue before, it is likely that the default column size for the file is incorrect. It will put a default size of 50 characters but the data you are working with is larger. In the advanced settings for your data file, adjust the column size from 50 to the table's column size.
I suspect the
or one or more characters had no match in the target code page
part of the error.
If you remove the rows with values in that column, does it load?
Can you identify, in other words, the rows which cause the package to fail?
It could be the data is too long, or it could be that there's some funky character in there SQL Server doesn't like.
If this is coming from SQL Server Import Wizard, try editing the definition of the column on the Data Source, it is 50 characters by default, but it can be longer.
Data Soruce -> Advanced -> Look at the column that goes in error -> change OutputColumnWidth to 200 and try again.
I've had this problem before, you can go to "advanced" tab of "choose a data source" page and click on "suggested types" button, and set the "number of rows" as much as you want. after that, the type and text qualified are set to the true values.
i applied the above solution and can convert my data to SQL.
In my case, some of my rows didn't have the same number of columns as the header. Example, Header has 10 columns, and one of your rows has 8 or 9 columns. (Columns = Count number of you delimiter characters in each line)
If all other options have failed, trying recreating the data import task and/or the connection manager. If you've made any changes since the task was originally created, this can sometimes do the trick. I know it's the equivalent of rebooting, but, hey, if it works, it works.
I have same problem, and it is due to a column with very long data.
When I map it, I changed it from DT_STR to Text_Stream, and it works
In the destination, in advanced, check that the length of the column is equal to the source.
OuputColumnWidth of column must be increased.
Path: Source Connection manager-->Advanced-->OuputColumnWidth