I can't figure out what's going on. I'm exporting data from our Oracle 12c db to Excel from a simple select query, using cfspreadsheet:
<cfset var_filenameis = "report.xlsx">
<cfset SpreadsheetObj = spreadsheetNew("true")>
<cfset SpreadsheetObj = spreadsheetNew("#var_filenameis#","yes")>
<cfspreadsheet action="write" filename="#var_filenameis#" query="get_data" overwrite="true">
<cflocation url = "#var_filenameis#">
One column of data contains catalog numbers in various formats. Some of them become truncated when exported to xlsx. For example, 02923F becomes 2923 and 08552D becomes 8552. However, 08566A stays 08566A and 02584C also stays the same. The data shows correctly when viewed in the browser. Direct export from the DB also shows correct data. I could understand the leading 0 disappearing, but not the letter.
I have re-entered the problem catalog numbers in the DB to make sure there were no extra characters, to no avail. If I add a single quote in front of the catalog number, the display is correct except that I can't have a single quote showing in the output.
I get the same result with CF9 and CF11. I can't even tell if the problem is with cfspreadsheet or xlxs. Any good ideas? Thanks!
(Too long for comments)
If you are using CF11, try using SpreadSheetAddRows instead of cfspreadsheet. It is a bit smarter and uses the data types of the query columns to determine the proper cell type. As long as the query column is some sort of VARCHAR, the string value will be preserved.
<!--- sample data --->
<cfset qData = queryNew("")>
<cfset queryAddColumn(qData, "StringValue", "varchar", ["02923F","08552D","08566A","02584C"])>
<!--- populate sheet --->
<cfset sheet = SpreadsheetNew("Sheet1", true)>
<cfset spreadSheetAddRows(sheet, qData)>
<!--- display results --->
<cfheader name="Content-Disposition" value="attachment;filename=testFile.xlsx">
<cfcontent type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" variable="#SpreadSheetReadBinary(sheet)#">
I can't figure out what's going on
It is because POI (or possibly CF) interprets the "F" and "D" as special literals indicating the values are floating point numbers:
Floating-Point Literals
A floating-point literal is of type float if it ends with the letter F
or f; otherwise its type is double and it can optionally end with the
letter D or d.
So your strings are being converted to numbers. That is why the leading zero and trailing "F" or "D" disappear.
Related
I previously thought that you can take an ordinary string and treat it as a list with "" as the delimiter, but CF doesn't seem to allow that. Doing something like this:
<cfloop list="abcdef" delimiters="" index="thisLetter">
<cfoutput>#thisLetter#</cfoutput><br>
</cfloop>
only results in 1 iteration of the loop, and an output of the whole string on one line. Not what's desired.
I did find you can use the Java string.split() method with a "" delimiter:
<cfset myArray = "abcdef".split("")>
<cfdump var="#myArray#">
But then the first element in the resulting array is an empty string, so there's an extra step in removing that from the array.
Is there a more elegant way to iterate over all the chars in this string without having to do any special string manipulation first, and without having to use left(), right(), or mid()?
I recommend using Java's String.toCharArray():
<cfset myString = "aäй漢か">
<cfset theChars = myString.toCharArray()>
<cfloop array="#theChars#" index="char">
<cfoutput>#char#</cfoutput>
</cfloop>
Looks like this is what I was looking for: https://www.bennadel.com/blog/307-ask-ben-iterating-over-the-characters-in-a-string.htm
Using mid() is simplest, and isn't too cumbersome. Still, would be nice to be able to use CF list loop with delimiters="".
An Excel table as data source may contain error values (#NA, #DIV/0), which could disturbe later some steps during the transformation process in Power Query.
Depending of the following steps, we may get no output but an error. So how to handle this cases?
I found two standard steps in Power Query to catch them:
Remove errors (UI: Home/Remove Rows/Remove Errors) -> all rows with an error will be removed
Replace error values (UI: Transform/Replace Errors) -> the columns have first to be selected for performing this operations.
The first possibility is not a solution for me, since I want to keep the rows and just replace the error values.
In my case, my data table will change over the time, means the column name may change (e.g. years), or new columns appear. So the second possibility is too static, since I do not want to change the script each time.
So I've tried to get a dynamic way to clean all columns, indepent from the column names (and number of columns). It replaces the errors by a null value.
let
Source = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
//Remove errors of all columns of the data source. ColumnName doesn't play any role
Cols = Table.ColumnNames(Source),
ColumnListWithParameter = Table.FromColumns({Cols, List.Repeat({""}, List.Count(Cols))}, {"ColName" as text, "ErrorHandling" as text}),
ParameterList = Table.ToRows(ColumnListWithParameter ),
ReplaceErrorSource = Table.ReplaceErrorValues(Source, ParameterList)
in
ReplaceErrorSource
Here the different three queries messages, after I've added two new column (with errors) to the source:
If anybody has another solution to make this kind of data cleaning, please write your post here.
let
src = Excel.CurrentWorkbook(){[Name="Tabelle1"]}[Content],
cols = Table.ColumnNames(src),
replace = Table.ReplaceErrorValues(src, List.Transform(cols, each {_, "!"}))
in
replace
Just for novices like me in Power Query
"!" could be any string as substitute for error values. I initially thought it was a wild card.
List.Transform(cols, each {_, "!"}) generates the list of error handling by column for the main funcion:
Table.ReplaceErrorValues(table_with errors, {{col1,error_str1},{col2,error_str2},{},{}, ...,{coln,error_strn}})
Nice elegant solution, Sergei
I am trying to pass a struct through a function, but the integers in it are converting to scientific notation.
Before deSerialization :
{"businessUnitValidList":2003051509034372557922
, "shortMessage":"Success"
, "longMessage":"Request Completed Successfully."
, "status":20001
}
After deSerialization:
businessUnitValidList 2.00305150903E+021
I have tried converting it into a string but it still gives me the same output. Any ideas?
Note: If I have more than one value in my businessUnitValidList, the numbers show up the way they are supposed to.
EDIT
This is the current code iteration:
<cfloop array="#businessUnitArray#" index="i">
<cfquery name="validatebusinessUnit" datasource="dbproduction">
select doctorid from survey.dbo.clientLocationMap
where clientbrandid = '#arguments.clientBrandid#'
and clientLocation = '#i#'
</cfquery>
<cfif validatebusinessUnit.recordcount gt 0>
<cfset businessUnitValidList = listAppend(businessUnitValidList,toString(validatebusinessUnit.doctorid),",")>
<cfelse>
<cfset businessUnitInValidList = listAppend(businessUnitInValidList,i,",")>
</cfif>
</cfloop>
<cfif businessUnitInValidList neq ''>
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitInValidList'] = "#businessUnitInValidList#">
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitValidList'] = "#businessUnitValidList#">
<cfreturn serializeJSON(ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse'])>
<cfelse>
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitSuccess']['businessUnitValidList'] = "#businessUnitValidList#">
<cfreturn serializeJSON(ResponseStruct['BusinessUnitCodes']['businessUnitSuccess'])>
</cfif>
ColdFusion's JSON serialization has issues and can vary between versions and even hotfixes. As Jedihomer Townend mentioned in the comments a leading space should force CF to treat it a string and not cast it.
I've just tried this on CF10, 11 and 2016 and preserves the input.
<cfscript>
a = {
"businessUnitValidList":" 2003051509034372557922",
"shortMessage":"Success",
"longMessage":"Request Completed Successfully.",
"status":20001
};
json = serializeJSON(a);
b = deserializeJSON(json);
writeDump(b);
</cfscript>
You can try it here:
http://trycf.com/gist/70b86fbb57f752125f35/acf?theme=monokai
(Too long for comments)
my code is producing the correct outcome is the deserializeJSON() that
is causing the issue.
Not quite. The JSON value is correct, but the serialization omits the surrounding quotes. That means the value will be handled as a numeric type during deserialization, causing the issue you observed. Unless you can force the serialization to treat the value as a string, the deserialized result will always be wrong. As Carl Von Stetten already mentioned, it is a bug. Jedihomer Townend's answer of appending a space character is probably the simplest work-around.
Longer answer:
Despite the improvements in CF11's JSON handling, CF is still a little too "helpful" ... As you noted, CF detects the value is numeric when serializing and omits the surrounding quotes. Consequently marking the value type as numeric.
{..."businessUnitValidList":2003051509034372557922 }
That all sounds great, until you try and deserialize. If the value was enclosed in quotes, it would be handled as a string, and the original value preserved. Unfortunately, without the quotes it is considered numeric, which means CF must stuff the value into one of its two numeric data types:
Real or floating point number, ie java.lang.Double or
32 bit Integer, ie java.lang.Integer
The maximum value of an Integer is 2147483647. Obviously your number is too large for that, so CF converts it into a java.lang.Double instead. That is a problem for two reasons. First, Double is an approximate type. Second, according to the rules of that class, scientific notation may be used when representing the number as a String, ie when the variable is displayed with cfoutput or cfdump. That is why the deserialized result looks different than what you were expecting. Unless you can force it to be treated as a string when serialized, the deserialized result will always be wrong.
In fairness, CF11 does contain a number improvements for JSON handling. Unfortunately most of them revolve around cfc's and query objects. Given your current structure, it won't quite work. However, if you were able to use a single query object you could resolve the issue by with the help of the new application level setting this.serialization.serializeQueryAs = "struct";. It forces a more sensible format for serialized queries than in earlier versions. Since CF11 respects column data types when serializing, the value would be preserved if the column data type is BIGDECIMAL, or you cast it as a VARCHAR. Unfortunately, CF still upper cases query column names, but the base values are preserved.
Result:
[ { "BUSINESSUNITVALIDLIST" : "2003051509034372557922",
"LONGMESSAGE" : "Request Completed Successfully.",
"SHORTMESSAGE" : "Success",
"STATUS" : 20001
} ]
Code:
qry = queryNew("");
queryAddColumn(qry, "businessUnitValidList", "varchar", ["2003051509034372557922"]);
queryAddColumn(qry, "shortMessage", "varchar", ["Success"]);
queryAddColumn(qry, "longMessage", "varchar", ["Request Completed Successfully."]);
queryAddColumn(qry, "status", "integer", [20001]);
json = serializeJSON(qry);
writeDump(deserializeJSON(json));
CF11 also introduced custom serializers/deserializers, which might work here. Though it is probably overkill for this specific task.
Having said all that, again the simplest option is to use the "append a non-numeric character" hack. Well .. either that or switch to a custom library which may do a more consistent job with JSON handling ;-)
Side Note / Efficiency:
Unless there is a specific reason you must execute a query within a loop, there are likely more efficient options (dbms specific, which you did not mention). Also, do not forget to use cfqueryparam on all variable query parameters. Among it is many benefits is boosting performance when the same query is executed multiple times - such as inside a loop.
In java we have BigInteger data type to store big values. In Coldfusion we can cast by using JavaCast to integer, long and double. But the problem with this is that even with 'long' datatype can only store 19 digits.
And by converting number to string it may be problematic to perform mathematical operations. Here we can use precisionEvaluate() while performing mathematical operations.
PrecisionEvaluate function lets you calculate arbitrarily long decimal
(BigDecimal precision) values. BigDecimal precision arithmetic accepts
and generates decimal numbers of any length.
In this example you can use like this :
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitInValidList'] = "#precisionEvaluate(businessUnitInValidList)#">
(using Coldfusion8/MySQL5.0.88)
I'm storing JSON strings in a database. Strings consist of an id (key) and a number of items in a list (values).
Values look like this:
LOCAL.dropRecall = {"994323":"596895,596871,596864,596888,596840abc,596833,596826","991234":"9999,8888,abced"}
My problem is trying to delete a key/value pair. I'm trying like this:
<cfif StructKeyExists(LOCAL.dropRecall,"#Session.id#")>
<cfdump output="e:\dump.txt" label="catch" var="detected">
<cfset StructDelete( LOCAL.dropRecall,"#Session.id#", "true")>
</cfif>
which correctly detects the key/value pair (detected is reported), but deleting it does not work.
Question:
What am I doing wrong? Why is the key/value pair not removed?
EDIT:
Ok. Found it. I queried the database for field items, set this to LOCAL.dropRecall and wrote items back into the database... so the structDelete worked, but I did not write the empty struct back to the database.
Check that the value of session.id is what you think it is (no extra whitespace or anything like that). I tried a modification if your code on CF8, and it works fine for me:
<cfset variables.id = 991234>
<cfset LOCAL.dropRecall = deserializeJson('{"994323":"596895,596871,596864,596888,596840abc,596833,596826","991234":"9999,8888,abced"}')>
<cfset LOCAL.safeCopy = duplicate(LOCAL.dropRecall)>
<cfif StructKeyExists(LOCAL.dropRecall,"#variables.id#")>
<cfset StructDelete( LOCAL.dropRecall,"#variables.id#", "true")>
</cfif>
<cfdump var="#LOCAL#">
Does that code not work for you?
I wonder if the problem is that your variable names inside LOCAL.dropRecall start with a number? ColdFusion variables should always begin with a letter, underscore, or Unicode currency symbol.
Updated version of Adam's solution for ColdFusion 2016
<cfscript>
variables.id = 991234;
LOCAL.dropRecall = deserializeJson('{"994323":"596895,596871,596864,596888,596840abc,596833,596826","991234":"9999,8888,abced"}');
LOCAL.safeCopy = duplicate(LOCAL.dropRecall);
LOCAL.dropRecall.delete(variables.id); // you don't have to test if it is there
writedump(LOCAL);
</cfscript>
I have a form that returns a list like this when submitted:
2009,9
I want to compare it to database pulled values but keep getting an error.
<cfif #FORM.month# eq #qGetDates.year#,#qGetDates.month#>
I know I probably have to cast it or convert it to a string for the comparison to work, how do I do this?
Thanks,
R.
<cfif FORM.month eq "#qGetDates.year#,#qGetDates.month#">
or
<cfif compare(FORM.month, "#qGetDates.year#,#qGetDates.month#") EQ 0>
You are overusing #. Unless variables are inside quotation marks or a cfoutput block, you don't use # as a general rule.
Another rule: You must use quotes around strings (the comma in this case). You can also include variables in your strings with the rule above (use #) as seen in Henry's example.
<cfif #FORM.month# eq #qGetDates.year#,#qGetDates.month#>
should have # removed and the comma needs the string concatenated
<cfif FORM.month eq qGetDates.year & "," & qGetDates.month>
Or as Henry said
If you want to get the second value (a value after first comma), then
<cfset x = "2009,7">
<cfoutput>
#listGetAt(x,2)#
</cfoutput>