I am trying to pass a struct through a function, but the integers in it are converting to scientific notation.
Before deSerialization :
{"businessUnitValidList":2003051509034372557922
, "shortMessage":"Success"
, "longMessage":"Request Completed Successfully."
, "status":20001
}
After deSerialization:
businessUnitValidList 2.00305150903E+021
I have tried converting it into a string but it still gives me the same output. Any ideas?
Note: If I have more than one value in my businessUnitValidList, the numbers show up the way they are supposed to.
EDIT
This is the current code iteration:
<cfloop array="#businessUnitArray#" index="i">
<cfquery name="validatebusinessUnit" datasource="dbproduction">
select doctorid from survey.dbo.clientLocationMap
where clientbrandid = '#arguments.clientBrandid#'
and clientLocation = '#i#'
</cfquery>
<cfif validatebusinessUnit.recordcount gt 0>
<cfset businessUnitValidList = listAppend(businessUnitValidList,toString(validatebusinessUnit.doctorid),",")>
<cfelse>
<cfset businessUnitInValidList = listAppend(businessUnitInValidList,i,",")>
</cfif>
</cfloop>
<cfif businessUnitInValidList neq ''>
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitInValidList'] = "#businessUnitInValidList#">
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitValidList'] = "#businessUnitValidList#">
<cfreturn serializeJSON(ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse'])>
<cfelse>
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitSuccess']['businessUnitValidList'] = "#businessUnitValidList#">
<cfreturn serializeJSON(ResponseStruct['BusinessUnitCodes']['businessUnitSuccess'])>
</cfif>
ColdFusion's JSON serialization has issues and can vary between versions and even hotfixes. As Jedihomer Townend mentioned in the comments a leading space should force CF to treat it a string and not cast it.
I've just tried this on CF10, 11 and 2016 and preserves the input.
<cfscript>
a = {
"businessUnitValidList":" 2003051509034372557922",
"shortMessage":"Success",
"longMessage":"Request Completed Successfully.",
"status":20001
};
json = serializeJSON(a);
b = deserializeJSON(json);
writeDump(b);
</cfscript>
You can try it here:
http://trycf.com/gist/70b86fbb57f752125f35/acf?theme=monokai
(Too long for comments)
my code is producing the correct outcome is the deserializeJSON() that
is causing the issue.
Not quite. The JSON value is correct, but the serialization omits the surrounding quotes. That means the value will be handled as a numeric type during deserialization, causing the issue you observed. Unless you can force the serialization to treat the value as a string, the deserialized result will always be wrong. As Carl Von Stetten already mentioned, it is a bug. Jedihomer Townend's answer of appending a space character is probably the simplest work-around.
Longer answer:
Despite the improvements in CF11's JSON handling, CF is still a little too "helpful" ... As you noted, CF detects the value is numeric when serializing and omits the surrounding quotes. Consequently marking the value type as numeric.
{..."businessUnitValidList":2003051509034372557922 }
That all sounds great, until you try and deserialize. If the value was enclosed in quotes, it would be handled as a string, and the original value preserved. Unfortunately, without the quotes it is considered numeric, which means CF must stuff the value into one of its two numeric data types:
Real or floating point number, ie java.lang.Double or
32 bit Integer, ie java.lang.Integer
The maximum value of an Integer is 2147483647. Obviously your number is too large for that, so CF converts it into a java.lang.Double instead. That is a problem for two reasons. First, Double is an approximate type. Second, according to the rules of that class, scientific notation may be used when representing the number as a String, ie when the variable is displayed with cfoutput or cfdump. That is why the deserialized result looks different than what you were expecting. Unless you can force it to be treated as a string when serialized, the deserialized result will always be wrong.
In fairness, CF11 does contain a number improvements for JSON handling. Unfortunately most of them revolve around cfc's and query objects. Given your current structure, it won't quite work. However, if you were able to use a single query object you could resolve the issue by with the help of the new application level setting this.serialization.serializeQueryAs = "struct";. It forces a more sensible format for serialized queries than in earlier versions. Since CF11 respects column data types when serializing, the value would be preserved if the column data type is BIGDECIMAL, or you cast it as a VARCHAR. Unfortunately, CF still upper cases query column names, but the base values are preserved.
Result:
[ { "BUSINESSUNITVALIDLIST" : "2003051509034372557922",
"LONGMESSAGE" : "Request Completed Successfully.",
"SHORTMESSAGE" : "Success",
"STATUS" : 20001
} ]
Code:
qry = queryNew("");
queryAddColumn(qry, "businessUnitValidList", "varchar", ["2003051509034372557922"]);
queryAddColumn(qry, "shortMessage", "varchar", ["Success"]);
queryAddColumn(qry, "longMessage", "varchar", ["Request Completed Successfully."]);
queryAddColumn(qry, "status", "integer", [20001]);
json = serializeJSON(qry);
writeDump(deserializeJSON(json));
CF11 also introduced custom serializers/deserializers, which might work here. Though it is probably overkill for this specific task.
Having said all that, again the simplest option is to use the "append a non-numeric character" hack. Well .. either that or switch to a custom library which may do a more consistent job with JSON handling ;-)
Side Note / Efficiency:
Unless there is a specific reason you must execute a query within a loop, there are likely more efficient options (dbms specific, which you did not mention). Also, do not forget to use cfqueryparam on all variable query parameters. Among it is many benefits is boosting performance when the same query is executed multiple times - such as inside a loop.
In java we have BigInteger data type to store big values. In Coldfusion we can cast by using JavaCast to integer, long and double. But the problem with this is that even with 'long' datatype can only store 19 digits.
And by converting number to string it may be problematic to perform mathematical operations. Here we can use precisionEvaluate() while performing mathematical operations.
PrecisionEvaluate function lets you calculate arbitrarily long decimal
(BigDecimal precision) values. BigDecimal precision arithmetic accepts
and generates decimal numbers of any length.
In this example you can use like this :
<cfset ResponseStruct['BusinessUnitCodes']['businessUnitMixResponse']['businessUnitInValidList'] = "#precisionEvaluate(businessUnitInValidList)#">
Related
So I have to go through a bunch of code to get some data from an iframe. the iframe has a lot of data but in there is an object called '_name'. the first key of name is 'extension_id' and its value is a big long string. the json object is enclosed in apostrophes. I have tried removing the apostrophes but still instead of 'extension_id_output' I get a single curly bracket. the json object looks something like this
Frame {
...
...
_name: '{"extension_id":"a big huge string that I need"} "a bunch of other stuff":"this is a valid json object as confirmed by jsonlint", "globalOptions":{"crev":"1.2.50"}}}'
}
it's a whole big ugly paragraph but I really just need the extension_id. so this is the code I'm currently using after attempt 100 or whatever.
var frames = await page.frames();
// I'm using puppeteer for this part but I don't think that's relevant overall.
var thing = frames[1]._name;
console.log(frames[1])
// console.log(thing)
thing.replace(/'/g, '"')
// this is to remove the apostrophes from the outside of the object. I thought that would change things before. it does not. still outputs a single {
JSON.parse(thing)
console.log(thing[0])
instead of getting a big huge string that I need or whatever is written in extension_id. I get a {. that's it. I think that is because the whole object starts with a curly bracket. this is confirmed to me because console.log(thing[2]) prints e. so what's going on? jsonlint says this is a valid json object but maybe it's just a big string and I should be doing some kind of split to grab whaat's between the first : and the first ,. I'm really not sure.
For two reasons:
object[0] doesn't return the value an object's "first property", it returns the value of the property with the name "0", if any (there probably isn't in your object); and
Because it's JSON, and when you're dealing with JSON in JavaScript code, you are by definition dealing with a string. (More here.) If you want to deal with the object that the JSON describes, parse it.
Here's an example of parsing it and getting the value of the extension_id property from it:
const parsed = JSON.parse(frames[1]._name);
console.log(parsed.extension_id); // The ID
I'm getting this message in Redshift: invalid input syntax for type numeric: " " , even after trying to implement the advice found in SO.
I am trying to convert text to number.
In my inner join, I try to make sure that the text being processed is first converted to null when there is an empty string, like so:
nullif(trim(atl.original_pricev::text),'') as original_price
... I noticed from a related post on coalesce that you have to convert the value to text before you can try and nullif it.
Then in the outer join, I test to see that there's a limited set of acceptable characters and if this test is met I try to do the to_number conversion:
,case
when regexp_instr(trim(atl.original_price),'[^0-9.$,]')=0
then to_number(atl.original_price,'FM999999999D00')
else null
end as original_price2
At this point I get the above error and unfortunately I can't see the details in datagrip to get the offending value.
So my questions are:
I notice that there is an empty space in my error message:
invalid input syntax for type numeric: " " . Does this error have the exact same meaning as
invalid input syntax for type numeric:'' which is what I see in similar posts??
Of course: what am I doing wrong?
Thanks!
It's hard to know for sure without some data and the complete code to try and reproduce the example, but as some have mentioned in the comments the most likely cause is the to_number() function you are using.
In the earlier code fragment you are converting original_price to text (string) and then substituting an empty string ('') if the value is NULL. Calling the to_number() function on an empty string will give you the error described.
Without the full SQL statement it's not clear why you're putting the nullif() function around the original_price in the "inner join" or how whether the CASE statement is really in an outer join clause or one of the columns returned by the query. However you could perhaps alter the nullif() to substitute a value that can be converted to a number e.g. '0.00' instead of ''.
Sorry I couldn't share real data. I spent the weekend testing small sets to try and trap the error. I found that the error was caused by the input string having no numbers, which is permitted by my regex filter:
when regexp_instr(trim(atl.original_price),'[^0-9.$,]') .
I wrongly expected that a non numeric string like "$" would evaluate to NULL and then the to_number function would = NULL . But from experimenting it seems that it needs at least one number somewhere in the string. Otherwise it reduces the string argument to an empty string prior to running the to_number formatting and chokes.
For example select to_number(trim('$1'::text),'FM999999999999D00') will evaluate to 1 but select to_number(trim('$A'::text),'FM999999999999D00') will throw the empty string error.
My fix was to add an additional regex to my initial filter:
and regexp_instr(atl.original_price2,'[0-9]')>0 .
This ensures that at least one number will be in the string and after that the empty string error went away.
Hope my learning experience helps someone else.
I'm using the Apache POI framework for parsing large Excel spreadsheets. I'm using this example code as a guide: XLSX2CSV.java
I'm finding that cells that contain just numbers are implicitly being treated as numeric fields, while I wanted them to be treated always as strings. So rather than getting 1.00E+13 (which I'm currently getting) I'll get the original string value: 10020300000000.
The example code uses a XSSFSheetXMLHandler which is passed an instance of DataFormatter. Is there a way to use that DataFormatter to treat all cells as strings?
Or as an alternative: in the implementation of the interface SheetContentsHandler.cell method there is string value that is the cellReference. Is there a way to convert a cellReference into an index so that I can use the SharedStringsTable.getEntryAt(int idx) method to read directly from the strings table?
To reproduce the issue, just run the sample code on an xlsx file of your choice with a number like the one in my example above.
UPDATE: It turns out that the string value I get seems to match what you would see in Excel. So I guess that's going to be "good enough" generally. I'd expect the data I'm sent to "look right" and therefore it'll get parsed correctly. However, I'm sure there will be mistakes and in those cases it'd be nice if I could get at the raw string value using the streaming API.
To resolve this issue I created my own class based on XSSFSheetXMLHandler
I copied that class, renamed it and then in the endElement method I changed this part of the code which is formatting the raw string:
case NUMBER:
String n = value.toString();
if (this.formatString != null && n.length() > 0)
thisStr = formatter.formatRawCellContents(Double.parseDouble(n), this.formatIndex, this.formatString);
else
thisStr = n;
break;
I changed it so that it would not format the raw string:
case NUMBER:
thisStr = value.toString();
break;
Now every number in my spreadsheet has its raw value returned rather than a formatted version.
I have no idea how to check for this. My method(if condition in method) should only work (execute) if the first argument passed in is a string. I know how to check other types, but I can't seem to find anything for checking for a string.
For a hash I would do something like;
if(ref eq 'HASH') {...}
If someone could provide a simple example I'm sure I would be able to apply it to what I'm doing. I will put up the code for the method and an explanation for the whole operational details of the method if needed.
Added Information
This is a method for handling different types of errors in the software, here are the 3 possible input formats:
$class->new("error string message")
$class->new("error string message", code => "UNABLE_TO_PING_SWITCH_ERROR")
$class->new("error string message", code => "UNABLE_TO_PING_SWITCH_ERROR", switch_ip => $ip3, timeout => $timeout)
There will always be an error message string first.
With the 1st case there is also a hashref to an error hash structure that is located in a library,
this method new will go into a template processing if the word "code" exists as an arg. where the longer detailed error message is constructed. (I already have the logic for this).
But I have to add logic so that the error message string is added to the hash, so the output is one hash, and not strings.
The second case is very similar to the first, where there are parameters eg. switch_ip , which are inserted into the string using a similar template processing logic, (already have this too).
So I think the first and second cases can be handled in the same way, but I'm not sure, so separated them in this question.
The last case is just can error message string by itself, which at the minute I just insert it into a one key message hash { message => "error string}.
So after all that how should I be checking or dividing up these error cases, At the minute my idea for the ones with code , is to dump the arguments into a hash and just use something like:
if(exists($param{code}) { doTemplateProcess()...}
I need to ensure that there is a string passed in first though. Which was my original question. Does any of my context information help? I hope I didn't go off the topic of my question, if so I'll open this a new question. Thanks.
Error hash - located in Type.pm
use constant ERROR_CODE => {
UNABLE_TO_PING_SWITCH_ERROR => {
category => 'Connection Error:',
template => 'Could not ping switch %s in %s minutes',
tt => {template => 'disabled'},
fatal => 1,
wiki_page => www.error-solution.com/,
},
}
From comments:
These will be called in the software's code like so
ASC::Builder::Error->new(
"Phase x this occured because y was happening:",
code => UNABLE_TO_PING_SWITCH_ERROR,
switch_ip => $ip3,
timeout => 30,
);
Putting the wisdom of your particular problem aside and channeling Jeff Foxworthy:
If you have a scalar and it's not a reference, you might have a string.
If your non-reference scalar doesn't look like a number, it might be a string.
If your non-reference scalar looks like a number, it can still be a string.
If your non-reference scalar has a different string and number value, it might be a dualvar.
You know that your argument list is just that: a list. A list is a collection of scalar values. A scalar can be a reference or not a reference. I think you're looking for the not a reference case:
die "You can't do that" if ref $first_argument;
Past that, you'd have to do fancier things to determine if it's the sort of value that you want. This might also mean that you reject objects that pretend to be strings through overloading and whatnot.
Perhaps you can make the first argument part of the key-value pairs that you pass. You can then access that key to extract the value and delete it before you use the remaining pairs.
You may easily check only whether the error string is a simple scalar value or a reference. You would do that with ref, but you must consider what you want to do if the first parameter isn't a string
You should write your constructor in the ASC::Builder::Error package along these lines
sub new {
my $class = shift;
my ($error, %options) = #_;
die if ref $error;
bless { string => $error }, $class;
}
This example simply dies, and so kills the program, if it is called with anything other than a simple string or number as the first parameter
You may call it as
ASC::Builder::Error->new('error')
or
ASC::Builder::Error->new(42)
and all will be well. If you try
ASC::Builder::Error->new('message', 'code')
then you will see a warning
Odd number of elements in hash assignment
And you may make that warning fatal
If there is anything more then you should explain
Supporting all of the following is simple:
$class->new("s")
$class->new("s", code => "s")
$class->new("s", code => "s", switch_ip => "s", timeout => "s")
All you need is the following:
sub new {
my ($class, $msg, %opts) = #_;
...
}
You can checks such as the following to examine what the called provided:
if (exists($opts{code}))
if (defined($opts{code}))
if ($opts{code})
Despite saying that the string will always be provided, you now ask how to check if was provided. As such, you are probably trying to perform validation rather than polymorphism. You shouldn't waste your time doing this.
Let's look at the hash reference example you gave. ref($arg) eq 'HASH' is wrong. That returns false for some hash references, and it returns false for some things that act like a reference to a hash. The following is a more proper check:
eval { %$arg; 1 }
The equivalent for strings would be the following:
eval { "$arg"; 1 }
Unfortunately, it will always return true! Every value can act as a string. That means the best thing you can do is simply to check if any argument is provided.
use Carp qw( croak );
croak("usage") if !#_;
It's rare for Perl subs to perform argument validation. Not only is it tricky, it's also expensive. It also provides very little benefits. Bad or missing arguments usually results in exceptions or warnings shortly after.
You might see suggestions to use croak("usage") if ref($arg); (or worse, die if ref($arg);), but keep in mind that those will cause the rejection of perfectly fine objects that overload stringification (which is somewhat common), and they will fail to detect the problem with ASC::Builder::Error->new(code => ...) because code produces a string. Again, performing type-based argument validation is an expensive and buggy practice in Perl.
I can't figure out what's going on. I'm exporting data from our Oracle 12c db to Excel from a simple select query, using cfspreadsheet:
<cfset var_filenameis = "report.xlsx">
<cfset SpreadsheetObj = spreadsheetNew("true")>
<cfset SpreadsheetObj = spreadsheetNew("#var_filenameis#","yes")>
<cfspreadsheet action="write" filename="#var_filenameis#" query="get_data" overwrite="true">
<cflocation url = "#var_filenameis#">
One column of data contains catalog numbers in various formats. Some of them become truncated when exported to xlsx. For example, 02923F becomes 2923 and 08552D becomes 8552. However, 08566A stays 08566A and 02584C also stays the same. The data shows correctly when viewed in the browser. Direct export from the DB also shows correct data. I could understand the leading 0 disappearing, but not the letter.
I have re-entered the problem catalog numbers in the DB to make sure there were no extra characters, to no avail. If I add a single quote in front of the catalog number, the display is correct except that I can't have a single quote showing in the output.
I get the same result with CF9 and CF11. I can't even tell if the problem is with cfspreadsheet or xlxs. Any good ideas? Thanks!
(Too long for comments)
If you are using CF11, try using SpreadSheetAddRows instead of cfspreadsheet. It is a bit smarter and uses the data types of the query columns to determine the proper cell type. As long as the query column is some sort of VARCHAR, the string value will be preserved.
<!--- sample data --->
<cfset qData = queryNew("")>
<cfset queryAddColumn(qData, "StringValue", "varchar", ["02923F","08552D","08566A","02584C"])>
<!--- populate sheet --->
<cfset sheet = SpreadsheetNew("Sheet1", true)>
<cfset spreadSheetAddRows(sheet, qData)>
<!--- display results --->
<cfheader name="Content-Disposition" value="attachment;filename=testFile.xlsx">
<cfcontent type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" variable="#SpreadSheetReadBinary(sheet)#">
I can't figure out what's going on
It is because POI (or possibly CF) interprets the "F" and "D" as special literals indicating the values are floating point numbers:
Floating-Point Literals
A floating-point literal is of type float if it ends with the letter F
or f; otherwise its type is double and it can optionally end with the
letter D or d.
So your strings are being converted to numbers. That is why the leading zero and trailing "F" or "D" disappear.