We have a mainframe file which we are trying to read using Complex Flat File Stage. The column has data type PIC X(1) which we are reading as char(1) and assigning to char (10). Problem is it converts to value "26" when the value should be 30. The value displayed in mainframe is x'30' (which seems to be hex). Is there a way to convert value correctly in datastage? Right now following is being implemented in the transformer:
DecimalToString(seq(link1.DAY),"suppress_zero")
Related
trying to out put a dataframe into txt file (for a feed). a few specific columns are getting automatically converted into float instead of int as intented.
how can i specific those columns to use int as dtype?
i tried to output the whole dataframd as string and that did not work.
the columns i would like to specify are named [CID1] and [CID2]
data = pd.read_sql(sql,conn)
data = data.astype(str)
data.to_csv('data_feed_feed.txt', sep ='\t',index=True)
Based on the code you provided, you turn all of your data to strings just before export.
Thus, you either need to turn some cols back to desired type, such as:
data["CID1"] = data["CID1"].astype(int)
or not convert them in the first place.
It is not clear from what you provided why you'd have issues with ints being converted to floats.
this post provides heaps of info:
stackoverflow.com/a/28648923/9249533
I am using f-strings for formatting strings and injecting variable values into the string, is there a way to set a format-spec for an entire module in python?
I am aware of Standard Format Specifiers which can be used to specify formats for each string, but how do I do it at once for the whole module?
Eg:
f""" Some random string {value1}, some more text {value2:.2f} ... """
Here I am specifying the format for value2, but I want to set a format for all globally.
f"""% profits are {profit}"""
Had I set format spec to {profit:.2f} this will be set to two decimal places, but I want to set that :.2f globally so that number decimal places can be changed with one variable update.
Something like format-spec = ':.2f' and all the f-string injected values should be displayed as floats with two decimals (if its a numbers).
If I understand you correctly, you want to define a string format one time that is usable throughout a script. I would create a function, callable from anywhere in your script that produces the formatted string as shown below:
def create_string(v_str: str, v_num: float) -> str:
return f""" Some random string {v_str}, some more text {v_num:.2f} ... """
You can then use the create_string function from anywhere in your script and produce the desired output as illustrated below:
print(create_string('Some words here', 2.56))
which yields:
Some random string Some words here, some more text 2.56 ...
QuotedPremium column is a string feature so I need to convert it to numeric value in order to use algorithm.
So, for that I am using Edit Metadata module, where I specify data type to be converted is Floating Point.
After I run it - I got an error:
Could not convert type System.String to type System.Double, inner exception message: Input string was not in a correct format.
What am I missing here?
As mentioned in comments, you must change column where numbers are handled as text to numeric type data and it shouldn't have any null values. Now answering the question of how to substitute NULL's in data using ML studio and converting to numeric type.
Substitute NULL's in data
Use Execute R Script module for that, and add this code in it.
dataset1 <- maml.mapInputPort(1); # class: data.frame
dataset1[dataset1 == "NULL"] = 0; # Wherever cell's value is "NULL", replace it with 0
maml.mapOutputPort("dataset1"); # return the modified data.frame
Image for same:
Convert to numeric data
As you have added in your answer, this can be done using the Edit Metadata module.
I'm using the Apache POI framework for parsing large Excel spreadsheets. I'm using this example code as a guide: XLSX2CSV.java
I'm finding that cells that contain just numbers are implicitly being treated as numeric fields, while I wanted them to be treated always as strings. So rather than getting 1.00E+13 (which I'm currently getting) I'll get the original string value: 10020300000000.
The example code uses a XSSFSheetXMLHandler which is passed an instance of DataFormatter. Is there a way to use that DataFormatter to treat all cells as strings?
Or as an alternative: in the implementation of the interface SheetContentsHandler.cell method there is string value that is the cellReference. Is there a way to convert a cellReference into an index so that I can use the SharedStringsTable.getEntryAt(int idx) method to read directly from the strings table?
To reproduce the issue, just run the sample code on an xlsx file of your choice with a number like the one in my example above.
UPDATE: It turns out that the string value I get seems to match what you would see in Excel. So I guess that's going to be "good enough" generally. I'd expect the data I'm sent to "look right" and therefore it'll get parsed correctly. However, I'm sure there will be mistakes and in those cases it'd be nice if I could get at the raw string value using the streaming API.
To resolve this issue I created my own class based on XSSFSheetXMLHandler
I copied that class, renamed it and then in the endElement method I changed this part of the code which is formatting the raw string:
case NUMBER:
String n = value.toString();
if (this.formatString != null && n.length() > 0)
thisStr = formatter.formatRawCellContents(Double.parseDouble(n), this.formatIndex, this.formatString);
else
thisStr = n;
break;
I changed it so that it would not format the raw string:
case NUMBER:
thisStr = value.toString();
break;
Now every number in my spreadsheet has its raw value returned rather than a formatted version.
I have to insert the string "johnmelling" value into a table which has the column as
[USERPASS] varbinary NOT NULL.
Please could any one suggest me, what would be the best conversion to insert "johnmelling"?
I tried to to insert as below,
Insert into table(column1)
Values(CONVERT(varbinary(1), 'johnmelling'))
Then I got the error
Line 1: String or binary data would be truncated.
Thank You,
You are converting to varbinary(1) so your target datatype is varbinary but the integer you have specified in parentheses is 1 which means your datatype will only have a length of 1; you are receiving that error because the length you have allocated to that datatype is too small. The literal, 'johnmelling' is 11 characters but you are trying to store it in a datatype that has a length of 1.
Simply change the integer in parentheses to 11, 50, 255, max or whatever you think is an appropriate length and you won't get that error.