I use the UDF.Javascript function to process the message,when after converting to json object ,I see the UDF.Javascript alias name getting added to the json.
{"Device":{"deviceId":"DJT3COE4","productFilter":"pcmSensor","SignalDetails":[{"Devicevalue":"72.04","DisplayName":"Valve Open Status","Description":"Machine Valve Open State Information","DataType":"BOOLEAN","Precision":"undefined","DefaultUoM":"undefined"},{"Devicevalue":"2.7","DisplayName":"Temperature","Description":"Temperature Sensor Reading","DataType":"TEMPERATURE","Precision":"2","DefaultUoM":"DEG_CELSIUS"},{"Devicevalue":"2.99","DisplayName":"Location","Description":"Location","DataType":"LOCATION","Precision":"undefined","DefaultUoM":"LAT_LONG"},{"Devicevalue":"15","DisplayName":"Valve Control","Description":"On / Off control","DataType":"BOOLEAN","Precision":"undefined","DefaultUoM":"undefined"}]}}
Remove the aliasname : {"Device": from the json.
Maybe you could use WITH...AS... in your sql,please see below example:
WITH
c AS
(
SELECT
udf.processArray(input)
from input
)
SELECT
c.processarray.item,c.processarray.name
INTO
output
FROM
c
Output:
My columns are very few,you need to define all of your columns which is a little bit tedious.But it does works,please have a try.
Related
I need to save an itab as an xstring or something like this and save it in dbtab.
Later I need to gather this xstring from dbtab and convert it in the itab before with exactly the same input from before.
I tried a lot of fuba´s like:
SCMS_STRING_TO_XSTRING or SCMS_XSTRING_TO_BINARY but I didn´t find something to convert it back.
Does somebody have tried something like this before and have some samples for me ?
Unfortunately I didn´t find something on other blogs or else.
An easy solution to convert into an xstring:
CALL TRANSFORMATION id SOURCE root = it_table RESULT XML DATA(lv_xstring).
Back would be like:
CALL TRANSFORMATION id SOURCE XML lv_xstring RESULT root = it_table.
For more information, see the ABAP documentation about data serialization and deserialization by using the XSL Identity Transformation.
use
import ... from data buffer
and
export ... to data buffer
to (re)store any variable as xstring.
Or you can use
import|export ... from|to database ...
I did some methods to do this:
First I loop at the table and concatenate it into a string.
Then convert the string into an xstring.
LOOP AT IT_TABLE ASSIGNING FIELD-SYMBOL(<LS_TABLE>).
CONCATENATE LV_STRING <LS_TABLE> INTO LV_STRING SEPARATED BY CL_ABAP_CHAR_UTILITIES=>NEWLINE.
ENDLOOP.
CALL FUNCTION 'SCMS_STRING_TO_XSTRING'
EXPORTING
TEXT = IV_STRING
IMPORTING
BUFFER = LV_XSTRING.
Back would be like:
Convert xstring back to string
String into table
TRY.
CL_BCS_CONVERT=>XSTRING_TO_STRING(
EXPORTING
IV_XSTR = IV_XSTRING
IV_CP = 1100 " SAP character set identification
RECEIVING
RV_STRING = LV_STRING
).
CATCH CX_BCS.
ENDTRY.
SPLIT IV_STRING AT CL_ABAP_CHAR_UTILITIES=>NEWLINE INTO: TABLE <LT_TABLE> .
READ TABLE <LT_TABLE> ASSIGNING FIELD-SYMBOL(<LS_TABLE>) INDEX 1.
IF <LS_TABLE> IS INITIAL.
DELETE TABLE <LT_TABLE> FROM <LS_TABLE>.
ENDIF.
I have a problem with the Function Module "GUI_DOWNLOAD" because of the date converting.
I want to get the date like I have it in my internal table but CSV (Excel) keeps converting it everytime.
The internal table contains the line like this: 12345678;GroupDate;2021-12-31;
The Output in the .csv-File should be "2021-12-31" but it keeps converting to "31.12.2021".
I also tried to put an ' (apostroph) before the date but the output will be '2021-12-31
Does anybode have an Idea ?
lv_conv = '2021-12-31'.
CONCATENATE TEXT-001 LV_CONV INTO LV_CONV.
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
FILENAME = IV_PATH
TABLES
DATA_TAB = LT_FILE.
LT_FILE is a string table.
Thanks for the help.
Like Suncatcher and Sandra said the file is right but it is only the settings from excel which convert the date.
If the Output file won´t be needed for other purposes than showing the code could be something like this
CONCATENATE '=("' LV_CONV '")' INTO LV_CONV.
The csv-Output would be a date like this 1960-01-01 but in the cell the value would look like =("1960-01-01").
I have a BUNCH of fixed width text files that contain multiple transaction types with only 3 that I care about (121,122,124).
Sample File:
D103421612100188300000300000000012N000002000001000032021420170012260214201700122600000000059500000300001025798
D103421612200188300000300000000011000000000010000012053700028200004017000000010240000010000011NNYNY000001000003N0000000000 00
D1034216124001883000003000000000110000000000300000100000000000CS00000100000001200000033NN0 00000001200
So What I need to do is read line by line from these files and look for the ones that have a 121, 122, or 124 at startIndex = 9 and length = 3.
Each line needs to be parsed based on a data dictionary I have and the output needs to be grouped by transaction type into three different files.
I have a process that works but it's very inefficient, basically reading each line 3 times. The code I have is something like this:
#121 = EXTRACT
col1 string,
col2 string,
col3 string //ect...
FROM inputFile
USING new MyCustomExtractor(
new SQL.MAP<string, string> {
{"col1","2"},
{"col2","6"},
{"col3","3"} //ect...
};
);
OUTPUT #121
TO 121.csv
USING Outputters.Csv();
And I have the same code for 122 and 124. My custom extractor takes the SQL MAP and returns the parsed line and skips all lines that don't contain the transaction type I'm looking for.
This approach also means I'm running through all the lines in a file 3 times. Obviously this isn't as efficient as it could be.
What I'm looking for is a high level concept of the most efficient way to read a line, determine if it is a transaction I care about, then output to the correct file.
Thanks in advance.
How about pulling out the transaction type early using the Substring method of the String datatype? Then you can do some work with it, filtering etc. A simple example:
// Test data
#input = SELECT *
FROM (
VALUES
( "D103421612100188300000300000000012N000002000001000032021420170012260214201700122600000000059500000300001025798" ),
( "D103421612200188300000300000000011000000000010000012053700028200004017000000010240000010000011NNYNY000001000003N0000000000 00" ),
( "D1034216124001883000003000000000110000000000300000100000000000CS00000100000001200000033NN0 00000001200" ),
( "D1034216999 0000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000" )
) AS x ( rawData );
// Pull out the transaction type
#working =
SELECT rawData.Substring(8,3) AS transactionType,
rawData
FROM #input;
// !!TODO do some other work here
#output =
SELECT *
FROM #working
WHERE transactionType IN ("121", "122", "124"); //NB Note the case-sensitive IN clause
OUTPUT #output TO "/output/output.csv"
USING Outputters.Csv();
As of today, there is no specific U-SQL function that can define the output location of a tuple on the fly.
wBob presented an approach to a potential workaround. I'd extend the solution the following way to address your need:
Read the entire file, adding a new column that helps you identify the transaction type.
Create 3 rowsets (one for each file) using a WHERE statement with the specific transaction type (121, 122, 124) on the column created in the previous step.
Output each rowset created in the previous step to their individual file.
If you have more feedback or needs, feel free to create an item (and voting for others) on our UserVoice site: https://feedback.azure.com/forums/327234-data-lake. Thanks!
I have U-SQL script where I need to process some data. The data is stored in blob, with ~100 files per day in this folder structure: /{year}/{month}/{day}/{hour}/filenames.tsv
Getting one day of data is easy, just put a wildcard in the end and it will pick out all the files for all the hours for the day.
However, in my script I want to read out the current day and the last 2 hours of the previous day. The naive way is with 3 extract statements in this way:
DECLARE #input1 = #"/data/2017/10/08/22/{*}.tsv";
DECLARE #input2 = #"/data/2017/10/08/23/{*}.tsv";
DECLARE #input3 = #"/data/2017/10/09/{*}.tsv";
#x1 = EXTRACT .... FROM #input1 USING Extractors.Tsv();
#x2 = EXTRACT .... FROM #input2 USING Extractors.Tsv();
#x3 = EXTRACT .... FROM #input3 USING Extractors.Tsv();
But in my case each extract line is very long and complicated (~50 columns) using the AvroExtractor, so I would really prefer to only specify the columns and extractor once instead of 3 times. Also, by having 3 inputs its not possible from the caller side to decide how many hours from the previous days that should be read.
My question is how can I define this in a convenient way, ideally using only one extract statement?
You could wrap your logic up into a U-SQL stored procedure so it is encapsulated. Then you need only make a few calls to the proc. A simple example:
CREATE PROCEDURE IF NOT EXISTS main.getContent(#inputPath string, #outputPath string)
AS
BEGIN;
#output =
EXTRACT
...
FROM #inputPath
USING Extractors.Tsv();
OUTPUT #output
TO #outputPath
USING Outputters.Tsv();
END;
Then to call it (untested):
main.getContent (
#"/data/2017/10/08/22/{*}.tsv",
#"/output/output1.tsv"
)
main.getContent (
#"/data/2017/10/08/23/{*}.tsv",
#"/output/output2.tsv"
)
main.getContent (
#"/data/2017/10/09/{*}.tsv",
#"/output/output3.tsv"
)
That might be one way to go about it?
I am tring to load a .xlsx file into an Oracle database table. I am getting an error for my code. I normally use this code for .csv files but need to use it for .xlsx I have edited my field names, table names etc
Is this possible?
Drop TABLE Temp_Info;
CREATE TABLE Temp_Info
(
Unique_Id varchar2(255) ,
Name varchar2(255),
Alt_Name varchar2(255)
)
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY SEPA_FILES
ACCESS PARAMETERS
(
records delimited by newline
skip 1
fields terminated by ','
missing field values are null
(
Unique ID -(filled automatically),Name,Alt Name
)
)
LOCATION ('Data_File.xlsx')
)
REJECT LIMIT UNLIMITED;
Select * From Temp_Info a;
Error Message: 9:16:55 ORA-29913: error in executing ODCIEXTTABLEOPEN callout
9:16:55 ORA-29400: data cartridge error
9:16:55 KUP-00554: error encountered while parsing access parameters
9:16:55 KUP-01005: syntax error: found "identifier": expecting one of: "comma, char, date, defaultif, decimal, double, float, integer, (, nullif, oracle_date, oracle_number, position, raw, recnum, ), unsigned, varrawc, varchar, varraw, varcharc, zoned"
9:16:55 KUP-01008: the bad identifier was: ID
9:16:55 KUP-01007: at line 6 column 30
9:16:55 ORA-06512: at "SYS.ORACLE_LOADER", line 14
9:16:55 ORA-06
1) Option [ Pure PL/SQL]
Xlsx document is zipped set of xml documents. You can change extension xlsx to zip, unzip and find out what is inside.
Here is description how to deal with xlsx document in oracle environment.
This solution works but implementation is very painful.
2) Option (PL/SQL + apache POI)
Create implementation in java. And use it in db.
3) Convert xlxs to csv.
You might want to take a look at ExcelTable SQL interface (Disclaimer : I'm the author).
It provides access to .xlsx (or .xlsm) files as an external table.
Here's an example based on your ext table definition :
SELECT t.*
FROM TABLE(
ExcelTable.getRows(
ExcelTable.getFile('SEPA_FILES','Data_File.xlsx')
, 'Sheet_name_goes_here'
, ' "UNIQUE_ID" for ordinality
, "NAME" varchar2(255)
, "ALT_NAME" varchar2(255) '
, 'A2'
)
) t ;
(I assumed UNIQUE_ID is some kind of autogenerated sequence)