Having an XML file like this:
<?xml version="1.0" encoding="UTF-8"?><outer>
<inner>Some text.</inner>
<inner>More text.</inner>
</outer>
and the following PowerBI script
let
Table0 = Xml.Tables(File.Contents("simple1.xml")){0}[Table]
in
Table0
you get this
Element:Text
Some text.
More text.
Now I'd like to add sub elements and keep inner.Element:Text
<?xml version="1.0" encoding="UTF-8"?><outer>
<inner>Some text.<secret>Don't care.</secret></inner>
<inner>More text.<secret>You know.</secret></inner>
</outer>
Using the same PowerBI script as above you get
secret
Don't care.
You know.
I already tried this script
let
Table0 = Xml.Tables(File.Contents("simple2.xml")),
Table1 = Table.ExpandTableColumn(Table0, "Table", {"secret"})
in
Table1
but got this
Name
secret
inner
Don't care.
inner
You know.
But I'd like to get this:
Element:Text
secret.Element:Text
Some text.
Don't care.
More text.
You know.
My current workaround (which I'd like to avoid) is to use sed to wrap the element text of an inner entry in its own sub element:
<inner><text>Some text.</text><secret>Don't care.</secret></inner>
Related
I have a situation where I need to concatenate long strings from multiple records in an Oracle database into a single string. These long strings are portions of a larger XML string, and my ultimate goal is to be able to convert this XML into something resembling query results and pull out specific values.
The data would look something like this, with the MSG_LINE_TEXT field being VARCHAR2(4000). So if the total message is less than 4000 characters, then there'd only be one record. In theory, there could be an infinite number of records for each message, although the highest I've seen so far is 14 records, which means I need to be able to handle strings that are at least 56000 characters long.
MESSAGE_ID MSG_LINE_NUMBER MSG_LINE_TEXT
---------- --------------- --------------------------------
17415414 1 Some XML snippet here
17415414 2 Some XML snippet here
17415414 3 Some XML snippet here
17415414 4 Some XML snippet here
The total XML for one MESSAGE_ID might look something like this. There could be many App_Advice_Error tags, although this specific example only contains one.
<tXML>
<Header>
<Source>MANH_prod_wmsweb</Source>
<Action_Type />
<Sequence_Number />
<Company_ID>1</Company_ID>
<Msg_Locale />
<Version />
<Internal_Reference_ID>17415414</Internal_Reference_ID>
<Internal_Date_Time_Stamp>2021-02-09 13:45:22</Internal_Date_Time_Stamp>
<External_Reference_ID />
<External_Date_Time_Stamp />
<User_ID>ESBUSER</User_ID>
<Message_Type>RESPONSE</Message_Type>
</Header>
<Response>
<Persistent_State>0</Persistent_State>
<Error_Type>2</Error_Type>
<Resp_Code>501</Resp_Code>
<Response_Details>
<Application_Advice>
<Shipper_ID />
<Imported_Object_Type>ASN</Imported_Object_Type>
<Response_Type>Error</Response_Type>
<Transaction_Date>2/9/21 13:45</Transaction_Date>
<Application_Ackg_Code>TE</Application_Ackg_Code>
<Business_Unit></Business_Unit>
<Tran_Set_Identifier_Code></Tran_Set_Identifier_Code>
<Transaction_Purpose_Code>11</Transaction_Purpose_Code>
<Imported_Message_Id></Imported_Message_Id>
<Imported_Object_Id>Reference Number Here</Imported_Object_Id>
<Additional_References>
<Additional_Reference_Info>
<Reference_Type>BusinessPartner</Reference_Type>
<Reference_ID></Reference_ID>
</Additional_Reference_Info>
</Additional_References>
<App_Advice_Errors>
<App_Advice_Error>
<App_Error_Text>Some error text here</App_Error_Text>
<Error_Message_Tokens>
<Error_Message_Token>Object that errored out</Error_Message_Token>
</Error_Message_Tokens>
<App_Err_Cond_Code>6100234</App_Err_Cond_Code>
</App_Advice_Error>
</App_Advice_Errors>
<Imported_Data></Imported_Data>
</Application_Advice>
</Response_Details>
</Response>
</tXML>
The values that I'm most interested in pulling out are the App_Err_Cond_Code, Error_Message_Token, and App_Error_Text tags. I had tried using something like this:
extractvalue(xmltype(msg_line_text), '//XPath of Tag')
This works beautifully for stuff where the entire XML is less than 4000 characters, i.e. the entire XML is stored in a single record. The problem comes when there are multiple records, because each individual snippet of XML isn't a valid XML string on its own, and so XMLTYPE throws an error, hence the reason I'm trying to concatenate them all into a single string, which I can then use with the above method.
I've tried a variety of ways to do this - LISTAGG, XMLAGG, SYS_CONNECT_BY_PATH, as well as writing a custom function something like this:
with
function get_messages(pTranLogID number) return string
is
xml varchar2;
begin
xml := '';
for msg in (
select r.msg_line_text
from tran_log_response_message r, tran_log t
where
t.message_id = r.message_id
and t.tran_log_id = pTranLogID
order by r.msg_line_number
)
loop
xml := xml || msg.msg_line_text;
end loop;
return 'test';
end;
select
tran_log_id, get_messages(tran_log_id)
from
tran_log
where
tran_log_id = '20633610';
/
The problem is that every one of these methods complained that the string was too long. Does anyone have any other ideas? Or maybe a better approach to this problem?
Thanks.
Hello I am parsing a xml document with contains bunch of CDATA sections. I was working with no problems till now. I realised that when I am reading the an element and getting the text abribute I am getting end of line characters at the beggining and also at the end of the text read it.
A piece of the important code as follow:
for comments in self.xml.iter("Comments"):
for comment in comments.iter("Comment"):
description = comment.get('Description')
if language == "Arab":
tag = self.name + description
text = comment.text
The problem is at element Comment, he is made it as follow:
<Comment>
<![CDATA[Usually made it with not reason]]>
I try to get the text atribute and I am getting like that:
\nUsually made it with not reason\n
I Know that I could do a strip and so on. But I would like to fix the problem from the root cause, and maybe there is some option before to parse with elementree.
When I am parsing the xml file I am doing like that:
tree = ET.parse(xml)
Minimal reproducible example
import xml.etree.ElementTree as ET
filename = test.xml #Place here your path test xml file
tree = ET.parse(filename)
root = tree.getroot()
Description = root[0]
text = Description.text
print (text)
Minimal xml file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description>
<![CDATA[Hello world]]>
</Description>
You're getting newline characters because there are newline characters:
<Comment>
<![CDATA[Usually made it with not reason]]>
</Comment>
Why else would <![CDATA and </Comment start on new lines?
If you don't want newline characters, remove them:
<Comment><![CDATA[Usually made it with not reason]]></Comment>
Everything inside an element counts towards its string value.
<![CDATA[...]]> is not an element, it's a parser flag. It changes how the XML parser is reading the enclosed characters. You can have multiple CDATA sections in the same element, switching between "regular mode" and "cdata mode" at will:
<Comment>normal text <![CDATA[
CDATA mode, this may contain <unescaped> Characters!
]]> now normal text again
<![CDATA[more special text]]> now normal text again
</Comment>
Any newlines before and after a CDATA section count towards the "normal text" section. When the parser reads this, it will create one long string consisting of the individual parts:
normal text
CDATA mode, this may contain <unescaped> Characters!
now normal text again
more special text now normal text again
I thought that when CDATA comes at xml they were coming with end of line at the beginning and at the end, like that.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description>
<![CDATA[Hello world]]>
</Description>
But you can have it like that also.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description><![CDATA[Hello world]]></Description>
It is the reason to get end of line characters when we are parsing the with the Elementtree library, is working perfect in both cases, you only have to strip or not strip depending how you want to process the data.
if you want to remove both '\n' just add the following code:
text = Description.text
text = text.strip('\n')
I have a structurised json file, containing a list of elements.
I would like to convert all of the elements (which have also a structure) to table format. I can manually do it from PowerQuery GUI for one element; example code below.
let
Source=
Json.Document(Web.Contents("<source_address>")),
elements = Source[elements],
elements1 = elements{0},
#"Converted into table" = Record.ToTable(elements1)
in
#"Converted into table"
I'd like to iterate through all elements (so from elements{0} to elements{x}) and keep them in one excel output table. All elements have the same structure (columns)
Would be useful to see some sample JSON, as it's difficult to say with certainty without looking at its structure. I've just used some dummy JSON string, which I'm assuming resembles yours (in terms of structure).
let
//Source = Json.Document(Web.Contents("<source_address>")),
Source = Json.Document("{""elements"": [{""someKey1"": ""someValue1""}, {""someKey2"": ""someValue2""}], ""someOtherKey"": ""cat""}"),
elements = Source[elements],
transformList = List.Transform(elements, Record.ToTable),
appended = Table.Combine(transformList)
in
appended
If you copy-paste the above to the Advanced Editor, replace the second Source = ... with the one from your original code, you can then check if it gives you what you're after.
Edit:
I think given the URL you've provided, the code below will save you pivoting the table later and give you desired output.
let
Source = Json.Document(Web.Contents("https://fantasy.premierleague.com/drf/bootstrap-static")),
elements = Source[elements],
toTable = Table.FromRecords(elements)
in
toTable
Which gives me the below:
I'm trying to extract CDATA content from an XML without the using GPath (or) node name. In short, i want to find & retrieve the innerText containing CDATA section from an XML.
My XML look like:
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
From the above XML, i want to get the CDATA content alone without using the reference of its node name 'Test2'. Because the node name is not always the same in my scenario.
Also note that the XML can contain innerText in few other nodes (Test1). I dont want to retrieve that. I just need the CDATA content out of the whole XML.
I want something like below (the code below is incorrect though)
def parsedXML = new xmlSlurper().parseText(xml)
def cdataContent = parsedXML.depthFirst().findAll { it.text().startsWith('<![CDATA')}
My output should be :
this is the CDATA section i want to retrieve
As #daggett says, you can't do this with the Groovy slurper or parser, but it's not too bad to drop down and use the java classes to get it.
Note you have to set the property for CDATA to become visible, as by default it's just treated as characters.
Here's the code:
import javax.xml.stream.*
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
def factory = XMLInputFactory.newInstance()
factory.setProperty('http://java.sun.com/xml/stream/properties/report-cdata-event', true)
def reader = factory.createXMLStreamReader(new StringReader(xml))
while (reader.hasNext()) {
if (reader.eventType in [XMLStreamConstants.CDATA]) {
println reader.text
}
reader.next()
}
That will print this is the CDATA section i want to retrieve
Considering you just have one CDATA in your xml split can help here
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
log.info xml.split("<!\\[CDATA\\[")[1].split("]]")[0]
So in the above logic we split the string on CDATA start and pick the portion which is left after
xml.split("<!\\[CDATA\\[")[1]
and once we got that portion we did the split again and then got the portion which is before that pattern by using
.split("]]")[0]
Here is the proof it works
My App is calling the free Weather Forecast web service found at this URL:
http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=Boston&CountryName=United+States
I'm using the usual NSURLConnection and NSXMLParser delegate methods to parse the incoming data (I've done this a million times before) but quite strangely, the NSMutableData that is returned is not getting converted to a string correctly via NSUTF8StringEncoding. Its basically failing to convert the "<" and ">" characters of the opening and closing XML tags, giving me "& l t;" and "& g t;" instead.
The problem seems to be in the connectionDidFinishLoading function:
func connection(connection: NSURLConnection, didReceiveData data: NSData) {
webServiceData!.appendData(data)
}
func connectionDidFinishLoading(connection: NSURLConnection) {
let XMLResponseString = NSString(data: webServiceData!, encoding: NSUTF8StringEncoding)!
println("XMLResponseString = \(XMLResponseString)")
}
The output I get from the println statement there is:
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www.webserviceX.NET"><?xml version="1.0" encoding="utf-16"?>
<CurrentWeather>
<Location>DALLAS EXECUTIVE AIRPORT, TX, United States (KRBD) 32-41N 096-52W 203M</Location>
<Time>Dec 30, 2014 - 08:53 AM EST / 2014.12.30 1353 UTC</Time>
<Wind> from the NE (050 degrees) at 12 MPH (10 KT):0</Wind>
<Visibility> 9 mile(s):0</Visibility>
<SkyConditions> overcast</SkyConditions>
<Temperature> 39.9 F (4.4 C)</Temperature>
<DewPoint> 34.0 F (1.1 C)</DewPoint>
<RelativeHumidity> 79%</RelativeHumidity>
<Pressure> 30.42 in. Hg (1030 hPa)</Pressure>
<Status>Success</Status>
</CurrentWeather></string>
So as you can see I'm getting the first 2 tags correctly - the "< ?XML >" and "< string xmlns >" tags, but the rest are all showing up as "& l t;" and "& g t;"
What's really strange is that its saying encoding="utf-8" for the first tag, but on the second line (towards the end) its saying encoding="utf-16".
So I tried using NSUTF16StringEncoding:
let XMLResponseString = NSString(data: webServiceData!, encoding: NSUTF16StringEncoding)!
and that basically gave me chinese looking characters.
I also tried running the parser directly on the url instead of the NSMutableData that's returned, like so:
myXMLParser = NSXMLParser(contentsOfURL:theURL!)!
(the original statement was this:
myXMLParser = NSXMLParser(data:webServiceData)
but neither of these worked.
So what's going on here? Any suggestions on how to get this to work properly?
This is actually the remote service being broken, rather than your code. Yes, the server really is sending XML in XML for no particularly good reason.
$ curl 'http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=Boston&CountryName=United+States'
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www.webserviceX.NET"><?xml version="1.0" encoding="utf-16"?>
<CurrentWeather>
<Location>BOSTON LOGAN INTERNATIONAL, MA, United States (KBOS) 42-22N 071-01W 54M</Location>