Hello I am parsing a xml document with contains bunch of CDATA sections. I was working with no problems till now. I realised that when I am reading the an element and getting the text abribute I am getting end of line characters at the beggining and also at the end of the text read it.
A piece of the important code as follow:
for comments in self.xml.iter("Comments"):
for comment in comments.iter("Comment"):
description = comment.get('Description')
if language == "Arab":
tag = self.name + description
text = comment.text
The problem is at element Comment, he is made it as follow:
<Comment>
<![CDATA[Usually made it with not reason]]>
I try to get the text atribute and I am getting like that:
\nUsually made it with not reason\n
I Know that I could do a strip and so on. But I would like to fix the problem from the root cause, and maybe there is some option before to parse with elementree.
When I am parsing the xml file I am doing like that:
tree = ET.parse(xml)
Minimal reproducible example
import xml.etree.ElementTree as ET
filename = test.xml #Place here your path test xml file
tree = ET.parse(filename)
root = tree.getroot()
Description = root[0]
text = Description.text
print (text)
Minimal xml file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description>
<![CDATA[Hello world]]>
</Description>
You're getting newline characters because there are newline characters:
<Comment>
<![CDATA[Usually made it with not reason]]>
</Comment>
Why else would <![CDATA and </Comment start on new lines?
If you don't want newline characters, remove them:
<Comment><![CDATA[Usually made it with not reason]]></Comment>
Everything inside an element counts towards its string value.
<![CDATA[...]]> is not an element, it's a parser flag. It changes how the XML parser is reading the enclosed characters. You can have multiple CDATA sections in the same element, switching between "regular mode" and "cdata mode" at will:
<Comment>normal text <![CDATA[
CDATA mode, this may contain <unescaped> Characters!
]]> now normal text again
<![CDATA[more special text]]> now normal text again
</Comment>
Any newlines before and after a CDATA section count towards the "normal text" section. When the parser reads this, it will create one long string consisting of the individual parts:
normal text
CDATA mode, this may contain <unescaped> Characters!
now normal text again
more special text now normal text again
I thought that when CDATA comes at xml they were coming with end of line at the beginning and at the end, like that.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description>
<![CDATA[Hello world]]>
</Description>
But you can have it like that also.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Description><![CDATA[Hello world]]></Description>
It is the reason to get end of line characters when we are parsing the with the Elementtree library, is working perfect in both cases, you only have to strip or not strip depending how you want to process the data.
if you want to remove both '\n' just add the following code:
text = Description.text
text = text.strip('\n')
Related
Having an XML file like this:
<?xml version="1.0" encoding="UTF-8"?><outer>
<inner>Some text.</inner>
<inner>More text.</inner>
</outer>
and the following PowerBI script
let
Table0 = Xml.Tables(File.Contents("simple1.xml")){0}[Table]
in
Table0
you get this
Element:Text
Some text.
More text.
Now I'd like to add sub elements and keep inner.Element:Text
<?xml version="1.0" encoding="UTF-8"?><outer>
<inner>Some text.<secret>Don't care.</secret></inner>
<inner>More text.<secret>You know.</secret></inner>
</outer>
Using the same PowerBI script as above you get
secret
Don't care.
You know.
I already tried this script
let
Table0 = Xml.Tables(File.Contents("simple2.xml")),
Table1 = Table.ExpandTableColumn(Table0, "Table", {"secret"})
in
Table1
but got this
Name
secret
inner
Don't care.
inner
You know.
But I'd like to get this:
Element:Text
secret.Element:Text
Some text.
Don't care.
More text.
You know.
My current workaround (which I'd like to avoid) is to use sed to wrap the element text of an inner entry in its own sub element:
<inner><text>Some text.</text><secret>Don't care.</secret></inner>
I am trying to read a PDF file with several pages, using iText 7 on a .NET CORE 2.1
The following is my code:
Rectangle rect = new Rectangle(0, 0, 1100, 1100);
LocationTextExtractionStrategy strategy = new LocationTextExtractionStrategy();
inputStr = PdfTextExtractor.GetTextFromPage(pdfDocument.GetPage(i), strategy);
inputStr gets the following string:
"\u0011\v\u000e\u0012\u0011\v\f)(*).=*%'\f*).5?5.5*.\a \u0011\u0002\u001b\u0001!\u0016\u0012\u001a!\u0001\u0015\u001a \u0014\n\u0015\u0017\u0001(\u001b)\u0001)\u0016\u001c*\u0012\u0001\u001d\u001a \u0016* \u0015\u0001\u0017\u0016\u001b\u001a(\n,\u0002>&\u00...
and in the Text Visualizer, it looks like that:
)(*).=*%' *).5?5.5*. !!
())* * (
,>&2*06) 2.-=9 )=&,
2..*0.5<.?
.110
)<1,3
2.3*1>?)10/6
(& >(*,1=0>>*1?
2.63)&*,..*0.5
206)&13'?*9*<
*-5=0>
?*&..,?)..*0.5
it looks like I am unable to resolve the encoding or there is a specific, custom encoding at the PDF level I cannot read/parse.
Looking at the Document Properties, under Fonts it says the following:
Any ideas how can I parse the document correctly?
Thank you
Yaniv
Analysis of the shared files
file1_copyPasteWorks.pdf
The font definitions here have an invalid ToUnicode entry:
/ToUnicode/Identity-H
The ToUnicode value is specified as
A stream containing a CMap file that maps character codes to Unicode values
(ISO 32000-2, Table 119 — Entries in a Type 0 font dictionary)
Identity-H is a name, not a stream.
Nonetheless, Adobe Reader interprets this name, and for apparently any name starting with Identity- assumes the text encoding for the font to be UCS-2 (essentially UTF-16). As this indeed is the case for the character codes used in the document, copy&paste works, even if for the wrong reasons. (Without this ToUnicode value, Adobe Reader also returns nonsense.)
iText 7, on the other hand, for mapping to Unicode first follows the Encoding value with unexpected results.
Thus, in this case Adobe Reader arrives at a better result by interpreting meaning into an invalid piece of data (and without that also returns nonsense).
file2_copyPasteFails.pdf
The font definitions here have valid but incomplete ToUnicode maps which only contain entries for the used Western European characters but not for Hebrew ones. They don't have Encoding entries.
Both Adobe Reader and iText 7 here trust the ToUnicode map and, therefore, cannot map the Hebrew glyphs.
How to parse
file1_copyPasteWorks.pdf
In case of this file the "problem" is that iText 7 applies the Encoding map. Thus, for decoding the text one can temporarily replace the Encoding map with an identity map:
for (int i = 1; i <= pdfDocument.GetNumberOfPages(); i++)
{
PdfPage page = pdfDocument.GetPage(i);
PdfDictionary fontResources = page.GetResources().GetResource(PdfName.Font);
foreach (PdfObject font in fontResources.Values(true))
{
if (font is PdfDictionary fontDict)
fontDict.Put(PdfName.Encoding, PdfName.IdentityH);
}
string output = PdfTextExtractor.GetTextFromPage(page);
// ... process output ...
}
This code shows the Hebrew characters for your file 1.
file2_copyPasteFails.pdf
Here I don't have a quick work-around. You may want to analyze multiple PDFs of that kind. If they all encode the Hebrew characters the same way, you can create your own ToUnicode map from that and inject it into the fonts like above.
I need replaceLast() method in the Groovy script - replace the last substring. It is available in Java, but not in Groovy AFAIK. It must work with regex in the same way as the following replaceFirst.
replaceFirst(CharSequence self, Pattern pattern, CharSequence replacement)
Replaces the first substring of a CharSequence that matches the given compiled regular expression with the given replacement.
EDIT: Sorry not being specific enough. Original string is an XML file and the same key (e.g. Name) is present many times. I want to replace the last one.
<Header>
<TransactionId>1</TransactionId>
<SessionId>1</SessionId>
<User>
<Name>Bob</Name>
...
</User>
<Sender>
<Name>Joe</Name>
...
</Sender>
</Header>
...
<Context>
<Name>Rose</Name>
...
</Context>
No idea what replaceLast in Java is...it's not in the JDK... If it was in the JDK, you could use it in Groovy...
Anyway, how about using an XML parser to change your XML instead of using a regular expression?
Given some xml:
def xml = '''<Header>
<TransactionId>1</TransactionId>
<SessionId>1</SessionId>
<User>
<Name>Bob</Name>
</User>
<Sender>
<Name>Joe</Name>
</Sender>
<Something>
<Name>Tim</Name>
</Something>
</Header>'''
You can parse it using Groovy's XmlParser:
import groovy.xml.*
def parsed = new XmlParser().parseText(xml)
Then, you can do a depth first search for all nodes with the name Name, and take the last -1 one:
def lastNameNode = parsed.'**'.findAll { it.name() == 'Name' }[-1]
Then, set the value to a new string:
lastNameNode.value = 'Yates'
And print the new XML:
println XmlUtil.serialize(parsed)
<?xml version="1.0" encoding="UTF-8"?><Header>
<TransactionId>1</TransactionId>
<SessionId>1</SessionId>
<User>
<Name>Bob</Name>
</User>
<Sender>
<Name>Joe</Name>
</Sender>
<Something>
<Name>Yates</Name>
</Something>
</Header>
I'm trying to extract CDATA content from an XML without the using GPath (or) node name. In short, i want to find & retrieve the innerText containing CDATA section from an XML.
My XML look like:
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
From the above XML, i want to get the CDATA content alone without using the reference of its node name 'Test2'. Because the node name is not always the same in my scenario.
Also note that the XML can contain innerText in few other nodes (Test1). I dont want to retrieve that. I just need the CDATA content out of the whole XML.
I want something like below (the code below is incorrect though)
def parsedXML = new xmlSlurper().parseText(xml)
def cdataContent = parsedXML.depthFirst().findAll { it.text().startsWith('<![CDATA')}
My output should be :
this is the CDATA section i want to retrieve
As #daggett says, you can't do this with the Groovy slurper or parser, but it's not too bad to drop down and use the java classes to get it.
Note you have to set the property for CDATA to become visible, as by default it's just treated as characters.
Here's the code:
import javax.xml.stream.*
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
def factory = XMLInputFactory.newInstance()
factory.setProperty('http://java.sun.com/xml/stream/properties/report-cdata-event', true)
def reader = factory.createXMLStreamReader(new StringReader(xml))
while (reader.hasNext()) {
if (reader.eventType in [XMLStreamConstants.CDATA]) {
println reader.text
}
reader.next()
}
That will print this is the CDATA section i want to retrieve
Considering you just have one CDATA in your xml split can help here
def xml = '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<root>
<Test1>This node contains some innerText. Ignore This.</Test1>
<Test2><![CDATA[this is the CDATA section i want to retrieve]]></Test2>
</root>'''
log.info xml.split("<!\\[CDATA\\[")[1].split("]]")[0]
So in the above logic we split the string on CDATA start and pick the portion which is left after
xml.split("<!\\[CDATA\\[")[1]
and once we got that portion we did the split again and then got the portion which is before that pattern by using
.split("]]")[0]
Here is the proof it works
My App is calling the free Weather Forecast web service found at this URL:
http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=Boston&CountryName=United+States
I'm using the usual NSURLConnection and NSXMLParser delegate methods to parse the incoming data (I've done this a million times before) but quite strangely, the NSMutableData that is returned is not getting converted to a string correctly via NSUTF8StringEncoding. Its basically failing to convert the "<" and ">" characters of the opening and closing XML tags, giving me "& l t;" and "& g t;" instead.
The problem seems to be in the connectionDidFinishLoading function:
func connection(connection: NSURLConnection, didReceiveData data: NSData) {
webServiceData!.appendData(data)
}
func connectionDidFinishLoading(connection: NSURLConnection) {
let XMLResponseString = NSString(data: webServiceData!, encoding: NSUTF8StringEncoding)!
println("XMLResponseString = \(XMLResponseString)")
}
The output I get from the println statement there is:
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www.webserviceX.NET"><?xml version="1.0" encoding="utf-16"?>
<CurrentWeather>
<Location>DALLAS EXECUTIVE AIRPORT, TX, United States (KRBD) 32-41N 096-52W 203M</Location>
<Time>Dec 30, 2014 - 08:53 AM EST / 2014.12.30 1353 UTC</Time>
<Wind> from the NE (050 degrees) at 12 MPH (10 KT):0</Wind>
<Visibility> 9 mile(s):0</Visibility>
<SkyConditions> overcast</SkyConditions>
<Temperature> 39.9 F (4.4 C)</Temperature>
<DewPoint> 34.0 F (1.1 C)</DewPoint>
<RelativeHumidity> 79%</RelativeHumidity>
<Pressure> 30.42 in. Hg (1030 hPa)</Pressure>
<Status>Success</Status>
</CurrentWeather></string>
So as you can see I'm getting the first 2 tags correctly - the "< ?XML >" and "< string xmlns >" tags, but the rest are all showing up as "& l t;" and "& g t;"
What's really strange is that its saying encoding="utf-8" for the first tag, but on the second line (towards the end) its saying encoding="utf-16".
So I tried using NSUTF16StringEncoding:
let XMLResponseString = NSString(data: webServiceData!, encoding: NSUTF16StringEncoding)!
and that basically gave me chinese looking characters.
I also tried running the parser directly on the url instead of the NSMutableData that's returned, like so:
myXMLParser = NSXMLParser(contentsOfURL:theURL!)!
(the original statement was this:
myXMLParser = NSXMLParser(data:webServiceData)
but neither of these worked.
So what's going on here? Any suggestions on how to get this to work properly?
This is actually the remote service being broken, rather than your code. Yes, the server really is sending XML in XML for no particularly good reason.
$ curl 'http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=Boston&CountryName=United+States'
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www.webserviceX.NET"><?xml version="1.0" encoding="utf-16"?>
<CurrentWeather>
<Location>BOSTON LOGAN INTERNATIONAL, MA, United States (KBOS) 42-22N 071-01W 54M</Location>