when read excel how to skip some invalid characters - apache-poi

Question

Read some excel using poi failed, encountered such an error
Caused by: org.xml.sax.SAXParseException; systemId: file://; lineNumber: 105; columnNumber: 147342; An invalid XML character (Unicode: 0xffff) was found in the element content of the document.
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
From xl/sharedStrings.xml, there exist <ffff> cause this problem.
How could read it successfully and just ignore these invalid characters? e.g.
aaa <ffff> bbb ==> aaa bbb

Answer 1

Those invalid characters should not be in the XML and Excel itself will not put them into there. So someone probably had done something wrong while creating that file using something else than Excel. That error should be avoided rather than trying to ignore the symptoms.
But I know how it feels to be depemdent on others work which will be done in far future, if even. So one needs improvising. But that is in this case only possible using ugly low level methods. Because the XML is invalid, parsing XML is not possible. So only String replacing will be possible.
In APACHE POI EXCEL XmlException: is an invalid XML character, is there any way to preprocess the excel file? I had schown this already. In that case to replace UTF-16-surrogate-pair numeric character references which also are invalid in XML.
In following I will show a code which is more flexible to add multiple other repairing actions to /xl/sharedStrings.xml if necessary.
The principle is using OPCPackage, which is the *.xlsx ZIP package, to get out the /xl/sharedStrings.xml as text string. Then do the needed replacings and put the repaired /xl/sharedStrings.xml back into the OPCPackage. Then do creating the XSSFWorkbook from that repaired OPCPackage instead of from the corrupt file.
import java.io.*;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.apache.poi.openxml4j.opc.*;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
class RepairSharedStringsTable {
static String removeInvalidXmlCharacters(String string) {
String xml10pattern = "[^"
+ "\u0009\r\n"
+ "\u0020-\uD7FF"
+ "\uE000-\uFFFD"
+ "\ud800\udc00-\udbff\udfff"
+ "]";
string = string.replaceAll(xml10pattern, "");
return string;
}
static void repairSharedStringsTable(OPCPackage opcPackage) {
for (PackagePart packagePart : opcPackage.getPartsByName(Pattern.compile("/xl/sharedStrings.xml"))) {
String sharedStrings = "";
try (BufferedInputStream inputStream = new BufferedInputStream(packagePart.getInputStream());
ByteArrayOutputStream sharedStringsBytes = new ByteArrayOutputStream() ) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
sharedStringsBytes.write(buffer, 0, length);
}
sharedStrings = sharedStringsBytes.toString("UTF-8");
} catch (Exception ex) {
ex.printStackTrace();
}
System.out.println(sharedStrings);
//sharedStrings = replaceUTF16SurrogatePairs(sharedStrings);
sharedStrings = removeInvalidXmlCharacters(sharedStrings);
//sharedStrings = doSomethingElse(sharedStrings);
System.out.println(sharedStrings);
try (BufferedOutputStream outputStream = new BufferedOutputStream(packagePart.getOutputStream()) ) {
outputStream.write(sharedStrings.getBytes("UTF-8"));
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
public static void main(String[] args) throws Exception {
try (XSSFWorkbook workbook = new XSSFWorkbook(new FileInputStream("./Excel.xlsx"))) {
System.out.println("success");
} catch (Exception ex) {
System.out.println("failed");
ex.printStackTrace();
}
OPCPackage opcPackage = OPCPackage.open(new FileInputStream("./Excel.xlsx"));
repairSharedStringsTable(opcPackage);
opcPackage.flush();
try (XSSFWorkbook workbook = new XSSFWorkbook(opcPackage);
FileOutputStream out = new FileOutputStream("./ExcelRepaired.xlsx");) {
workbook.write(out);
System.out.println("success");
} catch (Exception ex) {
System.out.println("failed");
ex.printStackTrace();
}
}
}

Answer 2

In my case below files all have invalid characters
xl/sharedStrings.xml
xl/worksheets/sheet1.xml
xl/worksheets/sheet8.xml
All these xml should be processed
opcPackage.getPartsByName(Pattern.compile("(/xl/sharedStrings.xml)|(/xl/worksheets/.+\\.xml)"))