Do not remove extra lines while processing the crawled text from webpages - nutch

Question

I searched the plugins of parse-html and I am not getting where to change the code so that it does not removes the extra line from html page.
while crawling with nutch, it is removing all the extra lines from the crawled text. I want to keep the text and whatever the new lines are present on the website. for example: on crawling this page https://www.modernfamilydental.net/,
the expected output is :\n\n\n\nSan Francisco, CA Dentist\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWould you like to switch to the accessible version of this site?\nGo to accessible site\n\nClose modal window\n\n\n\n\n\nDon\'t need the accessible version of this site?\nHide the accessibility button\n\nClose modal window\n\n\n\n\n\n\nAccessibility View\n\n\nClose toolbar\n\n\n\n\nJavascript must be enabled for the correct page display\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nModern Family Dental Hao Tran, DMD\nDentist located in Laurel Heights, San Francisco, CA\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n(415) 752-5244\n\n\n \n\n\n\n\n\n\n\n\n\nMenu\n\n\n\n\nHome\n\n\nServices\n \nLatest Equipment\n\n\nInsurance\n\n\nTeeth Whitening\n\n\nCrowns & Bridges\n\n\nSmile Makeovers\n\n\nResin Composite Bonding\n\n\nVeneers\n\n\nImplant Retained Dentures\n\n\nNight Guards\n\n\nMetal-Free Restoration\n\n\nInvisalign\n\n\nDental Examination
but the output from nutch is :
San Francisco, CA Dentist\nWould you like to switch to the accessible version of this site?\nGo to accessible site\nClose modal window\nDon\'t need the accessible version of this site?\nHide the accessibility button\n\nClose modal window\nAccessibility View\n\n\nClose toolbar\n\n\n\n\nJavascript must be enabled for the correct page display\nModern Family Dental Hao Tran, DMD\nDentist located in Laurel Heights, San Francisco, CA\n(415) 752-5244\nMenu\nHome\nServices\nLatest Equipment\nInsurance\nTeeth Whitening\nCrowns & Bridges\nSmile Makeovers\n\n\nResin Composite Bonding\nVeneers\nImplant Retained Dentures\nNight Guards\nMetal-Free Restoration\nInvisalign\nDental Examination
May I know which plugin code i should change or I should change code of parse_text.

Answer 1

As I have already Answered here in the comment section.
If you do not want to read from the /content folder from segments.
you can do the following things. I'm assuming you must be using parse-html|parse-tika plugins to parse HTML content.
If you are using any one of them. then Nutch plugins use DOMContentUtils API to extract the parsed Text from HTML.
**// this method extract text from Node object and append to
StringBuffer sb**
public boolean getText(StringBuffer sb, Node node,
boolean abortOnNestedAnchors) {
if (getTextHelper(sb, node, abortOnNestedAnchors, 0)) {
return true;
}
return false;
}
In getTextHelper method you can comment out line text = text.replaceAll("\\s+", " "); so that it will not replace multiple [ \t\r\n\f] with single occurrence.
private boolean getTextHelper(StringBuffer sb, Node node,
boolean abortOnNestedAnchors, int anchorDepth) {
boolean abort = false;
NodeWalker walker = new NodeWalker(node);
while (walker.hasNext()) {
Node currentNode = walker.nextNode();
String nodeName = currentNode.getNodeName();
short nodeType = currentNode.getNodeType();
Node previousSibling = currentNode.getPreviousSibling();
if (previousSibling != null
&& blockNodes.contains(previousSibling.getNodeName().toLowerCase())) {
appendParagraphSeparator(sb);
} else if (blockNodes.contains(nodeName.toLowerCase())) {
appendParagraphSeparator(sb);
}
if ("script".equalsIgnoreCase(nodeName)) {
walker.skipChildren();
}
if ("style".equalsIgnoreCase(nodeName)) {
walker.skipChildren();
}
if (abortOnNestedAnchors && "a".equalsIgnoreCase(nodeName)) {
anchorDepth++;
if (anchorDepth > 1) {
abort = true;
break;
}
}
if (nodeType == Node.COMMENT_NODE) {
walker.skipChildren();
}
if (nodeType == Node.TEXT_NODE) {
// cleanup and trim the value
String text = currentNode.getNodeValue();
**text = text.replaceAll("\\s+", " ");**
text = text.trim();
if (text.length() > 0) {
appendSpace(sb);
sb.append(text);
} else {
appendParagraphSeparator(sb);
}
}
}
return abort;
}