How to process two parts of a single xml by streaming? - xslt-3.0

Question

I have a really large XML that has two different sets of data. I need to stream through one and do a lookup on the other. My Data looks like below
<?xml version="1.0" encoding="utf-8"?>
<Root>
<EmployeePersonal>
<Employee>
<ID>21234</ID>
<Name>Jim Carrey</Name>
<Age>43</Age>
<City>Chicago</City>
<State>IL</State>
</Employee>
<Employee>
<ID>41876</ID>
<Name>Edward Norton</Name>
<Age>33</Age>
<City>New York</City>
<State>NY</State>
</Employee>
<Employee>
<ID>51239</ID>
<Name>Eli Roth</Name>
<Age>46</Age>
<City>Los Angeles</City>
<State>CA</State>
</Employee>
</EmployeePersonal>
<EmployeeEmployment>
<Empl>
<Emplid>21234</Emplid>
<Title>HR Partner</Title>
<HireDate>2008-12-29</HireDate>
</Empl>
<Empl>
<Emplid>41876</Emplid>
<Title>Comp Partner</Title>
<HireDate>1999-07-09</HireDate>
</Empl>
<Empl>
<Emplid>51239</Emplid>
<Title>Programmer</Title>
<HireDate>2004-12-06</HireDate>
</Empl>
</EmployeeEmployment>
</Root>
I would like to loop through the /Root/EmployeePersonal data and lookup the /Root/EmployeeEmployment by matching on the employee id.
I tried to loop through one and then load into a map and then loop through the other, but kept getting an error. Finally I tried loading one set into a variable and then tried to stream the other, but in vain. This is what I've tried so far.
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
exclude-result-prefixes="xs"
version="3.0">
<xsl:mode streamable="yes" on-no-match="shallow-skip"/>
<xsl:output indent="no"/>
<xsl:variable name="vEmploymentData" select="Root/copy-of(EmployeeEmployment)"/>
<xsl:template match="/Root/EmployeePersonal">
<AllEmployeeData>
<xsl:for-each select="Employee/copy-of()">
<Employee>
<xsl:copy-of select="./*"/>
<xsl:copy-of select="$vEmploymentData/Empl[Emplid=current()/ID]/*[local-name() ne 'Emplid']"/>
</Employee>
</xsl:for-each>
</AllEmployeeData>
</xsl:template>
</xsl:stylesheet>
I need to merge these two sets of data together. Since the data is huge, is there a way to stream both sets of data in?
Thanks

Answer 1

Given that you need to different sets of the data at the same time to compare and merge them I am not sure there is an easy single pass, streamable and therefore forwards-only approach, so some easy way around that would be to set up the same document twice as an xsl:merge-source for an xsl:merge:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
exclude-result-prefixes="#all">
<xsl:param name="input-uri" as="xs:string" select="'input1.xml'"/>
<xsl:output indent="yes"/>
<xsl:template name="xsl:initial-template">
<AllEmployeeData>
<xsl:merge>
<xsl:merge-source name="emp-details1" for-each-source="$input-uri" streamable="yes" select="Root/EmployeePersonal/Employee">
<xsl:merge-key select="ID"/>
</xsl:merge-source>
<xsl:merge-source name="emp-details2" for-each-source="$input-uri" streamable="yes" select="Root/EmployeeEmployment/Empl">
<xsl:merge-key select="Emplid"/>
</xsl:merge-source>
<xsl:merge-action>
<xsl:copy>
<xsl:copy-of select="current-merge-group('emp-details1')/*, current-merge-group('emp-details2')/(* except Emplid)"/>
</xsl:copy>
</xsl:merge-action>
</xsl:merge>
</AllEmployeeData>
</xsl:template>
</xsl:stylesheet>
As xsl:merge works with a snapshot, that has the advantage that the access to ID and Emplid child is not a problem. You also don't have to take any particular action to buffer data in maps or accumulators. The main disadvantage is that xsl:merge expects the sources to be ordered by the merge keys, that seems to be the case for your sample data but I am not sure that will be true for your complete data.
You would run such a stylesheet with Saxon 9 EE using the command line option -it to start without a primary input but with the initial template named xsl:initial-template instead.
As an alternative approach that only reads through the main input once but buffers the EmployeePersonal/Employee in an accumulator using a Saxon 9.9 EE specific saxon:capture attribute you could use:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:saxon="http://saxon.sf.net/"
exclude-result-prefixes="#all">
<xsl:output indent="yes"/>
<xsl:mode streamable="yes" on-no-match="shallow-skip" use-accumulators="emp-details1"/>
<xsl:accumulator name="emp-details1" streamable="yes" as="element(Employee)*" initial-value="()">
<xsl:accumulator-rule match="EmployeePersonal/Employee" phase="end" saxon:capture="yes" select="$value, ."/>
</xsl:accumulator>
<xsl:template match="/*">
<AllEmployeeData>
<xsl:apply-templates/>
</AllEmployeeData>
</xsl:template>
<xsl:template match="EmployeeEmployment/Empl">
<Employee>
<xsl:variable name="this" select="copy-of()"/>
<xsl:copy-of select="accumulator-before('emp-details1')[ID = $this/Emplid]/*, $this/(* except Emplid)"/>
</Employee>
</xsl:template>
</xsl:stylesheet>
This reads through the input only once but buffers all EmployeePersonal/Employee in memory and then uses the buffered/accumulated elements later one when EmployeeEmployment/Empl is matched to output the necessary corresponding data. This also of course, as the output is created when matching EmployeeEmployment/Empl, requires the presence of all ids in that data, I am not sure that is the case or whether the EmployeePersonal/Employee is the main input and you only look for matching EmployeeEmployment/Empl if it exists.
Finally, you could as well try to store the details of EmployeePersonal/Employee in an accumulator using a map and then access this from the template for EmployeeEmployment/Empl:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:map="http://www.w3.org/2005/xpath-functions/map"
xmlns:saxon="http://saxon.sf.net/"
expand-text="yes"
exclude-result-prefixes="#all">
<xsl:output indent="yes"/>
<xsl:mode streamable="yes" on-no-match="shallow-skip" use-accumulators="#all"/>
<xsl:accumulator name="current-emp-id" streamable="yes" as="xs:integer?" initial-value="()">
<xsl:accumulator-rule match="EmployeePersonal/Employee/ID/text()" select="xs:integer(.)"/>
</xsl:accumulator>
<xsl:accumulator name="emp-details" streamable="yes" as="map(xs:integer, map(xs:string, xs:string))" initial-value="map{}">
<xsl:accumulator-rule match="EmployeePersonal/Employee/ID/text()" select="map:put($value, xs:integer(.), map{})"/>
<xsl:accumulator-rule match="EmployeePersonal/Employee/*[not(self::ID)]/text()"
select="map:put($value, accumulator-before('current-emp-id'), map:put(map:get($value, accumulator-before('current-emp-id')), local-name(..), string()))"/>
</xsl:accumulator>
<xsl:template match="/*">
<AllEmployeeData>
<xsl:apply-templates/>
</AllEmployeeData>
</xsl:template>
<xsl:template match="EmployeeEmployment/Empl">
<Employee>
<xsl:variable name="this" select="copy-of()"/>
<ID>{$this/Emplid}</ID>
<xsl:apply-templates select="map:get(accumulator-before('emp-details'), xs:integer($this/Emplid))" mode="entry-to-element"/>
<xsl:copy-of select="$this/(* except Emplid)"/>
</Employee>
</xsl:template>
<xsl:template match=".[. instance of map(*)]" mode="entry-to-element">
<xsl:variable name="map" select="."/>
<xsl:for-each select="map:keys(.)">
<xsl:element name="{.}">{map:get($map, .)}</xsl:element>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>
Of course it will still buffer all the EmployeePersonal/Employee data, only this time using a light-weight map instead of a sequence of XML snapshot elements. The main disadvantage is that XSLT 3/XPath 3.1 maps have no order so unless you later spell out each entry you are looking for in the wanted order you get the results in a random order. In the example above I have simply used a for-each to convert map entries back to XML elements so the order is a random order and not the one of the input XML (that the xsl:merge approach or the saxon:capture approach would preserve).
As you asked in a comment about using xsl:merge but provide the input as a primary input source, the only way I can think of is to use it as a dummy streamable input and then to provide its document-uri() for the xsl:merge/xsl:merge-source, that is, to adapt the code from above to
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
exclude-result-prefixes="#all">
<xsl:mode streamable="yes"/>
<xsl:output indent="yes"/>
<xsl:template match="/">
<AllEmployeeData>
<xsl:variable name="input-uri" as="xs:anyURI" select="document-uri()"/>
<xsl:message select="$input-uri"/>
<xsl:merge>
<xsl:merge-source name="emp-details1" for-each-source="$input-uri" streamable="yes" select="Root/EmployeePersonal/Employee">
<xsl:merge-key select="ID"/>
</xsl:merge-source>
<xsl:merge-source name="emp-details2" for-each-source="$input-uri" streamable="yes" select="Root/EmployeeEmployment/Empl">
<xsl:merge-key select="Emplid"/>
</xsl:merge-source>
<xsl:merge-action>
<xsl:copy>
<xsl:copy-of select="current-merge-group('emp-details1')/*, current-merge-group('emp-details2')/(* except Emplid)"/>
</xsl:copy>
</xsl:merge-action>
</xsl:merge>
</AllEmployeeData>
</xsl:template>
</xsl:stylesheet>
Interestingly enough, that works with Saxon 9.7 EE (of course that opening the same input three times for streaming) but fails with an error " XTSE3430: Template rule is not streamable * The merge-source/#select expression is not striding" in Saxon 9.8 and 9.9 EE.
Not sure whether that helps, in earlier questions you seemed to indicate that the tool you use embeds Saxon 9.7 EE so it might be an option.