react-virtualized - how do I use this as a true infinite scroller - react-virtualized

I can't find any code example or docs that answers this:
Achieve almost complete infinite scroll -> unknown # of items, but there is a finite amount that may be infeasible to compute beforehand - e.g. at some point the list needs to stop scrolling
Can I trigger first load of data from within InfiniteScroller/List - it seems you need to pass in a data source that is populated with initial page
I am using this example:
https://github.com/bvaughn/react-virtualized/blob/master/docs/creatingAnInfiniteLoadingList.md
and:
https://github.com/bvaughn/react-virtualized/blob/master/source/InfiniteLoader/InfiniteLoader.example.js
along with CellMeasurer for dynamic height:
https://github.com/bvaughn/react-virtualized/blob/master/source/CellMeasurer/CellMeasurer.DynamicHeightList.example.js
The docs for InfiniteLoader.rowCount say:
"Number of rows in list; can be arbitrary high number if actual number is unknown."
So how do you indicate there are no more rows.
If anyone can post an example using setTimeout() to simulate dynamic loaded data, thanks. I can likely get CellMeasurer working from there.
Edit
This doesn't work the way react-virtualized creator says it should or the infinite loading example implies.
Calls:
render(): rowCount = 1
_rowRenderer(index = 0)
_isRowLoaded(index = 0)
_loadMoreRows(startIndex = 0, stopIndex = 0)
_rowRenderer(index = 0)
end
Do I need to specify a batch size or some other props?
class HistoryBrowser extends React.Component
{
constructor(props,context,updater)
{
super(props,context,updater);
this.eventEmitter = new EventEmitter();
this.eventEmitter.extend(this);
this.state = {
history: []
};
this._cache = new Infinite.CellMeasurerCache({
fixedWidth: true,
minHeight: 50
});
this._timeoutIdMap = {};
_.bindAll(this,'_isRowLoaded','_loadMoreRows','_rowRenderer');
}
render()
{
let rowCount = this.state.history.length ? (this.state.history.length + 1) : 1;
return <Infinite.InfiniteLoader
isRowLoaded={this._isRowLoaded}
loadMoreRows={this._loadMoreRows}
rowCount={rowCount}
>
{({ onRowsRendered, registerChild }) =>
<Infinite.AutoSizer disableHeight>
{({ width }) =>
<Infinite.List
ref={registerChild}
deferredMeasurementCache={this._cache}
height={200}
onRowsRendered={onRowsRendered}
rowCount={rowCount}
rowHeight={this._cache.rowHeight}
rowRenderer={this._rowRenderer}
width={width}
/>}
</Infinite.AutoSizer>}
</Infinite.InfiniteLoader>
}
_isRowLoaded({ index }) {
if (index == 0 && !this.state.history.length)
// No data yet, force load
return false;
}
_loadMoreRows({ startIndex, stopIndex }) {
let self = this;
for (let i = startIndex; i <= stopIndex; i++) {
this.state.history[startIndex] = {loading: true};
}
const timeoutId = setTimeout(() => {
delete this._timeoutIdMap[timeoutId];
for (let i = startIndex; i <= stopIndex; i++) {
self.state.history[i] = {loading: false, text: 'Hi ' + i };
}
promiseResolver();
}, 10000);
this._timeoutIdMap[timeoutId] = true;
let promiseResolver;
return new Promise(resolve => {
promiseResolver = resolve;
});
}
_rowRenderer({ index, key, style }) {
let content;
if (index >= this.state.history.length)
return <div>Placeholder</div>
else if (this.state.history[index].loading) {
content = <div>Loading</div>;
} else {
content = (
<div>Loaded</div>
);
}
return (
<Infinite.CellMeasurer
cache={this._cache}
columnIndex={0}
key={key}
rowIndex={index}
>
<div key={key} style={style}>{content}</div>
</Infinite.CellMeasurer>
);
}
}

The recipe you linked to should be a good starting place. The main thing its missing is an implementation of loadNextPage but that varies from app to app based on how your state/data management code works.
Can I trigger first load of data from within InfiniteScroller/List - it seems you need to pass in a data source that is populated with initial page
This is up to you. IMO it generally makes sense to just fetch the first "page" of records without waiting for InfiniteLoader to ask for them- because you know you'll need them. That being said, if you give InfiniteLoader a rowCount of 1 and then return false from isRowLoaded it should request the first page of records. There are tests confirming this behavior in the react-virtualized GitHub.
The docs for InfiniteLoader.rowCount say: "Number of rows in list; can be arbitrary high number if actual number is unknown."
So how do you indicate there are no more rows.
You stop adding +1 to the rowCount, like the markdown file you linked to mentions:
// If there are more items to be loaded then add an extra row to hold a
loading indicator.
const rowCount = hasNextPage
? list.size + 1
: list.size

Related

PDFKit split text into two equal columns while using for loop

Im trying to use PDFKit to generate a simple pdf, for the most part the pdf works but albeit in a very non useful way, what i have is a deck building API that takes in a number of cards, each of these objects i want to export to a pdf, its as simple as displaying their name, but as it is, the pdf only renders one card at a time, and only on one line, what id like to happen is to get it to split the text into columns so itd look similar to this.
column 1 | column 2
c1 c8
c2 c9
c3 c10
c4 c(n)
here is my code so far,
module.exports = asyncHandler(async (req, res, next) => {
try {
// find the deck
const deck = await Deck.findById(req.params.deckId);
// need to sort cards by name
await deck.cards.sort((a, b) => {
if (a.name < b.name) {
return -1;
} else if (a.name > b.name) {
return 1;
} else {
return 0;
}
});
// Create a new PDF document
const doc = new PDFDocument();
// Pipe its output somewhere, like to a file or HTTP response
doc.pipe(
fs.createWriteStream(
`${__dirname}/../../public/pdf/${deck.deck_name}.pdf`
)
);
// Embed a font, set the font size, and render some text
doc.fontSize(25).text(`${deck.deck_name} Deck List`, {
align: "center",
underline: true,
underlineColor: "#000000",
underlineThickness: 2,
});
// We need to create two columns for the cards
// The first column will be the card name
// The second column will continue the cards listed
const section = doc.struct("P");
doc.addStructure(section);
for (const card of deck.cards) {
doc.text(`${card.name}`, {
color: "#000000",
fontSize: 10,
columns: 2,
columnGap: 10,
continued: true,
});
}
section.end();
// finalize the PDF and end the response
doc.end();
res.status(200).json({ message: "PDF generated successfully" });
} catch (error) {
console.error(error);
res.status(500).json({
success: false,
message: `Server Error - ${error.message}`,
});
}
});
At Present this does generate a column order like i want, however theres and extreme caveat to this solution and that is, if the card text isnt very long, the next card will start on that same line, it'd be useful if i could find a way to make the text take up the full width of that row, but i havent seen anything to do that with.
I think the problem is that you're relying on PDFKit's text "flow" API/logic, and you're having problems when two cards are not big enough to flow across your columns and you get two cards in one column.
I'd say that what you really want is to create a table—based on your initial text sample.
PDFKit doesn't have a table API (yet), so you'll have to make one up for yourself.
Here's an approach where you figure out the dimensions of things:
the page size
the size of your cells of text (either manually choose for yourself, or use PDFKit to tell you how big some piece of text is)
margins
Then you use those sizes to calculate how many rows and columns of your text can fit on your page.
Finally you iterate of over columns then rows for each page, writing text into those column-by-row "coordinates" (which I track through "offsets" and use to calculate the final "position").
const PDFDocument = require('pdfkit');
const fs = require('fs');
// Create mock-up Cards for OP
const cards = [];
for (let i = 0; i < 100; i++) {
cards.push(`Card ${i + 1}`);
}
// Set a sensible starting point for each page
const originX = 50;
const originY = 50;
const doc = new PDFDocument({ size: 'LETTER' });
// Define row height and column widths, based on font size; either manually,
// or use commented-out heightOf and widthOf methods to dynamically pick sizes
doc.fontSize(24);
const rowH = 50; // doc.heightOfString(cards[cards.length - 1]);
const colW = 150; // doc.widthOfString(cards[cards.length - 1]); // because the last card is the "longest" piece of text
// Margins aren't really discussed in the documentation; I can ignore the top and left margin by
// placing the text at (0,0), but I cannot write below the bottom margin
const pageH = doc.page.height;
const rowsPerPage = parseInt((pageH - originY - doc.page.margins.bottom) / rowH);
const colsPerPage = 2;
var cardIdx = 0;
while (cardIdx < cards.length) {
var colOffset = 0;
while (colOffset < colsPerPage) {
const posX = originX + (colOffset * colW);
var rowOffset = 0;
while (rowOffset < rowsPerPage) {
const posY = originY + (rowOffset * rowH);
doc.text(cards[cardIdx], posX, posY);
cardIdx += 1;
rowOffset += 1;
}
colOffset += 1;
}
// This is hacky, but PDFKit adds a page by default so the loop doesn't 100% control when a page is added;
// this prevents an empty trailing page from being added
if (cardIdx < cards.length) {
doc.addPage();
}
}
// Finalize PDF file
doc.pipe(fs.createWriteStream('output.pdf'));
doc.end();
When I run that I get a PDF with 4 pages that looks like this:
Changing colW = 250 and colsPerPage = 3:

Using React Virtualized CellMeasurer with changing data order for List and dynamic heights

I am trying to create a chat application using React Virtualized. So far everything is working but I think think I am doing something wrong when using keyMapper and rowRenderer from the list.
The cache is using the id to store the height in the _rowHeightCache but it seems that the heights are looked up using the index and not the id. Im not sure if I should be passing the id as the rowIndex to CellMeasurer to get the right heights or something of that nature.
The issue is that the heights are wrong due to the changing order of the messages list so the index's don't have the proper heights and also that messages can be removed messing up the index orders. This I think should be fixed by using keyMapper to look up the heights but i must be doing it wrong.
The following is an edited down version of how I am using it.
constructor(props) {
this._cache = new CellMeasurerCache({
defaultHeight: 45,
fixedWidth: true,
keyMapper: (index) => _.get(this.props.messageList[index], ['msgId'])
});
}
render() {
const props = this.filterProps(this.props),
list = this.props.messageList,
rowCount = list.length;
return (
<InfiniteLoader
threshold={0}
isRowLoaded={this._isRowLoaded}
loadMoreRows={this._loadMoreRows}
rowCount={rowCount}>
{({onRowsRendered, registerChild}) => (
<AutoSizer>
{({width, height}) => (
<List
ref={(ref) => {
this.streamLane = ref;
registerChild(ref);
}}
height={height}
width={width}
onRowsRendered={onRowsRendered}
rowCount={rowCount}
rowRenderer={this._rowRenderer}
deferredMeasurementCache={this._cache}
rowHeight={this._cache.rowHeight}
/>
)}
</AutoSizer>
)}
</InfiniteLoader>
);
}
_rowRenderer({ index, key, parent, style }) {
return (
<CellMeasurer
cache={this._cache}
columnIndex={0}
key={key}
parent={parent}
rowIndex={index}>
<div style={{...style}}>
{this.props.messageList[index]}
</div>
</CellMeasurer>
);
}
_loadMoreRows({startIndex, stopIndex}) {
return new Promise((resolve) => {
if (!!stopIndex || !!startIndex || this.props.hasLastMessage || this.state.isPinned) {
return resolve();
}
this.props.loadOlder(() => resolve);
});
}
_isRowLoaded({index}) {
if (index === 0) return false;
return !!this.props.messageList[index];
}
}
Any help, suggestions or criticisms would be amazing.
The issue is that the heights are wrong due to the changing order of the messages list so the index's don't have the proper heights and also that messages can be removed messing up the index orders.
Your example code doesn't show how the order changes or what you do after it changes, so it's hard to say what you're doing wrong.
But when the order changes (eg you sort the data, or you add new items to the beginning of the list causing older items to shift) then you'll need to let List know that its cached positions may be stale. Do do this in your case, you'd need to call this.streamLane.recomputeRowHeights().
(You can pass the index of the first changed item to recomputeRowHeights but I assume in your case that maybe all items have shifted? I don't know, since I don't have all of your code.)
After you call recomputeRowHeights, List will re-calculate the layout using the heights reported to it by CellMeasurerCache. That cache uses your keyMapper function:
rowHeight = ({index}: IndexParam) => {
const key = this._keyMapper(index, 0);
return this._rowHeightCache.hasOwnProperty(key)
? this._rowHeightCache[key]
: this._defaultHeight;
};
So it won't need to remeasure items, even though their indices have changed.

Am I paginating through a huge Firebase collection the right way?

I am trying to find a way to iterate all objects from a large collection of data in Firebase Database.
My best attempt follows but I found it odd for several reasons:
startAt() values are always inclusive. So after fetching 100 elements, I had to use my last fetched key as an argument to startAt which results in the last item being fetched again
DataSnapshot's forEach method doesn't allow a callback with an index count as you would think it would based on JS's standards so I had to create a manual index - not sure it will work in every case as i'm not sure if forEach works perfectly synchronously
Here is my code, given the assumption my collection is located at users.
const mapAllTripsPaginated = function (database, childSnapshotCallback, start = '', limit = 100, totalNb = 0) {
return database.ref('/users').orderByKey().startAt(start).limitToFirst(limit).once('value').then((snapshot) => {
let childrenPromises = []
let lastChildKey = null
let idx = 0
snapshot.forEach((childSnapshot) => {
lastChildKey = childSnapshot.key
if (start !== '' && idx === 0) {
// console.log(`Skipping ${childSnapshot.key} as 1st element of page`)
} else {
childrenPromises.push(childSnapshotCallback(childSnapshot))
}
idx = idx + 1
})
return Promise.all(childrenPromises)
.then((result) => {
let newTotal = totalNb + result.length
if (snapshot.numChildren() === limit) {
console.log(`Paginating from ${lastChildKey}`)
return mapAllTripsPaginated(database, childSnapshotCallback, start = lastChildKey, limit = limit, totalNb = newTotal)
} else {
// Done paginating
return newTotal
}
})
})
}
Any idea on how I could make this method more elegant?
Firebase queries are inclusive both for their start and end conditions. You will indeed have to deduplicate the overlapping item on the client.
Firebase's Snapshot.forEach() is a synchronous operation.
I'd normally deduplicate based on already having the key of the item. That will also remove the need for the idx counter.
snapshot.forEach((childSnapshot) => {
if (lastChildKey !== childSnapshot.key) {
childrenPromises.push(childSnapshotCallback(childSnapshot))
}
lastChildKey = childSnapshot.key
})

Making realtime datatable updates

I built an app which consumes data from a redis channel(sellers) with socketio and push the data in realtime to the frontend. The dataset could contain up to a thousand rows so I'm thinking about using a datatable to represent the data in a clean way. The table elements will be updated regularly, but there will be no rows to add/remove, only updates.
The problem I'm facing is that I don't know which would be the proper way to implement it due to my inexperience in the visualization ecosystem. I've been toying with d3js but I think It'll be too difficult to have something ready quickly and also tried using the datatables js library but I failed to see how to make the datatable realtime.
This is the code excerpt from the front end:
socket.on('sellers', function(msg){
var seller = $.parseJSON(msg);
var sales = [];
var visits = [];
var conversion = [];
var items = seller['items'];
var data = [];
for(item in items) {
var item_data = items[item];
//data.push(item_data)
data.push([item_data['title'], item_data['today_visits'], item_data['sold_today'], item_data['conversion-rate']]);
}
//oTable.dataTable(data);
$(".chart").html("");
drawBar(data);
});
Using d3 to solve your problem is simple and elegant. I took a little time this morning to create a fiddle that you can adapt to your own needs:
http://jsfiddle.net/CelloG/47nxxhfu/
To use d3, you need to understand how it works with joining the data to the html elements. Check out http://bost.ocks.org/mike/join/ for a brief description by the author.
The code in the fiddle is:
var table = d3.select('#data')
// set up the table header
table.append('thead')
.append('tr')
.selectAll('th')
.data(['Title', 'Visits', 'Sold', 'Conversion Rate'])
.enter()
.append('th')
.text(function (d) { return d })
table.append('tbody')
// set up the data
// note that both the creation of the table AND the update is
// handled by the same code. The code must be run on each time
// the data is changed.
function setupData(data) {
// first, select the table and join the data to its rows
// just in case we have unsorted data, use the item's title
// as a key for mapping data on update
var rows = d3.select('tbody')
.selectAll('tr')
.data(data, function(d) { return d.title })
// if you do end up having variable-length data,
// uncomment this line to remove the old ones.
// rows.exit().remove()
// For new data, we create rows of <tr> containing
// a <td> for each item.
// d3.map().values() converts an object into an array of
// its values
var entertd = rows.enter()
.append('tr')
.selectAll('td')
.data(function(d) { return d3.map(d).values() })
.enter()
.append('td')
entertd.append('div')
entertd.append('span')
// now that all the placeholder tr/td have been created
// and mapped to their data, we populate the <td> with the data.
// First, we split off the individual data for each td.
// d3.map().entries() returns each key: value as an object
// { key: "key", value: value}
// to get a different color for each column, we set a
// class using the attr() function.
// then, we add a div with a fixed height and width
// proportional to the relative size of the value compared
// to all values in the input set.
// This is accomplished with a linear scale (d3.scale.linear)
// that maps the extremes of values to the width of the td,
// which is 100px
// finally, we display the value. For the title entry, the div
// is 0px wide
var td = rows.selectAll('td')
.data(function(d) { return d3.map(d).entries() })
.attr('class', function (d) { return d.key })
// the simple addition of the transition() makes the
// bars update smoothly when the data changes
td.select('div')
.transition()
.duration(800)
.style('width', function(d) {
switch (d.key) {
case 'conversion_rate' :
// percentage scale is static
scale = d3.scale.linear()
.domain([0, 1])
.range([0, 100])
break;
case 'today_visits':
case 'sold_today' :
scale = d3.scale.linear()
.domain(d3.extent(data, function(d1) { return d1[d.key] }))
.range([0, 100])
break;
default:
return '0px'
}
return scale(d.value) + 'px'
})
td.select('span')
.text(function(d) {
if (d.key == 'conversion_rate') {
return Math.round(100*d.value) + '%'
}
return d.value
})
}
setupData(randomizeData())
d3.select('#update')
.on('click', function() {
setupData(randomizeData())
})
// dummy randomized data: use this function for the socketio data
// instead
//
// socket.on('sellers', function(msg){
// setupData(JSON.parse(msg).items)
// })
function randomizeData() {
var ret = []
for (var i = 0; i < 1000; i++) {
ret.push({
title: "Item " + i,
today_visits: Math.round(Math.random() * 300),
sold_today: Math.round(Math.random() * 200),
conversion_rate: Math.random()
})
}
return ret
}

Results pagination in Cassandra (CQL)

I am wondering how can I achieve pagination using Cassandra.
Let us say that I have a blog. The blog lists max 10 posts per page. To access next posts a user must click on pagination menu to access page 2 (posts 11-20), page 3 (posts 21-30), etc.
Using SQL under MySQL, I could do the following:
SELECT * FROM posts LIMIT 20,10;
The first parameter of LIMIT is offset from the beginning of result set and second argument is amount of rows to fetch. The example above returns 10 rows starting from row 20.
How can I achieve the same effect in CQL?
I have found some solutions on Google, but all of them require to have "the last result from previous query". It works for having "next" button to paginate to another 10-results-set, but what if I want to jump from page 1 to page 5?
You don't need to use tokens, if you are using Cassandra 2.0+.
Cassandra 2.0 has auto paging.
Instead of using token function to create paging, it is now a built-in feature.
Now developers can iterate over the entire result set, without having to care that it’s size is larger than the memory. As the client code iterates over the results, some extra rows can be fetched, while old ones are dropped.
Looking at this in Java, note that SELECT statement returns all rows, and the number of rows retrieved is set to 100.
I’ve shown a simple statement here, but the same code can be written with a prepared statement, couple with a bound statement. It is possible to disable automatic paging, if it is not desired. It is also important to test various fetch size settings, since you will want to keep the memorize small enough, but not so small that too many round-trips to the database are taken. Check out this blog post to see how paging works server side.
Statement stmt = new SimpleStatement(
"SELECT * FROM raw_weather_data"
+ " WHERE wsid= '725474:99999'"
+ " AND year = 2005 AND month = 6");
stmt.setFetchSize(24);
ResultSet rs = session.execute(stmt);
Iterator<Row> iter = rs.iterator();
while (!rs.isFullyFetched()) {
rs.fetchMoreResults();
Row row = iter.next();
System.out.println(row);
}
Try using the token function in CQL:
https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useToken.html
Another suggestion, if you are using DSE, solr supports deep paging:
https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
Manual Paging
The driver exposes a PagingState object that represents where we were in the result set when the last page was fetched:
ResultSet resultSet = session.execute("your query");
// iterate the result set...
PagingState pagingState = resultSet.getExecutionInfo().getPagingState();
This object can be serialized to a String or a byte array:
String string = pagingState.toString();
byte[] bytes = pagingState.toBytes();
This serialized form can be saved in some form of persistent storage to be reused later. When that value is retrieved later, we can deserialize it and reinject it in a statement:
PagingState pagingState = PagingState.fromString(string);
Statement st = new SimpleStatement("your query");
st.setPagingState(pagingState);
ResultSet rs = session.execute(st);
Note that the paging state can only be reused with the exact same statement (same query string, same parameters). Also, it is an opaque value that is only meant to be collected, stored an re-used. If you try to modify its contents or reuse it with a different statement, the driver will raise an error.
Src: https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshPaging.html
If you read this doc "Use paging state token to get next result",
https://datastax.github.io/php-driver/features/result_paging/
We can use "paging state token" to paginate at application level.
So PHP logic should look like,
<?php
$limit = 10;
$offset = 20;
$cluster = Cassandra::cluster()->withContactPoints('127.0.0.1')->build();
$session = $cluster->connect("simplex");
$statement = new Cassandra\SimpleStatement("SELECT * FROM paging_entries Limit ".($limit+$offset));
$result = $session->execute($statement, new Cassandra\ExecutionOptions(array('page_size' => $offset)));
// Now $result has all rows till "$offset" which we can skip and jump to next page to fetch "$limit" rows.
while ($result->pagingStateToken()) {
$result = $session->execute($statement, new Cassandra\ExecutionOptions($options = array('page_size' => $limit,'paging_state_token' => $result->pagingStateToken())));
foreach ($result as $row) {
printf("key: '%s' value: %d\n", $row['key'], $row['value']);
}
}
?>
Although the count is available in CQL, so far I have not seen a good solution for the offset part...
So... one solution I have been contemplating was to create sets of pages using a background process.
In some table, I would create the blog page A as a set of references to page 1, 2, ... 10. Then another entry for blog page B pointing to pages 11 to 20, etc.
In other words, I would build my own index with a row key set to the page number. You may still make it somewhat flexible since you can offer the user to choose to see 10, 20 or 30 references per page. For example, when set to 30, you display sets 1, 2, and 3 as page A, sets 4, 5, 6 as page B, etc.)
And if you have a backend process to handle all of that, you can update your lists as new pages are added and old pages are deleted from the blog. The process should be really fast (like 1 min. for 1,000,000 rows if even that slow...) and then you can find the pages to display in your list pretty much instantaneously. (Obviously, if you are to have thousands of users each posting hundreds of pages... that number can grow quickly.)
Where it becomes more complicated is if you wanted to offer a complex WHERE clause. By default a blog shows you a list of all the posts from the newest to the oldest. You could also offer lists of posts with tag Cassandra. Maybe you want to inverse the order, etc. That makes it difficult unless you have some form of advanced way to create your index(es). On my end I have a C-like language which goes and peek and poke to the values in a row to (a) select them and if selected (b) to sort them. In other words, on my end I can already have WHERE clauses as complex as what you'd have in SQL. However, I do not yet break up my lists in pages. Next step I suppose...
Using cassandra-node driver for node js (koa js,marko js) : Pagination
Problem
Due to the absence of skip functionality, we need to work around. Below is the implementation of manual paging for node app in case of anyone can get an idea.
code for simple users list
navigate between next and previous page states
easy to replicate
There are two solutions which i am going to state here but only gave the code for solution 1 below,
Solution 1 : Maintain page states for next and previous records (maintain stack or whatever data structure best fit)
Solution 2 : Loop through all records with limit and save all possible page states in variable and generate pages relatively to their pageStates
Using this commented code in model, we can get all states for pages
//for the next flow
//if (result.nextPage) {
// Retrieve the following pages:
// the same row handler from above will be used
// result.nextPage();
//}
Router Functions
var userModel = require('/models/users');
public.get('/users', users);
public.post('/users', filterUsers);
var users = function* () {//get request
var data = {};
var pageState = { "next": "", "previous": "" };
try {
var userCount = yield userModel.Count();//count all users with basic count query
var currentPage = 1;
var pager = yield generatePaging(currentPage, userCount, pagingMaxLimit);
var userList = yield userModel.List(pager);
data.pageNumber = currentPage;
data.TotalPages = pager.TotalPages;
console.log('--------------what now--------------');
data.pageState_next = userList.pageStates.next;
data.pageState_previous = userList.pageStates.previous;
console.log("next ", data.pageState_next);
console.log("previous ", data.pageState_previous);
data.previousStates = null;
data.isPrevious = false;
if ((userCount / pagingMaxLimit) > 1) {
data.isNext = true;
}
data.userList = userList;
data.totalRecords = userCount;
console.log('--------------------userList--------------------', data.userList);
//pass to html template
}
catch (e) {
console.log("err ", e);
log.info("userList error : ", e);
}
this.body = this.stream('./views/userList.marko', data);
this.type = 'text/html';
};
//post filter and get list
var filterUsers = function* () {
console.log("<------------------Form Post Started----------------->");
var data = {};
var totalCount;
data.isPrevious = true;
data.isNext = true;
var form = this.request.body;
console.log("----------------formdata--------------------", form);
var currentPage = parseInt(form.hdpagenumber);//page number hidden in html
console.log("-------before current page------", currentPage);
var pageState = null;
try {
var statesArray = [];
if (form.hdallpageStates && form.hdallpageStates !== '') {
statesArray = form.hdallpageStates.split(',');
}
console.log(statesArray);
//develop stack to track paging states
if (form.hdpagestateRequest === 'next') {
console.log('--------------------------next---------------------');
currentPage = currentPage + 1;
statesArray.push(form.hdpageState_next);
pageState = form.hdpageState_next;
}
else if (form.hdpagestateRequest === 'previous') {
console.log('--------------------------pre---------------------');
currentPage = currentPage - 1;
var p_st = statesArray.length - 2;//second last index
console.log('this index of array to be removed ', p_st);
pageState = statesArray[p_st];
statesArray.splice(p_st, 1);
//pageState = statesArray.pop();
}
else if (form.hdispaging === 'false') {
currentPage = 1;
pageState = null;
statesArray = [];
}
data.previousStates = statesArray;
console.log("paging true");
totalCount = yield userModel.Count();
var pager = yield generatePaging(form.hdpagenumber, totalCount, pagingMaxLimit);
data.pageNumber = currentPage;
data.TotalPages = pager.TotalPages;
//filter function - not yet constructed
var searchUsers = yield userModel.searchList(pager, pageState);
data.usersList = searchUsers;
if (searchUsers.pageStates) {
data.pageStates = searchUsers.pageStates;
data.next = searchUsers.nextPage;
data.pageState_next = searchUsers.pageStates.next;
data.pageState_previous = searchUsers.pageStates.previous;
//show previous and next buttons accordingly
if (currentPage == 1 && pager.TotalPages > 1) {
data.isPrevious = false;
data.isNext = true;
}
else if (currentPage == 1 && pager.TotalPages <= 1) {
data.isPrevious = false;
data.isNext = false;
}
else if (currentPage >= pager.TotalPages) {
data.isPrevious = true;
data.isNext = false;
}
else {
data.isPrevious = true;
data.isNext = true;
}
}
else {
data.isPrevious = false;
data.isNext = false;
}
console.log("response ", searchUsers);
data.totalRecords = totalCount;
//pass to html template
}
catch (e) {
console.log("err ", e);
log.info("user list error : ", e);
}
console.log("<------------------Form Post Ended----------------->");
this.body = this.stream('./views/userList.marko', data);
this.type = 'text/html';
};
//Paging function
var generatePaging = function* (currentpage, count, pageSizeTemp) {
var paging = new Object();
var pagesize = pageSizeTemp;
var totalPages = 0;
var pageNo = currentpage == null ? null : currentpage;
var skip = pageNo == null ? 0 : parseInt(pageNo - 1) * pagesize;
var pageNumber = pageNo != null ? pageNo : 1;
totalPages = pagesize == null ? 0 : Math.ceil(count / pagesize);
paging.skip = skip;
paging.limit = pagesize;
paging.pageNumber = pageNumber;
paging.TotalPages = totalPages;
return paging;
};
Model Functions
var clientdb = require('../utils/cassandradb')();
var Users = function (options) {
//this.init();
_.assign(this, options);
};
Users.List = function* (limit) {//first time
var myresult; var res = [];
res.pageStates = { "next": "", "previous": "" };
const options = { prepare: true, fetchSize: limit };
console.log('----------did i appeared first?-----------');
yield new Promise(function (resolve, reject) {
clientdb.eachRow('SELECT * FROM users_lookup_history', [], options, function (n, row) {
console.log('----paging----rows');
res.push(row);
}, function (err, result) {
if (err) {
console.log("error ", err);
}
else {
res.pageStates.next = result.pageState;
res.nextPage = result.nextPage;//next page function
}
resolve(result);
});
}).catch(function (e) {
console.log("error ", e);
}); //promise ends
console.log('page state ', res.pageStates);
return res;
};
Users.searchList = function* (pager, pageState) {//paging filtering
console.log("|------------Query Started-------------|");
console.log("pageState if any ", pageState);
var res = [], myresult;
res.pageStates = { "next": "" };
var query = "SELECT * FROM users_lookup_history ";
var params = [];
console.log('current pageState ', pageState);
const options = { pageState: pageState, prepare: true, fetchSize: pager.limit };
console.log('----------------did i appeared first?------------------');
yield new Promise(function (resolve, reject) {
clientdb.eachRow(query, [], options, function (n, row) {
console.log('----Users paging----rows');
res.push(row);
}, function (err, result) {
if (err) {
console.log("error ", err);
}
else {
res.pageStates.next = result.pageState;
res.nextPage = result.nextPage;
}
//for the next flow
//if (result.nextPage) {
// Retrieve the following pages:
// the same row handler from above will be used
// result.nextPage();
//}
resolve(result);
});
}).catch(function (e) {
console.log("error ", e);
info.log('something');
}); //promise ends
console.log('page state ', pageState);
console.log("|------------Query Ended-------------|");
return res;
};
Html side
<div class="box-footer clearfix">
<ul class="pagination pagination-sm no-margin pull-left">
<if test="data.isPrevious == true">
<li><a class='submitform_previous' href="">Previous</a></li>
</if>
<if test="data.isNext == true">
<li><a class="submitform_next" href="">Next</a></li>
</if>
</ul>
<ul class="pagination pagination-sm no-margin pull-right">
<li>Total Records : $data.totalRecords</li>
<li> | Total Pages : $data.TotalPages</li>
<li> | Current Page : $data.pageNumber</li>
</ul>
</div>
I am not very much experienced with node js and cassandra db, this solution can surely be improved. Solution 1 is working example code to start with the paging idea. Cheers
a detailed blog.
Our use case was similar. Pull everything from a Cassandra table (cassandra does it smartly by fetching ~5000 in one go and return a cursor), heavy personalized processing on each row, and keep going. Once our iteration reaches close to 5000, it again fetches the next chunk of 5000 rows internally and adds it to the result cursor. It does it so brilliantly that we don’t even feel this magic happening behind the scene.
but It became a bottleneck for us.As iterating over the chunk took some time and till it reached the end of the chunk, Cassandra thought the connection was not being used and closed the connection automatically yelling, its timeout. So we implemented with page state.
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
from cassandra.query import SimpleStatement
# connection with cassandra
cluster = Cluster(["127.0.0.1"], auth_provider=PlainTextAuthProvider(username="pankaj", password="pankaj"))
session = cluster.connect()
# setting keyspace
session.set_keyspace("my_keyspace")
# set fetch size
fetch_size = 100
# It will print first 100 records
next_page_available = True
paging_state     = None
data_count     = 0
while next_page_available is True:
# fetches a new chunk with given page state
result = fetch_a_fresh_chunk(paging_state)
paging_state = results.paging_state
for result in results:
# process payload here.....
# payload processed
data_count += 1
# once we reach fetch size, we stop cassandra to fetch more chunk, internally
if data_count == fetch_size:
i = 0
break
# fetches a fresh chunk with given page state
def fetch_a_fresh_chunk(paging_state = None)
query = "SELECT * FROM my_cute_cassandra_table;"
statement = SimpleStatement(query, fetch_size = fetch_size)
results = session.execute(statement, paging_state=paging_state)

Resources