RxJs Observable Pagination - pagination

First: This is the first project in which I am using RxJs, I thought I will learn best by using it.
I found this answer: Turning paginated requests into an Observable stream with RxJs
But it says in the comments:
You're exceeding the maximum call stack still. At around 430 pages returned. I think recursion might not be the best solution here
I want to query the Youtube Data API, the results come back in pages and I need to paginate through them.
I imagined a work flow like this could work:
1)Initiate a call
2)Check if the response has a 'nextPageToken'
3)If it has, do another request to the Youtube API
4)If not, finish
So to do this I could Imagine the following Observables / streams:
FirstRequestStream -A-X--------------->
ResponseStream -A-A-A-A--X-------->
RequestStream -I-A-I-A----------->
A = Action
I = Info from upper stream
X = Termination
(Not sure if this diagram is correct the way I made it)
So the ResponseStream depends on FirstRequestStream and RequestStream(using the merge function). The RequestStream depends on the ResponseStream( is this called a circulating observable ?)
-Is this the right approach ?
-Are 'circulating observables' a good thing, are they even possible ?(I had problems creating one).
-Any other way I should try first?
-Is it possible to create interdependent observable streams ?
Thank you for your help.

You are overcomplicating this problem, it can be solved a lot easier using defer operator.
Idea is that you are creating deferred observable (so it will be created and start fetching data only after subscription) and concatenate it with the same observable but for the next page, which will be also concatenated with the next page, and so on ... . And all of that can be done without recursion.
Here is how the code looks:
const { defer, from, concat, EMPTY, timer } = rxjs; // = require("rxjs")
const { mergeMap, take, mapTo, tap } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
function fetchPage(page=0) {
return timer(100).pipe(
tap(() => console.log(`-> fetched page ${page}`)),
mapTo({
items: Array.from({ length: 10 }).map((_, i) => page * 10 + i),
nextPage: page + 1,
})
);
}
const getItems = page => defer(() => fetchPage(page)).pipe(
mergeMap(({ items, nextPage }) => {
const items$ = from(items);
const next$ = nextPage ? getItems(nextPage) : EMPTY;
return concat(items$, next$);
})
);
// process only first 30 items, without fetching all of the data
getItems()
.pipe(take(30))
.subscribe(e => console.log(e));
<script src="https://unpkg.com/rxjs#6.2.2/bundles/rxjs.umd.min.js"></script>

Here is my solution using the rxjs operators expand, reduce and empty using the HttpClient module:
Suppose your API response is an object containing shaped like the following
interface Response {
data: items[]; // array of result items
next: string|null; // url of next page, or null if there are no more items
}
You could use expand and reduce like so
getAllResults(url) {
return this.http.get(url).pipe(
expand((res) => res.next ? this.http.get(res.next) : EMPTY),
reduce((acc, res) => acc.concat(res.data), [])
);
}

I shamelessly reuse the code snippet from Oles Savluk, with its good fetchPage function, and I apply the ideas explained in the blog article linked to by Picci (in the comments), using expand.
Article on expand by Nicholas Jamieson
It gives a slightly simpler code, with recursion hidden in the expand call (and comments of the article show how to linearize it, if needed).
const { timer, EMPTY } = rxjs; // = require("rxjs")
const { concatMap, expand, mapTo, tap, toArray } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
const pageNumber = 3;
function fetchPage(page = 0) {
return timer(1000).pipe(
tap(() => console.log(`-> fetched page ${page}`)),
mapTo({
items: Array.from({ length: 10 }).map((_, i) => page * 10 + i),
nextPage: ++page === pageNumber ? undefined : page,
}),
);
}
const list = fetchPage().pipe(
expand(({ nextPage }) => nextPage ? fetchPage(nextPage) : EMPTY),
concatMap(({ items }) => items),
// Transforms the stream of numbers (Observable<number>)
// to a stream with only an array of numbers (Observable<number[]>).
// Remove if you want a stream of numbers, not waiting for all requests to complete.
toArray(),
);
list.subscribe(console.log);
<script src="https://unpkg.com/rxjs#6.2.2/bundles/rxjs.umd.min.js"></script>

LuJaks is definitively the simplest approach !
For a one line example, suppose you have a function that make a http request for a given page and that returns a (partial) array of data. We call that function until server returns empty array :
import { Observable, EMPTY, of } from "rxjs";
import { expand, reduce } from "rxjs/operators";
// Mock a request that returns only 5 pages...
function httpGet(p): Observable<number[]> {
if (p > 5) { return of([]); }
return of(new Array(10).fill(0).map((_, i) => p * 10 + i));
}
httpGet(0).pipe( // get the fist page
expand((value, index) => (value.length > 0 ? httpGet(index + 1) : EMPTY)), // other pages
reduce((a, v) => [...a, ...v], []), // optional if you want only one emit
).subscribe((x) => console.log(x));

Related

Update prop for dynamically inserted element

New to react... Really banging my head against it with this one... I'm trying to figure out how to get a dynamically inserted component to update when the props are changed. I've assigned it to a parent state object but it doesn't seem to re-render. I've read that this is what's supposed to happen.
I was using ReactDOM.unmountComponentAtNode to re-render the specific elements I needed to, but it kept yelling at me with red text.
I need to hide "chat.message" unless the user has the authority to see it (server just sends empty string), but I still need to render the fact that it exists, and reveal it should the user get authentication. I'm using a css transition to reveal it, but I really need a good way to update the chat.message prop easily.
renderChats(uuid){
let userState = this.state.userStates.find(user => {
return user.uuid === uuid;
});
const children = userState.chats.map((chat) => {
let ChatReactElement = this.getChatMarkup(chat.cuid, chat.message, chat.status);
return ChatReactElement;
});
ReactDOM.render(children, document.getElementById(`chats-${this.state.guid}-${uuid}`));
}
getChatMarkup() just returns JSX and inserts Props... I feel like state should be getting passed along here. Even when I use a for-loop and insert the state explicitly, it doesn't seem to re-render on changes.
getChatMarkup(cuid, message, status){
return(
<BasicChatComponent
key={cuid}
cuid={cuid}
message={message}
status={status}
/>
);
}
I attempted to insert some code line this:
renderChats(uuid){
let userState = this.state.userStates.find(user => {
return user.uuid === uuid;
});
const children = userState.chats.map((chat) => {
let ChatReactElement = this.getChatMarkup(chat.cuid, chat.message, chat.status);
if(chat.status.hidden)
this.setState({ hiddenChatRE: [ ...this.state.hiddenChatRE, ChatReactElement ] }); // <== save elements
return ChatReactElement;
});
ReactDOM.render(children, document.getElementById(`chats-${this.state.guid}-${uuid}`));
}
and later in my code:
this.state.hiddenChatRE.every(ReactElement => {
if(ReactElement.key == basicChats[chatIndex].cuid){
ReactElement.props = {
... //completely invalid code
}
}
});
The only response I see here is my ReactDOM.unmountComponentAtNode(); approach...
Can anyone point me in the right direction here?
Although perhaps I should be kicking myself, I read up on how React deals with keys on their components. So there's actually a fairly trivial answer here if anyone comes looking... Just call your render function again after you update the state.
In my case, something like:
this.setState(state =>({
...state,
userStates : state.userStates.map((userstate) => {
if(userstate.uuid == basicChats[chatIndex].uuid) return {
...userstate,
chats: userstate.chats.map((chat) => {
if(chat.cuid == basicChats[chatIndex].cuid){
//
return {
cuid: basicChats[chatIndex].cuid,
message: basicChats[chatIndex].message,
status: basicChats[chatIndex].status
}
}
else return chat;
})
}
else return userstate;
})
}));
and then, elsewhere in my example:
this.state.userStates.map((userstate) => {
this.renderChats(userstate.uuid);
});
Other than the fact that I'd recommend using indexed arrays for this example to cut complexity, this is the solution, and works. This is because even though it feels like you'd end up with duplicates (that was my intuition), the uid on the BasicChatComponent itself makes all the difference, letting react know to only re-render those specific elements.

how to push data in array in typescript

I want to create a dynamic menu bar by fetching data from two collections (supcat and cat) then combining the two to create a new array which i will access on page load for menu but the push() is not working.
ngOnInit() {
this.cattest();}
cattest(){
var x;
this.supcatobj.fetchsupcat().subscribe(
(res)=>
{
if(res.length!=0)
{this.supcat=res;
for(let x=0;x<this.supcat.length; x++)
{
this.catobj.fetchallcat(this.supcat[x]["_id"]).subscribe(
(resp)=>
{
this.allcat=resp;
for(let y=0;y<this.allcat.length;y++)
{
}
this.testarr[x].push(this.supcat[x]["supcatname"],this.allcat);
}
);
}
}
}
);}
Instead of nesting subscribe() calls, I would try to compose separate observables for your two different collections and then use the combineLatest() operator to combine those into your desired array. It is hard to see exactly what you are working for, but conceptually it would be something like this:
const supcat$ = this.supcatobj.fetchsupcat().pipe(filter(cat => cat.length > 0));
const allCat$ = this.catobj.fetchallcat();
const combinedCats$ = combineLatest(supcat$, allCat$);
const res$ = combinedCats$.pipe(map(res => {
// do operation that involves both data sets
});
Remember that map() will return a new array. This way you will only need to subscribe to the one variable, and if you put it at the class level you could use the async pipe (|) in your template so it will unsubscribe automatically.

React-native and Redux healthy way to call actions on props change

I've been using react-native with redux for a while, and the way i learn to call actions when something change on prop is using the componentWillReceiveProps, but when I use it I need to pass between if's and some times it goes to the wrong if, then I need to add more stuff to prevent it.
Here's an example I have done. I know this is not the best way to do it, but it is what I could think of.
componentWillReceiveProps(newProps) {
if(Object.keys(newProps.selected_product).length > 0) {
if(Object.keys(this.props.current_location).length > 0 || Object.keys(newProps.current_location).length > 0) {
this._handleNextPage(2);
this.props.verifyProductById(newProps.selected_product, newProps.current_location, this.props.token);
} else {
this.props.statusScanner(false);
this._handleNextPage(1);
}
} else if(Object.keys(newProps.historic_product_confirm).length > 0) {
if(newProps.historic_product_confirm.location._id == newProps.current_location._id)
this.props.handleModalConfirmPrice(!this.props.modal_confirmPrice_status)
} else if(newProps.scanResult != "") {
this.props.statusScanner(false);
if(Object.keys(newProps.current_location).length > 0) {
this._handleNextPage(2);
} else {
this._handleNextPage(1);
}
} else {
this._handleNextPage(0);
}
}
What I need is a healthy way to call my actions when the props change.
Edit:
Here i have the full OfferScene and an action file example:
OfferScene:
https://gist.github.com/macanhajc/0ac98bbd2974d2f6fac96d9e30fd0642
UtilityActions:
https://gist.github.com/macanhajc/f10960a8254b7659457f8a09c848c8cf
As mentioned in another answer, componentWillReceiveProps is being phased out, so I would aim for trying to eliminate it where possible. You'll be future-proofing your code and keeping your component logic more declarative and easy to reason about. As someone who has been responsible for (and been frustrated by) lifecycle method abuse like this, here are some things that have helped me.
Remember that when using redux-thunk, along with passing dispatch as the first argument, you can also pass getState as the second. This allows you to access state values in your action logic instead of bringing them into your component's props and adding clutter. Something like:
export const ExampleAction = update =>
(dispatch, getState) => {
const { exampleBool } = getState().ExampleReducer
if (exampleBool) {
dispatch({
type: 'UPDATE_EXAMPLE_STATE',
update
})
}
}
Using async/await in action logic can be a lifesaver when your action depends upon fetched results from an API call:
export const ExampleAction = () =>
async (dispatch, getState) => {
const { valueToCheck } = getState().ExampleReducer
, result = await someAPICall(valueToCheck)
.catch(e => console.log(e))
if (result.length > 0) {
dispatch({
type: 'UPDATE_EXAMPLE_STATE',
update: result
})
}
}
For cases where your component's rendering behavior depends upon certain state values after your state has been updated, I highly recommend reselect. A very basic example would be something like:
component.js
import React, { Component, Fragment } from 'react'
import { connect } from 'react-redux'
import { shouldDisplayItems } from '../selectors'
import MyListviewComponent from './myListview'
class ItemList extends Component {
render() {
const { shouldDisplayItems, items } = this.props
return (
<>
{shouldDisplayItems && <MyListviewComponent items={items} />}
</>
)
}
}
const mapStateToProps = ({ ListItems }) => shouldDisplayItems(ListItems)
export default connect(mapStateToProps)(ItemList)
selectors.js:
(Assuming your ListItems reducer has the params items and visibilityFilter)
import { createSelector } from 'reselect'
export const shouldDisplayItems = createSelector(
[state => state],
({ items, visibilityFilter }) => {
return {
shouldDisplayItems: visibilityFilter && items.length > 0,
items
}
}
)
I should mention that another option would be using higher-order components, but it can be tricky to use this approach before having a good grasp on how to keep too much imperative logic out of your components (I learned this the hard way).
I agree with #AnuragChutani and #Goldy in terms of clarity of the code; break it down some more into more components or functions.
Now after some review of your componentWillReceiveProps function, it is definitely not specific enough to narrow down exactly which prop changes. If any connected redux variable changes, the componentWillReceiveProps function will be invoked each time.
So e.g. if 'token' or 'selected_product' updates, componentWillReceiveProps will be triggered, even though you did not want it to trigger for token updates.
You can use a comparison for a specific variable update in the props.
E.g Using lodash
if(!_.isEqual( nextProps.selected_product, this.props.selected_product ))
// if props are different/updated, do something
Secondly, you can call actions/callbacks in your actions to narrow down navigation.
E.g.
takePicture = (camera, options){
...
//on success
dispatch(handleModalConfirmPrice())
...
}}

How to bind data to html in angular 2/4 by using thread or async

I have a Component, which have child component. at ngOnInit() I'm calling Web API and get list of data.
Initial point length of the list is 10, But it will have more.
Need to execute some method (task|process|job) in background to take rest of the data 10 by 10 in a loop which would run parallel to other task in background no matter what the user is currently doing, which component he/she is interacting with. And execute that method so that it doesn't block others.
What is the correct way to do this?
Seems like a recursive call to me !
firstResults: any[] = []; // First 10 results to show to your user
results: any[] = []; // All results
currentPosition = 0; // The current position of your last result fetched
getData() {
this.myService.getResults().subscribe(results => {
if(!this.firstResults.length) {
this.firstResults = results
}
this.results.push(...results);
this.currentPosition += results.length;
this.getData();
});
}
I don't imagine you're trying to do. But if the list is not so largger, you can get all the data and "paginate" the array
allData:any[];
page:number=0; //page is 0,1,2,3,4....
paginateData:any[]
this.httpClient.get("url").subscribe(res=>{
allData=res;
paginateData=allData.slice(10*this.page,10*(this.page+1));
}

Running node sqlite3 code synchronously

I'm having trouble adjusting to the async-first nature of node / js / typescript. This intent of this little function should be pretty clear: it takes a database and returns an array of courses that are listed in that database.
The problem is that the return statement gets run before any of the database operations have run, and I get an empty list. When I set a breakpoint inside the database each loop, I can see that the rows are being found and that courses are being put into ret one by one, but these courses never become visible in the scope where courseList() was called.
const courseList = (database: sqlite3.Database): Course[] => {
let ret = new Array<Course>();
database.serialize();
database.each("select ID, Title from Course", (err: Error, row: Object) => {
ret.push(new Course(
row.ID,
row.Title
))
})
return ret;
}
Suggestions?
The calling code just wants to print information about courses. For example:
let courses = courseList(db);
console.log(courses.length); // logs 0, even though the db contains courses
database.each takes a complete callback. Use that to resume e.g.
const courseList = (database: sqlite3.Database, complete): Course[] => {
let ret = new Array<Course>();
database.serialize();
database.each("select ID, Title from Course", (err: Error, row: Object) => {
ret.push(new Course(
row.ID,
row.Title
))
}, complete);
return ret;
}
let courses = courseList(db, () => {
console.log(courses.length);
});
More
There are better ways to write this. Use promises https://basarat.gitbooks.io/typescript/content/docs/promise.html
The documentation is horrible : https://github.com/mapbox/node-sqlite3/wiki I would be tempted to look elsewhere (TS First) for a database solution. Its not worth the pain for me personally. YMMV.

Resources