I am scraping for a sharable Google Map link.
On the first load, the map only shows 20 items max. After which, scrolling the scrollbar loads an additional 20.
The first 20 are first added to the results object/array. Then, to get the additional items not loaded:
I use the
await page.on('response', async (response) => {
// i will loop through each entry, and then append it to my results variable.
}
to check for the ajax request and extract the JSON result.
However, I realize that at the end of my code, the data from the ajax request do not get added to the results variable.
I believe this is because it is returned earlier, and the async call for page.on only runs after I've returned it.
How can I merge the results from my page.on into a variable, and then finally return the final results variable containing data from the async function as well?
Related
I'm using the query() to reference a lit element. For example
#query('#first')
_first;
However, that lit element hasn't been rendered to the dom yet and the #first returns null. When I do this._first = null. What can I do to make sure that first is not null? How can I wait for the lit element to be rendered to the dom?
As you mentioned, #query will return null before the initial update has completed because the element hasn't rendered yet.
One possible option is to use the #queryAsync decorator. Instead of returning your element or null synchronously, it returns a promise that resolves to the queried node after any pending render is completed.
Another option is to await this.updateComplete which also lets you wait for any pending updates to complete before querying the rendered DOM. E.g. from the linked updateComplete docs: "When writing tests you can await the updateComplete Promise before making assertions about the component’s DOM."
await this.updateComplete; // Wait for pending update to render #first
this._first // Result of querying for #first on rendered DOM in this.renderRoot.
I am new to SAP UI5 development. Currently the table is using "growing" and "growingThreshhold", then users can click more to see data of next page. Since we have thousands of data in that table, it takes user time to click more and more again to load next page data. we try to implement a function, that user can enter the page number then click a button and go to the specific page.
<Table id="genTable" growing="true" growingThreshold="60" fixedLayout="false" selectionChange="onHandleSelectChange"
backgroundDesign="Solid" updateFinished="onHandleGeneratorQueueUpdateFinished">
Expected UI:
I added a bar then UI display is good.
<Bar design="SubHeader">
<contentMiddle>
<Input type="Number" id="pageNumber" width="50px"></Input>
<Button id="goToButton" text="Go to" type="Emphasized" press="onHandleGoTo"></Button>
</contentMiddle>
</Bar>
For the backend logic, I refer to below articles, but still doesn't work.
https://blogs.sap.com/2016/12/14/sapui5-pagination-in-sap.m-table-on-button-click-using-odata-service/
https://sapyard.com/advance-sapui5-19-pagination-in-table-control-with-top-and-skip-query-options/
I tried to use read, the it can get the data back from odata service, but the data can't be refreshed in the table.
oModel.read("/ViewQueueSet", {
urlParameters: {
"$top": top,
"$skip": count
},
filters: [new Filter("RoleCode", FilterOperator.EQ, "G")],
useBatch: true,
success: function (tdata) { //successful Read in the server
var json = new JSONModel();
json.setData(tdata);
that.getView().setModel(json,"sapmodel");
sap.ui.core.BusyIndicator.hide();
},
error: function () {
sap.ui.core.BusyIndicator.hide();
}
});
}
also tried to call bindItems
//that.getView().setModel(json,"sapmodel");
//oTable.setModel(json); //JSON is preferred data format
//oTable.bindItems("/results",that.oGenQueueTemplate);
that.getView().byId("genTable").setModel(json);
that.getView().byId("genTable").bindItems("/results",that.oGenQueueTemplate);
Another approach I tried is to use bindItems, it call send the request to odata service, but it doesn't add the parameter top and skip parameter.
oTable.bindItems({
path: "/ViewQueueSet",
model: "sapmodel",
filters: [new Filter("RoleCode", FilterOperator.EQ, "G")],
template: this.oGenQueueTemplate,
// urlParameters: {
// "$top": top,
// "$skip": count
// },
parameters: {
"$top": top,
"$skip": count
}
});
Anyone has any idea about how to implement this functionality?
before I go into detail, please consider using other controls and/or ux patterns. imagine having thousands or millions of elements in backend and user equests to scroll to page 9292929 => for a responsive table (sap.m.Table) you would need to load all elements up to that page. Maybe filtering or some completely different approach could be tha right one.
The correct way to do this is by getting the listbinding and ask it to load more elements. how to ask the binding, may depend on the type of binding as well.
oTable = ... // get a reference on table
oItemsBinding = oTable.getBinding("items");
oItemsBinding.getLength() // will give you total number of elements
oItemsBinding.isLengthFinal() // will tell you if the length is final
oItemsBinding.getCurrentContexts() // will give you array of all loaded contexts.
now a few words to length and the length being final. If you have a binding implementation that knows the total number of objects (e.g. json - since it loads all elements to client, or OData, if cont is implemented in backend) then getLength will tell you the total number of objects.
if the backend doesnt have the count feature implemented, the length becomes final once you reach the end of the list (backend gives you less elements than you require - e.g. top=10,skip=90 returns 10 elements => length 100, not final; top=10,skip=100 returns 4 elements => length=104 becomes final)
Now, you can have a look at various binding implementations. But be aware that there is a lot to consider (direction of growing - upwards/downwards), at least you dont need to think about filtering/sorting - as this is part of the binding.
There is a nice (private) feature in sap.m.Table (or in sap.m.ListBase, to be more precise), which is called GrowingEnablement. you can use it like this:
// dont forget if _oGrowingDelagate is not undefined or similar
oTable._oGrowingDelegate.requestNewPage()
this will load one more page => you could start from reading the implementation of this method if you want to load several pages in one go.
you could also do a simple trick:
// assume you have 20 elements per page (default)
// and want to get to 7th page (elements 121 - 140)
// ckecks for 7th page exists and 7th page not yet loaded are omitted
oTable.setGrowingThreshold(70) // half of 140, so following load will load second page => 71 to 140
oTable._oGrowingDelegate.requestNewPage() // this will load the second page 71 - 140
// once loading is finished (take care of asynchronity)
oItemsBinding.attachEventOnce("dataReceived", function(oEvent){
// reset the growing threshold to 20
oTable.setGrowingThreshold(20)
// scroll to first element of 7th page (index 120, since count starts from 0)
oTable.scrollToInex(120)
})
I use firebase admin and realtime database on node.js
Data look like
When I want to get data where batch = batch-7, I was doing
let batch = "batch-7";
let ref = admin.database().ref('qr/');
ref.orderByChild("batch").equalTo(batch).on('value', (snapshot) =>
{
res.json(Object.assign({}, snapshot.val()));
ref.off();
});
All was OK!
But now i should create pagination, i.e. I should receive data on 10 elements depending on the page.
I use this code:
let page = req.query.page;// num page
let batch = req.params.batch;// batch name
let ref = admin.database().ref('qr/');
ref.orderByChild("batch").startAt(+page*10).limitToFirst(10).equalTo(batch)
.on('value', (snapshot) =>
{
res.json(Object.assign({}, snapshot.val()));
ref.off();
});
But I have error:
Query.equalTo: Starting point was already set (by another call to startAt or equalTo)
How do I get data in the amount of N, starting at position M, where batch equal my batch
You can only call one startAt (and/or endAt) OR equalTo. Calling both is not possible, nor does it make a lot of sense.
You seem to have a general misunderstanding of how startAt works though, as you're passing in an offset. Firebase queries are not offset based, but work purely on the value, often also referred to as an anchor node.
So when you want to get the data for a second page, and you order by batch, you need to pass in the value of batch for the anchor node; first item that you want to be returned. This anchor node is typically the last item of the previous page, since you don't know the first item of the next page yet. And for this anchor node, you need to know the value of the item you order on (batch) and usually also its key (if/when there may be multiple nodes with the same value for batch).
It also means that you usually request one item more than you need, which is the anchor node.
So when you request the first page, you should track the key/batch of the last node:
var lastKey, lastValue;
ref.orderByChild("batch").equalTo(batch).limitToFirst(10).on('value', (snapshot) => {
snapshot.forEach((child) => {
lastKey = child.key;
lastValue = child.child('batch').value();
})
})
Then when you need the second page, you do a query like that:
ref.orderByChild("batch").start(lastValue, lastKey).endAt(lastValue+"\uf8ff").limitToFirst(11).on('value', (snapshot) => {
snapshot.forEach((child) => {
lastKey = child.key;
lastValue = child.child('batch').value();
})
})
There's one more trick above here: I use startAt instead of equalTo, so that we can get pagination working. But it then uses endAt to ensure we still end at the correct item, by using the last known Unicode character as the last batch value to return.
I'd also highly recommend checking out some of the previous questions on pagination with the Firebase Realtime Database.
How do I limit .once('value') in firebase-admin?
Code:
async function GetStuff(limit, page){
const data = await ref.limitToFirst(parseInt(limit)).once('value')
return data.val();
}
I wanted to create a page system, where a it sends request for a limited amount of data, and the user can change the page to get different data, but for some reason, I can't get it to work.
The code above only gets the first 20(when limit is 20), but how can I make it start at 20, so I can make this page feature.
I thought:
Code:
async function GetStuff(limit, page){
const data = await ref.startAt(limit*page).limitToFirst(parseInt(limit)).once('value')
return data.val();
}
You might want to review the relevant documentation. It looks like you're trying to pass the offset of a child to startAt, but that's not how startAt works. It accepts the actual value of the child to start at. Pagination by offset index is not supported.
The way you use startAt is typically passing the last sorted value retrieved by the prior query (or, if you don't want to retrieve that value again, 1 + that value, or a string that is lexically greater than the last string received. As such, some data sets might actually be difficult to paginate if they have the same sorted value repeated many times.
I have a Google Spreadsheet with internal AppsScript code which process each row of the sheet and perform an urlfetch with the row data. The url will provide a value which will be added to the values returned by each row processing..
For now the code is processing 1 row at a time with a simple for:
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadsheet.getActiveSheet();
var range = sheet.getDataRange();
for(var i=1 ; i<range.getValues().length ; i++) {
var payload = {
// retrieve data from the row and make payload object
};
var options = {
"method":"POST",
"payload" : payload
};
var result = UrlFetchApp.fetch("http://.......", options);
var text = result.getContentText();
// Save result for final processing
// (with multi-thread function this value will be the return of the function)
}
Please note that this is only a simple example, in the real case the working function will be more complex (like 5-6 http calls, where the output of some of them are used as input to the next one, ...).
For the example let's say that there is a generic "function" which executes some sort of processing and provides a result as output.
In order to speed up the process, I'd like to try to implement some sort of "multi-thread" processing, so I can process multiple rows in the same time.
I already know that javascript does not offer a multi-thread handling, but I read about WebWorker which seems to create an async processing of a function.
My goal is to obtain some sort of ThreadPool (like 5 threads at a time) and send every row that need to be processed to the pool, obtaining as output the result of each function.
When all the rows finished the processing, a final action will be performed gathering all the results of each function.
So the capabilities I'm looking for are:
managed "ThreadPool" where I can submit an N amount of tasks to be performed
possibility to obtain a resulting value from each task processed by the pool
possibility to determine that all the tasks has been processed, so a final "event" can be executed
I already see that there are some ready-to-use libraries like:
https://www.hamsters.io/wiki#thread-pool
http://threadsjs.readthedocs.io/en/latest/
https://github.com/andywer/threadpool-js
but they work with NodeJS. Due to AppsScript nature, I need a more simplier approach, which is provided by native JS. Also, it seems that minified JS are not accepted by AppsScript editor, so I also need the "expanded" version.
Do you know a simple ThreadPool in JS where I can submit a function to be execute and I get back a Promise for the result?