Laravel version: 7.x
I have a test data file of 1161 rows, which needs to be uploaded via ajax and rendered in the editable format, in case of validations failed. So I am using Excel::toCollection(...). But, even with this small amount of data, the application hangs for a while, until the rendering is complete.
To handle this situation I reading only 50 rows at a time, validate and append in the form via ajax. However, it has reduced the load and also decrease the rendering time, but each time I have to read the entire file and get the next chuck out of it.
Here is my code:
$arrayPost = request()->only([
'start',
'limit',
]);
$arrayExcel = \Excel::toCollection(new SubscriberImport(), <uploaded_file_path>);
$arrayExcel = array_slice($arrayExcel, $arrayPost['start'], $arrayPost['limit']);
$validated = $this->validateExcel($arrayExcel);
if($validated->fails())
{
# return with error
return view('<form_path>', [
'arrayExcel' => $arrayExcel,
])->withErrors($validated->getMessageBag()->toArray());
}
else
{
...
}
Is there a way to fetch the specific rows from the file.?
Related
Some of the cytoscape layout is randomize where the position is not fix every time we launch it. I understand from multiple stack overflow questions where we can save the layout data (i.e. including its position x and y) into browser local storage or session storage so that we can display the same layout using the same data.
However, the problem with local storage or session storage is good for one users. But, imagine if there are thousands of users using the same app, the server will undergo mass computation for each user to store respective data to individual browsers. Can we save the data into a file format directly into app/web server so that 1000 users will see the same layout and this reduces the computation of different data set as well.
Thank you. Would like to know the possibility to convert data into a file and store in the web/app server.
Yes, you can store position data. Actually, there are 2 options in my mind.
Use cy.json(). You can store the elements as JSON like JSON.stringify(cy.json().elements) and then save this JSON string.
cy.json().elements is something like the below image
You can restore this data easily like cy.json({elements: JSON.parse(jsonStr));
As you could see cy.json().elements is a bit big thing. Position data is just a small object like {x: 0, y: 0}. Additional to position it contains many other data. So if you only need to restore the positions, you could store them manually easily with a code like below. You can use ele.id and node.position() functions.
function storePositions() {
const nodes = cy.nodes();
const nodePositions = {};
for (let i = 0; i < nodes.length; i++) {
nodePositions[nodes[i].id()] = nodes[i].position();
}
return nodePositions;
}
You can also restore node positions easily. You can use getElementById and node.position() functions.
function restorePositions(nodePositions) {
const nodes = cy.nodes();
const nodePositions = {};
for (let k in nodePositions) {
const node = cy.getElementById(k);
if (node && node.length > 0) {
node.position(nodePositions[k]);
}
}
return nodePositions;
}
I am new to SAP UI5 development. Currently the table is using "growing" and "growingThreshhold", then users can click more to see data of next page. Since we have thousands of data in that table, it takes user time to click more and more again to load next page data. we try to implement a function, that user can enter the page number then click a button and go to the specific page.
<Table id="genTable" growing="true" growingThreshold="60" fixedLayout="false" selectionChange="onHandleSelectChange"
backgroundDesign="Solid" updateFinished="onHandleGeneratorQueueUpdateFinished">
Expected UI:
I added a bar then UI display is good.
<Bar design="SubHeader">
<contentMiddle>
<Input type="Number" id="pageNumber" width="50px"></Input>
<Button id="goToButton" text="Go to" type="Emphasized" press="onHandleGoTo"></Button>
</contentMiddle>
</Bar>
For the backend logic, I refer to below articles, but still doesn't work.
https://blogs.sap.com/2016/12/14/sapui5-pagination-in-sap.m-table-on-button-click-using-odata-service/
https://sapyard.com/advance-sapui5-19-pagination-in-table-control-with-top-and-skip-query-options/
I tried to use read, the it can get the data back from odata service, but the data can't be refreshed in the table.
oModel.read("/ViewQueueSet", {
urlParameters: {
"$top": top,
"$skip": count
},
filters: [new Filter("RoleCode", FilterOperator.EQ, "G")],
useBatch: true,
success: function (tdata) { //successful Read in the server
var json = new JSONModel();
json.setData(tdata);
that.getView().setModel(json,"sapmodel");
sap.ui.core.BusyIndicator.hide();
},
error: function () {
sap.ui.core.BusyIndicator.hide();
}
});
}
also tried to call bindItems
//that.getView().setModel(json,"sapmodel");
//oTable.setModel(json); //JSON is preferred data format
//oTable.bindItems("/results",that.oGenQueueTemplate);
that.getView().byId("genTable").setModel(json);
that.getView().byId("genTable").bindItems("/results",that.oGenQueueTemplate);
Another approach I tried is to use bindItems, it call send the request to odata service, but it doesn't add the parameter top and skip parameter.
oTable.bindItems({
path: "/ViewQueueSet",
model: "sapmodel",
filters: [new Filter("RoleCode", FilterOperator.EQ, "G")],
template: this.oGenQueueTemplate,
// urlParameters: {
// "$top": top,
// "$skip": count
// },
parameters: {
"$top": top,
"$skip": count
}
});
Anyone has any idea about how to implement this functionality?
before I go into detail, please consider using other controls and/or ux patterns. imagine having thousands or millions of elements in backend and user equests to scroll to page 9292929 => for a responsive table (sap.m.Table) you would need to load all elements up to that page. Maybe filtering or some completely different approach could be tha right one.
The correct way to do this is by getting the listbinding and ask it to load more elements. how to ask the binding, may depend on the type of binding as well.
oTable = ... // get a reference on table
oItemsBinding = oTable.getBinding("items");
oItemsBinding.getLength() // will give you total number of elements
oItemsBinding.isLengthFinal() // will tell you if the length is final
oItemsBinding.getCurrentContexts() // will give you array of all loaded contexts.
now a few words to length and the length being final. If you have a binding implementation that knows the total number of objects (e.g. json - since it loads all elements to client, or OData, if cont is implemented in backend) then getLength will tell you the total number of objects.
if the backend doesnt have the count feature implemented, the length becomes final once you reach the end of the list (backend gives you less elements than you require - e.g. top=10,skip=90 returns 10 elements => length 100, not final; top=10,skip=100 returns 4 elements => length=104 becomes final)
Now, you can have a look at various binding implementations. But be aware that there is a lot to consider (direction of growing - upwards/downwards), at least you dont need to think about filtering/sorting - as this is part of the binding.
There is a nice (private) feature in sap.m.Table (or in sap.m.ListBase, to be more precise), which is called GrowingEnablement. you can use it like this:
// dont forget if _oGrowingDelagate is not undefined or similar
oTable._oGrowingDelegate.requestNewPage()
this will load one more page => you could start from reading the implementation of this method if you want to load several pages in one go.
you could also do a simple trick:
// assume you have 20 elements per page (default)
// and want to get to 7th page (elements 121 - 140)
// ckecks for 7th page exists and 7th page not yet loaded are omitted
oTable.setGrowingThreshold(70) // half of 140, so following load will load second page => 71 to 140
oTable._oGrowingDelegate.requestNewPage() // this will load the second page 71 - 140
// once loading is finished (take care of asynchronity)
oItemsBinding.attachEventOnce("dataReceived", function(oEvent){
// reset the growing threshold to 20
oTable.setGrowingThreshold(20)
// scroll to first element of 7th page (index 120, since count starts from 0)
oTable.scrollToInex(120)
})
I hope you can help me because I'm driving mad about this question.
I'm not experienced with Node, Ajax and DataTables, so this is likely a my fault.
I have an html table linked to Datatables. On load document I send a POST request to get data from an MySQL database. If I popolate DataTable with row.add() method all works fine, but if the table grows I could have some slowdowns with this system, so I would like to link the DataTable table with the database so that the program loads only the records to show. Now problem starts.
In my Node file, if I return sql data like so:
res.json(result);
my DataTable shows all the data, but table hasn't right number of record (below the table shows "Showing 0 to 0 of 0 entries (filtered from NaN total entries)" and, even worst, the table has infinite pages, all displaying the same data (the first 10 records).
If instead I return an ajax:
ret = JSON.stringify({
"draw": req.body.draw,
"recordsTotal": result[0]['COUNT(cf)'],
"recordsFiltered": result[0]['COUNT(cf)'],
"data": dati
});
res.send(ret);
DataTables shows the correct number of records in the info line at bottom of table, but no data is displayed.
In console I show the ret object and it seems all right. I'm no idea what to do now.
I resolved!!! It's incredible after near 2 days of attempts.
In html file I put in DataTables ajax :
[...]
dataSrc: 'dati',
[...]
Then in my Nodejs file:
ret = JSON.stringify({
"draw": req.body.draw,
"recordsTotal": result[0]['COUNT(cf)'],
"recordsFiltered": result[0]['COUNT(cf)'],
"dati": dati
});
res.send(ret);
and all works fine.
I have a Google Spreadsheet with internal AppsScript code which process each row of the sheet and perform an urlfetch with the row data. The url will provide a value which will be added to the values returned by each row processing..
For now the code is processing 1 row at a time with a simple for:
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadsheet.getActiveSheet();
var range = sheet.getDataRange();
for(var i=1 ; i<range.getValues().length ; i++) {
var payload = {
// retrieve data from the row and make payload object
};
var options = {
"method":"POST",
"payload" : payload
};
var result = UrlFetchApp.fetch("http://.......", options);
var text = result.getContentText();
// Save result for final processing
// (with multi-thread function this value will be the return of the function)
}
Please note that this is only a simple example, in the real case the working function will be more complex (like 5-6 http calls, where the output of some of them are used as input to the next one, ...).
For the example let's say that there is a generic "function" which executes some sort of processing and provides a result as output.
In order to speed up the process, I'd like to try to implement some sort of "multi-thread" processing, so I can process multiple rows in the same time.
I already know that javascript does not offer a multi-thread handling, but I read about WebWorker which seems to create an async processing of a function.
My goal is to obtain some sort of ThreadPool (like 5 threads at a time) and send every row that need to be processed to the pool, obtaining as output the result of each function.
When all the rows finished the processing, a final action will be performed gathering all the results of each function.
So the capabilities I'm looking for are:
managed "ThreadPool" where I can submit an N amount of tasks to be performed
possibility to obtain a resulting value from each task processed by the pool
possibility to determine that all the tasks has been processed, so a final "event" can be executed
I already see that there are some ready-to-use libraries like:
https://www.hamsters.io/wiki#thread-pool
http://threadsjs.readthedocs.io/en/latest/
https://github.com/andywer/threadpool-js
but they work with NodeJS. Due to AppsScript nature, I need a more simplier approach, which is provided by native JS. Also, it seems that minified JS are not accepted by AppsScript editor, so I also need the "expanded" version.
Do you know a simple ThreadPool in JS where I can submit a function to be execute and I get back a Promise for the result?
I am currently working with MEAN stack. I have around 99000 records in mongo dB. each record consist of an image array, which is containing image urls. maximum size of this image array can be 10. so every record can maximum have imageURL array length = 10 .
Now I want to fetch every records, and then compare images of every records with each other, using resemble js. then save their average value in that same record.
I used async module and tried to implement this, but it is taking too much time even with 5 records. also used async's forEachLimit but it won't help.
So basically How can I manipulate these kind of large amount of data with Node and mongo?
is there any way to do it in batches ? any other solution ?
loop1 ==> all records (response) {
loop2 == > convert all images of one record to base64 (resemble can't use images from urls)==> saved in new array = TempArray1 <==loop ends
loop3 == > TempArray1.length (TempArray1[i]) {
loop4 ==> TempArray1.length (TempArray1[j]){
count += resemble(TempArray1[i],TempArray1[j]);
}
avg[i] = count/(TempArray1.length -1);
}
}