I describe what I'm trying to do in VBA because it's the language I know best :
'Declare a recordset which will iterate through our records
Dim rs as DAO.recordset
'Setup a connection to our data store with the table/query we want to process records in
Set rs = dbengine.workspaces.opendatabase('path/to/database').openrecordset('select * from sometable')
rs.movefirst
Do Until rs.eof
'put record in edit mode
rs.Edit
rs!field = 'some value I want'
'put record in update mode
rs.update
rs.movenext
loop
It's looping and setting values. What I need is more complex (updating based on what is in the next row after and all of that), but I can figure that out myself once I understand how to iterate through records and perform updates at record level.
What I have now:
//this function is just is called by another function. processRecord is what does the individual record processing
async function processRecords(db: Database, SQL: string, processRecord: any) {
db.each(SQL, async(e,r) => {
if(e) {
reject(e);
} else {
try {
await processRecord(db,r);
resolve(true);
}
catch(e) {
reject(e);
}
}
});
}
async function processRecord(db: Database, r: Row) {
try {
//execute a query against the current row we are processing
//SQL for updates uses a select in SET clause to pull data from the record with the next rowid
//executeQuery is a "promisified" db.run
executeQuery(db,updateSQL,r.rowID);
} catch(e : any) {
throw e;
}
}
//I borrowed this from somewhere else and modified it a bit.
function executeQuery(db: Database, SQL: string, params: any = []) : Promise<number> {
return new Promise(resolve,reject) => {
db.Run(SQL,params,function(e) {
if(e) {
reject(e);
} else {
resolve(this.changes);
}
}
}
}
How to iterate through records and perform updates at record level?
Related
I'm trying to create a React app that:
sends data (json message) from backend to frontend using socket.io
if a json message with same is sent, update the existing list
This is how i'm implementing it right now but i'm not sure if this is a good design methodology or if there's a better way to achieve what I want to do.
function App(){
const [database, setDatabase] = useState([])
useEffect(() => {
socket.on('incoming_data', (data) => {
setDatabase((currentList) => {
if (currentList.length > 0){ //only check for update/delete if more than 1 item present
let exists = !!currentList.find((item) => item.ID === data.ID)
if (exists){ //if new item exists in database list
if (data.deleteFlag === true){ // incoming data will have a json field declaring whether to delete or not
//deleting item
var item = currentList.find(itm => itm.ID === data.ID)
let ind = currentList.indexOf(item)
return (currentList.splice(ind,1))
}
else{ // else if delete flag is not true... update fields
var item = currentList.find(itm => itm.ID === data.ID)
let ind = currentList.indexOf(item)
if (item.dataField !== data.dataField){
currentList[ind].dataField = data.dataField
}
return (currentList)
}
}
//if incoming data doesnt exist in list, add to it
else{ return([...currentList, data]) }
}
}
// if there are 0 items in list, add to list
else { return ([...currentList, data]) }
})
}, [socket])
return(/*using map to display list in front end*/)
}
Right now, this code works in the following ways:
Checks if there are 0 items in 'database', if so, it adds items to it.
What it's not doing:
updating items in database
deleting items properly. Sometimes it deletes items, other times it does nothing.
Any help would be great!
Use higher-order functions to simplify code like filter, findIndex, etc.
use findIndex method to check items exist and use the same index to update currentList.
use the filter function to delete items from the list.
function App() {
const [database, setDatabase] = useState([])
useEffect(() => {
socket.on('incoming_data', (data) => {
setDatabase((currentList) => {
if (currentList.length > 0) { //only check for update/delete if more than 1 item present
// Use same index to find item
let itemIndex = currentList.findIndex((item) => item.ID === data.ID)
if (itemIndex !== -1) { //if new item exists in database list
if (data.deleteFlag === true) { // incoming data will have a json field declaring whether to delete or not
// use filter for delete
return currentList.filter((item) => item.ID !== data.ID);
}
else {
let item = currentList[itemIndex]
const newItem = { ...item, dataField: data.dataField }
if (item.dataField !== newItem.dataField) {
currentList[itemIndex] = newItem;
return [...currentList]; // Set new value for updates
}
return (currentList)
}
}
//if incoming data doesn't exist in list, add to it
else { return ([...currentList, data]) }
}
// if there are 0 items in list, add to list
else { return ([...currentList, data]) }
});
});
}, [socket])
return (/*using map to display list in front end*/)
}
I'm trying to write, do...while statement. Inside the body generate random id and In a while condition check if exist in DB. If ID not exist, exit from while.
I'm little confused with callbacks and if someone can help me will be very grateful!
It doesn't matter if you have to generate numbers or a string...
funtion for do while loop, I have used async/await in place of callbacks.
async function checkUniqueId() {
do {
id = 'sadfasdrfawerwer';//some generated random id;
} while (await checkInDb(id));
}
function to check id in the db
async function checkInDb(id) {
try {
let idFound = await User.findOne({ id: id });
if (idFound) return true;
else return false;
}
catch (err) {
console.log(err);
return false;
}
}
I have this code in node js / firebase :
ref.child("recipts").once("value", function(usersSnap) {
usersSnap.forEach(function(reciptsSnap) {
reciptsSnap.forEach(function(reciptSnap) {
reciptSnap.ref.child("last_recipt").once("value", function(b) {
b.forEach(function(c) { //Here I fill some "product" object
});
});
reciptSnap.forEach(function(b) { //Here I fill some "product" object
});
});
});
});
I need to execute a function just when "reciptSnap" forEachs finished. How can I accomplish this, I try using a variable i++ and i-- but only work for one forEach iteration.
The function I call is for manipulating the product object I created with the filled data from the forEachs loops.
If I have understood correctly, you want to call a function when reciptsSnap.forEach is complete and all async tasks inside it are also complete.
For achieving this, you can use the index parameter and the original array that is passed to the callback function of forEach. (See Documentation)
The code will be like this:
(Note: The following code is without changing the current forEach loop structure used. However, re-writing the code with Promise or async would be a better & cleaner way to do it).
var loop1Done = false;
var loop2Done = false;
ref.child("recipts").once("value", function (usersSnap) {
usersSnap.forEach(function (reciptsSnap) {
reciptsSnap.forEach(function (reciptSnap, index, colA) {
const idx = index;
const col = colA;
reciptSnap.ref.child("last_recipt").once("value", function (b) {
const i = idx;
const c = col;
b.forEach(function (c, j, colB) { //Here I fill some "product" object
// Do what you want here
// Check if all done for this loop
if ((j >= colB.length) && (i >= c.length)) {
loop1Done = true;
// Check if all loops done
if (loop1Done && loop2Done) {
// Call final callback function
// e.g. myFinalCallback();
}
}
});
});
reciptSnap.forEach(function (b, k, colC) { //Here I fill some "product" object
const i = idx;
const c = col;
// Do what you want here
// Check if all done for this loop
if ((k >= colC.length) && (i >= c.length)) {
loop2Done = true;
// Check if all loops done
if (loop1Done && loop2Done) {
// Call final callback function
// e.g. myFinalCallback();
}
}
});
});
});
});
Try:
reciptSnap.child("last_recipt").forEach(function(b) {
b.forEach(function(c) {
//Here I fill some "product" object
});
});
This should work since all of your data should already have been fetched when you did "value" on the receipts node.
If this works, your code is no longer asynchronous and right after the last forEach, you can execute the function you wanted to.
reciptSnap.forEach(function(b) {
//Here I fill some "product" object
});
//Execute your function here
});
I have the following readable stream in typescript:
import {Readable} from "stream";
enum InputState {
NOT_READABLE,
READABLE,
ENDED
}
export class Aggregator extends Readable {
private inputs: Array<NodeJS.ReadableStream>;
private states: Array<InputState>;
private records: Array<any>;
constructor(options, inputs: Array<NodeJS.ReadableStream>) {
// force object mode
options.objectMode = true;
super(options);
this.inputs = inputs;
// set initial state
this.states = this.inputs.map(() => InputState.NOT_READABLE);
this.records = this.inputs.map(() => null);
// register event handlers for input streams
this.inputs.forEach((input, i) => {
input.on("readable", () => {
console.log("input", i, "readable event fired");
this.states[i] = InputState.READABLE;
if (this._readable) { this.emit("_readable"); }
});
input.on("end", () => {
console.log("input", i, "end event fired");
this.states[i] = InputState.ENDED;
// if (this._end) { this.push(null); return; }
if (this._readable) { this.emit("_readable"); }
});
});
}
get _readable () {
return this.states.every(
state => state === InputState.READABLE ||
state === InputState.ENDED);
}
get _end () {
return this.states.every(state => state === InputState.ENDED);
}
_aggregate () {
console.log("calling _aggregate");
let timestamp = Infinity,
indexes = [];
console.log("initial record state", JSON.stringify(this.records));
this.records.forEach((record, i) => {
// try to read missing records
if (!this.records[i] && this.states[i] !== InputState.ENDED) {
this.records[i] = this.inputs[i].read();
if (!this.records[i]) {
this.states[i] = InputState.NOT_READABLE;
return;
}
}
// update timestamp if a better one is found
if (this.records[i] && timestamp > this.records[i].t) {
timestamp = this.records[i].t;
// clean the indexes array
indexes.length = 0;
}
// include the record index if has the required timestamp
if (this.records[i] && this.records[i].t === timestamp) {
indexes.push(i);
}
});
console.log("final record state", JSON.stringify(this.records), indexes, timestamp);
// end prematurely if after trying to read inputs the aggregator is
// not ready
if (!this._readable) {
console.log("end prematurely trying to read inputs", this.states);
this.push(null);
return;
}
// end prematurely if all inputs are ended and there is no remaining
// record values
if (this._end && indexes.length === 0) {
console.log("end on empty indexes", this.states);
this.push(null);
return;
}
// create the aggregated record
let record = {
t: timestamp,
v: this.records.map(
(r, i) => indexes.indexOf(i) !== -1 ? r.v : null
)
};
console.log("aggregated record", JSON.stringify(record));
if (this.push(record)) {
console.log("record pushed downstream");
// remove records already aggregated and pushed
indexes.forEach(i => { this.records[i] = null; });
this.records.forEach((record, i) => {
// try to read missing records
if (!this.records[i] && this.states[i] !== InputState.ENDED) {
this.records[i] = this.inputs[i].read();
if (!this.records[i]) {
this.states[i] = InputState.NOT_READABLE;
}
}
});
} else {
console.log("record failed to push downstream");
}
}
_read () {
console.log("calling _read", this._readable);
if (this._readable) { this._aggregate(); }
else {
this.once("_readable", this._aggregate.bind(this));
}
}
}
It is designed to aggregate multiple input streams in object mode. In the end it aggregate multiple time series data streams into a single one. The problem i'm facing is that when i test the feature i'm seeing repeatedly the message record failed to push downstream and immediately the message calling _read true and in between just the 3 messages related to the aggregation algorithm. So the Readable stream machinery is calling _read and every time it's failing the push() call. Any idea why is this happening? Did you know of a library that implement this kind of algorithm or a better way to implement this feature?
I will answer myself the question.
The problem was that i was misunderstanding the meaning of the this.push() return value call. I think a false return value mean that the current push operation fail but the real meaning is that the next push operation will fail.
A simple fix to the code shown above is to replace this:
if (this.push(record)) {
console.log("record pushed downstream");
// remove records already aggregated and pushed
indexes.forEach(i => { this.records[i] = null; });
this.records.forEach((record, i) => {
// try to read missing records
if (!this.records[i] && this.states[i] !== InputState.ENDED) {
this.records[i] = this.inputs[i].read();
if (!this.records[i]) {
this.states[i] = InputState.NOT_READABLE;
}
}
});
} else {
console.log("record failed to push downstream");
}
By this:
this.push(record);
console.log("record pushed downstream");
// remove records already aggregated and pushed
indexes.forEach(i => { this.records[i] = null; });
this.records.forEach((record, i) => {
// try to read missing records
if (!this.records[i] && this.states[i] !== InputState.ENDED) {
this.records[i] = this.inputs[i].read();
if (!this.records[i]) {
this.states[i] = InputState.NOT_READABLE;
}
}
});
You can notice that the only difference is avoid conditioning operations on the return value of the this.push() call. Given that the current implementation call this.push() only once per _read() call this simple change solve the issue.
It means feeding is faster than consuming. The official approach is enlarge its highWaterMark, Default: 16384 (16KB), or 16 for objectMode. As long as its inner buffer is big enough, the push function will always return true. It does not have to be single push() in single _read(). You may push as much as the highWaterMark indicates in a single _read().
I have a problem that I'm importing some data and each new row depends on previous row being added (since each row has its order attribute set based on current maximum order from other objects). The flow is that I first try to find object with the same name, if not found I first check maximum order and create new object with order + 1 from that query.
I tried doing this with Q promises which are available under Waterline. I tried using all method as well as combining queries with then from Q docs:
var result = Q(initialVal);
funcs.forEach(function (f) {
result = result.then(f);
});
return result;
But all objects had the same order, just like they would be executed in parallel, instead of waiting for the first chain to finish.
I finally found a solution with recurrency, but I doubt it's the best way of working with promises. Here's the code that works (+ needs some refactor and cleaning etc.), to show the rough idea:
function findOrCreateGroup(groupsQuery, index, callback) {
var groupName = groupsQuery[index];
Group.findOne({ 'name' : groupName }).then(function(group) {
if (!group) {
return Group.find().limit(1).sort('order DESC').then(function(foundGroups) {
var maxOrder = 0;
if (foundGroups.length > 0) {
maxOrder = foundGroups[0].order;
}
return Group.create({
'name' : groupName,
'order' : (maxOrder + 1)
}).then(function(g) {
dbGroups[g.name] = g;
if (index + 1 < groupsQuery.length) {
findOrCreateGroup(groupsQuery, index + 1, callback);
} else {
callback();
}
return g;
});
});
} else {
dbGroups[group.name] = group;
if (index + 1 < groupsQuery.length) {
findOrCreateGroup(groupsQuery, index + 1, callback);
} else {
callback();
}
return group;
}
});
}