Is it possible to, with a Knex.js migration, copy data from one table to another ?
The use case is as follows:
I have a table A, which I want to split into two new tables B and C. Ideally, I would loop over the rows in A to create the appropriate rows in B and C and fill them with the right information.
Can this be done inside a migration file? Aport from this question, I feel this way of doing migrations in Node.JS is quite complex (e.g. compared to ActiveRecord). Is there any better, more managed way to do such migrations? Or is this the industry standard ?
There's nothing special about the query builder object passed in to your up and down functions inside the migration file. You can use it like you would use any other instance of a query builder in your app, that is run any queries you want as part of the migration.
Here's an extremely simple example. Given you have a table called accounts with 4 fields, 1 of which you want to split off into a table by itself:
// Starting promise chain with Promise.resolve() for readability only
exports.up = function(knex, Promise) {
return Promise.resolve()
.then(() => knex.schema.createTable('table_b', t => {
t.string('col_a')
t.string('col_b')
}))
.then(() => knex.schema.createTable('table_c', t => {
t.string('col_c')
t.string('col_d')
}))
.then(() => knex('table_a').select('col_a', 'col_b'))
.then((rows) => knex('table_b').insert(rows))
.then(() => knex('table_a').select('col_c', 'col_d'))
.then((rows) => knex('table_c').insert(rows))
.then(() => knex.schema.dropTableIfExists('table_a'))
};
exports.down = function(knex, Promise) {
return Promise.resolve()
.then(() => knex.schema.createTable('table_a', t => {
t.string('col_a')
t.string('col_b')
t.string('col_c')
t.string('col_d')
}))
.then(() => knex('table_b').select('col_a', 'col_b'))
.then((rows) => knex('table_a').insert(rows))
.then(() => knex('table_c').select('col_c', 'col_d'))
.then((rows) => knex('table_a').insert(rows))
.then(() => knex.schema.dropTableIfExists('table_b'))
.then(() => knex.schema.dropTableIfExists('table_c'))
};
In this case, you could also just keep table_a and instead of creating third table, just drop two columns and rename the table. Be mindful, however, that splitting your table like this will get messy if it has relationships to other tables in the DB already.
My understanding is that migrations deal only with performing CRUD operations in tables.
knex allows you to call a function after the migrations are finished:
`knex.migrate.latest()
.then(function() {
return knex.seed.run();
})
.then(function() {
// migrations are finished
});`
So you can add your code in either a seed file or simply as a function as shown.
Note that this function is called only after migrations complete, which means your table A still has to be present (can't be deleted).
Here's the relevant documentation
Related
I've a pipeline like this:
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.6.jar"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_connection_string => "${JDBC_CONNECTION_STRING}"
jdbc_user => "${JDBC_USER}"
jdbc_password => "${JDBC_PASSWORD}"
statement =>
# Select everything from the my_table and
# add a fake record to indicate the end of the results.
"SELECT * FROM my_table
UNION ALL
SELECT 'END_OF_QUERY_RESULTS' AS some_key;
"
}
}
filter {
ruby {
init => "
require 'net/http'
require 'json'
"
code => '
if event.get("some_key") == "END_OF_QUERY_RESULTS"
uri = URI.parse(ENV["MY_URL"])
response = Net::HTTP.get_response(uri)
result = JSON.parse(response.body)
if response.code == "202"
puts "Success!"
else
puts "ERROR: Couldn\'t start processing."
end
event.cancel()
end
'
}
}
output {
mongodb {
bulk => true
bulk_interval => 2
collection => "${MONGO_DB_COLLECTION}"
database => "${MONGO_DB_NAME}"
generateId => true
uri => "mongodb://${MONGO_DB_HOST}:${MONGO_DB_PORT}/${MONGO_DB_NAME}"
}
}
I simply grab all the data from a PostreSQL table to a MongoDB collection.
What I'm trying to achieve is: I want to call an API after loading ALL the data into MongoDB collection.
What I tried:
I tried the above approach to add a fake record at the end of the SQL query results to use as a flag to indicate the last event. The problem with this approach is LogStash does not maintain the order of events, hence, the event with 'END_OF_QUERY_RESULTS' string can become to the filter before it is actually the last one.
Setting pipeline.workers: 1 and pipeline.ordered: true, both don't seem to work.
I tried to sleep for a while in the Ruby filter and it works but I don't/can't really know how much time I should sleep.
I am trying to write a transaction that first query documents by documentId from a list of ids, then makes some updates.
I am getting the error:
The corresponding value for FieldPath.documentId() must be a string or a DocumentReference.
For example:
const indexArray = [..list of doc ids...]
const personQueryRef = db.collection("person").where(admin.firestore.FieldPath.documentId(), "in", indexArray)
return db.runTransaction(transaction => {
return transaction.get(personQueryRef).then(personQuery => {
return personQuery.forEach(personRef => {
transaction.update(personRef, { ...update values here })
//more updates etc
})
})
})
I am wanting to do this in an onCreate and onUpdate trigger. Is there another approach I should be taking?
Update
The error still persists when not using a transaction, so this is unrelated to the problem.
The problem does not occur when the query is .where(admin.firestore.FieldPath.documentId(), "==", "just_one_doc_id"). So, the problem is with using FieldPath.documentId() and in.
It sounds like the type of query you're trying to do just isn't supported by the SDK. Whether or not that's intentional, I don't know. But if you want to transact with multiple documents, and you already know all of their IDs, you can use getAll(...) instead:
// build an array of DocumentReference objects
cost refs = indexArray.map(id => db.collection("person").doc(id))
return db.runTransaction(transaction => {
// pass the array to getAll()
return transaction.getAll(refs).then(docs => {
docs.forEach(doc => {
transaction.update(doc.ref, { ...update values here })
})
})
})
I need to render in a single page several query returns, for example:
1) students per neighborhood
2) student average
app.get('/gerais',(req,res) => {
const client = new Client();
client.connect()
.then(() => {
return client.query('SELECT COUNT(name) studentsperneighborhood,'
+' neigh FROM student INNER JOIN adress ON student.adress_id ='
+'adress.id GROUP BY neigh');
})
.then((results) => {
console.log('results?',results);
res.render('general-info',results);
})
.catch((err) => {
console.log('error',err);
res.send('FAIL');
});
});
how could I modify the return, to return another query?
Try running the queries separately and pass the results of both the queries in different objects.
Then render your page and inject these objects there to use them.
for eg. if you are using ejs file:-
then
for eg:-
connect.query('query1',function(err1, data1){
connect.query('query2',function(err2, data2){
res.render('studentsinfo.ejs',{
studentsperneighborhood: data1,
studentaverage: data2
})})})
Now, you use use these 2 objects to display your data.
I would like to list currently deleted documents in order to provide the ability to undelete one or more.
How can I query couchdb for deleted documents? I am actually using pouchdb.
Although this POST nicely describes how to query for and undelete a document, it requires an id of an existing doc.
I am looking for a way to query for all documents that have been deleted. The POST cites making a query for all changes. That query returns all documents that have been deleted IN ADDITION to any that have been edited/changed.
I am looking only for documents that have been deleted. Think querying for documents in the 'trash bin'. :)
Starting from couch 2.1.0, you can add various selectors to _changes feed. So your request to output only deleted documents will be:
curl -X POST -H "content-Type: application/json" "http://adm:pass#127.0.0.1:15984/tracks/_changes?filter=_selector" -d '{"selector": {"_deleted": true}}'
You can add a filter to the _changes feed in PouchDB: https://pouchdb.com/api.html#filtered-changes
var changes = db.changes({
filter: function(doc) {
return doc._deleted;
}
}).on('change', function(change) {
console.log(change.id);
})
For an all-in-one solution combining the open_revs tip from this answer, here's the TypeScript code I came up with:
const db = new PouchDB('my-db');
async function deletedDocIds(): Promise<string[]> {
const ret: string[] = [];
return new Promise((resolve, reject) => {
db.changes({filter: d => d._deleted})
.on('change', c => ret.push(c.id))
.on('complete', () => resolve(ret))
.on('error', e => reject(e));
});
}
async function deletedDocs() {
const ids = await deletedDocIds();
return Promise.all(ids.map(id => db.get(id, {revs: true, open_revs: 'all'}).then(x => {
const revs = (x[0].ok as any)._revisions;
const lastRev = (revs.start - 1) + '-' + revs.ids[1];
return db.get(id, {rev: lastRev}); // with Pouchdb keys too
})));
}
Calling deletedDocs() will return a promise of an array of all deleted docs as per the rev just prior to deletion.
N.B., the elements of the array will include PouchDb metadata as well as your document's keys.
N.B. 2, version 6.1.3 of DefinitelyTyped's TypeScript bindings for pouchdb-browser which I'm using here (should work for #types/pouchdb too though) doesn't seem to know about the _revisions key, hence the as any escape hatch.
N.B. 3, this should be trivial to manually translate to plain JS, just delete the type declarations and coercions (:, as, and whatever token follows these).
I have three PostgreSQL tables defined in Sequelize: AccountTypes, Accounts and Users.
Using a base AccountType, I want to find all Accounts (hasMany) that ALSO belongs to a certain user. The appropriate table associations are defined.
In order words, I want to turn this...
return db.instances.AccountType.getLedger('USD').then(ledger => {
return ledger.getAccounts().then(accounts => {
accounts.forEach(account => {
account.getUser() // <-- horribly inefficient
})
});
});
Into some like this...
// assume 'user' contains a retrieved User model
return db.instances.AccountType.getLedger('USD').then(ledger => {
return ledger.getAccounts({where: user}).then(accounts => {
// now 'accounts' will be filtered by Accounts.UserId === user.id
});
});
My model associations allow user.getAccounts(), but then I'm stuck with the same issue -- wanting to filter a record's associations by model, without resorting to raw SQL.
I can do an ugly/hacky include, like:
return db.instances.AccountType.getLedger('USD').then(ledger => {
return ledger.getAccounts({include:{model: db.models.User, where: {UserId: user.id}}}).then(accounts => {
// now 'accounts' is joined by user, but I've also loaded a bunch of fields I don't need
});
});
But this is intended to join another table, which I don't want. I just want to return any Accounts row where Accounts.UserID === user.id, without looking up any other table.
Is there any function in Sequelize where I could just throw at it a ready-made model, and it'll figure out which fields should wire up to which?