My team has just started using bookshelf.js in our node app with knex. By default, the ORM object uses an id field which is created as a "serial" type in postgresql. We have existing data that we need to insert into the DB. My understanding is that the database "id" field will want to start with 1.
So is there a way to have a postgres serial number start at a particular number (also will there be issues with using a clustered database and the serial datatype)?
Is there a way to set the field to an "integer" field type through knex/bookshelf and use a sequence to nextval?
Any other options (not looking for extreme hacks here either)?
We are using the following stack:
Postgres: 9.x
NodeJs: 4.4.4 LTS
Knex: 0.11.8
Bookshelf: 0.10.0
Thanks!
Related
How to use sequelize with pgcrypto plugin in postgres.
How to encrypt and decrypt the values of a column using sequelize
How to use PGP_SYM_ENCRYPT and PGP_SYM_DECRYPT using nodejs and sequelize
I will walk through from starting what steps you need to follow :)
Steps to follow to get pgcrypto plugin into the Schema you are using
Login to postgres and goto the Schema used Or If you have pgadmin running... goto Schema... right-click... click Query Tool.
Run this query there to check available plugins in your postgres -
select * from pg_available_extensions
Above command will help you know what all plugins you already have in Postgres. Scroll and check there if pgcrypto is available. If yes... move on to 3rd point else download pgcrypto plugin first.
Run another query, this will help you to know what plugins are there with the Schema you have selected in point (1) -
select * from pg_extension
Above command will help you know what all pluings are supported by the Schema you have selected. Check if pgcrypto is there, if yes- skip to point (5), if no -continue to point (4)
Run this command to bring pgcrypto plugin from extensions to current Schema supported extension -
create extension pgcrypto
You can verify running cmd at point(3) to check if pgcrypto got pulled successfully to current Schema supported pluings.
Now we are ready to use pgcrypto in our Nodejs application
for the query which you want to encrypt make use of sequelize to encrypt it. Use below code to modify the text value of that column to encrypted value.
query: sequelize.fn("PGP_SYM_ENCRYPT", "data_to_encrypt",
"secret_key")
When you will save the data to the db using create- data will be encrypted using PGP_SYM_ENCRYPT which is a method provided by pgcrypto plugin.
To query or decrypt the values now, you can run this query in postgres
select PGP_SYM_DECRYPT(colum_name::bytea, 'secret_key') FROM table where PGP_SYM_DECRYPT(column_name::bytea, 'secret_key' LIKE '%search_string%';
To decrypt the value in Node application, use:
sequelize.findAll({
attribute: [
[
sequelize.fn(
'PGP_SYM_DECRYPT',
sequelize.cast(sequelize.col('column_name'), 'bytea'),
'secret_key'
),
"column_name"
]
]
}).then(data => console.log(data))
NOTE: To automate the 1st part(getting extension into Schema), you can use sequelize raw query, so that you don't have to do it manually each time required.
CREATE EXTENSION IF NOT EXISTS pgcrypto;
I'm trying to run my service with Micronaut and Cassandra (currently version 3.11.10) and store a column that is a set of tuples into Cassandra.
example code:
QueryBuilder
.insertInto(table)
.value("column", QueryBuilder.literal(items.map { it.toTuple() }.toSet())))
The toTuple() method is just an extension method that transfer the items into Term objects
When I'm doing so I'm receiving the following error:
Internal Server Error: Could not inline literal of type java.util.Collections$SingletonSet. This happens because the driver doesn't know how to map it to a CQL type. Try passing a TypeCodec or CodecRegistry to literal().
I've checked online in multiple sources but couldn't find a simple way to store a set of tuples into the database without implementing my custom TypeCodec. As I'm sure that I'm not the first person to have this issue, I'm probably doing something completely wrong, however I couldn't find any documentation regarding to what is the correct way of doing this.
I'm working with the Datastax Node.js driver and I can't figure out when to use a mapper vs. query. Both seem to be able to perform the same CRUD operations.
With a query:
const q = SELECT * FROM mykeyspace.mytable WHERE id='12345';
client.execute(q).then(result => console.log('This is the data', result);
With mapper:
const tableRow = await tableMapper.find({ id: '12345' });
When should I use the mapper over a query and vice versa?
Mapper is a feature from cassandra-driver released in 2018. Using mapper, cassandra-driver can make a map from your cassandra table to an object in nodejs and you can handle in your nodejs application like a set of document.
Using mapper you can make selects or inserts in your database like said in this article:
https://www.datastax.com/blog/2018/12/introducing-datastax-nodejs-mapper-apache-cassandra
With query method, if you need to use or reuse any property from your json you will need to make a Json.Parse().
The short answer is: whatever you find more comfortable.
The Mapper lets you deal with database data as documents (JavaScript objects), builds the CQL query for you, executes the query and maps the results.
On the other hand, the core driver only supports executing CQL queries that you have to write yourself.
I know there is no support for updateOnDuplicate for postgresQL by Sequelize sequelize doc, so is there a work around for this?
Can it be implemented via "SQL command".
New sequelize (v5) includes updateOnDuplicate feature for all dialects
Fields to update if row key already exists (on duplicate key update)?
(only supported by MySQL, MariaDB, SQLite >= 3.24.0 & Postgres >=
9.5). By default, all fields are updated.
Check here : Docs
You can use as
model.bulkCreate(dataToUpdate, { updateOnDuplicate: ["user_id", "token", "created_at"] })
There is some work around. See upsert function. When used in Postgresql it creates custom function in database. Unfortunately there is no bulkUpsert, so you either use it in some for-loop or execute raw SQL as suggested here.
I am using Spark-Solr connector 3.4.0 with Solr cloud version 7.6.0 in a Spark 2.2.1 Cluster . We have an existing Solr collection with a predefined schema for it. Most of the fields have the stored parameter set to true, but there are certain fields where we explicitly set stored=false. When we try to push data to Solr using the spark-solr connector, we get the following error-
org.apache.solr.api.ApiBag$ExceptionWithErrObject: error processing commands, errors: [{add-field={name=taxonomy, indexed=true, multiValued=true, docValues=true, stored=true, type=string}, errorMessages=[Field 'item_id_channel' already exists.
]}],
at org.apache.solr.handler.SchemaHandler.handleRequestBody(SchemaHandler.java:92)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
The error says the item_id_channel already exists, but this error is only raised for fields for which we have defined stored=false (in the Solr schema). I get that the connector wishes to create the schema again for some reason, but it sets the stored parameter to true which clashes with the predefined schema definition on Solr for this field.
My question is - Is there a way to tell the connector (probably through some option?) that we want the stored to be set to true for certain fields? And probably a generic way to define other solr parameters for the fields?
We found the issue that was causing the error. There was a bug in older versions of spark-solr connector, because of which the connector was trying to add existing fields to the solr schema in case the value of stored was true. This was fixed in 3.5.5 release. Hence, once we upgraded our connector to version 3.5.14, the ingestion stared working without any errors.