How to use postgres pgcrypto with Nodejs and Sequelize? - node.js

How to use sequelize with pgcrypto plugin in postgres.
How to encrypt and decrypt the values of a column using sequelize
How to use PGP_SYM_ENCRYPT and PGP_SYM_DECRYPT using nodejs and sequelize

I will walk through from starting what steps you need to follow :)
Steps to follow to get pgcrypto plugin into the Schema you are using
Login to postgres and goto the Schema used Or If you have pgadmin running... goto Schema... right-click... click Query Tool.
Run this query there to check available plugins in your postgres -
select * from pg_available_extensions
Above command will help you know what all plugins you already have in Postgres. Scroll and check there if pgcrypto is available. If yes... move on to 3rd point else download pgcrypto plugin first.
Run another query, this will help you to know what plugins are there with the Schema you have selected in point (1) -
select * from pg_extension
Above command will help you know what all pluings are supported by the Schema you have selected. Check if pgcrypto is there, if yes- skip to point (5), if no -continue to point (4)
Run this command to bring pgcrypto plugin from extensions to current Schema supported extension -
create extension pgcrypto
You can verify running cmd at point(3) to check if pgcrypto got pulled successfully to current Schema supported pluings.
Now we are ready to use pgcrypto in our Nodejs application
for the query which you want to encrypt make use of sequelize to encrypt it. Use below code to modify the text value of that column to encrypted value.
query: sequelize.fn("PGP_SYM_ENCRYPT", "data_to_encrypt",
"secret_key")
When you will save the data to the db using create- data will be encrypted using PGP_SYM_ENCRYPT which is a method provided by pgcrypto plugin.
To query or decrypt the values now, you can run this query in postgres
select PGP_SYM_DECRYPT(colum_name::bytea, 'secret_key') FROM table where PGP_SYM_DECRYPT(column_name::bytea, 'secret_key' LIKE '%search_string%';
To decrypt the value in Node application, use:
sequelize.findAll({
attribute: [
[
sequelize.fn(
'PGP_SYM_DECRYPT',
sequelize.cast(sequelize.col('column_name'), 'bytea'),
'secret_key'
),
"column_name"
]
]
}).then(data => console.log(data))
NOTE: To automate the 1st part(getting extension into Schema), you can use sequelize raw query, so that you don't have to do it manually each time required.
CREATE EXTENSION IF NOT EXISTS pgcrypto;

Related

node.js | Replacing DB client with Prisma keeping preserving DB structure - issue with uuid_generate_v4()

I am trying to add the Prisma DB client to the existing node.js project while preserving the DB structure.
Postgresql
Prisma 4.7.1
I've set up the initial Prisma configuration (env vars, etc.).
I've used the command npx prisma db pull to generate prisma.schema file according to the existing DB structure
I create the initial migration by using some empty DB npx prisma migrate dev
At this point it is expected that migration would create DB structure, but the command fails with the following error
✗ npx prisma migrate dev
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "service_prisma", schema "public" at "127.0.0.1:5432"
PostgreSQL database service_prisma created at 127.0.0.1:5432
✔ Enter a name for the new migration: … init
Applying migration `20221221095823_init`
Error: P3018
A migration failed to apply. New migrations cannot be applied before the error is recovered from. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
Migration name: 20221221095823_init
Database error code: 42883
Database error:
ERROR: function uuid_generate_v4() does not exist
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState(E42883), message: "function uuid_generate_v4() does not exist", detail: None, hint: Some("No function matches the given name and argument types. You might need to add explicit type casts."), position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_func.c"), line: Some(521), routine: Some("ParseFuncOrColumn") }
Follow up plan would be to:
Set DB back to the original one containing tables and data
Then mark initial migration as applied with the following command npx prisma migrate resolve --applied 20221221095823_init
So, main problem is that IDs in existing tables use uuid_generate_v4() to generate random UUID for new entries. The support on DB level is definitly there, because it simple works normally with slonik DB client.
model SomeTable {
id String #id #default(dbgenerated("uuid_generate_v4()")) #db.Uuid
}
Any idea how to solve this? Thanks in advance!
I struggled a bit, but then found a solution shortly after posting the question above...
I've found some tips but they were a bit unclear to me initially.
So if anyone runs into a similar problem, then there is a solution:
Ignore the error in step 3 - the migration file gets created anyway
Thenn add the CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; into the first line of the created migration file.
Run npx prisma migrate dev again, this time it should run without errors
Continue with step 4 of the list mentioned above.
Hope this helps :)

Running one time DML/Update scripts using Prisma

Does Prisma support running one-time DML statements, such as UPDATE, automatically?
For example, let's say we want all emails in a table to be lowercase. We make a change in our API so that all future accounts/emails are lowercase, however we want to update EXISTING emails to be lowercase too.
Running npx prisma generate and npx prisma migrate executes DDL to keep your schema in-sync. However, I do not see a place to hold database "patch" files. These files generally are run once in order to update existing records in a database.
Prisma doesn't support running one-time DML statements automatically.
You would need to use something like a cron if you want to run some function at specific intervals.
And for one time you could just invoke the function once.
As for your particular use case you could achieve it by using the below function.
async function main() {
const result = await prisma.$executeRaw`UPDATE "User" SET email=lower(email)`;
console.log(result);
}
You could know more about getting Raw Database Access from this Guide

Make a change to the database with Prisma.js without having to reset the whole thing

How can make a change to the database with Prisma.js without having to reset the whole thing?
if I have used this command
npx prisma migrate dev --name role-makeOptional-NormalizedName
I will lose all of the data in my database but I don't lose my data.
In my case I wanted to change String to String? in schema.prisma file
NormalizedName String? #unique #db.VarChar(64)
Is there a proper command to avoid losing the data?
Go into your schema and make the change:
NormalizedName String #unique #db.VarChar(64)
NormalizedName String? #unique #db.VarChar(64)
Then create a draft migration:
$ npx prisma migrate dev --name migration-name --create-only
Then edit the migration in SQL (this allow null values ie optional)
ALTER TABLE myTable ALTER COLUMN myColumn {DataType} NULL;
OR Postgres
ALTER TABLE myTable ALTER COLUMN myColumn DROP NOT NULL;
etc.
Then apply the modified SQL:
$ npx prisma migrate dev
I hope this works for you :)
In a development environment, Prisma Migrate sometimes prompts you to reset the database. Resetting drops and recreates the database, which results in data loss. The database is reset when:
You call prisma migrate reset explicitly
You call prisma migrate dev and Prisma Migrate detects drift in the database or a migration history conflict
I'm not sure why Prisma thinks that your change is breaking, but there is probably no other way to make schema change without data loss.
To recreate your database data consider using seeding script
If you are prototyping, consider using the db push command, although it will still result in data reset if Prisma considers that the change is breaking.
I ran into a similar issue. What happened was that an error occurred during one of my migrations and because of that it puts prisma into an error state. Prisma doesn't want to run any new migrations while in this error state.
To solve it:
Removed the broken migration folder that was auto generated inside /prisma/migrations
Remove the broken row that is created inside the _prisma_migrations table in your database

Sequelize bulkCreate updateOnDuplicate for postgresQL?

I know there is no support for updateOnDuplicate for postgresQL by Sequelize sequelize doc, so is there a work around for this?
Can it be implemented via "SQL command".
New sequelize (v5) includes updateOnDuplicate feature for all dialects
Fields to update if row key already exists (on duplicate key update)?
(only supported by MySQL, MariaDB, SQLite >= 3.24.0 & Postgres >=
9.5). By default, all fields are updated.
Check here : Docs
You can use as
model.bulkCreate(dataToUpdate, { updateOnDuplicate: ["user_id", "token", "created_at"] })
There is some work around. See upsert function. When used in Postgresql it creates custom function in database. Unfortunately there is no bulkUpsert, so you either use it in some for-loop or execute raw SQL as suggested here.

Making bookshelf.js work with existing data

My team has just started using bookshelf.js in our node app with knex. By default, the ORM object uses an id field which is created as a "serial" type in postgresql. We have existing data that we need to insert into the DB. My understanding is that the database "id" field will want to start with 1.
So is there a way to have a postgres serial number start at a particular number (also will there be issues with using a clustered database and the serial datatype)?
Is there a way to set the field to an "integer" field type through knex/bookshelf and use a sequence to nextval?
Any other options (not looking for extreme hacks here either)?
We are using the following stack:
Postgres: 9.x
NodeJs: 4.4.4 LTS
Knex: 0.11.8
Bookshelf: 0.10.0
Thanks!

Resources