Queries from NodeJS to PostgreSQL DB doesn't properly show UTF8 characters - node.js

I'm working on a project, and my mother tongue is spanish and inside my database I'm using characters such as "ñ" and "é". When I'm using the psql shell those characters properly show on the terminal, but when I make a query using node-postgress those characters doesn't show up, instead I get ¤ or ¢.
In my database I have both client_encoding and server_encoding to UTF8, and I even checked with node-postgress, using a query, to see if they also were set to UTF8 and they didn't change for some other reason.
My database connection it's set up like this
const { Pool } = require("pg");
const db = new Pool({
user: user,
password: password,
host: localhost,
port: 5432,
database: my_database,
});
module.exports = db;
And the code for my query is like this:
const router = require("express").Router(),
db = require("../database/db");
//GET A PLACE ROUTE
router.get("/", async (req, res) => {
try {
const place = await db.query("SELECT * FROM places");
console.log(place.rows[0].name);
res.status(200).json({
status: "success",
data: {
place_name: place.rows[0].name,
},
});
} catch (error) {
console.error(error.message);
res.status(500).send("Error del servidor");
}
});
And now, if the name of the place is for example "Salón de peñas", how it will show up both in the console.log and my json response will be like "Sal¢n de pe¤as".
At first, I thought that the problem was because I didn't correctly set up my json response charset, but then I sent these characters as a response, and they show up correctly. The problem is when these characters come from my database. I checked the database encoding (both the client and the server) and they're set to UTF8, and like I said before, these characters display correctly when I'm using the psql shell.
I'm basically having the exact same problem as this question that didn't get an answer

I think I found a workaround to this problem which may be hard to explain but I'll try my best.
So, I realized like for example a new user register and they have ñ or ó in their name, inside the database it shows as ├▒ and ├│. BUT if I do a query from my server and send a json response, "├▒" "├│" characters dissapear and "ñ" "ó" shows instead.
With this weird behavior I thought about inserting all my places through my backend into the database, instead of inserting it using the psql shell. That would be really annoying because I will need to create a route to insert data into my places table, and then do a post request with postman for every row that I need to insert, and then delete that post route because I don't really need it.
After that I realised that maybe using the \i command (the command to execute a .sql file) from the psql shell may cause a similar behavior like inserting data from the server with a post request. And it did! so now, I have a database.sql file with this inside:
CREATE DATABASE my_database;
\c my_database;
SET client_encoding = 'UTF8';
create extension if not exists "uuid-ossp";
CREATE TABLE users(
user_id UUID DEFAULT uuid_generate_v4(),
name VARCHAR(50) NOT NULL,
email VARCHAR(100) NOT NULL UNIQUE,
password VARCHAR(255) NOT NULL,
PRIMARY KEY (user_id)
);
CREATE TABLE places(
place_id SERIAL,
name VARCHAR(50) NOT NULL,
address VARCHAR(50) NOT NULL,
PRIMARY KEY(place_id)
);
INSERT INTO places(name, address) VALUES ('Salón de peñas', 'Calle 242');
INSERT INTO places(name, address) VALUES ('another place', 'another adress');
INSERT INTO places(name, address) VALUES ('another place', 'another adress');
INSERT INTO places(name, address) VALUES ('another place', 'another adress');
Now, if I need to add another row into my places table, I need to create a .sql file and execute it to add a row. Which may be annoying but it's just only needed when there are special characters in the row

Related

Postgres invalid input error on SELECT statement

I am using Supabase as my DB.
I have users table with id field being populated with uuid_generate_v4().
I also have table bank_ao with foreign key bankId pointed to users.id.
When I do following:
const uuid = req.params.id //this 100% works correctly and returns UUID every time!
await supabase
.from("bank_ao")
.select()
.eq("bankId", uuid);
I get this error
{"code":"22P02","details":null,"hint":null,"message":"invalid input syntax for type bigint: \"914eda70-2ecf-49b0-9ea6-87640944ed16\""}
I don't have BIGINT field in my entire DB. I have checked 100x.
I also tried to add ' around my uuid variable, but it still doesn't work.
Any ideas?
Could this be some error specific to Supabase?
You need to cast the UUID into text to compare it there:
const uuid = 'e1bee290-45d1-4233-ad96-b9924288d64c'
const supabase = createClient(SUPABASE_URL, SUPABASE_KEY);
const { data: ret, error } = await supabase
.from('bank_ao')
.select()
.eq('bankId::text', uuid);
console.log(ret);
Assuming table format as:
Output:
[
{
bankId: 'e1bee290-45d1-4233-ad96-b9924288d64c',
created_at: '2023-02-08T16:03:21.064501+00:00',
foo: 'bar',
bar: { bar: 'foo' }
}
]
Please note that for using BigInt in Javascript safely, then you can check it here (similar approach).
The problem was the following.
Before I used UUID as my id column value, it used to be BIGINT. When I changed it, it didn't change in cache, or some other type of memory. So, it threw an error.
I fixed it by re-creating entire database (I tried with table first, but it didn't work).
I have read online that if this is local version of Supabase you can just restart Postgres and it works fine (https://github.com/supabase/supabase/discussions/11891).

How do I use pgcrypto with knex inside a .select statement? (Postgres database)

I'm currently using knex to connect my node.js sever to a postgres database and I have started using pgcrypto to encrypt some of my data. I am a bit late to the game with encrypting my data, so I have several queries I'll need to update, and am looking for the most efficient way to not only swap over my queries, but to actually query the database. When I try to implement the PGP_SYM_DECRYPT directly inside the knex.select() query, I get an error saying the user can't be found. However, if I use the knex.raw() query, I can get it to work. Is there any way to use the PGG_SYM_DECRYPT inside the .select() query, or perhaps a way to pass the secret key alongside of the query so it will automatically decrypt any encrypted columns?
Example WORKING code:
const user = await knex("n_user AS u")
.where({
"u.uuid": uuid,
"su.site_id": site.id
})
.first()
.join("site_has_user AS su", { "su.user_id": "u.id" })
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
// "u.note", the note is the encrypted data
"u.disabled",
"su.role"
)
.select(
knex.raw(
`PGP_SYM_DECRYPT(u.note::bytea, '${process.env.SECRET_KEY}') as note`
)
);
Example DESIRED code (or some other variant):
const user = await knex("n_user AS u")
.where({
"u.uuid": uuid,
"su.site_id": site.id
})
.first()
.join("site_has_user AS su", { "su.user_id": "u.id" })
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
`PGP_SYM_DECRYPT(u.note::bytea, '${process.env.SECRET_KEY}') as note`,
"u.disabled",
"su.role"
);
Any thoughts?
You can add raw snippet inside select like this:
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
knex.raw("PGP_SYM_DECRYPT(??::bytea, ?) as note", ['u.note', process.env.SECRET_KEY]),
"u.disabled",
"su.role"
);
In raw syntax ?? is identifier replacement and ? is value binding so that secret key is passed to driver safely as binding without trying to interpolate it directly to SQL string.

Retrieve inserted row using pg-promise npm module in Express.js

I am inserting a user row into Postgres from Express.js like as follows:
db.none("INSERT INTO users (email, password) VALUES (${incoming.email}, crypt(${incoming.password}, gen_salt('bf')))",{
incoming: { email: qemail, password: qpwd}
}
).then(function (data) {
// data is null
});
The email and password come from the query parameters on the React.js frontend. I use this query for the signup action on my application and wanted to know if I could get this newly added user back from this same query. I just felt that making another get request right after the signup to get the user info was inefficient, which I do for my login action. Thanks in advance.
Append RETURNING * to your query, and replace none with method one.
const row = await db.one("INSERT INTO users(email, password) VALUES(${incoming.email},
crypt(${incoming.password}, gen_salt('bf'))) RETURNING *",
{
incoming: { email: qemail, password: qpwd}
}
);
// row = the newly inserted row
At the end of the query add RETURNING *. This should return the newly added query. (postgresql docs).
In your query:
"INSERT INTO users (
email, password
) VALUES (
${incoming.email}, crypt(${incoming.password}, gen_salt('bf'))
) RETURNING *"

Insert after another insert in transaction with pg-promise

Ok, I have looked but didn't find anything that worked for me. If you can point me to anything or help my find the solution it would be great.
Let's say I have a "users" table (columns: id, username and name) and a "users_items" table (columns: id_item, id_user, both Foreign Keys)
What I want to do: I want to insert a new user and assign him a default item that everyone should have. For that, I want to create a transaction where everything or nothing is saved. So if (for any reason) I can't give him that item, I want the user creation to fail.
How I do it:
const pgp = require('pg-promise')(
{
capSQL: true // generate capitalized SQL
});
const db = pgp(configuration);
saveUser = (username, name) =>
db.tx (t =>
t.one('INSERT INTO users (username, name) VALUES $1, $2 RETURNING *', [username, name]).then(user =>
t.none ('INSERT INTO users_items (id_user, id_item) VALUES $1, 1', [user.id]).then(()=>
console.log('Everything\'s alright :)');
)
);
What I expect: everything runs perfectly and we are all happy :).
What actually happens: The first instruction is OK, and returns correctly the user with the ID. However, the second one tells me that the constraint fk_id_user is being violated, and perform a ROLLBACK.
The part of it not commiting the first insert is working properly. However, shouldn't the second insert work too? Am I understanding something extremely wrong or is pg-promise not working as expected? Or maybe I need to do something different.. Any help would be appreciated.

Knex primary key auto increment

I'm using knex to create a simple table in postgres database:
function up(knex, Promise) {
return knex.schema.createTableIfNotExists('communities', (table) => {
table.increments('id').primary().unsigned();
table.string('name', 255).notNullable();
table.string('slug', 100).notNullable();
table.timestamp('createdAt').defaultTo( knex.fn.now() );
table.timestamp('updatedAt');
});
};
function down(knex, Promise) {
return knex.schema.dropTableIfExists(tableName);
};
module.exports = {
tableName,
up,
down
}
My problem is that table.increments('id').primary() creates a primary key that for default value has nextval('communities_id_seq'::regclass) and I can't do an insert without an id (even in raw sql).
Does anyone know how to make the id increment by default?
In js:
table.increments()
Output sql:
id int unsigned not null auto_increment primary key
check this out for more information
My problem was that the value for id was an empty string and not undefined or null, so that was braking the constraint for integer as data type.
Hope it helps!
A bit late to the party but I was having this issue with the same use case. However my solution was I didnt have all the correct permissions granted to my sequences, only my tables when i ceated the DB.
So something along the lines of "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO PUBLIC"

Resources