I'm using knex to create a simple table in postgres database:
function up(knex, Promise) {
return knex.schema.createTableIfNotExists('communities', (table) => {
table.increments('id').primary().unsigned();
table.string('name', 255).notNullable();
table.string('slug', 100).notNullable();
table.timestamp('createdAt').defaultTo( knex.fn.now() );
table.timestamp('updatedAt');
});
};
function down(knex, Promise) {
return knex.schema.dropTableIfExists(tableName);
};
module.exports = {
tableName,
up,
down
}
My problem is that table.increments('id').primary() creates a primary key that for default value has nextval('communities_id_seq'::regclass) and I can't do an insert without an id (even in raw sql).
Does anyone know how to make the id increment by default?
In js:
table.increments()
Output sql:
id int unsigned not null auto_increment primary key
check this out for more information
My problem was that the value for id was an empty string and not undefined or null, so that was braking the constraint for integer as data type.
Hope it helps!
A bit late to the party but I was having this issue with the same use case. However my solution was I didnt have all the correct permissions granted to my sequences, only my tables when i ceated the DB.
So something along the lines of "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO PUBLIC"
Related
I am using Supabase as my DB.
I have users table with id field being populated with uuid_generate_v4().
I also have table bank_ao with foreign key bankId pointed to users.id.
When I do following:
const uuid = req.params.id //this 100% works correctly and returns UUID every time!
await supabase
.from("bank_ao")
.select()
.eq("bankId", uuid);
I get this error
{"code":"22P02","details":null,"hint":null,"message":"invalid input syntax for type bigint: \"914eda70-2ecf-49b0-9ea6-87640944ed16\""}
I don't have BIGINT field in my entire DB. I have checked 100x.
I also tried to add ' around my uuid variable, but it still doesn't work.
Any ideas?
Could this be some error specific to Supabase?
You need to cast the UUID into text to compare it there:
const uuid = 'e1bee290-45d1-4233-ad96-b9924288d64c'
const supabase = createClient(SUPABASE_URL, SUPABASE_KEY);
const { data: ret, error } = await supabase
.from('bank_ao')
.select()
.eq('bankId::text', uuid);
console.log(ret);
Assuming table format as:
Output:
[
{
bankId: 'e1bee290-45d1-4233-ad96-b9924288d64c',
created_at: '2023-02-08T16:03:21.064501+00:00',
foo: 'bar',
bar: { bar: 'foo' }
}
]
Please note that for using BigInt in Javascript safely, then you can check it here (similar approach).
The problem was the following.
Before I used UUID as my id column value, it used to be BIGINT. When I changed it, it didn't change in cache, or some other type of memory. So, it threw an error.
I fixed it by re-creating entire database (I tried with table first, but it didn't work).
I have read online that if this is local version of Supabase you can just restart Postgres and it works fine (https://github.com/supabase/supabase/discussions/11891).
I'm currently using knex to connect my node.js sever to a postgres database and I have started using pgcrypto to encrypt some of my data. I am a bit late to the game with encrypting my data, so I have several queries I'll need to update, and am looking for the most efficient way to not only swap over my queries, but to actually query the database. When I try to implement the PGP_SYM_DECRYPT directly inside the knex.select() query, I get an error saying the user can't be found. However, if I use the knex.raw() query, I can get it to work. Is there any way to use the PGG_SYM_DECRYPT inside the .select() query, or perhaps a way to pass the secret key alongside of the query so it will automatically decrypt any encrypted columns?
Example WORKING code:
const user = await knex("n_user AS u")
.where({
"u.uuid": uuid,
"su.site_id": site.id
})
.first()
.join("site_has_user AS su", { "su.user_id": "u.id" })
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
// "u.note", the note is the encrypted data
"u.disabled",
"su.role"
)
.select(
knex.raw(
`PGP_SYM_DECRYPT(u.note::bytea, '${process.env.SECRET_KEY}') as note`
)
);
Example DESIRED code (or some other variant):
const user = await knex("n_user AS u")
.where({
"u.uuid": uuid,
"su.site_id": site.id
})
.first()
.join("site_has_user AS su", { "su.user_id": "u.id" })
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
`PGP_SYM_DECRYPT(u.note::bytea, '${process.env.SECRET_KEY}') as note`,
"u.disabled",
"su.role"
);
Any thoughts?
You can add raw snippet inside select like this:
.select(
"u.id",
"u.uuid",
"u.mobile_number",
"u.email",
"u.first_name",
"u.last_name",
"u.department",
knex.raw("PGP_SYM_DECRYPT(??::bytea, ?) as note", ['u.note', process.env.SECRET_KEY]),
"u.disabled",
"su.role"
);
In raw syntax ?? is identifier replacement and ? is value binding so that secret key is passed to driver safely as binding without trying to interpolate it directly to SQL string.
I am trying to delete and update records in cosmosDB using my graphql/nodejs code and getting error - "Entity with the specified id does not exist in the system". Here is my code
deleteRecord: async (root, id) => {
const { resource: result } = await container.item(id.id, key).delete();
console.log(`Deleted item with id: ${id}`);
},
Somehow below code is not able to find record, even "container.item(id.id, key).read()" doesn't work.
await container.item(id.id, key)
But if I try to find record using query spec it works
await container.items.query('SELECT * from c where c.id = "'+id+'"' ).fetchNext()
FYI- I am able to fetch all records and create new item, so Connection to DB and reading/writing is not an issue.
What else can it be? Any pointer related to this will be helpful.
Thanks in advance.
It seems you pass the wrong key to item(id,key). According to the Note of this documentation:
In both the "update" and "delete" methods, the item has to be selected
from the database by calling container.item(). The two parameters
passed in are the id of the item and the item's partition key. In this
case, the parition key is the value of the "category" field.
So you need to pass the value of your partition key, not your partition key path.
For example, if you have document like below, and your partition key is '/category', you need to use this code await container.item("xxxxxx", "movie").
{
"id":"xxxxxx",
"category":"movie"
}
I am trying to query dynamodb using the following code:
const AWS = require('aws-sdk');
let dynamo = new AWS.DynamoDB.DocumentClient({
service: new AWS.DynamoDB(
{
apiVersion: "2012-08-10",
region: "us-east-1"
}),
convertEmptyValues: true
});
dynamo.query({
TableName: "Jobs",
KeyConditionExpression: 'sstatus = :st',
ExpressionAttributeValues: {
':st': 'processing'
}
}, (err, resp) => {
console.log(err, resp);
});
When I run this, I get an error saying:
ValidationException: Query condition missed key schema element: id
I do not understand this. I have defined id as the partition key for the jobs table and need to find all the jobs that are in processing status.
You're trying to run a query using a condition that does not include the primary key. This is how queries work in DynamoDB. You would need to do a scan for the info in your case, however, I don't think that is the best option.
I think you want to set up a global secondary index and use that to query for the processing status.
In another answer #smcstewart responded to this question. But he provides a link instead of commenting why this error occurs. I want to add a brief comment hoping it will save your time.
AWS docs on Querying a Table states that you can do WHERE condition queries (e.g. SQL query SELECT * FROM Music WHERE Artist='No One You Know') in the DynamoDB way, but with one important caveat:
You MUST specify an EQUALITY condition for the PARTITION key, and you can optionally provide another condition for the SORT key.
Meaning you can only use key attributes with Query. Doing it in any other way would mean that DynamoDB would run a full scan for you which is NOT efficient - less efficient than using Global secondary indexes.
So if you need to query on non-key attributes using Query is usually NOT an option - best option is using Global Secondary Indexes as suggested by #smcstewart.
I found this guide to be useful to create a Global secondary index manually.
If you need to add it using CloudFormation here is a relevant page.
I was getting this error for a different scenario. Here is my scenario.
(It's very unlikely that anyone else ends up with this case, but incase)
I had a query working on a Table (say table A). Table A had a partition key m_id and sort key u_id.
I had a query to fetch data using m_id. The query was working.
'''
var queryParams = {
ExpressionAttributeValues: {
':m_id': mId
},
KeyConditionExpression: 'm_id = :m_id',
TableName: "A"
};
let connections = await docClient.query(queryParams).promise();
'''
I created another Table say Table B. I made some errors in naming keys so I simply deleted and created a table with the same name again, Table B. Table B had partition key m_id, and sort key s_id.
I copied pasted the same query which I was using for Table A, I changed Table name only because partition key had the same name.
To my shock, I get this expectation.
"ValidationException: Query condition missed key schema element"
I rechecked all the names, I compared the query with the working query. Everything was fine.
I thought maybe because, I was deleting recreating Table B, it could be something with that. So I create a fresh Table with a new Name Table B2 with the same key names as Table B.
In my query that was throwing exceptions, I changed only the Table name from B to B2.
And the Exception was gone.
If you are getting this on a fresh table, where no query has worked earlier, creating a new Table with a new name is an option.
If you delete a Table only to change partition key names, it may be safer to use a new name for Table as well (Dynamo could be referring metadata by table names and not by internal identifiers, it is possible that old metadata stays even if you delete a table. Just a guess given I faced this case).
EDIT:2022-July-12
This error does not leave me. My own answer was helpful but one more case, there was a trailing space in name of Key in the table. And Dynamo does not even check for spaces in key names.
You have to create an global secondary index for the status field.
Then, you code could look like smth like this:
dynamo.query({
TableName: "Jobs",
IndexName: 'status',
KeyConditionExpression: '#s = :st',
ExpressionAttributeValues: {
':st': 'processing'
},
ExpressionAttributeNames: {
'#s': 'status',
},
}, (err, resp) => {
console.log(err, resp);
});
Note: scan operation is indeed very costly, especially if you table is huge in size
i solved the problem using AWS.DynamoDB.DocumentClient() with scan, for sample (nodejs):
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "product",
FilterExpression: "#cg = :data",
ExpressionAttributeNames: {
"#cg": "categoria",
},
ExpressionAttributeValues: {
":data": category,
}
};
docClient.scan(params, onScan);
function onScan(err, data) {
if (err) {
// for the log in server
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
res.json(err);
} else {
console.log("Scan succeeded.");
res.json(data);
}
}
I'm using sequelize for node.js. I define this relationship:
// 1:M - Platform table with Governance status table
dbSchema.Platform_governance.belongsTo(dbSchema.Governance_status, {foreignKey: 'platform_status'});
dbSchema.Governance_status.hasMany(dbSchema.Platform_governance, {foreignKey: 'platform_status'});
So that means I have a table called platform_governance which has a foreign key that points to a governance_status row. A user wants to change this foreign key to point to a different governance_status row.
So I want to first search that this governance_status row the user selected actually exists and is unique, and then make the foreign key point to it. This is what I currently have:
// first I select the platform_governance of which I want to change it's foreign key
dbSchema.Platform_governance.findById(5, {include: [dbSchema.Governance_status]}).then(result => {
// Now I search for the user-requested governance_status
dbSchema.Governance_status.findAll({where:{user_input}}).then(answer => {
// I check that one and only one row was found:
if (answer.length != 1 ) {
console.log('error')
} else {
// Here I want to update the foreign key
// I want and need to do it through the associated model, not the foreign key name
result.set('governance_status', answer[0])
result.save().then(result => console.log(result.get({plain:true}))).catch(err => console.log(err))
}
The result.save() promise returns successfully and the object printed in console is correct, with the new governance_status correctly set. But if I go to the database NOTHING has changed. Nothing was really saved.
Oops just found the problem. When setting associations like this you shouldn't use the set() method. Instead, sequelize creates setters for each association. In my case I had to use setGovernance_status():
// Update corresponding foreign keys
result.setGovernance_status(answer[0])
If anyone finds anywhere in the documentation where this is documented I would appreciate :)