i am currently trying to implement my backend from my all conversations screen. i am finding all the conversations and including all the messages associated with each conversation.
In my react native frontend, in my all conversations screen, i am showing all the conversations as a flatlist and showing the last message sent between the 2 users by picking it out from the list of messages.
Should i continue using this approach or will it be better in terms of performance to include a link to the last message in Conversation table and update it every time a new message is sent in the conversation? If so, how do i change my code to do so?
Conversation Model:
const Conversation = db.define("Conversation", {},
{
paranoid: true,
timestamps: true,
deletedAt: "deletedAt",
}
);
module.exports = Conversation;
User.hasMany(Conversation, {
foreignKey: 'user1'
});
User.hasMany(Conversation, {
foreignKey: 'user2'
});
Conversation.belongsTo(User, { as: 'Creator', foreignKey: 'user1', allowNull: false })
Conversation.belongsTo(User, { as: 'Recipient', foreignKey: 'user2', allowNull: false })
Message Model:
const Message = db.define("Message", {
senderId: {
type: Sequelize.STRING,
},
receiverId: {
type: Sequelize.STRING,
},
message: {
type: Sequelize.STRING,
},
conversationId:{
type: Sequelize.INTEGER,
}
});
module.exports = Message;
User.hasMany(Message, {
foreignKey: 'senderId',
});
User.hasMany(Message, {
foreignKey: 'receiverId',
});
Message.associate = (models) => {
Message.belongsTo(models.Conversations,{
foreignKey: "conversationId",
}
);
};
Conversation.hasMany(Message,{
foreignKey: "conversationId",
}
);
Message.belongsTo(User, {
as:"Sender",
foreignKey: "senderId",
allowNull: false
});
Message.belongsTo(User, {
as:"Receiver",
foreignKey: "receiverId",
allowNull: false
});
Get All Conversations Query:
router.get('/', auth, async (req, res) => {
const conversations = await Conversation.findAll({
order: [[Message,"createdAt", "DESC"]],
include: [
{
model: Message,
where: {
[Op.or]: [
{ senderId: req.user.id },
{ receiverId: req.user.id },
],
},
required: true, // RIGHT JOIN
}]
})
if (!conversations) return res.status(404).send();
res.status(200).send();
});
You can try to limit included messages in each conversation to the last message only
const conversations = await Conversation.findAll({
order: [[Message,"createdAt", "DESC"]],
include: [
{
model: Message,
order: [["createdAt", "DESC"]],
limit: 1,
where: {
[Op.or]: [
{ senderId: req.user.id },
{ receiverId: req.user.id },
],
},
required: true, // RIGHT JOIN
}]
})
If the above query doesn't work you can always try to store the last message id in a conversation but this can lead to inconsistencies if you store a message and forget to update a link to the last message in a conversation or you delete the last message and again forget to update the link in a conversation.
You can cache the conversations and messages into the local database
using sqlite of the mobile device.
To do so, you don't have to query whole conversation for every login.
Meta data table can be even helpful to relieve the server stress
I suggest you to use sqlite in RN and add meta table in remote database.
Pattern.
Load Conversations from remote database with the count of messages when logining in.
( If meta data table exists, this part will be faster and lighter )
When displaying the conversions, use the lastest data in local database.
Comparing count of messages, if count is different, you can assume that user in this device didn't check the lastest message. So Render badge component to notify user that he or she didn't read it yet.
When user enter the conversation, fetch messages and store it into local database of the device.
Socket io part
Brief concept is like this.
ref. https://socket.io/docs/v3/server-api/#socket-on-eventName-callback
Client
const [ conversations, setConversations ] = useState([])
const [ syncState, setSyncState ] = useState(false)
const [ socket, setSocket ] = useState(null)
useEffect(() =>{
...
// you might wanna set conversation from local database here
...
io.on('connection', soc => {
soc.join("your room")
...
socket.on('synced', data =>{
//mapNewDataToConversationState
...
setSynced(true)
})
setSocket(soc)
})
...
}, [ ... ])
useEffect(() =>{//when loaded and socked established
if(!socket) return
if(syncState) return
socket.emit('sync', { data : conversation })
}, [ socket ])
Server
//let say you already set app.io as socket io connection
//add a event listener for client sync
socket.on('sync', async conversation =>{
const responseData = {}
for(const c of conversation){
const lastestMsg = c.Message[c.Message.length - 1]
const messages = await Message.findAll({
where : {
...//Maybe greater than createdAt, any cursor attribute in here
}
})
responseData[c.id] = messages
}
socket.emit('synced', responseData )
})
Related
So, I'm new to Node.JS, and this project is a simple petshop, with pets table, services offered, and appointments (when a certain service is applied to a certain pet in a certain date in the future).
Appointments have a petId and serviceId, each appointment must have only one of those, but pets and services can be referenced in multiple appointments.
This project was originally made with pure MySql, with separate repositories to handle queries, later I changed it to use sequelize ORM.
I've made the basics crud operations using the rest architecture, everything ok with the services and pets methods (post, get, patch, delete), but the problem began when trying to make post method with appointments, since appointments have serviceId and petId.
index.js
const customExpress = require('./config/customExpress');
const connection = require('./infrastructure/database/connection');
connection.sync()
.then(() => {
console.log('Successfully connected to database')
const app = customExpress()
app.listen(3000, () => console.log('Server running at port 3000'))
})
.catch(erro => console.log('Something went wrong while trying to synchronize to database'))
connection.js
here is where the associations between tables and models are made, I think the error might be here, but can't figure out where
const Sequelize = require('sequelize')
const config = require('config')
const { applyExtraSetup } = require('../../models/extraSetup')
const sequelize = new Sequelize(
config.get('mysql.database'),
config.get('mysql.user'),
config.get('mysql.password'),
{
host:config.get('mysql.host'),
dialect:config.get('mysql.dialect')
}
)
const modelDefiners = [
require('../../models/appointments'),
require('../../models/pets'),
require('../../models/services'),
]
// it gets the models definitions and pass the above instance of sequelize to them
for (const modelDefiner of modelDefiners){
modelDefiner(sequelize)
}
// and then make their relations with extra setup
applyExtraSetup(sequelize)
.then(module.exports = sequelize)
the extraSetup.js, makes the relations
// it exports the things the sequelize instance must do
async function applyExtraSetup(sequelize){
const { appointments, services, pets } = sequelize.models
await appointments.belongsTo(services)
await appointments.belongsTo(pets)
await services.hasMany(appointments)
await pets.hasMany(appointments)
}
module.exports = { applyExtraSetup }
and this is appointment.js, a controller that handles the /appointments route and call the sequelize methods
app.post('/appointment', async (req, res) => {
const data = req.body
const pet = await Pets.findAll({ where : { name : req.body.petId }})
const service = await Service.findAll({ where : { name : req.body.serviceId }})
const appointment = Appointment.create(data)
.then(result => {
result.setPet(pet)
res.end()
})
reading sequelize documentation, I saw that with this type of association, you get magic methods to handle data from the related tables, which you call from an instance of a model (setPet, getPet, and so on).
Can anyone help me?
edit, models definitions (they're together since I'm refactoring the code, but originally each one was in a separate module):
exports.appointment = (sequelize) => {
const now = new Date()
const Appointment = sequelize.define('appointments', {
client: {
type: DataTypes.STRING,
allowNull: false
},
creationDate: {
type: DataTypes.DATE,
allowNull: false,
defaultValue: now
},
date: {
type: DataTypes.DATE,
allowNull: false,
validate: {
isBefore: now
}
},
status: {
type: DataTypes.STRING,
allowNull: false
},
observations: {
type: DataTypes.STRING
}
})
return Appointment
}
exports.pets = (sequelize) => {
const Pets = sequelize.define('pets', {
name : {
type: DataTypes.STRING,
allowNull: false
},
image: {
type: DataTypes.STRING,
allowNull: false
},
})
return Pets
}
exports.service = (sequelize) => {
const Service = sequelize.define('services', {
name: {
type: DataTypes.STRING,
allowNull: false
},
price: {
type: DataTypes.FLOAT,
allowNull: false
}
})
return Service
}
I am trying to learn how to create a Normalized many to many Post request in express/router and Mongoose.
I have three collections: User, Building and Room which is also the order of parent to child documents.
My Building document schema includes both User and Room id's as follows:
const mongoose = require('mongoose');
const BuildingSchema = new mongoose.Schema({
user: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "user"
}
],
room: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "room"
}
],
My Room document schema includes the Building ID and another child document as follows:
const mongoose = require('mongoose');
const RoomSchema = new mongoose.Schema({
building: {
type: mongoose.Schema.Types.ObjectId,
ref: "building"
},
furniture: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "furniture"
}
],
I am having trouble understanding how to create a post request that allows a user/users to create a Building instance and multiple Room instances associated with it after...
My express code so far can create a new Building instance but I am unsure how to handle including the Room id's:
router.post('/createbuilding', [auth, [
check('name', 'Building Name is required').not().isEmpty(),
]
], async (req, res) => {
const errors = validationResult(req);
if (!errors.isEmpty()) {
return res.status(400).json({ errors: errors.array() });
}
const {
name,
type,
website,
location,
bio,
} = req.body;
const buildingFields = {
user: req.user.id,
//room: req.room.id,
name,
type,
website: website && website !== '' ? normalize(website, { forceHttps: true }) : '',
location,
bio
};
try {
let building = await Building.findOneAndUpdate(
{ user: req.user.id },
//{ room: req.room.id },
{ $set: buildingFields },
{ new: true, upsert: true }
);
res.json(building);
} catch (err) {
console.error(err.message);
res.status(500).send('Server Error');
}
}
);
Note: The references to room ID are commented out on purpose because this is what I am unsure of how to include in the Post request.
With Mongoose's .findOneAndUpdate() the first parameter is to query for the document, similar to .findOne(). Your second parameter is what fields to update. $set is redundant since Mongoose will do that for you.
If you want to add a room to the building rather than replacing all the associations, you'll want to add the room with a $push.
const buildingFields = {
$push: {
room: {
req.room.id
}
},
name,
type,
...
}
I am also assuming that you intended that the user field in BuildingSchema is a single association rather than a many associations. If not, you'd need to use a $elemMatch to query for that document:
Mongoose Mongodb querying an array of objects
We are writing an instant messenger for private use in our application. The planned load of the current version is hundreds of users, maybe a thousand or two.
As a database, I use Mongo. Messages are written in messages (linked via chats_id with user_chats and uid with an external table in the users database). The arrival of a message on the client creates a flurry of events in response (mark read). It is clear that I need to make a queue. How to do it better? Are there methods to prioritize queues?
How many users should the spherical process of a node in a vacuum withstand? How much memory should it use? I'm using pm2 for process management and node-ipc for broadcast.
Where to turn, who to ask to optimize requests to the database. Now there is a quick written construction of the kind illustrated below. This example requests all chats, for each chat it selects the last message, the number of unread messages and the last users seen. In MySQL I would do it with one big request. How is it better to do this in Mongo? My level with Mongo is still in the middle.
The architecture itself. Now the action from the client comes and goes to the handler who understands what to do with it and who needs to be notified about this action. Usually the handler notifies the corresponding manager, and it already writes to the database and, if necessary, notifies the other processes.
const loadUserChatsStatistic = async (chatIds, userId) => {
const startExecTime = new Date();
const userChats = await db()
.collection("user_chats")
.aggregate([
{ $match: { uid: userId, chats_id: { $in: chatIds.map(ObjectID) } } },
{
$lookup: {
from: "chats",
localField: "chats_id",
foreignField: "_id",
as: "chat"
}
},
{ $unwind: "$chat" },
{
$lookup: {
from: "user_last_seen",
localField: "chats_id",
foreignField: "chats_id",
as: "last_seens"
}
}
])
.toArray();
const chats = await Promise.all(
userChats.map(async userChat => {
const usersCount = await db()
.collection("user_chats")
.find({ chats_id: userChat.chats_id })
.count();
let unreadCountQuery = {};
if (userChat.last_read) {
unreadCountQuery = { _id: { $gt: userChat.last_read } };
} else {
unreadCountQuery = { _id: { $gt: userChat._id } };
}
const unreadCount = await db()
.collection("messages")
.find({ ...unreadCountQuery, chats_id: userChat.chats_id })
.count();
const message =
(await db()
.collection("messages")
.findOne({ chats_id: userChat.chats_id }, { sort: { _id: -1 } })) ||
{};
return { ...userChat, usersCount, unreadCount, message };
})
);
console.log(
"Execution time for loadUserChatsStatistic",
new Date() - startExecTime
);
return chats.sort((a, b) => {
if (!a.message._id) {
return true;
}
if (!b.message._id) {
return false;
}
return (
new Date(ObjectID(a.message._id).getTimestamp()) <
new Date(ObjectID(b.message._id).getTimestamp())
);
});
};
I'm trying to make a many to many table with matches of different teams and it works, but when I tried to make a join I got this error:
Unhandled rejection Error: team is not associated to match!
Here is my code:
var Sequelize = require('sequelize');
var sequelize = new Sequelize('cricket', 'root', '');
var Team = sequelize.define('team', {
name: Sequelize.STRING,
});
var Match = sequelize.define('match', {
id: {
type: Sequelize.INTEGER,
primaryKey: true,
autoIncrement: true
},
scoreHome: Sequelize.INTEGER,
scoreAway: Sequelize.INTEGER,
});
Team.belongsToMany(Team, {as: 'Home', through: Match, foreignKey: 'homeTeamId'});
Team.belongsToMany(Team, {as: 'Away', through: Match, foreignKey: 'awayTeamId'});
sequelize.sync().then(function () {
Match.findAll({
include: [ Team ]
}).then(function (matches) {
console.log(matches)
});
});
You also need to provide the reverse relationship for Match.
In your current setup Team is associated with Team, not with Match. Match model is your through model.
For example valid association for Match would be :
var HomeTeam = Match.belongsTo( Team, { as: 'HomeTeam', foreignKey: 'homeTeamId' });
var AwayTeam = Match.belongsTo( Team, { as: 'AwayTeam', foreignKey: 'awayTeamId' })
sequelize.sync().then(function () {
Match.findAll({
include: [ HomeTeam, AwayTeam ]
}).then(function (matches) {
console.log(matches)
});
});
// Makes the following query
// SELECT `match`.`id`, `match`.`scoreHome`, `match`.`scoreAway`, `match`.`createdAt`, `match`.`updatedAt`, `match`.`homeTeamId`, `match`.`awayTeamId`, `HomeTeam`.`id`
// AS `HomeTeam.id`, `HomeTeam`.`name` AS `HomeTeam.name`, `HomeTeam`.`createdAt`
// AS `HomeTeam.createdAt`, `HomeTeam`.`updatedAt` AS `HomeTeam.updatedAt`, `AwayTeam`.`id` AS `AwayTeam.id`, `AwayTeam`.`name` AS `AwayTeam.name`, `AwayTeam`.`createdAt` AS `AwayTeam.createdAt`, `AwayTeam`.`updatedAt` AS `AwayTeam.updatedAt` FROM `matches` AS `match` LEFT OUTER JOIN `teams` AS `HomeTeam` ON `match`.`homeTeamId` = `HomeTeam`.`id` LEFT OUTER JOIN `teams` AS `AwayTeam` ON `match`.`awayTeamId` = `AwayTeam`.`id`
I have the following code:
#!/usr/bin/env node
'use strict';
var Sequelize = require('sequelize');
var sequelize = new Sequelize('sqlite:file.sqlite');
var User = sequelize.define('User', { email: Sequelize.STRING});
var Thing = sequelize.define('Thing', { name: Sequelize.STRING});
Thing.belongsTo(User);
sequelize.sync({force: true}).then(function () {
return User.create({email: 'asdf#example.org'});
}).then(function (user) {
return Thing.create({
name: 'A thing',
User: user
}, {
include: [User]
});
}).then(function (thing) {
return Thing.findOne({where: {id: thing.id}, include: [User]});
}).then(function (thing) {
console.log(JSON.stringify(thing));
});
I get the following output:
ohnobinki#gibby ~/public_html/turbocase1 $ ./sqltest.js
Executing (default): INSERT INTO `Users` (`id`,`email`,`updatedAt`,`createdAt`) VALUES (NULL,'asdf#example.org','2015-12-03 06:11:36.904 +00:00','2015-12-03 06:11:36.904 +00:00');
Executing (default): INSERT INTO `Users` (`id`,`email`,`createdAt`,`updatedAt`) VALUES (1,'asdf#example.org','2015-12-03 06:11:36.904 +00:00','2015-12-03 06:11:37.022 +00:00');
Unhandled rejection SequelizeUniqueConstraintError: Validation error
at Query.formatError (/home/ohnobinki/public_html/turbocase1/node_modules/sequelize/lib/dialects/sqlite/query.js:231:14)
at Statement.<anonymous> (/home/ohnobinki/public_html/turbocase1/node_modules/sequelize/lib/dialects/sqlite/query.js:47:29)
at Statement.replacement (/home/ohnobinki/public_html/turbocase1/node_modules/sqlite3/lib/trace.js:20:31)
It seems that specifying {include: [User]} instructs Sequelize to create a new User instance matching the contents of user. That is not my goal. In fact, I find it hard to believe that such behaviour would ever be useful—I at least have no use for it. I want to be able to have a long-living User record in the database and at arbitrary times create new Things which refer to the User. In my shown example, I wait for the User to be created, but in actual code it would likely have been freshly loaded through User.findOne().
I have seen other questions and answers say that I have to explicitly specify the implicitly-created UserId column in my Thing.create() call. When Sequelize provides an API like Thing.belongsTo(User), I shouldn’t have to be aware of the fact that a Thing.UserId field is created. So what is the clean API-respecting way of creating a new Thing which refers to a particular User without having to guess the name of the UserId field? When I load a Thing and specify {include: [User]}, I access the loaded user through the thing.User property. I don’t think I’m supposed to know about or try to access a thing.UserId field. In my Thing.belongsTo(User) call, I never specify UserId, I just treat that like an implementation detail I shouldn’t care about. How can I continue to avoid caring about that implementation detail when creating a Thing?
The Thing.create() call that works but looks wrong to me:
Thing.create({
name: 'A thing',
UserId: user.id
});
Option 1 - risks DB inconsistency
Sequelize dynamically generates methods for setting associations on instances, e.g. thing.setUser(user);. In your use case:
sequelize.sync({force: true})
.then(function () {
return Promise.all([
User.create({email: 'asdf#example.org'}),
Thing.create({name: 'A thing'})
]);
})
.spread(function(user, thing) {
return thing.setUser(user);
})
.then(function(thing) {
console.log(JSON.stringify(thing));
});
Option 2 - does not work/buggy
It isn't documented, but from a code dive I think the following should work. It doesn't but that seems to be because of a couple of bugs:
// ...
.then(function () {
return models.User.create({email: 'asdf#example.org'});
})
.then(function(user) {
// Fails with SequelizeUniqueConstraintError - the User instance inherits isNewRecord from the Thing instance, but it has already been saved
return models.Thing.create({
name: 'thingthing',
User: user
}, {
include: [{
model: models.User
}],
fields: ['name'] // seems nec to specify all non-included fields because of line 277 in instance.js - another bug?
});
})
Replacing models.User.create with models.User.build doesn't work because the built but not saved instance's primary key is null. Instance#_setInclude ignores the instance if its primary key is null.
Option 3
Wrapping the Thing's create in a transaction prevents an inconsistent state.
sq.sync({ force: true })
.then(models.User.create.bind(models.User, { email: 'asdf#example.org' }))
.then(function(user) {
return sq.transaction(function(tr) {
return models.Thing.create({name: 'A thing'})
.then(function(thing) { return thing.setUser(user); });
});
})
.then(print_result.bind(null, 'Thing with User...'))
.catch(swallow_rejected_promise.bind(null, 'main promise chain'))
.finally(function() {
return sq.close();
});
I have uploaded a script demo'ing option 2 and option 3 here
Tested on sequelize#6.5.1 sqlite3#5.0.2 I can use User.associations.Comments.foreignKey as in:
const Comment = sequelize.define('Comment', {
body: { type: DataTypes.STRING },
});
const User = sequelize.define('User', {
name: { type: DataTypes.STRING },
});
User.hasMany(Comment)
Comment.belongsTo(User)
console.dir(User);
await sequelize.sync({force: true});
const u0 = await User.create({name: 'u0'})
const u1 = await User.create({name: 'u1'})
await Comment.create({body: 'u0c0', [User.associations.Comments.foreignKey]: u0.id});
The association is also returned during creation, so you could also:
const Comments = User.hasMany(Comment)
await Comment.create({body: 'u0c0', [Comments.foreignKey]: u0.id});
and on many-to-many through tables you get foreignKey and otherKey for the second foreign key.
User.associations.Comments.foreignKey contains the foreignKey UserId.
Or analogously with aliases:
User.hasMany(Post, {as: 'authoredPosts', foreignKey: 'authorId'});
Post.belongsTo(User, {as: 'author', foreignKey: 'authorId'});
User.hasMany(Post, {as: 'reviewedPosts', foreignKey: 'reviewerId'});
Post.belongsTo(User, {as: 'reviewer', foreignKey: 'reviewerId'});
await sequelize.sync({force: true});
// Create data.
const users = await User.bulkCreate([
{name: 'user0'},
{name: 'user1'},
])
const posts = await Post.bulkCreate([
{body: 'body00', authorId: users[0].id, reviewerId: users[0].id},
{body: 'body01', [User.associations.authoredPosts.foreignKey]: users[0].id,
[User.associations.reviewedPosts.foreignKey]: users[1].id},
])
But that syntax is so long that I'm tempted to just hardcode the keys everywhere.