I just lost 11,000 records from my database just running the command for sailsjs without the --prod part in it, So I thought I should ask whats the best way to change the tables on production server when the Model.js has been changed ?
Thanks
Automated migration should never be done in production. This a common-sense practice that applies to any production system with important data. There are a few solutions available for migrating a sails.js database.
sails-db-migrate: db-migrate integration for sails.js
db-migrate integration for Sails.js. This is a fairly simple wrapper, which provides grunt tasks for running and creating migrations.
sails-migrations: The missing, migrations arm of the octopus
sails-migrations provides an easy way to manage database migrations with sails, based on the amazing https://github.com/tgriesser/knex lib. This means you can have fine-grained control over your schema/data transformations between versions.
Sequelize migrations
Sequelize 2.0.0 introduces a new CLI which is based on gulp and combines sequelize-cli and gulp-sequelize. The CLI ships support for migrations and project bootstrapping. With migrations you can transfer your existing database into another state and vice versa
Related
Most of my dev experience is based on Ruby on Rails. The framework supports having a DB schema in two formats:
RoR DSL
SQL for cases when DSL is not enough. For instance, having an initially deffered unique constraint in PostgreSQL.
If it's needed to set up a DB from scratch, for instance in CI, it's possible to run a CLI task that will use either of the files and apply it without any further need to run migration files.
About two weeks ago we started a project that based on ExpressJS + PrismaJS and now we need to have a custom SQL for the DB structure.
After reading the Prisma docs I found that it's possible to write a custom SQL inside migration files and this is exactly what we need for our production. However, we also would like to have the same DB schema in our CI. Is there any other way to have the same schema for CI as we have in production without running migration files one by one as I can do with RoR?
In Typeorm there is a feature called synchronize. You can synchronize entities with a database, so there is no need for migirations. But as you know synchronize is dangerous for production.
Here is the question, when should I use the synchronize feature? Imagine at first (in the development environment) I started using the synchronize feature. If I disable it in production while I have no migration, how should my production database will going to be created?
Also, I'm going to deliver the project on some milestones. Should I disable it at the first milestone or at the end? And for long time maintenance, should I use synchronize disabled and use migration after the first production release?
Any idea would be appreciated.
Migrations in TypeORM
Even Though synchronization is a good option to synchronize your entity with the database, it is unsafe for production databases. Therefore migrations can be an alternative solution for safer migrations in production databases.
When doing a migration, you should follow the below steps.
1. Update the Typeorm config file and package.json file
You should change the synchronize attribute to false in Typeorm config file as the first step to prevent schema synchronization.
Then add the following command to the scripts attribute under the package.json file.
“typeorm”: “ts-node ./node_modules/typeorm/cli -f ./ormconfig.json”
2. Generate the migration
npm run typeorm migration:generate -n
Here you can give a name to your migration. After you run the command you will find a migration file under migrations with the name .
In the migration file, there are two functions namely up and down where up function responsible for running the migration and down for reverting the migration.
3. Run the migration
npm run typeorm migration:run
This command will run the migration which you have already created in the above command. When you run this command, it will execute the up function in the migration file.
4. Revert the migration
npm run typeorm migration:revert
This command will revert the migration which you have already executed in the above command. When you run this command, it will revert all the migrations which you have already done. Basically, it will run the down command of the migration file.
Synchronize is a great option to get up an running, but in my opinion you should always default to creating migrations. This is because it will enforce you to run your development environment similar to production, which is always key. You want to make your Dev environment run like production.
migration:generate is a great middle ground to building your migration files from your entities.
This was my question too, so I searched for it. I found out that as the documentation says:
Once you get into production you'll need to synchronize model changes into the database. Typically, it is unsafe to use synchronize: true for schema synchronization on production once you get data in your database. Here is where migrations come to help.
We can realize that once you have valuable data in your production database, you should turn synchronization off forever and start using migrations on development and production
source
I also have to decide that issue now. And i will use the syncronize option only for my e2e test db. As Roger King already mentioned, you want to have your dev and prod environments using the same tools to change the database. In this way you can prevent different behaviors between them.
I am trying to set up a NodeJS backend with a MySQL database. I want to eventually containerize the database and containerize the backend server. To containerize the database, I need migration files.
I am looking for a framework that handles the generation of these migrations well. I have tried Sequelize, but you couldn't update your migrations based on your models.
I wanted to use AdonisJS but it was a full MVC framework, while the goal is to use it alongside Express.
Am I thinking the wrong way here?
Actually you can generate migrations from models in sequelize using sequelize-auto-migrations.
Is there any tool that works similar to Django South, but for Node?
Now I'm working with Sequelize. If I got it right, Sequelize does not have an option to create migration files based on existing models.
So, to create a new model/table, the steps are:
Create model with sequelize model:create <model meta>.
Edit generated migration file - add actual code for creating tables
in DB under up section.
Run migration with sequelize db:migrate.
I'm looking for something that can create migration files based on existing models, manage it similar to what South can do for Django.
Is there any option?
I have written a step-by-step guide on how to auto-create migrations with Sequelize in another post. Here is a summary...
The closest thing with Sequelize is
Sequelize Auto Migrations.
It allows you to have an iteration cycle like the following:
Create/update model -- by hand or with sequelize-cli)
Run makemigrations to auto-generate up and down migrations
Repeat as necessary
While this is very helpful, I've found it to be lacking in some critical areas:
The down migrations can be created incorrectly. So it may try to drop a table before its dependent tables have been dropped.
There are certain configurations around multi-field indexes that it has not correctly output.
There are currently 10 outstanding PRs, so it seems like a few additional contributors are attempting to make it more production-ready... but I've yet to find anything as clean and reliable as Django Migrations (formerly Django South).
TypeORM supports model based migrations. It can sync your db to your models directly, it can also create migration files too.
I think prisma is another option. It doesn't seem like that popular but it's a promising one.
Either way, it's just ridiculous that there are no solid tools for this. I've worked on django and .net projects in the last years and creating migrations is just easy with them. But when you try to use node.js for backend, you get stuck at a lot of points.
I was using sequelize and when I saw there was no official way to create automatic migrations from models, I've dropped using it. Maintaining your models with manually written migrations becomes very hard in my experience.
Now my only alternative is TypeORM and it just bugs me there is no other alternative in case TypeORM just goes unmaintained or if I want to use another library etc.
I'm seriously thinking dropping using node.js for backend. Yet, there are good tools to create projects integrated with modern front-end tools (like Next.js), finding a good orm is a big problem.
Take a look at https://typeorm.io/#/migrations/generating-migrations. I am in the same situation you was 4 years ago.
My options:
Waterline just for ORM and diff tool (like dbdiff) to make a file with differences from a new schema (generated by waterline migration with 'drop') vs production schema. With that output, you run query by query in a safe mode.
Prev option plus knex migrations. But you have to make your own migrations files. Knex doesnt have a schema file to compare but there is a request feature https://github.com/knex/knex/issues/1086.
Using sails but change waterline for sequalize and give a try at the #paulmest answer.
Use sails but change waterline for typeorm an use that auto generated.
Over the years, my team has tried Sequelize and TypeORM, but neither of them were great experiences. TypeORM had tons of cryptic problems and thousands of issues are unaddressed/open for years on end. Sequelize was generally fine, but lack of type support made it time-consuming to work with.
For the last 1.5 years, my team has been using Hasura. It's honestly been a breath of fresh air. Hasura's migration system is pretty barebones compared to Django South, but it is straight-forward and has a clean happy path.
You can use the Hasura Console (a localhost web editor) to create tables and add/remove columns. Those changes will be automatically distilled into schema changes stored in .sql files in a migration directory. You can modify these files. You can run a migration command from the command line to apply them when you want.
Since Hasura is a GraphQL engine, it also comes with a schematized SDK that is TypeScript compatible so you get incredible editor support.
All of this and much more is available in the open source version of their product. We have not needed to pay for any higher tier.
You can find more information here: https://hasura.io/docs/latest/graphql/core/migrations/index.html
I am exploring writing a Sails.js application that talks to a mongodb data store. I've got it all working just fine but I'd like to be able to add some sort of initialisation process that runs at the npm install point that runs analogously to rake admin:create in the Rails world.
I'm guessing that this could be done by via a grunt task somehow, but I figured, before I roll my own solution, I'd ask other Sails / MongoDB users what they actually do in the real-world and see if a consistent, or best-practice solution has emerged.
Database initialization for Sails apps is typically done in the config/bootstrap.js file. A simple one would be something like:
module.exports = function(cb) {
User.findOrCreate(
// Search for user with "admin" flag
{admin: true},
// Create one if no such user is found
{admin: true, name: 'Admin User', ...}
).exec(cb);
}
The bootstrap will run only when Sails is lifted, and the server won't start until the callback is executed. If you send anything as the first argument to the callback, it'll be considered an error (like most standard Node callbacks) which will halt lifting of the server.
Note that you can't create the actual database with Waterline. A Grunt task would be a fine place for that, but you have to consider that Grunt doesn't have access to the Sails config, so you'd have to hard-code the name of the database to use, or find some other way to make it configurable. This could have implications for different environments (i.e. development vs. production). In our projects at Balderdash, we typically leave database creation as a manual step for the developer after they check out the code, and then use bootstrap.js to initialize the data. This allows the developer to maintain their own database settings using config/local.js.
All that being said, I can see the value in setting up a Grunt task for creating a db for production, to make it easier to deploy your Sails app to its final destination.
I personally think it was a wise decision by the sails.js team to use Grunt. Grunt provides a reliable and flexible solution to whatever automation you need.
I use grunt for everything in my sails.js projects including....
Importing data from legacy databases
Compiling sass (even though sails uses less by default)
Compiling coffeescript
Assembling knockout templates
Loading indexes into elasticsearch
You wont be disappointed using Grunt, It's pretty easy to get started with....So get to it!