Is there any tool that works similar to Django South, but for Node?
Now I'm working with Sequelize. If I got it right, Sequelize does not have an option to create migration files based on existing models.
So, to create a new model/table, the steps are:
Create model with sequelize model:create <model meta>.
Edit generated migration file - add actual code for creating tables
in DB under up section.
Run migration with sequelize db:migrate.
I'm looking for something that can create migration files based on existing models, manage it similar to what South can do for Django.
Is there any option?
I have written a step-by-step guide on how to auto-create migrations with Sequelize in another post. Here is a summary...
The closest thing with Sequelize is
Sequelize Auto Migrations.
It allows you to have an iteration cycle like the following:
Create/update model -- by hand or with sequelize-cli)
Run makemigrations to auto-generate up and down migrations
Repeat as necessary
While this is very helpful, I've found it to be lacking in some critical areas:
The down migrations can be created incorrectly. So it may try to drop a table before its dependent tables have been dropped.
There are certain configurations around multi-field indexes that it has not correctly output.
There are currently 10 outstanding PRs, so it seems like a few additional contributors are attempting to make it more production-ready... but I've yet to find anything as clean and reliable as Django Migrations (formerly Django South).
TypeORM supports model based migrations. It can sync your db to your models directly, it can also create migration files too.
I think prisma is another option. It doesn't seem like that popular but it's a promising one.
Either way, it's just ridiculous that there are no solid tools for this. I've worked on django and .net projects in the last years and creating migrations is just easy with them. But when you try to use node.js for backend, you get stuck at a lot of points.
I was using sequelize and when I saw there was no official way to create automatic migrations from models, I've dropped using it. Maintaining your models with manually written migrations becomes very hard in my experience.
Now my only alternative is TypeORM and it just bugs me there is no other alternative in case TypeORM just goes unmaintained or if I want to use another library etc.
I'm seriously thinking dropping using node.js for backend. Yet, there are good tools to create projects integrated with modern front-end tools (like Next.js), finding a good orm is a big problem.
Take a look at https://typeorm.io/#/migrations/generating-migrations. I am in the same situation you was 4 years ago.
My options:
Waterline just for ORM and diff tool (like dbdiff) to make a file with differences from a new schema (generated by waterline migration with 'drop') vs production schema. With that output, you run query by query in a safe mode.
Prev option plus knex migrations. But you have to make your own migrations files. Knex doesnt have a schema file to compare but there is a request feature https://github.com/knex/knex/issues/1086.
Using sails but change waterline for sequalize and give a try at the #paulmest answer.
Use sails but change waterline for typeorm an use that auto generated.
Over the years, my team has tried Sequelize and TypeORM, but neither of them were great experiences. TypeORM had tons of cryptic problems and thousands of issues are unaddressed/open for years on end. Sequelize was generally fine, but lack of type support made it time-consuming to work with.
For the last 1.5 years, my team has been using Hasura. It's honestly been a breath of fresh air. Hasura's migration system is pretty barebones compared to Django South, but it is straight-forward and has a clean happy path.
You can use the Hasura Console (a localhost web editor) to create tables and add/remove columns. Those changes will be automatically distilled into schema changes stored in .sql files in a migration directory. You can modify these files. You can run a migration command from the command line to apply them when you want.
Since Hasura is a GraphQL engine, it also comes with a schematized SDK that is TypeScript compatible so you get incredible editor support.
All of this and much more is available in the open source version of their product. We have not needed to pay for any higher tier.
You can find more information here: https://hasura.io/docs/latest/graphql/core/migrations/index.html
Related
Most of my dev experience is based on Ruby on Rails. The framework supports having a DB schema in two formats:
RoR DSL
SQL for cases when DSL is not enough. For instance, having an initially deffered unique constraint in PostgreSQL.
If it's needed to set up a DB from scratch, for instance in CI, it's possible to run a CLI task that will use either of the files and apply it without any further need to run migration files.
About two weeks ago we started a project that based on ExpressJS + PrismaJS and now we need to have a custom SQL for the DB structure.
After reading the Prisma docs I found that it's possible to write a custom SQL inside migration files and this is exactly what we need for our production. However, we also would like to have the same DB schema in our CI. Is there any other way to have the same schema for CI as we have in production without running migration files one by one as I can do with RoR?
Apologies in advance as I am unable to find a good solution regarding this.
Background: We are using Typeorm in express app and have too many migrations because its a little old project.
Question: Is there a good way to manage migrations? Ideally I would want to delete all migrations(generated till now) and have only single file for schema creation(updated automatically after every migration run) and another file for seeding master data. Using these two files any new developer can setup the database locally in no time.
I worked in Ruby on rails and there used to be a clean approach like ruby maintains the database schema all the time in schema.rb and database could be generated using that.
I researched a lot but couldn't find a maintainable solution. May be the experts here could help me.
Thanks in advance
All your schemas describe your data model at current time, you can just delete all migrations from code and database, it will be as if you had no migrations at all. The problem will be only with old versions of your database which needs to be migrated.
We're using Contentful CMS, and are interested in programmatically migrating our schema. Contentful has a migration guide available here - https://www.contentful.com/developers/docs/tutorials/cli/scripting-migrations/
The guide describes how to write migration scripts and run them against Contentful to alter its schema. However, it doesn't provide guidance around how the scripts should be run. After a migration is executed, it seems like you have to manually remember to never run it again (if executed again it will produce errors). This makes automating the schema changes difficult. Are there best practices that exist around this?
Heyooo, Contentful DevRel here.
We have a blog post about that.
What it describes is that you could set up a content type that holds the version of the last ran migration. After you ran this migration you could then update then entry and apply some logic.
When you're running bigger migrations you could also have a look at environment aliases which allow you to promote environments to master after a successful migration. This gives you much flexibility and safety.
Hope that helps. :)
Both ActiveRecord (from Rails) and Sequelize (another node.js ORM) provide a way to initialize a database, creating the table structures from the model definitions. Rails does that through the rails db:schema:load command, while Sequelize has the sync() method that does the same. By using that, we don't need to run the entire migration stack of the application to start a fresh database, neither save SQL dumps on the project repository.
Coming from this background, I was expecting Bookshelf.js or Knex.js to have some kind of similar functionality, but I couldn't find it on the documentation of both projects.
I decided then to take a look at the source code of the Ghost blogging engine, which uses Bookshelf, and I found out that they treat the database initialization inside their own codebase:
https://github.com/TryGhost/Ghost/blob/e40290a/core/server/data/schema/schema.js
https://github.com/TryGhost/Ghost/blob/e40290a/core/server/data/migration/populate.js
https://github.com/TryGhost/Ghost/blob/e40290a/core/server/data/schema/commands.js
I'd like to avoid having to write my own code to treat things like this specially because other options like Sequelize offer this out of the box.
Is there any common practice, plugin or library recommended for database schema loading on Bookshelf?
I think you were on the right track and maybe just missed the docs.
http://knexjs.org/#Migrations
For loading a schema, you can run the migration stack if the migrations are written to be idempotent.
Another option is to use the database's export and import features. For example: https://www.postgresql.org/docs/9.1/static/app-pgdump.html
Rails achieves its migrations by having two rake tasks db:schema:dump and db:schema:load. Dump will dump your db schema to a local schema.rb file in a custom ruby format. db:schema:load loads that file.
So you could achieve something similar using pg_dump and pg_restore and an npm package script. Just make the script use node's exec method to call pg_dump and dump to the same file every time.
I just lost 11,000 records from my database just running the command for sailsjs without the --prod part in it, So I thought I should ask whats the best way to change the tables on production server when the Model.js has been changed ?
Thanks
Automated migration should never be done in production. This a common-sense practice that applies to any production system with important data. There are a few solutions available for migrating a sails.js database.
sails-db-migrate: db-migrate integration for sails.js
db-migrate integration for Sails.js. This is a fairly simple wrapper, which provides grunt tasks for running and creating migrations.
sails-migrations: The missing, migrations arm of the octopus
sails-migrations provides an easy way to manage database migrations with sails, based on the amazing https://github.com/tgriesser/knex lib. This means you can have fine-grained control over your schema/data transformations between versions.
Sequelize migrations
Sequelize 2.0.0 introduces a new CLI which is based on gulp and combines sequelize-cli and gulp-sequelize. The CLI ships support for migrations and project bootstrapping. With migrations you can transfer your existing database into another state and vice versa