I'm new to the world of data migration and looking into ways to migrate the data of customers using OurApp 1.0 to a new database with a schema compatible with OurApp 2.0. I've seen a more than a few people recommending Liquibase for database change management and database refactoring tasks, which to my newbie ears sounds like it may be close to what we need.
However, after reading through the material on www.liquibase.org, I get the feeling that Liquibase is more about keeping the schema up to date than it is about converting lots of already existing data so that it can be persisted in the new schema.
Say I wanted to split a column in my Employee table called name into a firstname and lastname column. Liquibase would be able to alter the table by dropping the name column and adding a firstname and lastname column. However, I get the feeling Liquibase isn't really built for me to plug in conversion code that would parse the name field of existing records in the database into a firstname and lastname and store those in their respective columns.
For example, say my table looked like this
id | name | position
*********************************
12 Horace Slughorn Professor
13 Albus Dumbledore Headmaster
After I ran Liquibase the name column would be replaced by a firstname and lastname column so my database schema would be correct. But I'm guessing Liquibase is NOT a framework that lets me plug in some code that parses "Horace Slughorn" into "Horace" and "Slughorn" and stores these values in the firstname and lastname columns for that record.
id | firstname | lastname | position
*****************************************
12 Horace Slughorn Professor
13 Albus Dumbledore Headmaster
So Liquibase keeps your schema up to date, but isn't designed to help you convert your existing data so that it it matches the new schema. Is that right?
The goal of Liquibase is to allow you to move your database from one version to the new version. It has many built in database "refactorings", but it does not have one to split your data as you are describing because how you break up existing strings is so dependent on your implementation.
Liquibase does allow you to create custom change classes (http://www.liquibase.org/manual/custom_refactoring_class) in 1.9 as well as a much more powerful custom change support in 2.0 (http://liquibase.org/extensions). It is java based, so your script would need to be written in java, but you can do any data reading and manipulation you need.
Note: if you have a lot of data, reading the records into java, then writing them back out will be slow. In that case, you can use liquibase's tag and create database-based sql statement(s) and/or stored procedure(s) that will do the splitting in-database. Liquibase will still be very helpful to track that the splitting has been done and should not be done again.
You can also use dbdeploy if you want to write custom SQL. Liquibase supports writing custom SQL too, there is DBMaintain or iBatis Migrator.
Related
I have Liferay 6.2.
I have a portlet that use MySql database.
I have a table persons:
Id | Name | Info
Id is auto-increment, so in service.xml i have:
<column name="Id" type="long" primary="true" id-type="increment" />
I have a development machine.
I have 100 rows already available in the mysql table persons.
I entered information within the table persons via a sql query and I used the id starting from 300 up to 600.
I used a Liferay backend tool to update the database cache. When I enter a new row with the application (portlet), the id is 601. It's correct.
I have a production machine, I performed the same operation except for emptying the database cache because for some reason I can not do it.
When I insert a new row the with the portlet id is 101 and not 601.
When will I come to ** id ** 199 what will happen when I insert a new line?
What can I do to solve the problem?
Given that you see the ids on "hundreds", you might just see the default increment value for Liferay entities:
From portal.properties:
# Set the number of increments between database updates to the Counter table.
# this value to a higher number for better performance.
# Defaults:
counter.increment=100
You can test for this behavior by either creating 101 objects (where object #101 wou then get 700 as an id) or by shutting down the server that you used to create the #101 object and restart, then create a new object: The granularity of 100 would mean that the next 100 ids will be allocated and #700 will be picked next.
When will I come to ** id ** 199 what will happen when I insert a new line?
It's easy to find out...
What can I do to solve the problem?
Validate that there is a problem. There might be no need to solve anything
I have a development machine. I have 100 rows already available in the mysql table persons. I entered information within the table persons via a sql query and I used the id starting from 300 up to 600
It's a quite bad idea to do so when you otherwise rely on automatic id construction. Liferay's Service Builder can't rely on id generation in the database, as every database behaves slightly different in that regard.
Liferay's id generation - when it follows the mechanism that I've described above - is done through CounterLocalService. You can make calls to this service to advance the Ids for your entity (use your entity's class name as key) in case you do something manually to your database.
You should never change any Liferay entity on the database directly. If you have your own servicebuilder-generated entities, you might have a point, but my recommendation would be to choose one method of writing/accessing the data and stick with it.
I'm developing an ASP.NET MVC 4.5 project using EF 6 Code First to an Existing Database. I would like to create some new tables with foreign key relationships to one of the tables in the dbcontext I've created. I've altered and added columns in that original table, creating several migrations. There is real data in that table.
I would prefer to create the new tables in the database, but don't see how EF would generate a model for me. I can code the model myself, but don't see any documentation about how I would add it to the context class generated by EF. And then the migrations would be out of whack.
So I'm thinking that the best thing to do would be to delete all the migrations, delete the context class and drop the migrations table. Then I could start from scratch with an initial migration. Am I missing some gotcha? Is there a better way?
FWIW to others facing this dilemma, I figured it out. First I got rid of all the migrations, following the 100+ up-voted answer here: Reset Entity-Framework Migrations
Second, I created new the tables and constraints I needed in the database.
Third, I created a new entity in my solution and generated model classes from the database. I changed calls from the old entity to the new entity.The generator overwrote the model for the original table, but since I have all the annotations in version control, it is trivial to paste them in.
If I need to, I can enable migrations again.
Hope this helps.
With Entity Framework it is possible to enable migrations and create migration steps. But is there an intermediate way where it is possible to change the models, and take care of database schema changes yourself? I don't want to drop the database, because there are future production schenario's.
Now - without enable migrations - I use a code first, and when I create another property in a DbSet - lets assume for example in table 'ExistingTable' int NewField {get; set;}
And when in SQL I update my schema with
Alter table ExistingTable add column NewField int not null
the database knows existence of the new field, the Entity Framework / C# knows the property, but when running, there is some hidden check that still want's to drop my database because of the model change.
Question: can I overwrite a certain setting, in such a way that intial 'Code First' can be transformed to database first?
Removing the __MigrationHistory table from the database (Azure) did work fine for me. I made my (simple) database changes myself and published the code. It all runs fine. There is an alternative see EF Code First Migrations Deployment to an Azure Cloud Service. For a simple one-way patch (and no change history needed) removing the __MigrationHistory works fine.
I have found out that SubSonic SimpleRepository expectes plural table names however I have started working on a database that doesn't have this and I am unable to change the table names.
Is there a way I can use Subsonic without making database changes?
I have seen one suggestion but I don't fancy that too much.
I'm not tied to using the SimpleRepository I just thought it would be easiest as I need the ability to swap database connections (SQL & Oracle) based on the clients requirements. The schema is the same on both. With SimpleRepository I can just swap out the connection string in the web.config.
You can apply the SubSonicTableNameOverride attribute on your classes you use with Simple Repo and use an arbitrary table name!
I have a data model that includes common columns like addedBy, editedby (user), addedDate, editedDate.
Is there a setting I can use in SubSonic 2.1 that will automatically update these fields appropriately instead of having to explicitly specify in every update?
Check out http://subsonicproject.com/setup/subsonic-conventions/. SubSonic uses a convention-based approach when it comes to audit columns. If you can change your column names to CreatedBy, ModifiedBy, CreatedOn, and ModifiedOn respectively, then they will be updated automatically. I don't think there's a way to change what those names can be without making changes to the SubSonic source code.