I want to create a development environment running Movable Type 5.
To create a separate development environment, it is necessary to copy and paste to reflect production.
How would I go about building a good environment?
There are many ways to build a development environment, and an experienced Movable Type developer would need to know more about your goals in order to make a good recommendation.
All of the following guidance assumes that Movable Type has been installed and is ready to run
on the development server.
Here are a few basic tips:
Although some of the key configuration details for a Movable Type instance are kept in mt-config.cgi, there are website-level and blog-level settings that are of equal importance that are kept in the underlying database.
Since most Movable Type 5 instances use MySQL as the database backend, it's possible to dump the entire contents of the Movable Type database using the mysqldump utility or a more visual tool like the Export function of phpMyAdmin. This produces a large text file with MySQL CREATE TABLE and INSERT statements.
Once the database is dumped to a file, the file can be moved to another server, modified, and reconstituted. One of the tasks we commonly perform at that point is to go through the database using an editor, the UNIX sed command, or some similar process, and perform global searches and replacements for the URLs and file system paths that are embedded in the database dump.
This is necessary in many cases because your production website may be http://www.mysite.com/, but your development environment may be http://dev.mysite.com/ or even http://localhost/. Similarly, the file system paths in production may be /var/www/mysite/htdocs/... while development may be /opt/local/apache2/htdocs/mysite/....
Once changes of this nature are made and the modified file is saved, the database is reconstituted on the development server by using a UNIX shell command like:
cat mysite.sql | mysql -u mt_user -p mt_password
Or by importing the database into another copy of phpMyAdmin.
Once all of this is done, the mt-config.cgi file from production needs to be copied into the Movable Type working directory and rewritten so that several important elements are changed:
CGIPath
StaticWebPath
Database
DBUser
DBPassword
DBHost
These Movable Type Configuration Directives are discussed in the online documentation.
All of the non-database assets have to be copied from production to development. Things like files containing jpeg, png, and gif images, files that have been placed in the production file system either manually, or using the Asset Manager. There may be other files that need to be copied from production, depending on how you use Movable Type.
Once all of this is done, and you are able to login to the Movable Type development server successfully, you will probably want to the websites and blogs to ensure that all of the content has been copied into development.
I hope these instructions are somewhat helpful to people needing to setup a development environment. I would be happy to get comments or edits if anyone thinks I left out anything significant.
By saying that you need a development environment for Movable Type, what exactly do you need to develop?
If you are developing a plugin? or theme? a website? content?
It is possible to assign a different mt-config.cgi file to each virtual server, and working on different database, for the same installation.
If you are developing a plugin, you will want to use the PluginSwitch directive so the developed plugin won't be loaded on the real website.
http://www.movabletype.org/documentation/installation/managing-multiple-instances-of.html
Eslar, you may like to consider also this documentation resource:
http://www.movabletype.org/documentation/mt41/rsync.html
Alternatively, you may like to consider:
http://www.cis.upenn.edu/~bcpierce/unison/
If you go with the 'rsync' solution described in the movable type documentation, you may like to check also these configuration directives mentioned there:
http://www.movabletype.org/documentation/appendices/config-directives/rsyncoptions.html
http://www.movabletype.org/documentation/appendices/config-directives/synctarget.html
Related
Can someone explain what's benefits of environment variables in Node.js over regular config file?
In my project I have config.js files with DB details, AWS keys, etc. This file is added to .gitignore and never shared on repository, instead there is demo.config.js file with all required parameters filled with fake creditentials, so you can just copy it as config.js and fill it with correct details after fresh install.
This file is "required" in every file when I need credentials in my project and on my development machine this config file is configured with test server details and with actual production server details on production machine.
Lately I read everywhere that everyone should use environment variables to store credentials safely, but I don't see any benefit to doing so in my project.
I'm not saying it's bad and my approach is better, I just want to know what actual benefit (security or otherwise) will I get with environment variables over my setup?
For me it is more like a common standard than anything else. The way how you use config.js is practically the same as using environment variables. But instead of storing the configuration in environment variables, you store it in js file.
The main difference is how you read that config. All mainstream languages I know, will easily allow you to read from environment variables, there is really wide support for it. Reading from config files brings additional complexity as you need to know the structure of that file, how to parse etc. In some languages (maybe node.js) it is probably easy to read from js file, but in others it could be difficult task.
That's why using environment variables is just a common standard and language agnostic. You can even read it in bash scripts etc.
Edit: adding reference to The Twelve-Factor App, the Config section is particularly connected with above question:
https://www.12factor.net/config
One benefits i see when you are using docker for local development and kubernetes or any container orchestration for SIT/UAT etc where config setting is there . In local development we keep all env variable required and move the same on container based system
Had started my typical EE build (using a bootstrapped config) for a client when they announced they wanted another additional site using the MSM module (le sigh).
So added the MSM module, I commented out the $config['site_url'] and $config['cp_url'] and set those in index.php instead using $assign_to_config.
That's when I discovered this bug where MSM config file settings are not recognized, which is a pain but I can work around it. However, I noticed that when I created the secondary site, it wouldn't recognise my custom location for add-ons and so I had to add that to index.php as well to $assign_to_config['third_party_path'] = "../assets/third_party/";.
Then I discovered that when I create or modify a template file, it won't automatically sync and so I need to manually do that each time which is a real PITA.
Why would my templates not be syncing to the database? Is this related to the MSM config bug?
While I haven't tried bootstrapping the third party path yet, I've definitely been able to bootstrap the template path for MSM sites... What bootstrap method are you using?
Are your sites on subdomains or subfolders? I've only had experience with subfolders so perhaps that makes a difference (although it shouldn't).
Could you maybe walk through in a bit more detail what's happening? Your first site (site_id = 1) templates sync automatically from filesystem edits, but your second site does not? Yet if you go to CP > Design > Synchronize Templates, that works?
The $assign_to_config portion of MSM setup is definitely a weakspot when it comes to bootstrapping... I wonder if we need to work up an additional bootstrap for MSM+CP environment, where it looks at the cp cookie ($_COOKIE['exp_cp_last_site_id']), and sets values based on that.
It may be helpful if you let us know which bootstrap you are using. For example, if you look at this bootstrap the site_url and cp_url are set using the HTTP_HOST server variable, so this shouldn't clash with your MSM install (and multiple domains) at all.
Perhaps you could try using that boostrap file instead, and see if it fixes your issue with template syncing?
Finally, if you're going to use the EE template manager, you don't really need to store templates as files. Conversely, if you want to save templates as files, it's probably much easier editing them using Sublime Text or another editor, rather than the clunky built-in editor (which is really only useful for small/simple changes).
What are the best steps to take to prevent bugs and/or data loss in moving servers?
EDIT: Solved, but I should specify I mean in the typical shared hosting environment e.g. DreamHost or GoDaddy.
Bootstrap config is the smartest method (Newism has a free bootstrap config module). I think it works best on fresh installs myself, but ymmv.
If you've been given an existing EE system and need to move it, there are a few simple tools that can help:
REElocate: all the EE 2.x path and config options, in one place. Swap one URL for another in setup, check what's being set and push the button.
Greenery: Again, one module to rule them all. I've not used this but it's got a good rating.
So install, set permissions, move files and and DB, and then use either free module. If you find that not all the images or CSS instantly comes back online, check your template base paths (in template prefs) and permissions.
I'm also presuming you have access to the old DB. If not, and you can't add something simple like PHPMyAdmin to back it up, try:
Backup Pro(ish): A free backup module for files and db. Easy enough that you should introduce it to the site users (most never consider backups). All done through the EE CP. The zipped output can easily be moved to the new server.
The EE User Guide offers a reasonably extensive guide to Moving ExpressionEngine to Another Server and if you follow all of these steps then you will have everything you need to try again if any bugs or data loss occur.
Verify Server Compatibility
Synchronize Templates
Back-up Database and Files
Prepare the New Database
Copy Files and Folders
Verify File Permissions
Update database.php
Verify index.php and admin.php
Log In and Update Paths
Clear Caches
As suggested by Bitmanic, a dynamic config.php file helps with moving environments tremendously. Check out Leevi Graham's Config Bootstrap for a quick and simple solution. This is helpful for dev/staging/prod environments too!
I'd say the answer is the same as any other system -- export your entire database, and download all of your files (both system and anything uploaded by users - images, etc). Then, mirror this process by importing/uploading to the new server.
Before I run my export, I like to use the Deeploy Helper module to change all of my file paths in EE to the new server's settings.
Preventing data loss primarily revolves around the database and upload directories.
Does your website allow users to interact with the database? If so at some point you'll need to turn off EE to prevent DB changes. If not that you don't have too much to worry about as you can track and changes on the database end between the old and new servers.
Both Philip and Derek offer good advice for migrating EE. I've also found that having a bootstrap config file helps tremendously - especially since you can configure your file upload directories directly via config values now (as of EE2.4, I think).
For related information, please check out the answers to this similar Stack Overflow question.
In a typical enterprise scenario with in-house development, you might have dev, staging, and production environments. You might use SVN to contain ongoing development work in a trunk, with patches being stored in branches, and your released code going into appropriately named tags. Migrating binaries from one environment to the next may be as simple as copying them to middle-ware servers, GAC'ing things that need to be GAC'ed, etc. In coordination with new revisions of binaries, databases are updated, usually by adding stored procedures, views, and adding/adjusting table schema.
In a Sharepoint environment, you might use a similar version control scheme. Custom code (assemblies) ends up in features that get installed either manually or via various setup programs. However, some of what needs to be promoted from dev to staging, and then onto production might be database content that supports the custom code bits.
If you've managed an enterprise Sharepoint environment, please share thoughts on how you manage promotion of code and content changes between environments, while protecting your work and your users, and keeping your sanity.
I assume when you talk about database content you are referring to the actual contents contained in a site a or a list.
Probably the best way to do this is to use the stsadm import and export commands to export and import content from one environment to another. (Don't use backup/restore when going from one environment to another.)
For any file changes (assemblies, aspx) you can use Features and then keep track of the installers. You would install the feature and do an upgrade to push changes.
There's no easy way to sync the data...you can use stsadm import/export commands as John pointed out. But this may not be straight-forward, especially if the servers are configured differently.
There's also Data Sync Studio product (http://www.simego.net/DataSync_Studio.aspx) you can try.
Depending on what form the database content takes, I would keep the creation of it in code so it's all in one place (your Visual Studio project) and can also be managed via source control. Deployment of the content could either be via a console application or even better feature receiver.
You might also like to read this blog post and look at the tool mentioned there for another approach.
The best resource I can point you to is Eric's paper:
http://msdn.microsoft.com/en-us/library/bb428899.aspx
I was part of a team working to better the story around development of WSS and MOSS solutions with TFS, but I don't know where that stands.
I'm looking for something that can copy (preferably only changed) files from a development machine to a staging machine and finally to a set of production machines.
A "what if" mode would be nice as would the capability to "rollback" the last deployment. Database migrations aren't a necessary feature.
UPDATE: A free/low-cost tool would be great, but cost isn't the only concern. A tool that could actually manage deployment from one environment to the next (dev->staging->production instead of from a development machine to each environment) would also be ideal.
The other big nice-to-have is the ability to only copy changed files - some of our older sites contain hundreds of .asp files.
#Sean Carpenter can you tell us a little more about your environment? Should the solution be free? simple?
I find robocopy to be pretty slick for this sort of thing. Wrap in up in a batch file and you are good to go. It's a glorified xcopy, but deploying my website isn't really hard. Just copy out the files.
As far as rollbacks... You are using source control right? Just pull the old source out of there. Or, in your batch file, ALSO copy the deployment to another folder called website yyyy.mm.dd so you have a lovely folder ready to go in an emergency.
look at the for command for details on how to get the parts of the date.
robocopy.exe
for /?
Yeah, it's a total "hack" but it moves the files nicely.
For some scenarios I used a freeware product called SyncBack (Download here).
It provides complex, multi-step file synchronization (filesystem or FTP etc., compression etc.). The program has a nice graphical user interface. You can define profiles and group/execute them together.
You can set filter on file types, names etc. and execute commands/programs after the job execution. There is also a job log provided as html report, which can be sent as email to you if you schedule the job.
There is also a professional version of the software, but for common tasks the freeware should do fine.
You don't specify if you are using Visual Studio .NET, but there are a few built-in tools in Visual Studio 2005 and 2008:
Copy Website tool -- basically a visual synchronization tool, it highlights files and lets you copy from one to the other. Manual, built into Visual Studio.
aspnet_compiler.exe -- lets you precompile websites.
Of course you can create a web deployment package and deploy as an MSI as well.
I have used a combination of Cruise Control.NET, nant and MSBuild to compile, and swap out configuration files for specific environments and copy the files to a build output directory. Then we had another nant script to do the file copying (and run database scripts if necessary).
For a rollback, we would save all prior deployments, so theoretically rolling back just involved redeploying the last working build (and restoring the database).
We used UnleashIt (unfortunate name I know) which was nicely customizable and allowed you to save profiles for deploying to different servers. It also has a "backup" feature which will backup your production files before deployment so rollback should be pretty easy.
I've given up trying to find a good free product that works.
I then found Microsoft's Sync Toy 2.0 which while lacking in options works well.
BUT I need to deploy to a remote server.
Since I connect with terminal services I realized I can select my local hard drive when I connect and then in explorer on the remote server i can open \\tsclient\S\MyWebsite on the remote server.
I then use synctoy with that path and synchronize it with my server. Seems to work pretty well and fast so far...
Maybe rsync plus some custom scripts will do the trick.
Try repliweb. It handles full rollback to previous versions of files. I've used it whilst working for a client who demanded its use and I;ve become a big fan of it, partiularily:
Rollback to previous versions of code
Authentication and rules for different user roles
Deploy to multiple environments
Full reporting to the user via email / logs statiing what has changed, what the current version is etc.