andthennothing.net: Tag writeup

rake migrate

2005-10-22T14:19:00+00:00

This is the first time I’m updating my Typo installation, and the update contains changes to the database schema as well to make things a little bit more interesting.

I start by backing up my MySQL database:

$ mysqldump -C -u username -p --opt --lock-tables=false --skip-add-locks --skip-extended-insert database > dump.sql

I fetch the dump to my laptop and bring it into my local MySQL server (using SQLyog, but could just as well have used MySQL directly). Updating to latest version of Typo.

svk sync --all
svk update andthennothing-dev
svk smerge //typo/mirror andthennothing-dev
svk commit andthennothing-dev

When I try to run Typo now it doesn’t work since the database schema has changed. This is where Active Record’s Migrations come in. It should be as easy as rake migrate, but it wasn’t for me. Not this time. First I try it out on my laptop and since the Typo user doesn’t have all permissions needed some of the migration steps don’t succeed (that’s at least what I think went wrong). However, after changing the permissions of the user, changing schema_info.version to 18 (the version it was before I tried to migrate) and re-restoring the dumped database it works as expected.

Since I don’t want the same problem on my production server I create a test database and checkout the new Typo version in a temporary directory on the server. But it failed in another way:

$ rake migrate RAILS_ENV=test
rake aborted!
WARNING:  You have a nil object when you probably didn't expect it!  Odds are you want an instance of Array instead.

Look in the callstack to see where you're working with an object that could be nil.
Investigate your methods and make sure the object is what you expect!
./rakefile:218

The reason was that the schema_info table somehow was empty, don’t know how it happened. But adding a record with version = 18, the migration works.

After some testing locally of the new version I’m ready to push the new version to the production server. Merge dev to trunk locally:

svk update andthennothing-trunk
svk smerge //andthennothing/branches/dev andthennothing-trunk
svk commit andthennothing-trunk

On the server (I still have the database dump if anything should go wrong):

$ killall lighttpd # I think I'll manage some downtime
$ svn update
$ rake migrate RAILS_ENV=production
Trying to add whiteboard to: articles... comments... pages... done.
Renaming Articles table
Copying article data
Adding a type column
Extending Content table
Converting comments
Extending Content table
Converting trackbacks
Extending content table
Converting pages
Updating all articles
$ #restart lighttpd

Everything seems to work just fine! The update took something like one or two minutes I think, which is quite an acceptable downtime on a personal web site. I will look into how to update without any downtime some other time since I think it’s possible with Rails.

It was a quite painless experience once I figured out how to get around the problems, and I’m glad I tried it out locally first. Rails Migrations rocks! The only objection I have is why it didn’t perform the migration in a transation so when it failed the first time it wouldn’t have screwed up my database?

Update: The transaction problem is fixed by scott.

Relocating svk mirrors

2005-10-22T12:37:00+00:00

When I created my svk mirrors I used http to connect to my svn repository. However since then I activated https support on my server. Due to some stupidity on my part (and how hard it is to debug webserver configuration problems) I didn’t get it to work until this week (thanks to the TextDrive support). To relocate a mirror you should just have to do like this:

C:\>svk mirror --list
Path                    Source
============================================================
//andthennothing        http://urltorepository
//typo/mirror           svn://leetsoft.com/typo/trunk
C:\>svk mirror --relocate //andthennothing https://urltorepository

But I wouldn’t write this if it was that painless. I got this error instead:

C:\>svk mirror --relocate //andthennothing https://urltorepository
Error validating server certificate for 'https://hostname:443':
<snip />
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Type error in argument 2 of svn_auth_cred_ssl_server_trust_t_accepted_failures_set. Expected _p_apr_uint32_t at C:/Development/svk/lib/SVN/Base.pm line 80, <STDIN> line 1.

After a while of searching darix at the #svn helped me out.

C:\>svn ls https://urltorepository
Error validating server certificate for 'https://hostname:443':
<snip />
(R)eject, accept (t)emporarily or accept (p)ermanently? p
branches/
trunk/
C:\>svk mirror --relocate //andthennothing https://urltorepository
Authentication realm: <https://hostname:443> realm
Password for 'user':
Committed revision 633.
C:\>svk mirror --list
Path                    Source
============================================================
//andthennothing        https://urltorepository
//typo/mirror           svn://leetsoft.com/typo/trunk

Yay! So I guess there’s some problems with adding a certificate through svk at least on Windows (more specifically svn’s Perl bindings on Windows). But as soon as the certificate is cached (through svn) everything works fine.

I feel so much secure now using https instead of http :-)

Update: To relocate a svn checkout you just type svn switch https://urltorepository. Easy peasy!

Handling multiple repositories with svn and svk

2005-10-01T19:46:00+00:00

On a day-to-day basis I’m using Perforce at work for my SCM needs. I’ve used CVS a few times back in the days but I’ve tried to forget about it :-). But as I’m migrating to my new web host I have to learn how to use Subversion (svn). The main issue I saw was how to manage multiple branches and repositories.

I asked at the Typo mailing list, and a similar thread appeared on the Rails mailing list, and the general recommendation was to use svk (especially this mail won me over). Other reference material I used was this, that, and the other. This is a writeup of how I went about (it might not be that easy to understand and a little more explanations are probably needed for this to be helpful, but just ask me if there is something that you don’t understand):

Installing svk from here.

svk depotmap --init
svk mkdir //typo
svk mirror svn://leetsoft.com/typo/trunk //typo/mirror
svk sync //typo/mirror
svk co //typo/mirror typo-mirror

Seems to work.

I’m a little bit worried at this stage. I’d like to use the Eclipse plugin subclipse when developing. Perhaps I should only use svk when migrating between different trees. Yes, I think I’ll go for that, and to make it smooth I’ll keep all svk checkouts (svk co --list) in one directory called svk. So let’s relocate:

svk co --relocate typo-mirror svk\typo-mirror

I might change my mind further down the line and want to use svk exclusively, but I don’t expect that to be a problem since svk appears to be quite flexible.

Time to move into svn land. I need to create a trunk and a development tree on my Textdrive repository.

svn mkdir http://urltorepository/trunk
svn mkdir http://urltorepository/branches
svn mkdir http://urltorepository/branches/dev

Time to chechout the svn branches:

svn co http://urltorepository/trunk site-trunk
svn co http://urltorepository/branches/trunk site-dev

Back in svk land.

svk mirror http://urltorepository //site
svk sync //site
svk co //site/branches/dev site-dev

Time to do a baseless merge from Typo to the dev branch:

svk smerge --baseless //typo/mirror site-dev

Since the dev branch was empty before it’s safe to commit the changes without reviewing: svk commit.

Time to update Typo with the changes I’ve done previously. svn update in my site-dev checkout. Firing up a favourite open source utility of mine: WinMerge. Takes a while to merge all the files, but now I’m done, just have to try out if it works alright. It seems to work fine. svn status. Hmm, most of my customizations of Typo is in the views, and looking at this ticket Typo is apparently supporting themed views, have to try it out… Yes it works, neato! svn add on all the new files and svn commit.

Back to svk to integrate to the trunk.

svk sync //site
svk co //site/trunk andthennothing-trunk

Migrating from the dev branch to trunk:

svk smerge --baseless //site/branches/dev site-trunk
svk commit

I think that’s it and hope it will be quite easy to manage in the future. There are two main cycles I’ll be performing. One is getting updates from the Typo repository, and then migrate the changes to the dev branch to the trunk. The other one is migrating changes I’m doing to the dev branch to the trunk. Both of those should be quite easy to manage in svk. I think that the dev branch might be unnecessary, but I’m used to working with a lot of different branches (release, import, team, personal etc.) so that’s how I’m feeling comfortable at the moment. But it will probably be quite easy to remove the dev branch if deemed unnecessary.

Both svn and svk are new and welcomed acquaintances. Working in the command prompt with SCM tools were nicer than I thought and remembered. Hopefully I will have the same opinion in a couple of weeks when I have a little more experience with merging and resolving conflicts (I’m somewhat worried that it will be cumbersome). svn’s poor support for branches and merging is my main pet peeve, but svk handled that nicely. I would go for a svk only (on the client side) if there were a Eclipse plug-in.

Nice learning experience!

MoveableType installation exercise

2005-02-03T20:16:00+00:00

So here is my “guide” to installing MoveableType (MT). I have actually installed MT once before and remeber it beeing somewhat cumbersome. I’m about to install MT not to use it but as an exercise.

So I have a quite freshly installed computer with WinXP Pro. I have installed XAMPP before, which is as easy as it gets. I didn’t get the Perl add-on, so that’s my first step. A hefty download and a lot of small files to install, so I can relax to the sound of the Daily Sourcecode (DSC) podcast… But the install is (almost) one-click, so no worries.

No idea if Perl (and Perl-Apache collaboration) works yet, but there’s no reason why it shouldn’t. So I head off to MT’s site and download… and downloaaaad… Hmm… Their site is really slow. And download the “Limited Free Version” (1 author, 3 weblogs), which will do since I’m just trying it out. Need to register (or “register”) for the download. (And now DSC is over)

Looking at the /docs/mtinstall.html file in the .zip file. Hmmm… I see. By the way, I have a little itch, that I didn’t check if Perl works alright, so I go to http://localhost/xampp/perl-info.php and see that everything seems to work. Now it’s time to continue with the installation.

It’s seems like the Perl installation installed three ways of running Perl scripts. modperl which I guess is quite efficient, modperlasp (Apache::ASP I think) which seems to have a ASP-like syntax, and than old trustworthy CGI. The main difference between modperl and running CGI is that I suspect that modperl fires up one (or more?) instances of Perl which Apache uses for every page view, whereas it has to start a new instance of Perl for each page view when using CGI. I don’t know much about Perl and Apache, but that’s my bet. But since the MT scripts are .cgi I think CGI is the (only?) way to go. As XAMPP’s perl-info.php page states, the CGI directory is \xampp\cgi-bin\. So as I understand it now, the .cgi files should go in to the cgi-bin directory, and the other files should be elsewhere. You could probably also create a new directory and enable CGI on that directory, but let’s at least start with just using the existing cgi-bin directory (I’m not that fond of messing with Apache’s httpd.conf file).

But then again, I’m just installing MT to try it out, and the documentation is a bit vague on which file to place where, so let’s create a new directory \xampp\htdocs\mt and extract everything from the installation .zip file. Then I update the shebang (the first line) of the .cgi files to #!I:\development\xampp\perl\bin\perl -w. Without it, Apache won’t know where to find Perl to execute the scripts.

I will use the MySQL server functionality, which shouldn’t be a problem since XAMPP installed it for me earlier. (You have to do some extra steps here if you use the other options). Hmm… The guide talks something about setting up “[y]our weblog directories”. I guess that’s where the actual blog(s) will be generated (i.e. the output directory/ies). But I skip that step. :-)

So let’s start editing mt.cfg:

CGIPath http://localhost/mt/
ObjectDriver DBI::mysql
Database mt
DBUser username
DBHost localhost

I called the database mt so let’s start PHPMyAdmin and add the database. http://localhost/phpmyadmin/ –> Create new database –> mt –> Create. Done. I think that should be enough, so I save the mt.cfg. And change the database password in mt-db-pass.cgi (you can set the MySQL password in http://localhost/xampp/xamppsecurity.php). I don’t like saving the password in plain text :-(.

Hmm… So now I have to set the permissions on the files. Not really sure what to do with this one. I’m on Windows remember? Not that there are no permissions in Windows, but I skip this step for not.. Hush, don’t tell anyone :-)

Let’s check the mt-check.cgi to see if everything is ok. http://localhost/mt/mt-check.cgi. Yikes. It seems like it worked. “Movable Type System Check Successful”. I thought I had to do something for Apache to execute the script, you see my Apache fu is weak. A few of the optional modules were missing, so I’ll look at that later perhaps.

Now I’m supposed to run mt-load.cgi. Fingers crossed. http://localhost/mt/mt-load.cgi. Finally, an error, everything worked too smoothly.

Bad ObjectDriver config: Connection error: Client does not support authentication protocol requested by server; consider upgrading MySQL client

Hmmm…. Didn’t see this one coming though. Ouch, my Perl fu is weak as well. I wonder if there is anything like irb for Ruby or Python interactive shell.

Found a little script that I changed somewhat:

#!I:\development\xampp\perl\bin\perl -w
use DBI;
my $db;
$db = DBI->connect ("DBI:mysql:host=localhost;database=mt", 
    "username", "password", {PrintError => 0, RaiseError => 1});
$db->disconnect ();
exit (0);

and I got the following error:

File ‘C:\mysql\share\charsets\?.conf’ not found (Errcode: 2)
Character set ’#48’ is not a compiled character set and is not specified in the ‘C:\mysql\share\charsets\Index’ file
DBI connect(‘host=localhost;database=mt’,’username’,...) failed: Client does not support authentication protocol requested by server; consider upgrading MySQL client at mysqltest.cgi line 7

Hmm… Same error message. The easy way would be to give in and use Berkley DB. But not yet. So somewhere there is a hardcoded path to C:\mysql\\share\charsets\?.conf. I don’t even have a drive on the C drive (don’t ask). Looking through the Perl\bin directory I found the script dbish (DBI::Shell) where I get the same error when connecting to MySQL. There is also a mt script, which is kind of weird.

Now it seems like I found something. It seems like DBD::mysql tries to install something to c:\mysql\ by default. And since I don’t have a C drive (don’t ask) I guess something went wrong. So I guess this is quite unique problem for my computer. Most Windows computers have C drives (like 100% – 1). Wonder if I can install it manually…

This is taking waaaay too much time, I was just going to install this quickly, work tomorrow. TheRightThing would be to use Berkley DB and go on with my life. But a Perl installation with a faulty MySQL connection, is like… wrong. Wait a minute. MySQL Reference Manual :: 5.8.7 Problems With Character Sets seems to be my friend this late hour. It seems like my charset is not compiled into the binaries. The number #48 in the error message means latin1_general_ci, which I don’t have compiled support for. From the manual, enter the SQL query “show collation;” to see which charsets are compiled. I changed the following line in \xampp\mysql\bin\my.cnf from “collation-server = latin1_general_ci” to “collation-server = latin1_swedish_ci”. Now I don’t get that error anymore. But I still got the rest of the error (”... upgrading MySQL client”). Ok, I tried to run PPM to upgrade DBD::mysql, but of course PPM doesn’t start because of some version of some module is wrong :-(. That’s all I needed for be able to give in. Bye bye Perl-MySQL.

So now it’s 1:30 a.m. So I head of to bed, and continue tomorrow.

Ok, 16 hours later it’s time to get this thing to work. Let’s (nah, I haven’t used “let’s” too many times yet) back up a bit and go for the Berkley DB option. So first I create a the directory \xampp\htdocs\mt\db, then I comment out the stuff I wrote in mt.cfg and set the DataSource to \xampp\htdocs\mt\db, and remove the mt-db-pass.cgi. I guess it’s time to try running mt-load.cgi again. Fingers crossed… “System Initialization Complete”. Then I’m a nice boy and remove mt-load.cgi. Login at http://localhost/mt/mt.cgi as user Melody, password Nelson. Everything seems to works as it should. “Create Entry”. Enter some text. Save. “Rebuild site”, get the error “Use of uninitialized value in substitution iterator at I:/Development/xampp/htdocs/mt/lib/MT/Util.pm line 146.” It publishes some .html files, but my blog entry is not there. Hmm… Changing some settings for the blog. Creating the archives directory. Still the same problem. Oups, I’m such a noob, as the MSN kids would say. Didn’t change the status of the entry to “Publish”. Now I get a nice blog (but still get the error, but I don’t care since I’m just setting this thing up as an exercise).

So now you just have to set it up, and start blogging. I guess. But this set-up is probably quite insecure (remeber that I disregarded the permissions).

The setup was all in all quite smooth, albeit not a bit fuzzy at times. The main problem for me was, as you can see above, that the Perl-MySQL interface didn’t work. And that is not MT’s fault, it’s what you get when you try to use Windows. I guess that the problem wouldn’t appear, or be as large, had I installed everything manually instead of using XAMPP. But that would take much more time (disregarding my Perl troubleshooting session). However, I can understand people using other blogging engines.