svn:externals with WordPress

I’ve mentioned this a few times in conversations, so I’ve been meaning to do a blog post on this for a while, but I really wanted to tidy up the theme before I got back into posting more, so now that I’ve done that, here goes…

This is an example of how to use the “externals” feature of Subversion in order to simplify the maintenance of your software project when it contains references to other libraries or projects that are hosted on their own repositories, and you want to keep them all updated from their respective sources without having to do any tedious copying / exporting, etc.

This particular long, rambling example will use a somewhat typical WordPress installation, with a couple minor organizational tweaks to keep things simple and clean.

Working Backwards

I’ll start by laying out what the end result will look like, and then walk through the steps of how to get there.

The goal in this case is to have a single SVN repo (under your control) which you can use to manage your complete package (in this case a WP install), but internally have it contain your own stuff but also pull from several different “external” SVN repositories (WP core + various separate plugin and theme repos) that you don’t need to manage or copy things from.

The obvious benefit here is that you end up with a single SVN source that you can put on your server(s) with a single svn co http://your.repo.url and keep updated with a simple svn up or svn switch http://your.repo.url/tags/1.2.3 if you use tags to manage your releases (which I recommend, but that’s a topic for another post).

Basic Directory Structure

This is the part that may be a bit different than you’re used to, but I find this structure to be much easier to work with when doing stuff like this, and recent versions of WordPress have supported this (moving wp-content and wp-config.php somewhere other than their standard locations) for quite a while.

So assuming we’re starting with a blank directory that is a local checked out copy of your trunk/ directory, here is what the end result will look like once we’re done (just so you’re ready). Assuming we’re under something like /var/www/yoursite/, you’d have three items in that directory:
*NOTE: See this comment below about making sure you start with an empty dir, and that the local target dirs you reference in externals do not exist before SVN creates them by loading the external.

core/
wp-content/
wp-config.php
Where

  • core will be a copy of the WordPress core files, linked via an svn:externals reference to the latest tagged release of WordPress, and fetched automatically from their servers. This is the dir that your apache vhost should map to as its root.
  • wp-content is where all your themes and plugins (among other things) will go. This will contain a mix of plugins and themes from other sources (via svn:externals) plus whatever themes and plugins we add to this repo itself.
  • wp-config.php This will be a copy of your WordPress config file. I usually keep it on it’s own out here, and exclude it from the subversion repository by adding it to svn:ignore. That’s perhaps another post for another time, but I just don’t add it to the repo since my local and server passwords are different, and I don’t want either sitting out in a repo that I may or may not want to make public at some point. Also it helps to stick to the rule that you don’t need to modify any files (other than .htaccess) under the core directory.

OK, so far so good. If you are not cool with that project layout, it’s possible to use other structures, but I wouldn’t recommend it, and won’t answer any questions supporting it. :-)

Step by Step – How To Get There

So, starting in a blank directory (checked out copy of your trunk, which should be empty at this point), first you want to set your SVN_EDITOR if you haven’t already.
export SVN_EDITOR=vi, substituting vi for your text editor of choice.

Step 1 – Core WP files

Within this directory, type the following command to add the first external reference to WordPress itself:
svn propedit svn:externals .and add the following (single) line of text:
core http://svn.automattic.com/wordpress/tags/2.8.2then save and close the file.

To get a preview of the magic now at work, go ahead and check in your changes and then update:
svn ci
svn up
You should notice that when your local copy goes to update from your SVN repo it now also knows to go pull down the contents from the Automattic WP repo you linked to and put them in a directory called “core”.

Step 2 – wp-content

Now that we have the core dir handled, we want to make a wp-content dir, under which we’ll put all of our custom content, in addtion to other external plugins and themes.

Again assuming you are in the same directory as above (which should now only contain the “core” directory, issue the following command to get a clean copy of the wp-content that was retrieved from WP:svn export core/wp-content wp-content* Do not just do a normal copy here, because that will copy the SVN info from the dir as well, which will screw you up.

Now we’ll add a couple themes from other external repos. svn propedit svn:externals .and append the following two lines of text (after the existing core line):
wp-content/themes/redoable-lite http://svn.automattic.com/wpcom-themes/redoable-lite

wp-content/themes/sandbox http://sandbox-theme.googlecode.com/svn/trunkthen save and close the file.

svn ci
svn up
and we will see that those two themes right where they belong inside of our wp-content directory.

I could make this post even longer by illustrating linking to additional external plugins, etc., but it would work the same way as what I just did for themes. I’m sure you’ve got the hang of that by now.

All your other changes within this wp-content directory (your own themes and plugins, etc.) will be handled just like you’re used to with SVN; it’s just that now you’ll also have other content in there that you don’t need to manage yourself.

Step 3 – wp-config

This step isn’t strictly related to the svn:externals thing, but you’ll need it in order to follow the structure I laid out above.

cp core/wp-config-sample.php wp-config.php This is now your official config file, and you’ll need to edit it as normal to point it at your database, etc.

WP is smart enough to find this file here (assuming you don’t have one under the core dir), but you’ll also just want to add an entry in that wp-config file to tell it to use the alternate wp-content directory we were working with in step 2, instead of the one under the core directory. In wp-config, add the line:
define( 'WP_CONTENT_DIR', '/var/www/yoursite/wp-content' );
Depending on your configuration, you may or may not also need to add an entry for ‘WP_CONTENT_URL’, but I wouldn’t bother with that unless you find you need it. You may also want to add Alias /wp-content /var/www/yoursite/wp-content to your Apache vhost config as well.

Upgrading – AKA The Payoff

I should start by clarifying that I do not use the built in WP updating mechanisms for core WP or plugins, because I prefer to manage everything from SVN this way.

I find it easier to work with anyway, and it also has the added benefits of supporting more restrictive file permissions on the server, as well as allowing you to work with and test a local copy that you know is exactly the same combination of core files, plugins, etc. as what you will eventually put on your server, because you are pulling it all from the same place: your SVN repo.

Let’s say WP releases 2.8.3. You will simply svn propedit svn:externals .and change the URL reference for the core dir to point to the 2.8.3 release:
core http://svn.automattic.com/wordpress/tags/2.8.3then save and close the file. You can also change any other externals refs (plugins & themes) to their latest versions.
svn up will then update your local copy with all the latest references from your external sources. Test everything out locally, and then once you are happy svn ci and your latest config is checked in, ready to be updated on your server, via svn up or svn switch.

Closing Remarks

If you’re more of a hands on learner and want to follow along but also explore a reference, here is the trac view of the trunk of the repo I was using as a demo for this article, and the corresponding SVN repo itself as well.

Thanks for taking the time to read this (assuming anyone got this far). Any questions?

This entry was posted in Blog Posts. Bookmark the permalink.

20 Responses to svn:externals with WordPress

  1. Dan says:

    I forget if it’s new in SVN 1.5 but you should mention the version that externals started as a new feature. I know by experience it’s a big problem if you’re working with other developers whom an unsupported SVN version.

  2. JB says:

    Good thought, although the syntax I list here will work with all released versions of Subversion, with the possible exception of pre-1.0, which no one in their right minds should be using today.

    Subversion 1.5 did introduce some new features and syntax related to the externals property, specifically support for referencing specific revision numbers and using relative URLs, so it is important to be aware of that if you want to dive deeper into the world of svn:externals.

    That being said, I’ve never used those new features, and the examples I list in this post should be fine in any version of Subversion, as far as I’m aware.

  3. Dan says:

    Actually, if you have that setup with someone trying to checkout the same repo with 1.4 s/he will get errors and the external will never get checked out. But they could check it out normally and just make sure not to check it in (ignore).

  4. JB says:

    Are you sure about that? Sounds like a weird bug to me, but if it does occur, I suspect that it has something to do with conflicting version numbers between the server and multiple clients (some old and some new).

    In other words, if the server and all clients are 1.0 (or any other released versions), it should work fine. If they are mixed and matched, you can probably expect bugs to creep in, but that is not a smart setup anyway.

    Generally speaking, there is no good reason for everyone not to be running the latest client version, and it’s my understanding that there will be no issues if that is the case.

    But it’s good to note that there are (or may be) some issues out there in those odd cases, if for no other reason than to push people towards not working in such a silly manner… :-)

  5. JB says:

    It’s also worth mentioning that as of 1.6, 1.4 is no longer supported (will not receive future patches, etc.) so everyone should get the heck off of 1.4 if they haven’t already.

  6. JB says:

    Oh, in re-checking the docs on externals, I did find one thing that could catch you (regardless of versions): make sure that the target directory that the external is referencing does not already exist on your local machine, otherwise it will not be checked out.

    If you follow the directions in this post, that won’t be a problem, but I will append this info to the main article just in case.

  7. JB says:

    UDPATE: I did confirm (and by “confirm”, I mean quickly searched on google) that you can run into a variety of issues when trying to use old Subversion clients and newer ones against the same repo.

    Again, like I said above, if all your versions are in sync, it doesn’t matter whether you’re still on 1.0, the above will work.

    However, there is no reason to keep using ancient versions, so… (like I said above) don’t do that!

  8. Thanks for the post. I’m a noob and you’re clearly not. However, I like being structured and I like the structure you’ve created. I want to know if this will likely work and if there are things I should watch for if:

    a) I’m developing on localhost to publish online at a later time.
    b) I want to use the same setup once everything is online
    c) I’m developing in WordPress MU instead of WordPress
    d) I’m using tortoise svn and don’t want to spend too much time on command lines if I can avoid it.

    Thanks in advance for your help,
    Freddy

  9. JB says:

    A-C all apply to my personal use of the things in this article, so you’re OK there.

    As for D (tortoise svn), it should allow you to do everything that I’ve outlined here, but I’m not too familiar with it, so I couldn’t tell you how exactly.

  10. Thanks for this. I’ve literally read this 5+ times and I at least understand the goal. There is precious little about this online and so getting help has been excruciating. I’m using Aptana and the subclipse plugin, both of which have minimal documentation and Googling hasn’t helped.

    I’m new to Aptana, Subclipse, and SVN altogether, so the learning curve is STEEP (can’t I just get back to PHP??). That said, I got svn:externals to work, but am getting lost in combining repositories with Aptana’s “projects” and understanding the relationships between my repository, the Aptana “workspace”, my XAMPP server for testing and my online server for deployment. Once I get this worked out, I hope to come back and post my success.

    Oh yeah, could you explain the line “just like you’re used to with SVN” regarding my own plugins/Themes. I imagine each would need separate space within a repository to track versions, no?

    Thanks again.

  11. JB says:

    So, taking the last thing first: the “just like you’re used to…” reference is referring to the standard practice of modifying your own custom theme, and having it be under the wp-content folder.

    The difference is that in this setup, the wp-content folder is not a child (in the folder hierarchy) of the main WP folder, so all the core files are contained in a directory structure that you never need to change, and are handled by a single externals reference to the core repo itself (as opposed to being copied into your own repo).

    In other words, the main goal of this technique is to not clutter up your repo with copies of things that come from other repos, and then have to try to keep them in sync. You use externals to pull in all of those things from their official repos, so that the only actual files you need in your repo are the ones you’re actually changing yourself (custom themes, plugins, etc.).

    As for the rest, I can’t offer support for any particular combination of IDE plugins, etc. just because I’m not familiar with them and don’t have the time to invest in it. I can say that personally, when I do use an IDE for PHP stuff, it is NetBeans, and it has a SVN plugin that works perfectly well with the setup I’ve laid out here. I imagine most others would too (or should), but I wouldn’t guarantee it. Good luck!

    If you are interested in getting a better grip on the SVN part of this (without the other IDE stuff – projects, workspaces, etc. – muddying the waters), I would recommend getting familiar with it just using the command line svn client with no IDE involved. That way you will at least have a better feel for what is going on, even if you do later decide to use an IDE that hides some of that complexity (while adding a bit of its own).

  12. Sometimes my handicap as a coder is that I’m a visual learner. I tried to show what you wrote for my own benefit. Let me know if I’ve got the picture right. http://designodyssey.org/sharing/structure1.ppt

    I assume in this that I checkout the trunk of my own plugins and themes to the local working copy wp-content and that I svn:externals to the branch/release of those plugins/themes.

    I think I figured out how to do this with Tortoise and I’ll maybe management with Subclipe/Aptana once I learn that client better.

    Thanks again for all your help.

  13. JB says:

    Yes, that looks good.

  14. For others looking for more help with this, I found http://codex.wordpress.org/Installing_WordPress_With_Clean_Subversion_Repositories in the codex to help with the structure part. For a newbie, this was a helpful addition.

  15. I’ve tried this with an WPMU install and wpmu is putting a blogs.dir file in the main wp-content directory after my svn update. Don’t quite know what to do about it because it’s not apart of the install.

    Also have some path issues issues I’ve posted in the WPMU forums.
    http://mu.wordpress.org/forums/topic/16051

  16. JB says:

    You don’t have to do anything about it. MU creates that directory to hold uploaded files, etc. If you want it to be handled under version control, you can allow it to be checked in to your repo. If you want to keep it out of the repo, you can add it to svn:ignore.

  17. Hey,

    This is absolutely the best article on this that I have come across, and as someone else mentioned, finding info on doing this is not easy at all.

    Thank you for the info, it’s been helpful. And everything’s working, with one massive exception that has nothing to do with svn and using svn:externals… WordPress absolutely won’t run for me w/wp-config.php in the directory above core. I tried this the way it’s referenced in this article as well: http://codex.wordpress.org/Installing_WordPress_With_Clean_Subversion_Repositories (designodyssey pointed to this one above as well), and it does not work.. I just get a directory listing.

    I am using Xampp…. could this be an apache thing? Clearly this works for everyone else, so I know it’s got to be something I have improperly configured… any insight from ANYONE reading this thread is very welcome.

    Thanks all, and thanks again JB… great info!

    Best,

    PJM

  18. FYI.. managed to get it to install using the codex article mentioned above, but I still cannot get to the site.. just get a 404. Any insight is welcome. The subversion stuff works perfectly, just WordPress that won’t go! ;-)

    Thanks!

    PJM

  19. I simply HAD to move the index.php up to the root directory of the site to get this to work. No amount of editing .htaccess and wp-config would produce the index. Do you foresee any issues with having to have the index file at this level? I know this is supposed to work, and your info AND the tutorial at wordpress.org indicate you do not have to move this file up, but I cannot get it to work w/just wp-config and the edits therein.

  20. Pingback: Version controlling Wordpress

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>