Self-Hosted Repositories

One of the benefits of Mercurial, so I thought, was that it was easy to get self-hosted repositories up and running, and I’d been privately hosting repositories on my VPS for some time, since I used to host Subversion ones on there too, I assumed this was something that was what everyone did. I had about 10 private repos stored on my server and developed on each one of them on-and-off on various machines, so having the central repository and multiple local clones meant even if I forgot to push from one machine, I could still get on with some development. What I didn’t realise I was missing out on was all the project management tools I really needed, such as issue tracking, wiki entries, commit tracking and so on. This is when I started looking for an alternative solution…

Using Bitbucket

This is when I found Bitbucket, a project hosting service provided by Atlassian, that ran using Mercurial as it’s source control. This was the perfect setup, being able to manage my projects online using their tools, and delegating the responsibility of my repositories to a third-party service. Better yet, by this point, Bitbucket was completely free for unlimited private repos, so it really was the best choice. It gave me easily editable wikis, issue tracking, basically everything I needed.

Problems

Or so I thought. I did wonder if, by being a free service, Bitbucket’s support and stability was going to be lacking a little bit, since I knew it used to be a private service that charged. Turns out my fears were not completely unfounded. When trying to migrate all of my repositories over by creating new repositories on Bitbucket, something that should’ve taken a few seconds took a few hours. Of course, a support email later and a 24 hour wait and the problem was completely rectified, but I was a bit worried that I’d had to wait 24 hours, and that the problem happened in the first place, especially since the support staff quoted that “this happens sometimes, we just need to create them manually”. Dismissing a problem so casually tends not to be in my nature, so I tweeted in case anyone had any suggestions…

Migrating to Git

Of course, the first suggestion was moving to Git and trying out github. I was still a bit sceptical, but decided to give a go anyway, all I had to lose was a couple of hours in trying it out. It turned out there was a wonderful plugin for Mercurial, called hg-git, that would allow Mercurial repos to be pushed to Git servers, meaning I didn’t even have to lose any of my commits if I moved over! This cheered me up, and by creating a testing a couple of repos I found that Git wasn’t that hard to use at all, and was pretty much the same as Mercurial, with just a couple of subtle differences. By this time I’d signed up to github on their smallest paid-for package and started to migrate my most active repositories over, having no trouble at all.

Using Github

I still compare github to Bitbucket, and I apologise for that, because they are completely different services, but it’s a natural habit. Github has very similar features to Bitbucket, such as wiki entries and issue-tracking, but the subtle differences make life a lot easier. For example, github has labels for issue tracking, which lets you create your own management for these problems, instead of relying on the priorities and constrained problem types like on Bitbucket. It also uses live-loading a lot more often in it’s interfaces, which means navigating the website is a lot quicker and simpler. I’ve even started using branches to keep track of changes more conveniently!

Git GUI Clients

MacHg had been my client of choice for Mercurial for a good while, and I didn’t fancy venturing back to the command-line for my day-to-day source control, I just can’t stomach it. So I needed a nice GUI to sit on my Macs for Git too.Git has it’s own GUI, gitk, which comes with it and runs in Tcl/Tk. I have never been a fan of Tcl/Tk, it looks like something from 1992, and only seems to fit in on minimal Linux desktops like Blackbox and the like, so I wasn’t going to put up with that on the Mac. Luckily, I don’t have to. Tower is a fantastic free GUI that, despite still being in the beta testing phase, feels very polished and very well thought out. The interface is superb, and the features, such as using Git’s stashing feature, and staging, link in wonderfully, so I’ll be happily sticking with it for a good long while. If you use Git with a Mac, I strongly urge you to give it a try, it’s a beautiful client.

This year is going to be a big one, I can feel it. I have a lot of plans for a lot of projects, and think I’m finally going to be able to enact my plan of organising things for the better a certain company. A big part of the New Year is making sure I keep on top of this blog, and to make sure I keep posting what I consider to be semi-interesting topics (I doubt you’ll agree).

These are a few of the things I’m looking forward to this year:

• LINGsCARS - As ever, I love working at LINGsCARS, and love all the different places it seems to take me to, but I’m hoping this year is going to be better than ever. We have a lot of plans for various projects, which I’m looking forward to being able to plan, implement and deliver. I’ve done a lot of learning in the years I’ve been in the company, and I’m hoping it’s all going to pay off very soon.
• DBlib - I’ve been planning a release of this lightweight MySQL-driven database library for quite a while now, but haven’t devoted as much time to it as I would have liked to. I’m hoping to add a few more features to make it usable to a wider audience, and get a proper version released very very soon.
• Monkey Boards – This is another MASSIVE project that I’ve put on the sidelines for one too many times… I’ll be devoting more time to Monkey Boards to make sure it’s all cleaned up and ready for a 1.0 release, while still keeping with it’s small stature and lightweight principles.
• And More… – I’ve also got an idea for a massive project, that might see me branching out on my own a bit more, but I’m not sure whether the time is right just yet… It certainly isn’t the time to reveal it!

So I wish you all the best to everything you put your mind to this year, and hope you’ve got a year as fully-planned and ambitious as mine! I’ll leave you with a small photo of a toy Ford Capri I got for Christmas, which cheered me up immensely and now has pride of place on my desk

There are a lot of folk on Twitter and such I know that use Stack Overflow, and who find it very useful, but I’ve just never given it a try, I’ve always avoided it. So what I thought might be a neat idea is to document my experiences with Stack Overflow from the point of view as a new user, and see how I get on with using it and whether it comes across as being any use to me.

First Impressions

Firstly, the main interface. I absolutely love websites that don’t force you to sign up before they let you use them, and Stack Overflow fits that category very nicely. The first page is nicely designed. It’s very easy to understand with clear tags for each question, and a simplistic but effective interface with clear buttons and choices for what to do next. Despite the fact that SO doesn’t require a user account, I feel I should have one since I’m going to be using it more and more, so I want to sign up. There isn’t an obvious place to do this, but trying the “Log In” link at the top of the page leads me to where I want to be.

So, instead of signing up with an individual SO account, I can instead create one from one of the other services I probably already use, such as Google, Facebook or OpenID. I use all three of these, but decide to use Google, because it’s the most focused and already holds all of the information I’d want to pass over. I love the choice of using an existing service to register with SO, because having multiple accounts on multiple sites is tricky at the best of times; I’m always forgetting with email address or username I used to sign up to things, but never the password, strangely. So now I’m registered with Google and have an account, what’s next?

So now I have a very basic profile, with a username of “user555182“, probably about time I changed that. After setting up a bit of a better profile with my name and very quick and brutally honest “About Me” section, it’s about time I got on with trying to find some questions to answer!

Answering

After browsing some PHP questions for a few minutes, one of the first things that grasps me is the sheer size of the SO community. How did I miss this? Every question seems to get a very quick and very reasonable response. After searching through the new PHP questions for a bit, I managed to post a couple of answers, which I hope were helpful, although I haven’t quite found that out yet. I’ll have to wait and see if I get any response for them.

Some of the questions are very simplistic, like asking about foreach loops, nesting MySQL queries and the like, but a lot of the questions are far more advanced, and looking at those I do feel a little bit out of my depth. My limited PHP experience with only a tiny bit of Zend Framework programming is starting to show. Maybe this is the boot up the arse I need to start advancing my abilities a bit…

Anyway, I digress. What I’m trying to do now is think of a question I can ask on SO that will actually help me in some way, and help to solve a problem I’ve been having while coding. That one might have to wait for another day though, since nothing is immediately coming to mind, and I’m not back at work in the office until next week.

One Last Point…

After a bit of searching and use, I just found this question, specifically on the problems with forcing plain text pasting in TinyMCE, which I think so far I’ve spent half my coding life trying to solve, and someone has very kindly posted the answer to this ridiculous problem on SO. If I’d been using SO when I had this problem, I would’ve had it solved in two minutes, instead of two years. That pretty much illustrates this entire idea of trying these new services. I didn’t know I needed SO until I started using it. From now on, I’ll be continuing to use SO to get answers to questions I need, and helping those in need by trying my best to provide a solution to their woes.

In short, Stack Overflow is a fantastic service for ANY programmer who occasionally gets stuck on a piece of code, or who feels they can contribute by making the lives of other coders that little bit easier.

At LINGsCARS, we use TinyMCE for composing messages in the LINGO system, on both sides, admin and customer, so it gets used fairly heavily each day. On the old system (LINGO v1) we just had a textarea that would end up being filled with tags like <b>, <br />, etc. Not particularly pretty to look at and not much use either, so we decided to give a WYSIWYG editor a go, to make typing the message look exactly like a message, and I suggested TinyMCE straight away. We ended up building a couple of quick in-house plugins for templates, image uploading and storage and all was going very well initially when we turned the system on. But then, the mysterious bugs started appearing…

What was Google Chrome Doing?

Most of the team at LINGsCARS use Google Chrome for all their admin work, with a few exceptions who use Firefox, so the first bug we noticed was one that only showed up in Chrome. Every time some plain text content was pasted into Chrome, the content would come in upside-down! Not back-to-front, all the letters were still the right way around, but the bottom line was at the top and so forth! I found it such a strange bug that I couldn’t believe it at first, it must’ve been the way that Chrome itself was handling the plain text content before passing it to the paste plugin. I managed to debug it in the paste plugin source code all the way back to the event trigger, so I knew it wasn’t an easily fixed problem…

Of course, by the time this bug had cropped up, so had many others. Pasting from Outlook was causing massive problems for some staff, with <style> tags being included, a plethora of various pointless HTML tags being passed through with no explanation or conversion, and random spaces being omitted from the ends of lines when pasting from MS Word! We were using the BBCode plugin to translate any HTML into strict BBCode so that we weren’t contaminating the database with HTML like we were in the previous system, and to make sure no admin staff or customers could post anything remotely injectable, so a lot of the tags weren’t going to be used anyway. Trying to ascertain whether TinyMCE was going to return HTML filled with <p>s or <br>s was basically guess-work, as the option for setting either one wasn’t just in one place… By this time, I’d realised the pit of snakes I’d fallen into and was beginning to regret it…

First step: Upgrade

Of course, after having a system in active development for such a long time, the first thing I noticed was that we were running a version of TinyMCE that was of a fair age. I think we were running version 3.2.x, and version 3.3.x was out at the time. I know that when LINGO development started, Chrome was a fairly new idea, so maybe the upside-down bug had been picked up and eliminated? Yep, correct. Upgrading TinyMCE fixed that particular problem, which was good news! I didn’t fancy sitting there debugging all the way through a huge library like that, and to be perfectly frank, I didn’t have the time! This helped me ease my fears over TinyMCE and helped me develop a resolve to get rid of further bugs…

Look at the Paste Plugin

The next step was to get rid of some of the strange HTML pasting problems. I was using the paste plugin that comes with TinyMCE to differentiate between plain-text and HTML-formatted text, so this was the code I focused my efforts in. There didn’t seem to be a consistent way of getting the HTML to work between various browsers, applications and operating systems, so I made the decision early on to abandon HTML support for pasting and merely support plain-text. Looking back on this, I could’ve made a better decision at this point, as now we’ve been in situations where being able to paste tables, bulleted lists and the like would be an advantage, but hindsight is a wonderful thing, and I’m happy with the outcome of the plain-text work.

Modifications needed to be made to the paste plugin source code, trying to get rid of the strange HTML tags and converting all content to plain-text. I even roped in one of the core developers of TinyMCE on Twitter in sorting a bug with <br /> and <p> inconsistency between Webkit-based browsers and others. That pretty much sorted the <p> to <br /> problem I mentioned earlier, and after about a day and a half of debugging and swearing, I had the plain-text isolation sorted, so pasting was finally working in a consistent manner.

Style Tags?

Now, even though all content was being pasted as plain-text and forced into the editor in that guise, there were still a few tags slipping through full of content, one of which was a <style> tag. This only ever happened when pasting from MS Outlook, and only when pasting messages into the system from one particular supplier, who happened to include a <style> tag in his message signature above his name. The <style> tag itself was getting removed, but the inner content, including all the CSS definitions and comments, was being pasted every time! This was a very strange bug relating to the way the paste plugin actively searches for and removes special tags such as <style> tags. It must have been an unintentional oversight, but after careful examination of the code, it really does leave the content in! Having discovered this, I quickly removed this problem and finally got the last of the reported bugs ironed out…

The Finished Article

After spending almost a week on TinyMCE, I was very pleased with the results I got. Now I have a very good understanding of how the library works, and have integrated it a lot quicker into subsequent projects, taking the modified plugins and knowledge I’ve built along the way. A lot of Twitter users have recommended I look into alternative WYSIWYG editors, such as CKEditor, or just abandon them altogether in favour of a solution such as Markdown, but for now I’m sticking with TinyMCE after putting all that effort in. Plus, when it works, it really does the job.

I do plan on submitting some code examples of what I changed to get the paste plugin to work in plain-text only mode, and am still planning on submitting a couple more patches to the latest TinyMCE builds from these bugs I’ve found, but I need to make sure I’ve clearly tested these changes first before I submit anything…

I hope some of you found this useful, it just shows that even with these massive libraries, they aren’t always perfect and any developer can make a difference to these projects and help to improve them for their own needs and everyone else’s.

Everyone had already sorted out travel by the time I was going, and the Megabus was the cheapest and easiest option (note the lack of “comfiest” in that list), so we all went with it; this just meant two seven hour trips with stops at every major city in the north-east. Consequently, we ended up trying to sort out a bit of entertainment for the trip. On the way down, I caught up with the sixth Harry Potter film while everyone else slept, and on the way back, I wrote most of this while Chris built some more of his new website while Lauren and Richard seemed to spend the entire trip on Twitter. Craig#mce_temp_url# built an iPhone app based on Richard’s vocals for “Hey Jude” (we’re all hoping for App Store submission soon!) and played Scrabble with Kat.

This was also the first time I’d ventured forth towards the capital city, normally being a typical northerner and never going further south than Sheffield at most, and the sheer size and complexity of London did overwhelm me slightly. I have no idea how anyone navigates the Underground, or how postmen manage to do a serviceable job, when we got lost in a group of six people, five of which own iPhones with GPS and Google Maps capability, and one with an Android just to be safe. The tube seems stupidly expensive to me, we spent £10 each for two-days travel, so how anyone affords it when they’re commuting every day is beyond me. I wasn’t prepared for how many people it was physically possible to cram on a train either, or the fact that personal space is a privileged option unavailable to the users of London’s public transport, nevertheless, it seems to work! We got around easily and without much swearing…

We were stopping overnight in the EasyHotel in Victoria, and had little idea what to expect, other than a minimalist room with bright orange walls. Unsurprisingly, that’s what we got, however this didn’t prove to be as bad as we all thought. The room was comfortable enough, the facilities all worked, and most of us slept reasonably well after a night of boozing, so I was happy enough.

The actual awards ceremony started at 7pm, so we got the down to Elephant & Castle, stumbled around trying to find the Ministry of Sound, and then went straight in when we found it. Underbelly had pitched up outside with a photo booth that included a wind machine, so after we’d all messed up our carefully sculpted hair, we went into the main club, where we were greeted by free champagne with strawberries, and free beer. Two hours later, I was quite nicely hammered when the main awards ceremony started, with Jack Whitehall as host. Jeffrey Zeldman managed to bag three awards! The awards announcements were followed by a trip to a free burger van and some old-skool PacMan gaming, then, when the free beer ran dry, we headed back to hotel in a typically expensive London taxi.

I’d like to thank the .net Magazine for putting on such a great awards night and being such fantastic hosts, and for providing a night of drunkenness and debauchery for a lot less cost than normal! I’ve made a quick list of all the award winners below:

• Web App of the Year – Typekit
• Community Site of the Year – Ravelry
• Best API Use – Flipboard
• Interactive Site of the Year – Chrome Fastball
• Blog of the Year – Smashing Magazine
• Podcast of the Year – SitePoint
• Video Podcast of the Year – The Big Web Show
• Design Agency of the Year – Happy Cog
• Innovation of the Year  - Node.js
• Web Personality of the Year – Jason Santa Maria
• Viral Campaign of the Year – Old Spice
• Standards Champion – Jeffrey Zeldman
• Open Source Application of the Year – Modernizr
• Redesign of the Year – BBC News
• Mobile Site of the Year – Flickr
• Mobile App of the Year – Dropbox

Everything else on the .net Magazine Awards website.

What I’ll do is gather together all the notes and topics I took away from each of the sessions I attended, and why I think each one will help me in my day-to-day development duties. Obviously, I didn’t attend all of the talks, and hand-picked the ones I was interested in, and the ones I thought would help the most; so I ended up going to a lot of Zend Framework and integration-based talks, solely for the purpose of furthering my own understanding! Anyway I’m rambling… On to the topics.

Saturday

First up was Lorna Mitchell, who I’ve seen a talk from before at a SuperMondays-based PHP event. Her talk was entitled “Teach a Man to Fish”, with some tips on improving both your own learning and understanding as a developer, but also those of your team. I was looking forward to this, especially since we’ve had a recent addition in the office, and communicating across your experiences is not an easy task… Lorna explained some interesting ideas, such as defining the skills that each team member has, and plugging gaps that were evident. Making sure that the person who knows everything doesn’t “go under a bus” is also pretty important, so I recommend at least one minder for your top developer, rather than spreading the knowledge around, which is clearly far less sensible…

Next was a look at Zend Framework 2.0, and the new features the team have planned to integrate from Rob Allen, the author of this very nice Zend Framework tutorial and a ZF contributor. It’s nice to know that a lot of the problems in the current ZF 1.0 implementation (my goodness, where do I start?) are going to be addressed, and that PHP 5.3 is being used extensively, which will push the advancement of production servers where needed. I could go on and on about why I don’t use PHP 5.3 yet in production, but I’d rather save that for another time… Rob’s talk included a lot of neat new tricks for ZF, such as HUGE performance increase benchmarks, classmap files, and the use of interfaces over all of the ZF abstract classes. Apparently they’re looking towards an Easter release for ZF 2.0, but he didn’t seem particularly optimistic, and neither am I to be perfectly honest…

ZF was the theme of the next talk too, with Ryan Mauger introducing some of the features ZF is best at, and where you need to be careful when beginning with Zend for the first time. It’s only very recently, in the last year, that I’ve been using Zend Framework on-and-off myself, and Ryan’s presentation helped confirm my understanding of some of the aspects I’ve been learning, and I’m glad I now know I’m going in the right direction with things! He also showed off a neat tool, BeachPHP, for sandboxing PHP code LIVE in a browser! I might be a bit behind the times, but I’d never seen anything like this before and was very impressed! I’ll be using it a lot on the future…

Continuous Integration was the theme of the next talk, given by Sebastian Marek, and I have to say, this was just fantastic. I got so many ideas and so much information from this talk on how best to move from a live coding environment into a fully integrated system with automated builds, testing, coding standards checking and tons more. Admittedly, at PlusNet they have a HUGE codebase (gigabytes and gigabytes apparently) and need some phenomenal power to drive their TDD environment, but the improvements they’ve made since undergoing this approach are evident. I’ll be looking at tools such as PHPUnit, phpUnderControl, PHP_CodeSniffer, PHP Depend, PHP Mess Detector, Capistrano and Selenium all thanks to this talk!

Back to familiarity with another Zend-based talk, this time on Zend_Acl from Rowan Merewood. The ACL is something that’s always puzzled me, especially since the manual misses out some of it’s best features and it’s a bit hard to find any good tutorials for, so Rowan’s talk was very useful for talking through the implementations of a static and dynamic ACL in Zend, and the pitfalls and problems that can arise too. However, I’m still not entirely convinced the ACL is a library to be used by THAT many developers, and think that it has been a bit over-engineered to solve a specific set of problems rather than providing a wider solution.

For my final choice on Saturday I went to see Harrie Verveer introduce his thoughts on database version control, something even he said there wasn’t a “silver bullet” for in terms of a solution. Harrie described a number of approaches, such as patch files, especially with undo files too, creating an install.sql file that will deploy an instantly-working version of the database setup, possibly with some dummy test data too, and he went on to mention tools such as Phing and DB Deploy, both of which attempt to solve the version control nightmare surrounding MySQL. I’m not sure which idea I like best from Harrie’s talk, but when implementing these solutions I’ll be going for a hybrid approach I think, making sure all avenues are covered off with patch files, undo files and an overall installation file too.

The final session on Saturday was a framework shootout, a very interesting idea where four advocates of some of the major PHP frameworks, namely Kohana, Agavi, Apache Zeta Components and Zend Framework, would first answer some general questions and then any questions put to them. This helped me to get a very good understanding of the major differences between some of the mainstream frameworks, and why there are so many, since each framework has one or more core goals that it attempts to achieve, where others may fall down. I had actually only heard of Zend Framework before this presentation, so this gave me some information on the other frameworks, but still hasn’t convinced me to change! What I especially took from this session was the sheer passion that the developers have for their particular outlook on framework ideals.

The evening involved a free bar at the conference centre for a while, followed by a swift march into Manchester with Paul Geraghty and his wife Dominique, Clair, Andrew Taylor and Anthony Sterling to line our stomachs with delicious curry and a quick stop off at an Irish bar, before heading back to the hotel for a couple more drinks before a well deserved sleep.

Sunday

Sunday started off with a group of us missing the obvious sign for the entrance to the conference at the MOSI, so after feeling like a set of pillocks, we entered inside and sat ourselves down for another round of slightly quicker talks for those of us who were still around.

Michael Peacock kicked things off with his talk on abstracting functionality using centralised content, which explained how to design a database for a project in such a way that you can have versioning applied to it easily, and have most, if not all of your different types of content extending from a centralised table. The content would also have designated searchable fields that could easily be integrated into an ORM to continue the theme. I found Michael’s talk to be very interesting, especially since this is the approach he deploys with client projects at Peacock Carter, although it clearly isn’t applicable to every single situation. Michael’s example was a simple standard database-driven website that incorporated products, a blog, comments, and ratings.

Another Zend talk, how surprising! This talk involved Kathryn Reeve doing some live demonstrations and basic explanations of the principles and features that Zend_Tool allows you to use, such as easily setting up projects, controllers, actions and modules within a new Zend project. I must admit, I didn’t find this talk too in-depth, as I’d already been familiar with Zend_Tool for some time, although I’m sure it was very useful for those in the audience who were just starting out with Zend and needed to get a grip on it’s basic features.

Exploring Nginx and the differences and advantages it has over Apache was something I was really looking forward to, as we’re rolling out Nginx on our new server deployment at LINGsCARS very soon, and there are quite a few aspects of it I’m still not familiar with at all. Errazudin Ishak has travelled over all the way from Malaysia to give his talk, so I made sure I paid particular attention. Nginx runs PHP in a completely different way than Apache, making use of the FastCGI implementation within PHP, and from v5.3.3 onwards, PHP-FPM. The speed increases were instantly noticeable, and Errazudin demonstrated a test on his local laptop that proved it all. The only problem I’ve found with Nginx are that it’s Rewrite rules are quite different to how Apache works, so that’ll take some getting used to.

The final talk we saw before setting off on the long journey back to Newcastle was from Thijs Feryn, a Belgian PHP Benelux advocate who works at a hosting company managing their PHP clients. Thijs explained some of the security he has to put in place, such as strict memory limits, disallowing override tags within .htaccess files, and the problems he has with both PEAR and PECL implementations. Although this talk wasn’t too relevant to me, Thijs is a very good speaker, and did explain some very interesting ideas, so I still felt that I took a lot from it.

Thanks!

Well, that’s about it! All I have left to do is thank all the wonderful team who did such a brilliant job organising PHPNW 2010, especially Jeremy Coates and the team at Magma Digital (enjoying a cup of coffee from my Magma Digital mug now!), I’m very sorry we had to leave a talk early, but I look forward to seeing you all again next year at PHPNW 2011! BY then I hope to have taken a lot of the ideas on board and be able to demonstrate how they’ve helped us at LINGsCARS!

When your system looks like a knackered Vauxhall Cavalier, it's time to change...

Upgrading to a new system is something I’ve only really had to get to grips with in the last few months. I normally spend my day-today working life either making minor website changes, developing completely new ideas that don’t tie into anything else, or maintaining our current systems, but never building new ones. This is pretty rare since the definition of being a software engineer alone carries the fact that most of the time you’ll be building a replacement system for something or other, even if the current system isn’t in any way computerised.

So, this article is going to detail my personal experiences with migrating from out-of-date systems. This means that it will contain a lot of opinions, some of which not everyone in the industry my agree with, and some points of view that may be fundamentally wrong. However, I’m not going to make any apology for this, this is just exactly how I see system migration, so it’s how I’ll tell it!

When is the Best Time to Start Building a New System?

The first problem is knowing when the time is right to sit down and start planning your new system. If the current system you’ve got isn’t throwing up any problems, and there aren’t any features you or your clients are gagging for, then I would suggest to stick with what you’ve got. However, if you’re finding that the bug and feature requests are piling up, and the system has either stopped making money, or worse, has become a money-pit, then it’s time to get out your pen and start drawing some mind maps. Bear in mind though, that you have to be able to justify the planning, development and migration costs that will be involved with a new system. If you can’t justify this to yourself, how are you going to justify it to your boss?

With regards to my experiences, we built the first incarnation of our system and turned most of it on in Summer 2008. For a long time, we were very happy with it, but it soon became apparent that a few little problems were going to re-appear as we continued to use the system, and when we identified a whole new side to the system that would be absolutely impossible to bolt-on to the current incarnation, it was time to sit down and start drawing up plans for version 2. When we built version 1, it was more than adequate for what we needed, but as ever, times change and new situations evolve that cause complications and mean that systems need re-evaluating…

Planning and Designing the New System

OK, so you’ve decided a new system is in the pipeline. Now, you need to sit down and design it, using the flaws and missing bits out of the current system as a foundation. One thing to remember is not to get hung-up on how the current system is designed, you want a version 2, not a version 1.01… The best way to help this process is to bring people on board who weren’t involved with the last system. These people will be able to provide a fully independent view of the problems that befell the last system, and how they see the problems as being fixed in the newer version. I know today’s software engineers are all about agile, but still try and plan as much as you can before beginning development. It’s one thing to start the first iteration of a build, and then to find out the plan was no good and start again, but how many times are your paymasters going to let you do that?

We used the first system’s design, or lack of it, as a basis to begin the design for the new system. This time we knew exactly what we wanted from it, so designing a framework and basic architecture wasn’t too demanding.

Development

I like tho think that the actual development of the new system is the easy bit, normally. If you’ve planned it well and know exactly what you want from the new build, then park yourself in front of an IDE and code away furiously until it’s done. Occasionally though, it isn’t that simple. Feature requirements will change throughout development, and you’ll need to be on your toes to keep up with the demands of your stakeholders!

Feature creep has been a major problem in the development of our new system, and we had to get to the eventual stage of saying “No more!” and admitting to developing them after the system has gone live. That’s the beauty of a web-based system, development doesn’t end with a final release, it just keeps going and going

Testing: the Important Bit

I can’t stress enough how important testing is for a new system. However you decide to deploy it, everyone, and I mean EVERYONE, is going to expect it to work perfectly straight away. If it doesn’t, you’re in for a really hard time. Make sure you right the tests as you’re developing the system, it’s no good waiting till the end and planning the tests around the finished system, save that for the basic black-box tests. Test every function and feature as you’re adding it, and write unit tests that you can run every time you add something new, in case it breaks something old.

The Final Migration

Finally, your system is built! Rejoice! You’ve designed it, built it, and tested it within an inch of it’s life…

But now comes the hard part.

The way I’ve seen it so far in my short time on this world (even shorter as a developer!), there are two approaches to switching your systems over. Both ways have their advantages and disadvantages, and however you do it, something might inevitably go wrong, so I suggest weighing up the two options and choosing based on your current circumstances.

Approach One: Run Two Systems in Parallel

This method is the easiest from your point-of-view as a developer. You get to see any problems you’ve missed out in the new system, and improve on them as you go before the final switchover. The problem is that everyone who actually uses the system has an immediate doubling of their workload. They might not be too happy if it takes you a month to iron out all the bugs before you get the new system fully online by itself. If you go for this approach, make sure you only do it for a short period, otherwise you may end up having no work friends!

Approach Two: Complete Switchover

Although you may sweat blood with this approach, you should be fairly confident! You should’ve already tested the system thoroughly, all that remains is finding a suitable time to switch when the system is either not being used, or being used in a minimal fashion. The pitfalls of this method are bugs: if they creep up now AFTER you’ve switched over, they can cause a lot of misery.

You need to weigh up the two options and decide which one besets your position at this point, I can only provide the options, not make you choose

Update

Read Aidan Garnish’s comments below on his third method! A hybrid between the two that may just be the best for your situation!

MySQL works wonderfully as a stand-alone database server, but has the flexibility so that it can be deployed in a multi-database server environment. Commonly, this is with one database server as the “master”, which holds the original copy of the data, and one or more “slaves”, which the master is responsible for keeping upto date. The following diagram illustrates how a MySQL multi-server setup plays out:

As shown above, the master server is normally used for all of the client writes to the databases, and the client will normally read from the various slave servers. In this quick tutorial, I’ll describe how to set-up a MySQL environment with one master server that is already in use, and a brand new single slave server using one database called exampledb. This presumes you’ve already got MySQL installed and working on both machines, as normally that’s a simple task of installing a package in Linux, an installation package in Windows, or a disk image in Mac OS X.

MySQL’s reference manual also some pretty cool info on replication.

Master Configuration

So first off, you need to make some configuration changes to the MySQL server that will become your master. These changes will be made on the main MySQL configuration file, commonly, on Linux/Mac at /etc/my.cnf or /etc/mysql/my.cnf. Firstly, find the following two lines and comment them out by adding a leading # character:

#skip-networking
#bind-address            = 127.0.0.1

Then add the following lines at the bottom of the file to tell MySQL where to store the log files that will be used to keep the slaves up-to-date:

server-id=1
log-bin = /var/log/mysql/mysql-bin.log
binlog-do-db=exampledb

Now you need to restart MySQL to make sure it recognises itself as a master, and to trigger the new configuration changes. Normally this is done with the following command, but this may vary depending on your system:

/etc/init.d/mysql restart

Next, the slave will need to be able to authenticate with the master as a user to receive the updates it requires. This means you’ll need to login to MySQL using the client as follows:

mysql -u root -p

Make sure to type your root password when prompted. Then create a user with the privileges the slave server will need, and make sure you replace example_password with an actual password, as follows:

GRANT REPLICATION SLAVE ON *.* TO 'replicate'@'%' IDENTIFIED BY 'example_password';
FLUSH PRIVILEGES;

We’re finished with the master server for now, but don’t shut your client because we’ll need to come back to it later on to extract the data from the master to transport to the slave. We aren’t dumping the data now because it means locking the database into read-only mode until the slave is up-to-date. It’s best to finishing setting up the slave first.

Slave Configuration

OK, now you need to log into the server that holds the MySQL installation you want to set-up as a slave server. You need to tell the server it is a slave in mostly the same way the master knows it’s a master, so add the following lines to your MySQL configuration file, changing the IP address, username and password where required:

server-id=2
master-host=192.168.2.1
master-user=replicate
master-password=example_password
master-connect-retry=60
replicate-do-db=exampledb

Then restart the server in the same way you did for the master. That’s it! The slave is now set-up and ready to receive data from the master.

Data Synchronisation

There are a few ways to transfer the initial data from the master to the slave, but I always find the easiest way is to read lock the database and make a dump of the current tables, import it into the slave and start replicating from there. Before you start this, make sure you have a client for both MySQL servers open, ready to input commands, and a separate terminal window on each server too for entering commands not in the MySQL client.

On the master, we need to read lock the tables and get the info ready for replication. To do this, lock the tables using the following commands:

USE exampledb;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

That will give you some information similar to the following:

+---------------+----------+--------------+------------------+
| File          | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.002 | 754      | exampledb    |                  |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Make sure you take a note of the log file AND position, as you’re going to need this info to synchronise the slave! Next, use the terminal on the master to run the following command to create a dump of the current database:

mysqldump -u root -pexample_password --opt exampledb > exampledb.sql

And then run the following command in the MySQL client:

UNLOCK TABLES;

This will unlock the read lock from the tables meaning your master database will operate normally again. Now, you need to copy the SQL dump file over to the slave. I’ll leave it up to you how to accomplish this, there are far too many ways to list. Once copied, create the database on the slave’s MySQL client:

CREATE DATABASE exampledb;

And use the terminal to import the dumped master data like this, replacing the password and path as needed:

mysql -u root -pexample_password; exampledb < /path/to/exampledb.sql

Now, finally, go into the slave’s MySQL client again, and stop the slave from running to update the config:

SLAVE STOP;

Update the master configuration on the slave to the log file and position you noted above by running the following command in the client, replacing the password, IP, etc. as required:

CHANGE MASTER TO MASTER_HOST='192.168.2.1', MASTER_USER='replicate', MASTER_PASSWORD='example_password', MASTER_LOG_FILE='mysql-bin.002', MASTER_LOG_POS=754;

Now start the slave again:

SLAVE START;

And that’s it! Your slave should now be receiving data from the master, and will update whenever a change on the master is pushed to it automatically.

More Tips

A very quick tip, use the following command to check how the slave is doing in relation to the master, and to make sure there are no I/O or SQL errors that have propagated:

SHOW SLAVE STATUS\G

The \G forces the client to output the parameters line-by-line instead of in a tabular make-up, making it a lot easier to read.

If you want to add more slaves to the master, simply increment the server-id configuration you set-up, and dump the data from the master in the same format as shown above.

There are more configuration parameters you can give to the slave, such as replicate-do-table which will specify only the tables that are allowed to be replicated, instead of an entire database.

I hope you’ve found this tutorial helpful, and if, as ever, you have any comments or suggestions, please feel free to leave them!

Mercurial Basics

The first thing you’ll need to do is download the Mercurial binary. As far as I’m aware, it isn’t included on any major system distributions, certainly not on Windows or Mac in any case, although it may be packaged up with some Linux offerings. So, your first stop is Mercurial’s website, which will guide you through the simple installation process. You may also need to install Python if you don’t already have the language set-up on your system.

Create a Repository

Once Mercurial is installed, you’ll have access to the hg command-line tool. This is Mercurial’s core, and where you’ll handle all of your version control needs. To create a new repository for a new project, first you need to create a new directory (normally mkdir on most systems), then cd into that directory and run the following command:

hg init

This tells Mercurial to set up the directory as a repository, creating all the hidden files it needs to store the history of the repo and keep track of all the changes. Now create some files that you might want to store in the repo such as some source code, and add them to the repository using this command:

hg add (files)

Finally, you need to commit the changes you’ve made to the repository, so run this command

hg commit -m 'My first commit'

Congratulations! Your repository is now at revision 1!

Managing the Repository

This is exactly how version control works. You make some changes to the files in the repository, add some new files and possibly remove some old ones, and then commit the changes with a helpful message to remember what you did at a later date. There are just a couple more commands that will help to get you started with your own repository. To remove a file from the repository, use this command:

hg remove (files)

And, to quickly commit all changes, adding and removing any files automatically without using the hg add and hg remove commands, use the following syntax:

hg commit -Am 'Message'

Cloning

Of course, one of Mercurial’s more powerful abilities is to clone other repositories, allowing the user to make changes locally before pushing the changes back to the central server. To clone an existing repository, use the following command:

hg clone (central-repository) (destination)

Then you can work on the repository as normal. When you’ve committed some changes and are ready to synchronise with the central server, use the following command:

hg push

This pushes all outstanding changes to the server, ensuring it is kept up to date. However, the server could be more up-to-date than your local repository, which would mean you’d need to pull the changes and then update your local repository to the current revision, just like this:

hg pull
hg update

And that’s the basics of using Mercurial’s cloned repository feature. As ever, I’m only showing you the very basics here. There are a lot of different circumstances you might want to foresee, such as merging changes between repositories, updating or rolling back to a previous revision, tagging and branching, and more. To read up on these features and how they are jused in the Mercurial tool, the best place is always Mercurial’s website.

Another problem that might arise is how to serve your repositories across the internet. Cloning from a local source works fine, but defeats the purpose of working in a decentralised environment, to clone effectively, you need to be able to do it across the internet. Normally this is done in one of two ways: using SSH as a wrapper for the Mercurial command, or using HTTP to serve the repositories. There are a lot of details to go into with both of these methods, and the best way to do this is search on Google, as I don’t want to baffle everyone with the details in this simple post. If you’re not so inclined as to search on Google, here is an example of how to setup Mercurial to serve using Apache.

Helpful Clients

Personally, I hate using the command-line unless I absolutely have to, so if you’re like me, you’ll probably want to know if there are any decent clients for Windows, Mac and Linux out there, and luckily, there are!

Windows – TortoiseHg
Based on the brilliant TortoiseSVN, which integrates nicely with Windows Explorer meaning no nasty gigantic client windows cluttering everything up, TortoiseHg can handle everything from committing and updating, to merging, diff’ing and more! When I was back using Windows, this was my tool of choice, and did the job very nicely!

Mac – MacHg
This has only been a recent discovery myself, but MacHg is simply fantastic for managing tons of repos in a GUI environment! The history graphs, simple directory window and helpful icons mean it is really easy to use and live with. It does have a couple of things I would change, such as TextMate integration, but I might have a go at writing a couple of scripts myself to sort out things like that!

Linux
I’ll be honest, I don’t use Linux with a desktop very often at all, so I’m not completely aware of any simple GUI-native interface for Mercurial on this platform. Have a look at Mercurial’s own list of tools and see whether there is something there that jumps out!

Bitbucket

Just one final piece of information. Bitbucket is a fantastic repository centre that will manage projects using Mercurial for you. It only gives you one free private repository currently, but an unlimited number of public ones, so it can be extremely handy for tracking a project and making it available online for others to use. I highly recommend giving this website a look for starting up your Mercurial lifestyle through, as it really does make things a lot easier with wikis, issue tracking, user management and more!

We appeared in the Sunderland Echo and Shields Gazette on Thursday, Jonny even seemed to get his own special article in the Gazette, stepping into the limelight and chucking me and Mark behind him! We’re all extremely pleased, and I’m personally looking forward to getting recognised all over Tyne and Wear as a local celebrity! I’ve highlighted the parts of the article that either mention me or mention the specific disasters we seem to create… (We actually have over 320 database tables now, around 340 at last count, and we’re always improving the stability, contrary to popular belief!) Ling has put her own take on the events on her website too.

Incidentally, I am now a full-time web developer at LINGsCARS.com, and Mark and Jonny are both off to do Masters degrees: Jonny at Southampton and Mark possibly at Newcastle or Bristol! I’ve decided to bow out of the education system once and for all, I don’t think I could manage another year of juggling everything doing a Masters degree, and in my opinion I won’t really learn any more out of it, I’ll learn much more staying at the wacky company that I’ve been at for almost three years now!

So here’s to all of our futures!