Rebooting Local.ch

A migration story, 2011-2013

I started working with local.ch in 2011 as a contractor, for some short-term Ruby on Rails support. Soon after, I took a full-time position as a Senior Engineer with the Web Business department and became team lead.

Back in 2011, local.ch had a very large legacy codebase written in PHP that relied on XSLT for rendering. It was a complex, custom framework and there was a strong desire to move to something more agile and modern.

Over the next 9 months, we would migrate the entire website to Ruby on Rails, using the existing backend services. I helped architect a plan to do this with nearly zero downtime. Features were relesed in phases, and as we introduced each new piece, we would use the load balancer to map the application paths to the new code.

To help make the migration go smoothly, we switched the traffic over by browser type. We started with Firefox, moved on to Chrome, and even targeted specific platforms. This allowed us to buy more time to fix problems with legacy IE browsers, and to scale up our new infrastructure.

To ensure reliability, I created performance test tools, using real user activity and log replay. The logs were transformed and stored on S3. Scripts were created for benchmarking, as well as to ensure stability and correctness by verifying response codes. Using these tools, we were able to check for performance regressions and estimate capacity.

After the migration was complete, we were able to spend time on design updates and some new features. Towards the end of 2012, we started work on an ambitous project to add a restaurant guide to the website. This involved restaurant partner data imports, a new CMS, new search services and APIs, and a whole new design team.

In early 2013, after having been online with the new system for one year, I gave a talk at Web Tuesday called "Local.ch: A Year on Rails".

A few months later, we were nominated for the Best of Swiss Web Award in the Technology category, and won the Silver prize.

Starting in late 2012, I became team lead for the web engineers, and started working closely with product owners in designing new features. The new local.ch map was challenging, because it involved coordinating the user stories and engineering tasks, together with external design, new search features, and a whole new approach to the front-end stack (namely, Google Maps and backbone.js .)

In the summer of 2013 we began a project to redesign the website of local.ch, using modern web techniques, emotional images, and improvements to usability.

To develop the overall approach, we started with paper, and eventually moved on to wireframes and design mockups. Early in the process, we built a cheap prototype and did several rounds of user testing. For the city images and new content, we extended our open-source CMS to act as a cheap API endpoint, so we could quickly serve content to the front-end applicaitons.

Angular.js played a big part in interactive features like autocomplete. I extended the image service I had developed to support more options and provide dynamically resized images. We took the grid and styles and packed them into something we call "Skeleton", a frontend framework that provided consistent style, common elements, and components.

We needed a flexible process for engineering that was compatible with the pace of change at local.ch. As the web engineering team, we had to answer to many stakeholders within the company - marketing, sales, customer support, products. I adopted an "agency model", where we acted as an internal web shop, and the company stakeholders were our clients. Clients worked directly with engineers on features, and tasks were prioritized and organized into weekly sprints. Our team often deploy several times a week!

The sprint process, taken from a presentation that I gave to the department in early 2013.

For new features, we used Trello, which is very flexible for backlog management and sprint planning. For bugs and inter-departmental changes, we used JIRA, which was great at managing support tickets. To make the two tools work better together, I created a tool that would track JIRA tickets within Trello automatically. This allowed us to connect everything to a single sprint board.

We wanted to be visible in the Zurich community - it's a great source for hiring. Our team gave presentations at tech events and meetups, for instance: Web Tuesday, JS Zurich, Frontend Conference, Angular.js Zurich meetup, RailsHöck Zurich

The hiring process for new engineers was something that we took very seriously. We formalized the interview process and created an intensive all-day program that included live coding and one-on-ones interviews. We avoided group interviews, or puzzles. Sometimes candidates would return for a "test day" afterwards.

The efforts were worthwhile - we grew a vibrant and highly skilled front-end team at local.ch, that worked with joy together.

I firmly believe in an open development process. Always transparent about what we're doing, and invite others to participate in the process. Code reviews are mandatory and pairing is encouraged. We published open source and contributed back to projects we used. We built relationships with open source authors, and found ways to work together.

I encouraged team specialty roles, cross-training, pairing, offsites, workshops, and hack days. These are important parts of engineering culture.

In 2013, I had a crazy idea to do an internal tech conference, and pitched it to my managers and fellow engineers, who embraced it. There was a strong need for the different teams to share what they were doing, inspire each other, and learn new tools and techniques. Thus, the Local Engineering Conference was born.

I have always been a fan of data visualization, and became quite interested in Graphite in 2012. I began implementing dashboards for server and app performance. Together with the systems group, I helped set up the infrastructure with StatsD and Graphite across our two datacenters. By adding support to the core web libraries and deployment scripts we used, metrics became default for things like service timings, errors, and resource usage.

With our front-end load balancer wired into Graphite (using Python), we were able to monitor response codes on an application-level basis, and quickly narrow in on problems. We installed several TVs and monitors around the office to monitor various things.

The Web Dashboard - local.ch front end applications

Other tools that I used at local.ch included Splunk, New Relic, AirBrake, Pingdom, Puppet, Distimo, Capistrano, and many custom scripts. These were the very early days of Devops, and we were ahead of the times.