Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

Twaves: How ZeroTurnaround follows team, app and business activity using Twitter

Disclaimer: ZeroTurnaround is a commercial vendor, but Twave is not a commercial product or in fact a product or project of any kind. It’s an approach we use at ZeroTurnaround to keep tabs on our team and business activity using Twitter that we would like to share with the world.

It may not be obvious to an outsider, but ZeroTurnaround has exploded during the last year. Our team went from 4 to 12 engineers in half a year and is continuing to grow every month. This may not seem much to someone from a large corporation, but for us it was a pain. Mainly, because everyone on our team is a star in his own right and we didn’t want to compromise productivity even a little bit.

To achieve that we started by cleaning up our tools and process. We switched from SVN to Mercurial to better accomodate  the team size and code complexity. In fact we are using a hosted FogBugz & Kiln solution from the Joel Spolsky’s FogCreek and are pretty happy with the productivity it offers (I have a lot more to say here, but it’s a story for another day). We also use Maven and Hudson to ensure a continuously stable nightly build and a rock solid release.

Unfortunately even with those tools in place we keep making mistakes (and what’s worse not catching them in time). E.g. we rewrote our testing framework three times. Granted it’s a big pile of awesomeness managing a farm of cloud instances, but we did lose a lot of time on going down the wrong path and if we paid more attention we could have avoided it.  We had a few grandiose and embarrassing failures right after the release, because the tests were missing and nobody noticed the changes made. We had people waste time overengineering a bicycle, when a guy sitting in three meter radius would have known a simple and easy solution.

You may notice that all these problems were caused more or less by lack of communication. More precisely, it’s what Alistair Cockburn describes as Osmotic Communication, background information sharing that happens without people even realizing it. We value communication very high at ZeroTurnaround and use our own brand of agilish process. Some of it is like Scrum, except that daily stand-up meetings are replaced with daily lunches, much more nourishing for both body and soul. But largely it’s inspired by Crystal Clear methods of Alistair Cockburn and our own experiences in the field. But clearly we were missing something in our process that would enable the busybodies like me to interfere when something smells wrong.

What we needed was some way to follow the team activity without too much overhead and spike conversations when something goes wrong before long. One appealing idea in that area is repeteadly expressed in Malcolm Gladwell’s Blink: The Power of Thinking Without Thinking: it’s basically well known that people are good at pattern matching at break necking speeds (mind you, Blink generally makes a lot of unscientific claims, but this one is fairly well supported by the current evidence). E.g. just scrolling through a bunch of text you can get the gist of the meaning. Same way by looking at the team activity even for a fraction of a second some things stand out. That doesn’t mean that whenever something wrong comes in it’ll be noticed by everyone, but with the whole team watching there’s a good chance someone will notice.

We looked for solutions around the web and didn’t find anything we liked too much (IBM Jazz purports to enable a similar style of information sharing, but come’on). However we found inspiration in several Open Source efforts. Specifically cia.vc allows for great insight into the Open Source projects linked to it. You can follow activity of specific developers or the project as a whole and have instant conversation starters when something goes wrong. We also liked the  the IRC bots and the coolness of having conversations in the same medium.

Twaves

After some investigation we decided to use Twitter to follow the team activity stream (or twave, name having no connection to me being a Firefly fan). There are several reasons for that:

  • Twitter is really easy to use. In fact most of the people on our team were already using it. It has a multitude of desktop and mobile clients, it has a great web site and it keeps improving, but still stays simple.
  • Twitter is real easy to integrate with. Besides the OAuth, which can be a pain, but needs to be done only once, the API is all simple REST and JSON. We hacked on it mostly from PHP, but it’s easy to use “as is” in any language (except Java, but it has a nice library).
  • Twitter has a great ecosystem around it with a bunch of ready-made apps we could use. In fact our first integration was just using twitterfeed.com to feed a bunch of RSS we already had into our protected Twitter account.
  • Finally, Twitter is short. Considering that we want to take advantage of pattern matching machinery of the human mind it’s important to keep it as short as possible, but not shorter :)

There are also specific reasons we didn’t choose other mediums:

  • The classical way to follow such activity is email. Email is deceivingly better, as it can have unlimited amount of content and comes to the inbox where it surely will be read. Reality, however, is that the sheer amount of content makes the email hard to scan quickly and generally automated emails quickly overwhelm the recipients and get filtered in a nice dusty folder that nobody ever reads.
  • Another established way is to have an IRC bot. It’s a great alternative for those who are used to it, though probably a little harder to integrate and maintain. The main reason we didn’t go with it is that nobody on our team uses it. Jabber would also be a nice option, but it’s support for group chat is still not in mainstream use. And Skype does not have an API to hack on.
  • Yammer, as a walled off Twitter alternative was also evaluated as an option. However as Twitter allows protected accounts, has an established and easy-to-use API, has a multitude of clients and was already used in the team we didn’t see much reasons to use it. But it is definitely a reasonable alternative if you are already using it or if you need centralized administration or other enterprizy options.

So after the medium was decided it was time to build the twave. We registered a protected account and decided to include the following events:

  • Commits, including the message and the committer, naturally.
  • Bug tracker activity. Initially we used RSS so only new bugs were shown, but later we included assignments, closed bugs and updates.
  • Support emails. We use FogBugz for support so it was actually included in the bug tracker activity.
  • Hudson build notifications.

Additionally we also included purchases to keep the team motivated as well as some web statistics. In your case, depending on the app and team responsibilities, you might want to include:

  • Monitoring alerts: hardware failures, low resources, slowdowns and whatever else.
  • Failures: exceptions, lost user sessions, etc.
  • Heartbeat: just as important can be for the servers to report that all’s well and include some running stats.
  • Business events: purchases, complected workflows, etc

In the end we found it important to include relevant team and business events, that allow to follow the activity of the team, the apps and the business as a whole. However the reasons for that were not solely to prevent failures, but also to react to them quickly, align the team goals and increase motivation.

So did it work for us? You betcha! It’s not used in equal measure by everyone on the team, but it became an invaluable tool for the leads at the very least. Some things that we noticed:

  • The beauty of Twitter is that everyone uses it differently. Some prefer the desktop client, some prefer the web. Some have it notifying in real-time via Growl or similar, some scroll through a queue of tweets a few times a day. A common pattern for smartphone owners is to go through canned tweets while on the can :)
  • We noticed quite a few weird things that would otherwise slip our notice. E.g. a well intending engineer renamed all plugins to follow the same convention of “xxx_plugin” (whereas previously some of them used the dash, like “xxx-plugin”. Unfortunately since the plugin names spill into the user configuration it would have been a much larger and more dangerous change than he intended. More dangerous things were noticed and prevented as well, some that would cause crashes.
  • In some cases things that someone thought were dangerous or bad, were actually well motivated. But the discussions spiked by the activity stream allowed us to improve them to be better understood by others and make sure that everyone is on the same page.
  • Finally, the fact that we can follow sales and other business events gives the team a day-to-day feedback on their work and their success. We are very thankful to everyone who purchases our products and are proud that we can add value to the Java community. This pride is what motivates us to continue innovating.

One issue we haven’t touched is security. In some companies it might be impossible to keep any data in the cloud, but there are a few tricks to make it less sensitive:

  • The data includes mostly titles, urls and maybe some emails. URLs are internal, so they cannot be accessed even if known. Some companies don’t like to have any data about their netwoks outside their tight control, but that’s security by obscurity :)
  • You can run a nightly script that will delete all history longer than a day/week. This ensures that even if account is broken into it only has the latest history available.

Outer Rim

So now that we’ve explained our reasons and the basic setup we used let’s dig deeper and examine how exactly we integrated with Twitter and how the twave is shared, aggregated and accessed:

  • Although in the previous section we were talking about a single protected accounts, we eventually created several. They corresponded to different twaves produced by our team, by our business and by our web site (e.g. correspondingly dev, sales, web). This was done because not everyone was interested in all events equally and this way they had a choice to subscribe to the relevant ones. This also allows us to easily create in the future additional “noisy” twaves without adding noise to the current ones. This is also important for access control to “sensitive” twaves.
  • It’s important to meaningfully aggregate events that would otherwise produce too much noise. E.g. we get a lot of downloads, so producing an event for each download would quickly drown the twave in noise. So instead we can have hourly/daily stats visible in the twave and only include anomalous events (e.g. a download from Somalia) that may be of interest to us.
  • We would also like to be able to discuss the stream in the same medium. It’s quite possible to register protected accounts for this purpose and freely discuss this on Twitter, but this experiment is still ongoing. Usually we just discuss the stream events in real life, so it’s not a big problem for us. It’s also fairly easy to copy paste the events into the group chat (we use Skype) so I’m not sure if it’s a real issue.
  • Sometimes you want the alerts to be delivered immediately, not wait until you go through the Twitter queue. For this purposes we use Prowl that delivers push notifications to iPhone. However in countries where Twitter will send SMSes you can also register a custom account associated with your phone and push the events there. You can also use Twilio to script a creepy robotic call in the middle of the night when something occurs.

Unification

When it comes to the actual implementation there are quite a few annoyances that you can copy-paste from us instead of solving yourself again.

To gain access to the Twitter API you will need to hack through the gordian knot of OAuth. It’s a bit of a hassle to use, even with the help of the libraries. Since we need to access accounts we control we can take a shortcut. Namely, sign in at dev.twitter.com with your twave account and register a new app at dev.twitter.com/apps/new. You can register a “Client” app, but make sure it has “read and write” access. Once it’s done go to your app and copy “Consumer key” and “Consumer secret”. The click on the “My Access Token” and copy the “Access Token” and “Access Token Secret”. You will need all four for each twave as shown in the the following PHP example:

function notify_dev($message) {
  $connection = new TwitterOAuth(
    DEV_CONSUMER_KEY,
    DEV_CONSUMER_SECRET,
    DEV_OAUTH_TOKEN,
    DEV_OAUTH_TOKEN_SECRET);
 
  $parameters = array('status' => "$message");
  $status = $connection->post('statuses/update', $parameters);
 
  if ($connection->http_code != "200")
    error_log(print_r($status, true));
}

We are using the twitteroauth library, which is a very thin PHP wrapper over the Twitter API, but any other will behave similarly.

The next problem we faced is that Twitter does not automatically shorten URLs. However bit.ly provides an extremely easy to use API that can be accessed like this:

function shorten_url($url) {
  $ch = curl_init("http://api.bit.ly/v3/shorten?"
            ."login=$login&apiKey=$apiKey&domain=j.mp"
            ."&longUrl=".urlencode($url));
  curl_setopt($ch, CURLOPT_HEADER, false);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  $result = json_decode(curl_exec($ch));
  curl_close($ch);
 
  return $result->data->url;
}

All you have to do is register a user to get an API key and decode the returned JSON.

An additional thing we do at ZeroTurnaround to help twitterize the events, is to replace names, usernames and emails in the twave to Twitter @usernames (they are highlighted nicely, easy to reply and show up in mentions). We do that with a simple search & replace:

function normalize_email_address($email) {
  $pos = strpos($email, "<");
  if ($pos !== false) {
    $email = substr($email, $pos + 1, -1);
  }
  return $email;
}
 
function contact_to_twitter($contact) {
  $ZT_CONTACTS = array (
    array (
        'name' => ''Malcolm Reynolds",
        'email' => 'mal@zeroturnaround.com',
        'twitter' => '@malcolmreynolds'
    ));
 
  $contact = normalize_email_address($contact);
  foreach ($ZT_CONTACTS as $c) {
    $contact = str_replace($c['name'], $c['twitter'], $contact);
    $contact = str_replace($c['email'], $c['twitter'], $contact);
  }
  return $contact;
}

One thing that you need to know here is that Twitter will hide the tweet starting with @username if you do not follow that user. Therefore you need to add something in the beginning of the tweet to avoid that. It can be as simple as putting a dot in the beginning “.@jrebel” or you can make use of that and add a unicode symbol like “☯” that distinguishes the event type.

To make sure that events come in realtime we either post them directly when the action happens (e.g. purchases or downloads) or use callback scripts. A very useful feature in many cloud-based services (like the hosted FogBugz & Kiln we use) is WebHooks. Basically you get a callback HTTP REST request to a URL of your liking with payload as GET parameters or in the POST body. So in the final code snippet of the article, here’s the whole of our commit hook:

<?php
 if(!$_REQUEST['payload']) exit();
 require_once 'zeroturnaround/lib/notify.php';
 
 $payload = json_decode(stripslashes($_REQUEST['payload']), true);
 
 foreach ($payload['commits'] as $c) {
   $message = "☯ ".contact_to_twitter($c['author'])
   ." committed ".$c['message']
   ." to ".$payload['repository']['name'].": ";
 
  if (strlen($message) > 120)
    $message = substr($message, 0, 118)."..";
 
  $message .= " ".shorten_url($c['url']);
  notify_dev($message);
 }
?>

As you can see from the script it is important to keep the Twitter message short. The total size is 140 symbols, so we reserve 20 symbols for the shortened URL and make sure that the rest of the message is 120 symbols or less. The script is called with a JSON payload in the POST “payload” parameter as documented in Kiln and the rest is straightforward.

Serenity

It’s my hope that the solution described here will be of help to those feeling the stress of the communication failures. As we came up with this approach a while ago I spent some time searching the web for similar efforts. I couldn’t find any, which means that either this stress is unique to us or the issue doesn’t have the prominence it deserves (or perhaps I suck at googling, which is totally an option as well). I’d like this to change, so I ask that if you have done or planning to do something similar (or want to adopt this solution), please write to jevgeni at zeroturnaround dot com. Also if you know of any public records of similar approaches be nice enough to link to them in the comments.

Can’t stop the signal!

Jevgeni KabanovJevgeni Kabanov is the founder and CTO of ZeroTurnaround (www.zeroturnaround.com), a development tools company that focuses on productivity. Jevgeni has been speaking at international conferences for over 5 years, including JavaPolis/Devoxx, JavaZone, JAOO, QCon, TSSJS, JFokus and so on. He also has an active research interest in programming languages, types and virtual machines, publishing several papers on topics ranging from category theoretical notions to typesafe Java DSLs. You can follow Jevgeni on Twitter as @ekabanov.

  • http://twitter.com/shmonder Anton Litvinenko

    Hey! We are building a devpulse app (http://devpulse.com/), which has very similar idea and intentions. At the moment it is in private beta and we are cooking a huge update that, we think, should solve at least some of the problems you’ve mentioned. Especially, the noize/signal ratio and individuality when it comes to the stream consumption.

    Nice to hear that we are not the only ones looking forward for such tool/service.

  • http://twitter.com/toomasr Toomas Römer

    When I read the first drafts of the article I thought of devpulse too. Would like to see devpulse in action.

  • http://www.nuxeo.com/ sfermigier

    We built something similar for Nuxeo: http://www.nuxeo.org aggregates information from the forum, the blogs, Jira, etc. And @nuxeodev on twitter tweets commits.

    This is a WIP.

  • http://twitter.com/mutru Otto Hilska

    Interesting to see Twitter being used for something like this. How does it work when your team would like to discuss something that appeared in the feed?

    We had a pretty similar situation when we started working on Flowdock (http://www.flowdock.com/). We wanted to see a stream of information from many sources, and we also wanted to connect our Skype chat discussions to them. Also, we felt that the IM discussions were too temporary, and the important bits were lost. This is what Flowdock solves.

    You can check out the developer screencast from Youtube: http://www.youtube.com/watch?v=Rz7pEKZwzqY

    Our own team brings in Twitter keyword search for our brand, customer feedback, Git pushes, random emails, server monitoring alerts, Pivotal Tracker events, wiki edits etc. Then we discuss them when needed.

    I’m not expecting you to switch now that you seem to have a pretty decent setup with Twitter, but I’d like to hear your comments about Flowdock. :)

    Btw, Yammer in my opinion Yammer is more about organization-wide networking, while Flowdock is for team’s internal collaboration.

    Thanks for the great post!

  • http://zeroturnaround.com Jevgeni Kabanov

    I’ll take a deeper look at FlowDock later, but at least as far as I discovered so far from the work flow, it’s easy as cheese to copy-paste the event you want to comment to Skype group chat without polluting it with every event out there. I think it’s very much a non-problem.

  • http://zeroturnaround.com Jevgeni Kabanov

    Beautiful!

  • http://twitter.com/texmex5 Jaana

    As Jevgeni said, integrating with Twitter is really easy, everybody is using it already anyways. I looked at flowdock.com that was recommended here in the comments, and showed it to my team. Moving away from Skype where our chat is right now seemed impossible… Developers like things they know well…introducing them to something new and taking something away usally needs to be really well argumented and benefits to the developer need to be easily seen… Just adding some twitter feeds and growl notification is a lot smaller stepp :).

  • http://zeroturnaround.com Jevgeni Kabanov

    The moment Skype APIs are available I’ll write a bot to publish Twitter feeds to Skype group chat. That’ll solve the problem as far as I’m concerned.

    In fact I’m not even sure it is necessary, as people are perfectly fine copy-pasting relevant events to the chat, this way there’s no information overload there.

  • http://zeroturnaround.com Jevgeni Kabanov

    I did know about @devpulse, but I’m not sure I see too much usefulness in developing a product for that, as all it took us is a a few day’s work and it’ll take others even less thanks to the info presented here. That said 37signals did a living making overly simple products, so I might be wrong.

  • http://twitter.com/markkofman Mark Kofman

    there will always be a holly war between people who think you should build internal tools yourself and those who believe buying those is a smarter alternative. I personally like to focus and spend 100% of time developing my own product, instead of creating, maintaining and administering internal tools.

  • http://twitter.com/markkofman Mark Kofman

    guys, are you planning to open source the code of twaves? if yes, let us know – we have a plan of creating open source project to contribute connectors to different development tools. would be cool to have your contribution there too…

  • http://zeroturnaround.com Jevgeni Kabanov

    Sorry for that, I didn’t mean to start a holy war and as a commercial vendor myself I wish all the best of luck to you. This was a personal opinion and I know better than to think that my personal coincides with a majority or even a significant minority of people. To make it up to you and other products omitted from the original post I’ll write a follow-up article reviewing the Twave alternatives and similar initiatives.

  • http://zeroturnaround.com Jevgeni Kabanov

    We might do that, but I’d like to see the response first.

  • Bob Holmstrom

    Hey, Jevgeni! Very well written article — easy for a total outsider to comprehend (except for the software specific stuff, of course). Your English prose is excellent and interesting to read!

  • Sohbet

    thanks very much from http://www.sohbetbak.com/

  • http://pulse.yahoo.com/_NBQOBOQNDUVJN4TH3SXEUMDDAY trevor

    Twitter is getting more an more populat these days, I got a page of my site for JPG to PDF converter http://openpdfconverter.com/ on Twitter and provide updates via tweets.