Skip to content

Optimizing Sales for Happiness

Looking for GitHub's sales team? Please Contact Us or email sales@github.com.


10/17/14 Update from @pjhyett: Building products that people love paired with a customer-focused sales team hasn't changed, but the specifics of our approach have evolved over time as with all things at GitHub. For engineering-centric startups looking to build a sales team, this post is for you.


We've spoken about how we try to optimize for happiness at GitHub. You're also likely hear the term first principles thrown around if you spend a few days at our office. We insist on taking the time to research and discuss various solutions when faced with a new challenge rather than blindly accepting the status quo. How we approach sales at GitHub is no different.

GitHub Has a Sales Team?

We sure do, but figuring out what sales means within a very developer-centric company wasn't all unicorns and octocats. The siren song of big revenue gains can easily disrupt your team's ultimate goals and culture if you're not careful. It wasn't difficult to spot the slippery slope after hiring even one dedicated salesperson, so how do we optimize sales for happiness?

By putting our product and people first. If that means turning down revenue opportunities to remain true to this philosophy, so be it. Making gobs of money has never been our main focus, but we can operate this way and enjoy massive revenue growth across all of our products.

The first (and most important) thing we had going for us was a product that sells itself. With an unwavering focus on building the best products possible, we're in a wonderful position that 99% of our customers require very little handholding. People just want to use GitHub, so we make it easy to pay for it, and essentially get out of their way.

Sales our Way

The remaining 1% of customers is where sales comes into play. Much in the same way support guides folks through technical questions, we needed people to guide customers through business questions. Not only that, developers within larger organizations sometimes need help convincing the people with the purchasing authority to buy the products they really, really want to use.

We still call this role sales, but our team likes to think of themselves as developer liaisons. The role is akin to a sales engineer or technical account manager. It's definitely not your prototypical sales person that's cold-calling people all day to reach their quota.

In fact, all of our sales people have technical backgrounds, they just happen to also love the support and business side of things. They enjoy speaking with customers, building long-term relationships, and are comfortable navigating the inner workings of organizations to ultimately find a way to make sure those developers get to use the products they're asking us for.

No commissions

Another traditional sales tool that doesn't really makes sense for us is paying commissions. Commissions are an incentive to churn and burn through orders as fast as possible, regardless of the consequences. They also introduce massive overhead and logistic problems amongst the salespeople and the company as a whole, so we're happy to avoid all of that crap.

We want our sales people compensated just like everyone else in our organization: great salaries, great benefits, and stock options. We expect everyone to work hard, but we're all in this together and no one should feel marginalized because sales is reaping the rewards from years of hard work done by others.

At the end of the day, the key to sales at GitHub is the key to GitHub's success in general: build an awesome product and treat people well. We don't claim to have unlocked some huge business secret with this idea, but we're excited about our sales team at GitHub because we feel that doing it this way is best for our company and our customers. We're in this for the long haul!

Surviving the SSHpocolypse

Over the past few days, we have had some issues with our SSH infrastructure affecting a small number of Git SSH operations. We apologize for the inconvenience, and are happy to report that we've completed one round of architectural changes in order to make sure our SSH servers keep their sparkle. :sparkles:

As we've said before, we use GitHub to build GitHub, so the recent intermittent SSH connection failures have been affecting us as well.

Before today, every Git operation over SSH would open its own connection to our MySQL database during the authentication step. In the past this wasn't a problem, however, we've started seeing sporadic issues as our SSH traffic has grown.

Realizing we were potentially on the cusp of a more serious situation, we patched our SSH servers to increase timeouts, retry connections to the database, and verbosely log failures. After this initial pass of incremental changes aimed to pinpoint the source of the problem, we realized this piece of our infrastructure wasn't as easily modified as we would have liked. We decided to take a more drastic approach.

Starting on Tuesday, I worked with @jnewland to retire our 4+ year-old SSH patches and rewrite them all from scratch. Rather than opening a database connection for each SSH client, we call out to a shared library plugin (written in C) that lives in our Rails app. The library uses an HTTP endpoint exposed by our Rails app in order to check for authorized public keys. The Rails app is backed by a web server with persistent database connections, which keeps us from creating unbounded database connections, as we were doing previously. This is pretty neat because, like all code that lives in the GitHub Rails app, we can redeploy it near-instantly at any time. This gives us tremendous flexibility in continuing to scale our SSH services.

@jnewland deployed the changes around 9:20am Thursday and things seem to be in much better shape now. Below is a graph that shows connections to the mysql database. You can see a drastic reduction in the number of database connections:

You can also observe an overall smaller number of SSH server processes (they're not all stuck because of contention on the database server anymore):

Of course, we are also exploring additional scalability improvements in this area.

Anywho, sorry for the mess. As always, please ping our support team if you see any further issues on github.com where Git over SSH hangs up randomly.

Designing GitHub for Windows

Today, I thought it would be fun to give some insight into how we do native application design at GitHub. Many people are surprised to hear the breadth of design and code that happens at GitHub on a daily basis. We love Ruby and Rails, but we are far from a single language dev shop. We write code in Ruby, Python, JavaScript, CoffeeScript, Objective-C, C#, C, C++, Java, and shell of various flavors from Bash to PowerShell. On the design side of things we obviously build web and native applications, but we also do print, animation and motion graphics. We design booths, stamps, whisky glasses, dodgeball uniforms, business cards and many many octocats. Most of our designers work in Adobe products but we did branch out a bit as a part of creating GitHub for Windows.

Even though we shipped GitHub for Mac almost a year before GitHub for Windows, we've known for a long time that we wanted to build platform-specific native applications. In fact, both applications were conceived at the same time and share some core design principals. They also share a lot of low level code in the form of libgit2, but by and large they are entirely separate entities. We specifically made the decision to write each application in a language native to the platform, and this has turned out to be hugely beneficial to us. Because of this separation we've been free to tackle the problems that are most pressing for each platform and work in the best possible tools instead of being constrained to the lowest common denominator.

So how does an application like GitHub for Windows get built from a design standpoint? It all starts with an idea and convincing someone else to work on it.

Metro

In the end, Metro was my secret weapon in convincing Cameron McEfee to work on this project with me. Cameron was coming from a design background in print media and the layout and typography of Metro really caught his eye. The rigid grid system and Swiss design principals, along with a very modern and clean feel made the prospect of designing a native GitHub client in this style very exciting. After sharing a few sketches and some initial prototyping work I had done, Cameron was hooked and the application started to take shape.

Concept

Despite the existence of incredible mocking tools, I still like to start my design work with a pencil and my notebook. Paper is still a very powerful medium for concept work.

notebook mockup

This is my original sketch of the dashboard in GitHub for Windows. You can see one of my first XAML mocks kept some similar ideas.

xaml mockup

Here is one of Cameron's first takes:

mockup

At this point, we were still discovering what is useful to display on this view and trying to understand how this would actually be implemented in WPF. At GitHub, we work in a very fast, iterative design process. In this particular case, Cameron and I would trade mocks and do design reviews dozens of times a day. We try to get design work out as early as possible for peer review and iteration. It can initially be intimidating to get feedback on incomplete work, but once you learn to take things in stride it's hard to imagine working any other way.

A key part of this iterative cycle is that a single person generally owns the design. We don't create A, B, C versions and present them to the CEO. We don't have dedicated product managers (or managers of any sort). Instead, we do work in small self-forming teams and those teams generally end up naturally surfacing someone who owns the design. The result is consistent, but opinionated design.

Just for reference, here is how the dashboard ended up in the final release. We slowly refined our idea of Metro and began to add in more GitHub elements while actually moving away from how applications like Zune interpret Metro.

screenshot

In the end you can see that the design is clean - living on the strict Metro grid system and definitely focusing on content over chrome. However, we have very specifically made this native app have some of the feel of GitHub.com. A side-by-side perspective of the app and the website provides an interesting example of the affinity.

side by side

XAML

One of the wonderful things about WPF applications is that you can basically build anything that you can create in a Photoshop or Illustrator mock-up. Cameron tended to work largely in Photoshop and I would come in to tweak things and then re-create them in XAML (with little or no underlying functionality). We would keep doing design as new visual elements of the application came up. So, for instance, we didn't actually design a progress bar until I had need of one in the application.

As we got further along, I started committing completely to XAML and used the Photoshop mocks merely as reference points. I tend to work half in Blend and half in Visual Studio. If you've never used one of the XAML designers before I have to tell you that one of my favorite things is how you can write markup and watch the design change, or use the color palette and watch the markup change. It is like changing HTML and CSS on the fly in the Chrome inspector except you are actually editing the real code and various other assets in a full fidelity editor. Practically, this means that you can really quickly tweak colors, alignment, and even basic components of a control without a large debug or refresh cycle. And in the end, you check in a version-able document to your repository so that when I change the foreground color of a text field, it is an obvious diff on GitHub.com. Try doing that with a Photoshop document!

xaml diff

Styling and Templating

The other amazing thing about doing design in XAML is that you have control over not just how your controls are styled, but also the basic graphic components of a control. Standard buttons in WPF look like this:

standard button

But we redefined what a button looks like entirely. This can be done in a way where buttons have to opt into the style/template, but it can also be done in a way where you override every button. This is subtly powerful. Here is how we define one of our buttons:

github button

Along with doing basic graphic design, Blend also lets you do interaction design. In Photoshop, you might have to manually lay out a series of buttons that represent what a button looks like in the various states of hover, pressed, normal, disabled, focused, etc. In XAML this is natively supported through something called the VisualStateManager, and from a design-time perspective you actually get to 'record' what each of those states look like and then play with the behavior of the object. You can even control how the movement between states happens by animating properties as part of state transition.

Borderless Window

One of the most striking parts of GitHub for Windows is that it ditches the traditional Windows chrome almost entirely. We have a small control box for the traditional min/max/close buttons, but otherwise we killed all the rest of the chrome. Here is how we did that.

Our borderless Window is implemented as a behavior that can be attached to any Window in XAML like so:

<Window xmlns:win="clr-namespace:GitHub.Extensions.Windows"
        xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity">
    <i:Interaction.Behaviors>
        <win:BorderlessWindowBehavior />
    </i:Interaction.Behaviors>
  ...
</Window>

The min/restore/close buttons are just custom XAML buttons where we've written the appropriate code to make them behave as you would expect. The meat of the BorderlessWindowsBehavior happens in the callback where we handle a few specific window messages to control the display of the window. We got a lot of help and inspiration from MahApps which has a similar BorderlessWindowBehavior.

I don't know if killing the chrome is going to be widely adopted on Windows, but I hope that it is. For me, those 5px borders and the massive title area on each window are simply artifacts of a past life. They are distracting, largely unnecessary, and take up space without providing functionality. Plus, even the standard window chrome on to-be-released Windows 8 makes it look like your computer is 20 years old.

This doesn't necessarily mean that every application should or needs to be a metro application, but it has been fun to see even Visual Studio move towards this style. I think this is very welcome change.

visual studio 2012

Vectors

We leverage vector graphics as much as possible in GitHub for Windows. This makes the application resolution independent and makes it really easy to move designs from Photoshop/Illustrator to WPF. For instance, the maximize button is implemented with this path:

Performance

There was a lot of criticism in the early days of WPF about performance and some of it was warranted in the initial versions of the framework. These days, the tools are at your disposal to make great WPF applications. The only major thing we've noticed is if you are running old hardware and old operating systems (ahem, XP, I'm looking at you), then WPF has to fall back to software rendering and this tends to be a bit more memory and processor intensive. To the credit of the framework we were able to release a single application that supports a huge range of operating systems from a single code base with almost zero platform specific code.

Some notes about performance. XAML files (an XML format) are actually compiled down to a binary format (BAML) which makes them wicked fast to load at runtime (sort of how nibs/xibs work on Mac/iOS). What's most important to GUI applications is perceived performance and a responsive UI, which we achieve using Rx and making sure that we don't perform blocking operations on the UI thread. This is the same kind of thing you have to watch out for when writing a node application, for instance. In our case, Visual Studio also comes with some great tools for measuring what threads are doing work and what code is causing the UI to block (and even what code eventually caused it to unblock).

performance

What this is showing is all the threads in my application and the state that they are in. Generally you want that main UI thread to be green, and all those places where you see red rectangles is where the UI thread is waiting on another thread. When the UI thread blocks like that the application is frozen for the user, so you want to look at what code is running that is blocking and then what code eventually ran to unblock the thread. If you want to use this tool yourself, make sure to follow Paul's instructions to get real stack traces.

What you don't see

So far, we've really focused on the visual and constructional aspects of the design, but a lot of our effort in creating GitHub for Windows went into understanding how to make Git and the distributed collaborative nature of GitHub accessible to people who have no desire to use a command line tool. From a design standpoint this means we thought hard about everything from how you download/install the application to how we present version control concepts like making a commit or rolling back changes. We even went so far as to package our own distribution of msysGit and a curated command-line experience as part of the application. Design is often most important for the things that you don't see.

The Making of Octicons

In our last post we announced Octicons, our new icon font. We put a lot of work into the font and gained a lot of knowledge in the process. With five different designers working to make it happen, this was one of our bigger collaborations. We thought we'd detail how we built Octicons and what we learned along the way.

Making the Icons

In most cases, a designer would begin to work in Illustrator to create vector icons, but we chose Photoshop as our start place. From the outset we knew we wanted to design icons for specific sizes, so optimizing for those pre-defined pixels was paramount. With the recent release of Photoshop CS6, Photoshop has become a fairly powerful vector tool for pixel projects.

One handy feature that is a little hard to come across, but is great for creating icons is a 2-up view of a file. With the application frame visible Window > Application Frame you can create a second instance of your working document Window > Arrange > New Window For …psd. After that you can have the windows sit side-by-side with Window > Arrange > 2-up Vertical. Each window is a view of the same document, but can be moved and zoomed independently. Best of all, when working with paths in one, the path outlines and handles don't appear in the other. This lets you work zoomed into one and at 100% to proof your icon in the other.

2-up

In Photoshop CS6, Adobe introduced pixel snapping vectors. This is an amazing feature if you work with paths a lot. From time to time, however, it's necessary to turn this feature off. There isn't a hotkey that does it, but it is possible to record an action for turning this setting on and off which can be hotkeyed.

prefs

With Photoshop set up for maximum icon awesomeness, we began to audit all the icons on our site. We created new Octicons for each, as well as adding a few extras we thought might be useful down the road.

Building the Font

Design is in the details. With all our icons designed, it was time for us to create our font. We decided we needed two sizes of each icon. One size, 16px, would be optimized for its exact size. At 16px the details are limited so every pixel was important. Since the icons were designed for such a small space, they don't really scale well. To take care of that our second size, 32px, would be designed with more detail so that it could be scaled up for many purposes.

Having planned out or icon strategy, we embarked on the adventure that was putting together our font. For this, we used an application called Glyphs.

To begin, we needed to set up our font file so that it would work with our specific sizes. Since a font technically only has one size, we decided to set it up to be optimized for 32px, with the 16px icons actually scaled down from larger versions.

In the case of our icon font, the font's metrics include a em width, cap height, and grid. This roughly translates to the idea of width, height, and resolution.

We initially tried setting our cap height, x-height, and em width to 32 to make a 32x32 unit square. This technically worked, but we found when exporting the font any nodes not placed on an exact integer unit would be shifted to the nearest unit. This resulted in a lot of ugly icons.

With this knowledge in hand, we decided to adjust things a bit. Using a 2048x2048 unit square, we were able to ensure that when the shifts occurred during export the effect on the overall shape was minor. This was mostly caused by being able to change the ratio of the grid to the total size of the font character. When the nodes snapped to the 1 unit grid in a 32 unit square, the jump was significant. With a 1 unit grid in a 2048 unit square, the jump was far less noticeable.

Here are our final metrics in Glyphs.

fontmetrics

The flow after we got setup was as follows.

  1. Our icons were built in Adobe Photoshop.

    Design your icons in whatever you're comfortable with. Building it in Photoshop did add an extra step, but we preferred the control it gave us.

  2. We scaled in Adobe Illustrator to 2048 units.

    We first opened the Photoshop file in Illustrator. When copying the paths out of Illustrator, Glyphs seemed to use Illustrator's coordinates for positioning. Having all the icons on one sheet, this caused alignment issues when simply copying and pasting icons into Glyphs.

    To resolved this, we created a second 2048x2048 document. To get Illustrator to match Glyphs' grid, we had to shift the ruler zero point -1792pts. We then copied in a character (enclosed in a 2048x2048 square) into the new document and ran an action that scaled it up to fill the document and set it to (0,0). From here we were able to copy the vector path into Glyphs in the correct position.

    This process was repeated for every character. In the end, each character fits a 32px square, sitting 4px below the baseline to line the icon up with any text it is pared with.

  3. We exported our font as an OTF.

    You may ask yourself why we chose 2048 as our magic number. We did quite a few tests at other sizes like 512 and 1024. What we found is that the size of the characters didn't have much of an effect on the file size. What did change it was the grid size ratio. The larger the grid size (fewer grid squares), the smaller the file size got. At 2048 units, we had much more flexibility to adjust the grid scale without impacting the shapes of the font. In the end we chose a grid of 32px, which resulted in a 20% smaller file on export than a grid of 0 units. Larger grids like 64 units and 128 units, produced files that, while slightly smaller, didn't reduce the file size enough to make it worth the loss of vector quality.

  4. We uploaded the exported .otf file to the Font Squirrel @font-face generator.

    Since we didn't want any of the standard characters and used our own special unicode characters, we specified our own subset range, and turned pretty much everything off. You can grab our config settings here.

With our font built and ready for the browser there were a few items left we had to polish before calling the icons complete.

Sub-Pixels

off-grid

The grid matters! Each one of the grid squares above represents a pixel in Glyphs. This is equal to 128 units in the app (We actually used a 32 unit grid, but this helps illustrate our point). When the lines don't line up with this grid, the font rendering engine tries to compensate. It does this by aliasing which renders the half pixels as grey pixels. The Create New Repo icon above is mis-aligned. The result is below.

Not Aligned

not-optimized

Aligned

optimized

Fine Tuning

The last thing our font needed was a little something-something to adjust how the font was aliased. WebKit's -webkit-font-smoothing property was our answer.

Subpixel-antialiasing

20120430-txpcd66jfdtmidy6fmdqj44n8s

Browsers use subpixel rendering by default. This looks great for text, but left the icons looking a little heavy as a result of the extra pixels it adds.

No antialiasing

20120430-eii3htkain6jr7i6tcibi54wac

By turning off font-smoothing entirely, we could preserve every pixel perfectly. This was great for straight lines, but curved or angled lines ended up looking blocky and pixelated.

Antialiasing

20120430-c34d57pyb63khe6f4utxkdfkkb

Antialiased text was a good middle ground. It provided both the sharp edges when the font was built correctly, and smooth, lightweight curves when they were needed.

While only a feature of WebKit at this point, we'll update the icons to support font-smoothing in other browsers as it becomes available.

Credit

We're pretty proud of these icons. They are the result of a lot of hard work from @bryanveloso, @jonrohan, @jasoncostello, @kneath, and @cameronmcefee. We hope you've found this breakdown of the process interesting.

ReactiveCocoa for a better world

Native apps spend a lot of time waiting and then reacting. We wait for the user to do something in the UI. Wait for a network call to respond. Wait for an asynchronous operation to complete. Wait for some dependent value to change. And then they react.

But all those things—all that waiting and reacting—is usually handled in many disparate ways. That makes it hard for us to reason about them, chain them, or compose them in any uniform, high-level way. We can do better.

That's why we've open-sourced a piece of the magic behind GitHub for Mac: ReactiveCocoa (RAC).

RAC is a framework for composing and transforming sequences of values.

No seriously, what is it?

Let's get more concrete. ReactiveCocoa gives us a lot of cool stuff:

  1. The ability to compose operations on future data.
  2. An approach to minimize state and mutability.
  3. A declarative way to define behaviors and the relationships between properties.
  4. A unified, high-level interface for asynchronous operations.
  5. A lovely API on top of KVO.

Those all might seem a little random until you realize that RAC is all about handling these cases where we're waiting for some new value and then reacting.

The real beauty of RAC is that it can adapt to a lot of different, commonly-encountered scenarios.

Enough talk. Let's see what it actually looks like.

Examples

RAC can piggyback on KVO (key-value observing) to give us a sequence of values from a KVO-compliant property. For example, we can watch for changes to our username property:

[RACAble(self.username) subscribeNext:^(NSString *newName) {
    NSLog(@"%@", newName);
}];

That's cool, but it's really just a nicer API around KVO. The really cool stuff happens when we compose sequences to express complex behavior.

Let's suppose we want to check if the user entered a specific username, but only if it's within the first three values they entered:

[[[[RACAble(self.username) 
    distinctUntilChanged] 
    take:3] 
    filter:^(NSString *newUsername) {
        return [newUsername isEqualToString:@"joshaber"];
    }] 
    subscribeNext:^(id _) {
        NSLog(@"Hi me!");
    }];

We watch username for changes, filter out non-distinct changes, take only the first three non-distinct values, and then if the new value is "joshaber", we print out a nice welcome.

So what?

Think about what we'd have to do to implement that without RAC. We'd have to:

  • Use KVO to add an observer for username.
  • Add a property to remember the last value we got through KVO so we could ignore non-distinct changes.
  • Add a property to count how many non-distinct values we'd received.
  • Increment that property every time we got a non-distinct value
  • Do the actual comparison.

RAC lets us do the same thing with less state, less boilerplate, better code locality, and better expression of our intent.

What else?

We can combine sequences:

[[RACSignal 
    combineLatest:@[ RACAble(self.password), RACAble(self.passwordConfirmation) ] 
    reduce:^(NSString *currentPassword, NSString *currentConfirmPassword) {
        return [NSNumber numberWithBool:[currentConfirmPassword isEqualToString:currentPassword]];
    }] 
    subscribeNext:^(NSNumber *passwordsMatch) {
        self.createEnabled = [passwordsMatch boolValue];
    }];

Any time our password or passwordConfirmation properties change, we combine the latest values from both and reduce them to a BOOL of whether or not they matched. Then we enable or disable the create button with that result.

Bindings

We can adapt RAC to give us powerful bindings with conditions and transformations:

[self 
    rac_bind:@keypath(self.helpLabel.text) 
    to:[[RACAble(self.help) 
        filter:^(NSString *newHelp) {
            return newHelp != nil;
        }] 
        map:^(NSString *newHelp) {
            return [newHelp uppercaseString];
        }]];

That binds our help label's text to our help property when the help property isn't nil and after uppercasing the string (because users love being YELLED AT).

Async

RAC also fits quite nicely with async operations.

For example, we can call a block once multiple concurrent operations have completed:

[[RACSignal 
    merge:@[ [client fetchUserRepos], [client fetchOrgRepos] ]] 
    subscribeCompleted:^{
        NSLog(@"They're both done!");
    }];

Or chain async operations:

[[[[client 
    loginUser] 
    flattenMap:^(id _) {
        return [client loadCachedMessages];
    }]
    flattenMap:^(id _) {
        return [client fetchMessages];
    }]
    subscribeCompleted:^{
        NSLog(@"Fetched all messages.");
    }];

That will login, load the cached messages, then fetch the remote messages, and then print "Fetched all messages."

Or we can trivially move work to a background queue:

[[[[[client 
    fetchUserWithUsername:@"joshaber"] 
    deliverOn:[RACScheduler scheduler]]
    map:^(User *user) {
        // this is on a background queue
        return [[NSImage alloc] initWithContentsOfURL:user.avatarURL];
    }]
    deliverOn:RACScheduler.mainThreadScheduler]
    subscribeNext:^(NSImage *image) {
        // now we're back on the main queue
        self.imageView.image = image;
    }];

Or easily deal with potential race conditions. For example, we could update a property with the result of an asynchronous call, but only if the property doesn't change before the async call completes:

[[[self 
    loadDefaultMessageInBackground]
    takeUntil:RACAble(self.message)]
    toProperty:@keypath(self.message) onObject:self];

How does it work?

RAC is fundamentally pretty simple. It's all signals all the way down. (Until you reach turtles.)

Subscribers subscribe to signals. Signals send their subscribers 'next', 'error', and 'completed' events. So if it's all just signals sending events, the key question becomes when do those events get sent?

Creating signals

Signals define their own behavior with respect to if and when events are sent. We can create our own signals using +[RACSignal createSignal:]:

RACSignal *helloWorld = [RACSignal createSignal:^(id<RACSubscriber> subscriber) {
    [subscriber sendNext:@"Hello, "];
    [subscriber sendNext:@"world!"];
    [subscriber sendCompleted];
    return nil;
}];

The block we give to +[RACSignal createSignal:] is called whenever the signal gets a new subscriber. The new subscriber is passed into the block so that we can then send it events. In the above example, we created a signal that sends "Hello, ", and then "world!", and then completes.

Nesting signals

We could then create another signal based off our helloWorld signal:

RACSignal *joiner = [RACSignal createSignal:^(id<RACSubscriber> subscriber) {
    NSMutableArray *strings = [NSMutableArray array];
    return [helloWorld subscribeNext:^(NSString *x) {
        [strings addObject:x];
    } error:^(NSError *error) {
        [subscriber sendError:error];
    } completed:^{
        [subscriber sendNext:[strings componentsJoinedByString:@""]];
        [subscriber sendCompleted];
    }];
}];

Now we have a joiner signal. When someone subscribes to joiner, it subscribes to our helloWorld signal. It adds all the values it receives from helloWorld and then when helloWorld completes, it joins all the strings it received into a single string, sends that, and completes.

In this way, we can build signals on each other to express complex behaviors.

RAC implements a set of operations on RACSignal that do exactly that. They take the source signal and return a new signal with some defined behavior.

More info

ReactiveCocoa works on both Mac and iOS. See the README for more info and check out the Mac demo project for some practical examples.

For .NET developers, this all might sound eerily familiar. ReactiveCocoa essentially an Objective-C version of .NET's Reactive Extensions (Rx).

Most of the principles of Rx apply to RAC as well. There are some really good Rx resources out there:

How we use Pull Requests to build GitHub

We recently shipped a new About section. It has all sorts of stuff like high resolution logos, pictures of the GitHub team, a little bit about our story, recent press mentions and maybe most importantly positions we're hiring for. It's awesome.

But that's not the point of this post. Instead, let's take a look at how we used a massive Pull Request to ship this feature.

We talk a lot about how GitHub works in blog posts and talks and this is a great example of how we use Pull Requests.

Here is what the PR looked like for the new About page:

You're looking at 130 commits and 91 comments from 10 different people over a two month timespan. The discussion ranged from the original idea and HTML mock-up, to Skitch mock ups from developers, to content strategy. There are designs posted for review at various points. And of course, every commit to the branch is tracked and ready for code review.

If you've ever talked to a GitHubber you'll know we think Pull Requests are the greatest thing ever. And not just because we invented them.

They are a great way to generate discussion around new ideas and recruit people to help out. Because we don't have major organizational decisions, Pull Requests let people see what's being worked on and they hop in where they think they'll add the most value. It works a lot like an Open Source project.

Some tricks to make Pull Requests more awesome for your project:

  • Open a Pull Request as early as possible

    Pull Requests are a great way to start a conversation of a feature, so start one as soon as possible- even before you are finished with the code. Your team can comment on the feature as it evolves, instead of providing all their feedback at the very end.

  • Pull Requests work branch to branch

    No one has a fork of github/github. We make Pull Requests in the same repository by opening Pull Requests for branches.

  • A Pull Request doesn't have to be merged

    Pull Requests are easy to make and a great way to get feedback and track progress on a branch. But some ideas don't make it. It's okay to close a Pull Request without merging; we do it all the time.

Akavache is now open source

Today, we're open-sourcing a library that we have been using at GitHub: Akavache.

Akavache is an asynchronous, persistent key-value cache created for writing native desktop and mobile applications in C#. Think of it like memcached for desktop apps.

What does that mean?

Downloading and caching remote data from the internet while still keeping the UI responsive is a task that nearly every modern native application needs to do. However, many applications that don't take the consideration of caching into the design from the start often end up with inconsistent, duplicated code for caching different types of objects.

Akavache is a library that makes common app patterns easy, and unifies caching of different object types (i.e. HTTP responses vs. JSON objects vs. images).

It's built on a core key-value byte array store (conceptually similar to a Dictionary<string, byte[]>), and on top of that store, extensions are added to support:

  • Arbitrary objects via JSON
  • HTTP Requests
  • Fetching and loading Images
  • Securely storing User Credentials

An example - consider Twitter for Mac

When you open Twitter for Mac, you immediately see content, even before the application finishes talking to Twitter. This content is cached, but the pattern of when to refresh the data might be different. For the tweets themselves, the logic might be something like, "Load the cached data, but always fetch the latest data".

However, for the avatar images, you might have logic like, "Always load the cached image, but if the avatar is older than six hours, refresh the image."

In Akavache, the former might be:

var tweets = await BlobCache.UserAccount.GetAndFetchLatest("tweets", DownloadSomeTweets());

And the latter might be something like:

var image = await BlobCache.LocalMachine.GetAndFetchLatest(tweet.AvatarUrl, 
    DownloadUrl(tweet.AvatarUrl), 
    createdAt => DateTimeOffset.Now - createdAt > TimeSpan.FromHours(6));

No (Thread.)Sleep 'till Brooklyn

Akavache is non-blocking, via a library called the Reactive Extensions - any operation that could delay the UI returns an Observable, which represents a future result.

Akavache also solves several difficult concurrency problems that simplify UI programming. For example, in the image above, a naive cache implementation could end up loading the GitHub avatar icon multiple times if the requests were issued at the same time (which is easy to do if you try to load the avatar for each tweet). Akavache ensures that exactly one network request is issued.

Learn More

Check out the GitHub Repository for more examples, and install the package via NuGet. Akavache is also used extensively in Play for Windows, a desktop client for Play, your employee-powered office DJ.

Fileserver Maintenance Wednesday Night

We'll be performing some scheduled maintenance on a fileserver pair this Wednesday (April 25th, 2012) at 9PM PDT. A small percentage of repositories will be placed in maintenance mode for around 30 minutes during this process, impacting access to these repos via HTTP, Git, and SSH.

Update: The maintenance is complete. Thanks for your patience.

All of the Hooks

Over three years ago, @pjhyett launched GitHub Services with just four services: Campfire, IRC, Lighthouse, and Twitter. Since then, 124 other people contributed to a total of 68 third-party services. We and many others depend on these services to power our workflow and ship awesome features.

There have been two main requests for Hooks: an API to manage the Hooks, and support for other GitHub events besides just push.

Today, we're announcing two APIs for Hooks and eleven new possible events, so you can build tighter integrations around GitHub.

The new events are:

  • issues - Any time an Issue is opened or closed.
  • issue_comment - Any time an Issue is commented on.
  • commit_comment - Any time a Commit is commented on.
  • pull_request - Any time a Pull Request is opened, closed, or synchronized (updated due to a new push in the branch that the pull request is tracking).
  • gollum - Any time a Wiki page is updated.
  • watch - Any time a User watches the Repository.
  • download - Any time a Download is added to the Repository.
  • fork - Any time a Repository is forked.
  • fork_apply - Any time a patch is applied to the Repository from the Fork Queue.
  • member - Any time a User is added as a collaborator to a non-Organization Repository.
  • public - Any time a Repository changes from private to public.

In addition to an HTTP Hook API that's consistent with the rest of API v3, we also support a PubSubHubbub endpoint to manage Hooks. The core Web hook already accepts hooks from all twelve events. The Campfire and IRC hooks support the new pull request and issue events as well. These changes are available in the API only, so eager developers can take a crack at them. Expect a future update to the Service Hook admin in the near future once these changes have been proven in battle.

Gist creation in 5 lines with Java v3 API

As part of the Mylyn connector announcement last week a new stand-alone Java library was released for accessing GitHub API v3.

Below is a 5 line example showing how to create a new private Gist that has a single file containing a simple one line Java example:

Running the above example would create the following Gist:

Many more examples can be found within the GitHub Mylyn connector source base which uses it exclusively for all API calls. The library currently supports creating and fetching issues, labels, milestones, and gists. More services will be added to the library as they become available through the v3 API.

The org.eclipse.egit.github.core library requires Google Gson 1.6+ and Apache HttpComponents 4.1+ and the latest builds can be found on the GitHub Eclipse Hudson page.

Have you created something interesting with the GitHub Java API library? Let us know!

Those are some big numbers

Every night, our friendly Hubot pops into one of our Campfire rooms and posts some numbers. Turns out we passed some pretty significant numbers in the past couple days. And numbers are fun, so we thought we'd share them with you.

That's a lot of git repositories

What's even more staggering is that 70% have been created in the past year. We're getting around 4,500 new GitHub projects a day.

How it breaks down

Many people assume GitHub is filled with Ruby and Javascript projects. Let's look at the numbers.

The most prominent project language on GitHub? Everything else. Remember that our languages page counts the amount of code — the chart above counts the number of projects with a primary language.

On to the next one

Anyhow — these are some pretty crazy numbers to me. See you guys next time.

Reply to Comments from Email

You should notice a small change to the From address on your email notifications now: they're no longer from no-reply@github.com.

We're now accepting replies from most email notifications that you'll receive:

  • Issue comments
  • Commit comments
  • Pull Requests
  • Direct messages

The biggest change in all this is how these replies are displayed. We figured out early on in testing that we couldn't expect people to write Markdown. People are going to be dumping code or stacktraces, and will expect them to look nicely. Also, we need to accomodate the various top posters and bottom posters among our users (while holding judgement).

The new email comment formatting still has some quirks, but we think that the power to reply to emails was too much to hold back. Try it out, and let us know in support if you have problems.

A few caveats:

  • Plain text email is vastly preferred. Essentially, we strip all HTML from HTML emails before formatting.
  • Email attachments are currently ignored. There's no place to upload assets to issues or comments anyway.
  • You need to send from an email address that is attached to your GitHub account.
  • If you have problems, giving us a URL and/or the email Message-ID is helpful.

Scheduled Maintenance Tonight at 22:00 PST

We will be performing scheduled maintenance tonight from 22:00 to 22:20 PST. During this window GitHub and all Git access will be entirely unavailable for a short period while we perform MySQL and Redis maintenance. GitHub Pages and GitHub Jobs will be unaffected.

Recent Services Interruptions

Here's a summary of the outages we encountered this week and what we're doing to prevent this from happening again.

Monday January 3rd

Monday marked the first "real" workday for most people in 2011. Our wonderful users all hopped online and got back to hacking. As North American work hours came around our Pacemaker application failed over one of our xen machines which happened to host our primary load balancer. This is something that happens really rarely and most of our users notice because the load balancer is the machine that everyone hits when accessing GitHub. This exposed a few problems in our infrastructure that we'll be addressing.

Our internal routing had issues that we hadn't experienced before due to our growing internal network. We specifically had problems with internal DNS resolution after the failover as well as routing certain traffic to some of our frontend machines.

New Relic was great in helping us diagnose this issue.

network outage

Something was taking WAY too long compared to how things normally look. Everything was essentially timing out.

Unfortunately it took us a little while to figure out the real issues were with networking. We know this can happen now and the team has a much better understanding over the networking overall.

We're now aware that under our current configuration, certain services on our load balancers must be located on different hosts to prevent this particular routing issue. We have a plan in place to reconfigure that part of our networking setup to remove the issue. In the meantime, we're also setting up a third load balancer to restore our n+1 redundancy.

During all of the networking insanity we had a fileserver, fs7, failover during this bumpy outage. We use a high availability setup for the fileservers, and they fail over a lot more often than you'd think. We kind of chalked it up to general insanity inside the cluster and our trusty sysadmin, Tim, went off to make sure we didn't have another day like Monday.

We had intermittent service between 8:30AM PST and about 3PM PST.

Tuesday January 4th

Around 7AM PST on Tuesday we started to notice high load and an abnormally high number of http connections. By 8AM fs7, the same machine with problems the previous day, had failed over. The failover machine is usually online within a few minutes but due to the high load it hobbled along for a little over an hour. Shortly after that it kernel panicked which required Tim to spend some quality time with it. We realized that the kernel the failed fileserver was running was older than most of the rest of our fileservers so we decided to upgrade it. This took us a little bit and service was restored on fs7 by 3PM PST. Keep in mind that this only impacted a subset of our customers but a second shaky day obviously isn't what we want for our users.

Everything was back to normal but two straight days of issues impacting one fileserver left us a little spooked and focusing hard on what was wrong with fs7 specifically. Everything seemed to corrolate around north american business hours starting in EST, so we camped out and waited for wednesday morning.

Wednesday January 5th

Wednesday we saw the heightened load start around 5AM PST and resulted in a bumpy two hours. The system went in and out of swap before swapping itself to death shortly after 7am PST.

06:58:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:10:01 AM    124428  16347368     99.24   1195892   7180972   1036700     11868      1.13         8
07:11:01 AM     90832  16380964     99.45    865808   4479240   1036700     11868      1.13         8
07:12:01 AM     96648  16375148     99.41    205644    939236   1036676     11892      1.13        36
07:13:10 AM     81588  16390208     99.50     36040    104276         0   1048568    100.00      9632
07:14:10 AM     83004  16388792     99.50     29232    100256         0   1048568    100.00      3812
07:15:10 AM     81992  16389804     99.50      2324     67620         0   1048568    100.00      3212

You can see it die off in collectd graphs.

swapdeath

Once again fs7 failed over and this time it had a lot of queued requests to handle when the failover was promoted. As the failover came up its load stayed extremely high but started to settle after 20-30 minutes of hammering it. We were unhappy that it happened again, but we were glad that we'd avoided another prolonged outage.

Around 8:30AM PST we saw another burst of activity on the fileserver, luckily we were watching the system closely and kept the system in check. You can see the memory start to rise here.

stomp it

We noticed something happening on the system that never should though, dozens of 'git pack-objects' calls running. Normally Librato keeps these processes in check but something seemed to be ignoring this. We made it through the second onslaught and had time to really dig into what might be causing the issue.

We started looking into what networks were on the fileserver, I'm sure you recognize a few of them.

We were investigating whether or not this specific fileserver might be overloaded due to popular projects when something else popped up. Joe from Librato pointed us to some really awkward behavior we were seeing in system resource usage on the server. Something that we weren't managing with Librato really grew out of control during the times we saw service interruptions and high load.

librato-rss

Memory grew linearly from around 3PM PST the day before until 5am where it maxed out and eventually lead to the box swapping itself to death.

You can also see the virtual memory follow a similar trend here.

librato-vss

With this information we were able to quickly identify that the git-http service that's running on the fs servers was not under Librato's policy management. We've been slowly pushing more and more people to use git-http by default and we hadn't experienced such a spike in traffic as we've seen over the past few days. We put git-http into a Librato container and we had to wait for Thursday morning to really test it.

Problem Solved

This morning went smoothly. Librato kept all of our git-http processes in check despite another morning of enormous git-http traffic. We're excited to get back to work on making GitHub better, not keeping GitHub running. We're really sorry for any inconvenience our users experienced due to the insanity over the past few days. We hope this run down of the events gives our users some insight into how we handle problems. Having metrics around as many things as possible really helped us identify a difficult problem to diagnose.

A big thanks go out to Saj from Anchor for waking up in the middle of the night for three days straight to help us out with systems issues. Thanks to Joseph Ruscio for the Librato insight that revealed the real fix.

The Tree Slider

Those of you running recent versions of Safari, Chrome, or Firefox 4 may have noticed some changes to tree browsing on GitHub.

The new HTML5 History API (which really has nothing to do with HTML — it's a JavaScript API) allows us to manage the URL changes while CSS3 transitions handle the sliding. Permalinks are always maintained, your back button works as expected, and it's much faster than waiting for a full page load.

Basically we intercept your click, call pushState() to change the browser's URL, load in data with Ajax, then slide over to it.

When you hit the back button, an onpopstate handler is fired after the URL changes, making it easy to send you "back".

Want more? Check out the HTML History API Demo and MDC's Manipulating the browser history documentation. Facebook has blogged about their use of this stuff, and Flickr has been doing it for months on their lightbox view.

There's also some hot replaceState() action over on our new Features page and the Pull Requests dashboard.

We're still getting all the kinks out of the Tree Slider, but we hope you like it!

Something went wrong with that request. Please try again.