July 12, 2023

Data Exploration: The Missing Ingredient to Reduce Development Bottlenecks

Welcome to my blog series on Data Exploration! In this monthly column, I explore the common challenges and pitfalls that functions within an organization often run into and how they can solve for it with Sigma. We are a company of innovators who believe in pushing the boundaries of what’s possible with data, and we also pride ourselves in being the model Sigma customer. Read on.

Every few months, I get pinged by a new startup with the magical answer my engineering managers (EMs) need in order to manage their teams better: Using statistics about pull requests (PRs) in GitHub, the companies will let our managers know how our employees are doing. All this for the small small price of $x per user per month.

For the non-engineering folks, PRs or pull requests are a way for contributors to ask maintainers of a repository to review, approve, and integrate code changes or a new feature into the main project.

PR and code statistics are a horrible way to understand performance. The pitches have improved over time and are now trying to help my managers understand who is blocked, who might need help, etc, but honestly, these are things the EMs should be able to get from team stand-ups and discussion with the employees.

This led us to look at what interesting tidbits we could find in our own GitHub data, and get a bit more creative with what we can identify. Our ability to go from question to answer is seconds, not a product management cycle which translates to a team sprint of weeks.

Here are a few things we were able to identify. I’ll caveat all of this with the fact that data here gives an indicator that something interesting might be happening, and you need to go ask questions to understand what. You are dealing with people, and these are all just signals, not absolute truths:

We can obviously see if people are ramping up or have a sudden drop in productivity. This is mostly a signal for managers to ask questions.
Time to close submitted PRs: Who seems to be taking longer than average?
We can see the average number of comments being given to the person on PRs. Over time, this shows whether they’re gaining mastery of the area of the code they’re working on.
For reviewers, we can see their contributions: How many code reviews do they do, how many PR approvals, but more importantly, how deep are their reviews? Are these one liners, or longer texts explaining things?
We can also see how developers structure their PRs; if they are turning in small PRs which help with faster reviews.

Laying the Groundwork

Sigma allows us to do some incredible data analysis without writing any SQL. Our customers like to think of it as a spreadsheet on steroids. You can use the data in your cloud data warehouse to get incredible insights, but first the data needs to be in the warehouse. For this analysis, we needed to get our GitHub statistics into Sigma. We used Fivetran to move the data over from GitHub to Snowflake, making it available in Sigma. We could now go analyze. Fivetran creates a star schema database in our warehouse, reflecting GitHub’s data representation of Issue, Pull_Request, User, Repository, etc. These are very easily connectable using Excel-style Lookup mechanics in Sigma’s spreadsheet-like interface.

An Overall Productivity View

Let’s start with the obvious. We can use the Issue table (which is GitHub’s way of listing both tasks and PRs) to filter for just PRs, and further for closed ones. We can then display a pivot table that shows the number of PRs per month by submitter. What emerges is a view over a period of time of how many PRs the person submitted and closed. The interesting thing here is mostly analyzing the change from month to month for a specific person which might be an indication that something is going on. I can’t emphasize enough that your job as a manager is to talk to your people, not expect a computer to do your job. Don’t just act on the numbers you see. Use them as a signal that something might be going on.

Here, every row represents a developer’s accepted pull requests. You can see major changes in productivity (up or down), track how their integration into the company or the team is going, and chat with them to understand if they are blocked for some reason.

This is where I have to list all the obvious warnings:

Code output and impact aren’t the same.
Different people work on different complexity problems.
Different people work in different languages, on different systems, etc.
Different people submit different sized PRs despite any company policy you might have.
Remember the law of unintended consequences: If you start using this as a measure of performance, expect people to submit a lot of very simple PRs, or create a script that keeps reverting the last PR. Their stats will go through the roof. Be smart! This is not an arms race you will win.

Are People Working Efficiently?

A more interesting measure is how long it takes people, on average, to get a PR through approvals. We have the creation and close date for each PR, which makes this easy enough to calculate. We can use the same pivot table, adding another view that shows the average PR close time.

It turns out average is not a great measure. One PR that gets held back because the team is discussing approaches could easily screw up averages. We switched to looking at medians instead.

Here, each row represents a single developer. You can see the median number of days it takes for a developer’s PRs to be approved or rejected and closed. Standouts take longer across a longer period of time, and are worth exploring. 0 indicates the same day.

The standouts quickly come out. Next, the manager’s responsibility is to understand what’s going on. Code reviews can become the bottleneck very quickly and this is a constant source of pain in most companies. For example, PR size could be the cause. By structuring the PRs efficiently, submitters have a better chance of getting responses faster.

Are People Gaining Mastery?

In the table above, the left represents the total number of comments a developer received in a particular month, which also scales with the number of pull requests they’ve made. The right shows how many distinct comments they get per commit on average. People who are learning will start higher on average, then go lower. People who fail to learn will tend to remain high.

For every PR, we have the actual comments provided within Sigma. This lends itself to some interesting analysis. For determining if engineers are gaining experience, we removed any comments provided by the author of the PR. We then looked at two things:

The average amount of text commented on a person’s PRs monthly
The number of comments provided on a person’s PRs monthly

These numbers provide an indication of whether they understand the area they are working in well. When someone is new to the company or switching to a new area, the number of comments and amount of text tend to be high. As they gain mastery (usually 2-3 months) both of these numbers tend to go down and reach equilibrium. If those numbers remain high, that’s a good signal for the manager to go explore further. We need to take into consideration that this would still depend on the complexity of the PRs. Plus, it could sway depending on the type of work being done.

Interns also stood out in these two numbers, with very high numbers on both counts. Considering they are still learning how to work in a professional environment and tend to shy away from merging to the main branch despite being encouraged to do so, this isn’t that surprising.

Is Our Community Strong?

Developers provide reviews to others’ pull requests. Typically as their mastery grows, so do the number of reviews they do. Those who don’t participate, or who are preoccupied elsewhere are visible in this kind of table.

Part of our engineering culture is to review code, provide feedback, and help people grow. Code reviews help both in making sure we avoid mistakes, but also in teaching people and sharing knowledge. Getting a view into who is providing feedback, at what depth, and even to whom, isn’t always easy.

We took our PR comments data and flipped it around. Instead of looking at data per PR submitter, we collated it per reviewer, to see what we can glean. What emerged was a view of reviewing activity and insights:

Who are the most prolific commenters in our engineering org? Were there standouts? Were there senior engineers who should be stepping up more?
Who are the most frequent PR approvers?
Who provides the longest / deepest feedback? PR feedback is not just about “LGTM.” It’s a good way to teach people about coding practices, techniques, etc.
Did someone once submit the first 5000 decimal points of PI as a review comment and change the heat map of average review comment size we were creating?

Data Makes the World Go Round

We found that it’s enough to look at this data once every few months to get a sense of how people are doing. Managers should be actively working with their teams. We don’t need to pay a startup to tell us what we should already know. We were able to see trends in productivity, who might be stuck in getting PRs through, and who is providing a lot of mentorship to the team.

There’s more we can easily do with Sigma, including creating conditional weekly reports to the team about PRs that are stuck or that have been languishing for more than a certain amount of time. We did not find the need. What do you do to understand the efficiency and performance of your team?

Want to learn more? Schedule a demo today!

Read about the missing ingredient to improving customer support.

‍