Trying DX Core 4 on Skeptical Developers: First Impressions

We tried the new DX Core 4 framework. Did it help us overcome our long-time skepticism of ‘developer productivity’ metrics? Kind of.

Measuring the effectiveness of an engineering organization is an age-old problem. Are my engineers efficient, productive, focused on the right things, producing high quality work - and how do I improve things if they are not?

Many frameworks have come and gone over time, most recently with DORA rising as an easy-to-grok but very difficult to implement north star for many teams. That last part about being difficult to implement is key - even when the frameworks make sense, they have to be achievable. DX Core 4 aims to solve this problem by unifying aspects of  the popular DORA, SPACE, and DevEx frameworks and address head on the benefits and limitations of each.

How it works: Core 4 outlines four key dimensions of productivity, which is a useful number to have to make sure the framework title rhymes – Speed, Effectiveness, Quality, and Impact – each with a specific top-level metric. We won’t go through it in too much granular detail here, we highly recommend going to the source for that. What we wanted to investigate, as folks who wake up every day building a tool focused on improving engineering velocity and performance, was two questions:

#1 - What do we think of DX Core 4? Is it helpful for leaner teams like ours? We’ll try to answer that question in this post.

#2 - How does what we build (API virtualization) impact Core 4? This will be the topic of our next article.

Taking DX Core 4 for a spin

Everyone on the WireMock team has been around more than their fair share of attempting in vain to quantify the health of an engineering org, so as we picked up the exercise we had a few reservations. Measuring PRs? Semi-arbitrary. DevEx index? Yellow flags, at least.

After going through it though, I think it’s safe to say that we think they work on balance within the framework in a way that feels directionally correct. I should also acknowledge, the DX team and the folks they worked with also did some heavyweight research on this, with pretty good sample sizes and methodologies – so while we’re giving our overall gut feelings, credit where it's due that they tried very hard and transparently to back this up with more than just vibes.

Trying the self assessment: Let’s take a quick look at our own scores from the self-reported assessment DX recommends as a starting point:

It’s entirely possible our bias to liking this framework is due to the fact that we performed well on it. In my experience, everyone who says I’m doing great is very wise. But, most important we felt like it put targets on how we all generally felt but sometimes struggled to articulate during retros and process discussions:

• Our developers are mature and ramped, and we are good at addressing workflow issues as they come up
  
  
• But, for a time now we’ve spent a lot of our energy on important, but not feature-generating work. This wasn’t the wrong call, it was intentional - but we also aim to change it in future quarters.

We think this delivers a helpful insight about our development efforts. The truth of that is right there in black and white, and we feel good about that because it means over the remainder of the year we can judge our intentions for what we want to be seeing differently in the team against something concrete. For instance, we actually expect as new capability work increases, PRs/week may somewhat drop, as our current velocity reflects a specific type of work that has been happening in smaller batch sizes than larger new capabilities might. And with Core 4, we’ll get to see if that’s true.

The other thing we liked about Core 4 is that it’s easy. The survey is quick, the metrics you want to monitor are in most cases readily available, and overall the effort is about as non-interruptive as a process like this can be just to get a baseline in progress. That was not the case with systems like DORA, in the past.

Are we going all on in on DX Core 4?

All this being said, we may or may not adopt Core 4 as an orthodox part of how we measure our organization going forward, but we certainly are glad we have this point in time comparison, and we have every expectation that we will compare ourselves against it at least a few times throughout the year.

TLDR: Core 4 is a welcome addition to a tricky, complicated and long-simmering discussion about how to measure teams, and may be the best effort yet to balance comprehensiveness, subjectivity and

Coming soon: How does WireMock Cloud / API simulation improve your DX Core 4 metrics?