Measuring for Impact

Measure what matters, not what is convenient or comfortable

When money is at stake, people will go to great lengths to get their share. There is nothing nefarious in that desire. We all want our fair compensation for our work, and we all want a fair price for the goods we buy. What happens though when a seemingly positive goals creates the seeds for less desirable behaviors?

In order to promote greater cross-sell of their banking products, one bank promoted the mantra “8 is Great”. This meant that every one of their customers should have eight accounts with the bank. While the mantra had a nice ring to it, realizing this goal was much harder. This sales goals were brutal, leading some managers and sales reps to open up fake accounts for customers just to meet ever higher goals.

These misdeeds eventually caught up to Well Fargo, one of the largest US banks. The scandal decimated customer trust, resulted in significant fines, and led to a massive restructuring. Metrics are only useful if you are measuring the right things.

The ability to measure what you do helps you understand where and how to improve. Without that understanding, decision-making is based more on whim than reality. The subtle point here is knowing what to measure so that your “where to improve” results in affecting meaningful outcomes and avoiding an uncomfortable hearing before a Congressional committees.

We often lose sight of this idea of measuring for impact rather than activity or vanity. This leads to situations where we measure lines of code or bugs per release instead of more valuable metrics such as customer engagement or satisfaction. The former leads to people gaming the system. The later gives us a much better lens into business results.

Measure what matters, not what is convenient or comfortable
Measure what matters, not what is convenient or comfortable

I have a book coming out next week on the topic of building and scaling communities called “Community-in-a-Box”. I wanted to share my own story about measuring the wrong things in my journey to build a global community and share an excerpt from the book along the way.

The story begins with a fundamental truth, something I only came to understand much later once the community was rapidly scaling:

Having a large number of members is not the success criteria. Success is in the growth of value created within and by the community.

The mistake that is often made by most community builders is in using signups or memberships as proxies of community health. Signups are absolutely not a useful metric to assess the strength of a community. In fact, obsessing over the aggregate number of people in your community can actually lead to a false sense of security that obscures warning signs.

I made this very same mistake with the Enterprise Sales Forum. As I was launching chapters, I was also struggling to understand what would lead to sustainable communities. After all, going through the effort to build chapters just to have them fold would be a huge waste.

I relied on this post about “true fans” to guide my thinking early on. If each chapter built up to 1,000 members and 10% of that group would attend events any given month, that would lead to a strong monthly showing of 100 people to create excitement around the event and create a self-sustaining flywheel through word of mouth.

The mistake I made was on two fronts. First, I was caught up in every chapter needing to reach 1,000 members as a litmus test for community health. Second, I assumed each event needed 100 people to generate excitement.

Getting to 1,000 members is incredibly difficult. It was easy in NYC to build to 1,000 people because it is a city of over 8 million and a metropolitan area of over 20 million people. It also happened to have a thriving meetup culture, a rising tech startup ecosystem, and lots of younger professionals willing to spend their evening at an event. Many other cities did not have that same culture or the sheer population to attract 1,000 B2B enterprise salespeople.

As the community hosted more and more events, I started to see the value in smaller groups. Enterprise Sales Forum events felt comfortable at 75 attendees, but there were also events with 20 people that were amazing, such as the Sales Manager Roundtables or talks with 35 people in places like Singapore and Hong Kong that formed tight-knit communities.

What I came to understand was that absolute size meant less than magnitude of impact. Things tend to be easier at a certain scale, like in promotions and recruiting. The success and long-term viability of a community does not require lots of people though. Success comes from the strength of the alignment to the vision. For that alignment to take hold, it can require as few as a handful of people.

There is a theory in social networking called the “90–9–1 Rule.” It states that for any community built around a network, 90% will be lurkers, 9% will be engaged, and 1% will be heavily involved super users. Specific numbers are less important though than the magnitudes they represent. For communities built around in-person events, these numbers can look like 70–25–5 or 60–30–10. Every community has sporadically engaged lurkers, but what is important is the “engaged” and “super users” are involved and feel motivated by the vision of the community.

So how does any of this relate to measuring impact? Another theory in the social networking world often cited is “Metcalfe’s Law.” The basic premise is that the more people in a network, the greater the value for everyone in the network. Therefore, based on the theory, more value is created from 100 people than from 10. This is what powers Facebook, LinkedIn, Twitter, and other social networks.

Determining what to measure then becomes an exercise in understanding the impact of the community to create and foster that value. Success is in how that value in the community grows. Therefore, measurement requires understanding the value members receive from the community and how they weigh those values.

The community can be evaluated along three key metrics that enable organizers to evaluate and take actions that support the health of the community:

● Level of satisfaction of benefits community members receive

● Weighting of the importance of those benefits

● Likelihood of recommending the community to their network

I call these community pulse metrics, as they speak most directly to the attitudes and feelings of members. The value of these metrics is they give the most unvarnished and transparent view of both the experience and quality of community. Just as an EKG gives doctors an indicator of a patient’s health, community pulse metrics do the same for community organizers.

Community-in-a-Box, a book about building & managing communities
Community-in-a-Box, a book about building & managing communities

I go into more details about the specific metrics in “Community-in-a-Box”. You can also follow updates about the book and additional materials on my website. Most of all though, I hope my story gives you inspiration to look at the things you are measuring and consider this thought, are you measuring activity or are you measuring impact?

Want to learn more about DEVBIZOPS and read more hot takes about IT, technology, and working smarter. Receive our weekly newsletter by signing up to our Substack!

Written by

Thoughts on developers, digital transformation, enterprise agility, community building & software engineering culture. Author 👉 https://twitter.com/marksbirch

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store