Measuring the Health of Ecosystems

I gave a keynote recently that pulled together a number of different threads I've been working on over the last year. The key message is simple -- how do you measure the health of a complex Ecosystem like Eclipse?

Almost every ecosystem I know of (including ours) spends a lot of time measuring size. We measure the number of download requests, number of members, number of attendees at events, number of posts and news releases, etc, and we try to compare and do all kinds of wonderful things with these numbers. But is measuring size really a measure of health? I think it's clear that we would rather have a very healthy ecosystem with steady growth than an unhealthy ecosystem. As an example, I don't know many people who look at a graph like this and claim that earth is getting more healthy:

Measures of health of the earth might be better quantified by looking at global per capita cereal production, or worldwide burden of disease trends. The bottom line is that although we hope to see our ecosystem growing, growth is really a symptom of health - not a measure of it.

So how can we measure the health of a technology Ecosystem? And more importantly to me, how do we operationalize this? Spending 100 hours a month gathering data just will not scale. Well, turns out there is quite a bit research going into topics like this around the world, including some work sponsored by the Eclipse Foundation.

As I blogged last fall, there is a great HRB article called "Strategy as Ecology" which describes three key measures of the health of an ecosystem:

• Productivity of the Ecosytem -- how much value is being created by the ecosystem.
• Robustness -- how durable and able to adapt is the ecosystem to external events.
• Niche Creation -- the ability to expand the ecosystem with meaningful diversity.

For the rest of the keynote, I gave some examples that I think prove how healthy the Eclipse Ecosystem is, within the framework of productivity, robustness and niche creation.

Productivity

It's important to be clear what is meant by productivity here in the context of measuring health -- we are talking about the ability of the Eclipse Ecosystem to create economic value, not the individual productivity of developers. So how do we measure economic activity (and hopefully for me, not spend 100's of hours researching)? I have two quick-and-dirty measures. First, I believe strongly that job postings are a great indicator of economic activity. If a framework or tool is generating economic activity, then I would expect to see more and more organizations seeking to pay employees for those skills. Sure enough, Eclipse appears to be generating ever increasing economic activity, about 2.5x over the last 2 years:

If the Eclipse Ecosystem was productive, I would also expect to see an increasing number of products based on the Eclipse Platform. Measuring this precisely could be extremely time consuming, but luckily there are some tools that help spot trends. I would argue that as we see more products based on Eclipse, we would see more product related press releases mentioning Eclipse. Sure enough, over an 4 year period we see an 8x increase in the number of press releases mentioning Eclipse technology (this graph plots number of press releases at Google news archives per year mentioning Eclipse technology):

Finally, it's encouraging to see that Wall Street also agrees that Eclipse is a productive Ecosystem. Take a look at Brent William's EclipseCon presentation and the commentary surrounding it to see evidence that Wall Street bumps up stock multiples for companies that have a clear strategy around Eclipse.

Robustness
A healthy ecosystem is able to adapt and respond to external events. For example, thriving blueberry industries appeared in the north east of US and Canada in the late 1800's after devastating forest fires whipped through. It turns out that blueberries thrive after forest fires -- allowing the local ecosystem to adapt and respond to external events.

Although we haven't had any real devastating forest fires in the Eclipse Ecosystem (maybe a couple of brush fires), I would argue the Eclipse Ecosystem is prime to handle any number of negative events. A great example of this is the CDT project. The left hand side of this diagram shows just some of the organizations that are contributing to the CDT project, and the right side shows just some of the organizations that are shipping products and services based on or with the CDT project. There is clearly a significant level of diversity and if any one organization experienced a "forest fire", the CDT ecosystem would very likely continue strong.


Another good example of the robustness of the Eclipse Ecosystem is with respect to Ruby support. In early March many people had commented that the RadRails project looked stalled and that the developers had moved on. In fact, Aptana has taken on a leadership position and now the Eclipse Ruby ecosystem is stronger than ever.

Niche Creation

If the Eclipse Ecosystem is healthy, then we should see rapid adoption of Eclipse in new technology niches. For example, through 2006 I noted a few of the many Eclipse related product announcements specific to Ajax on a graph showing some of the commercial uptake of Ajax:


Moreover, if the Eclipse ecosystem is healthy, we should see it be at least "neck and neck" competitive in new technology niches versus longer established ecosystems such as Visual Studio. For example, we can see that Visual Studio has a lead in economic activity over Eclipse (but it is narrowing):


But in the context of new technology niches, Eclipse is neck and neck with Visual Studio:


Conclusions

Measuring the size of an Ecosystem is not sufficient, we know we need to track the health of the Ecosystem also. Operationalizing the tracking of this information can be very difficult and time consuming, but there are a number of "back of the napkin" tools and calculations we can use to demonstrate our overall health.

[edit: fixed broken image of world population]