How often have we heard the phrase "We don't want to be reactive, we want to be proactive" with the implication that unless we're able to take action in anticipation of events which will impact our business, we're going to be in trouble?

It may be true that taking pre-emptive action seems like a great thing to do when the information enables one to do so.  But as we seem to relearn every few years, past performance is no guarantee of future results, and models which use the past to predict the future are vunerable to changes which make those predictions invalid (just ask anyone at Bear Stearns).  Given that, I think it's worth asking what's wrong with developing really good reactions as an alternative (or compliment) to being proactive?

In responding to events once they've happened, we have the advantage of potentially full information on them, and beyond that, we're not locked in by a system built on what we thought would happen, but are free to react to what actually has happened.  We're given the freedom to fully analyze and understand the situation.

This needs to be balanced against the fact that if we wait too long to react, it's not much better than not reacting at all, but with the proper tools, I believe that reacting in real time can be even better than committing yourself to moving in one direction and hoping that reality accommodates you.

What makes for tool set that allows for real-time reactions?  Though there may be others, in my mind there are two key features of such a tool set:

1. A responsive, flexible analysis environment.  Given that one of the advantages of reactive analysis is that we're not limited by preconceived models, it's important that we not be limited by our tools.  The idea is to know what's happened as fast as possible, and if your tools limit the sorts of analysis you can perform, or can't perform them quickly, your reactions aren't as good as they could be.
2. An infrastructure which supports real-time data capture and distribution.  Obviously to do any sort of analysis, appropriate input data is required.  If that data needs to go through a lengthy process of being captured, cleansed and transformed before it's available, reacting in real-time is essentially impossible.  The data needs to be available when it's fresh and it needs to be easy for analysts to get to.

 I'll go into more detail on the details of each of these items in subsequent posts.

I've got an article in DM Review which hits on one of the things about analytics that's often overlooked--the state of mind and organizational culture that their use and popularization encourages:

One of the things that adoption of analytics across the enterprise does is encourage everyone in an organization to think like an analyst.  This can have effects a bit like those of the Kaizen process improvement practices that have helped to revolutionize manufacturing organizations, but extended to areas of the organization where the rigor of statistically controlled processes hasn't been found (or possible).  Recently CIO magazine focused on how the Oil & Gas industry uses BI, but which featured a quote that I thought captures the idea of an anaytic culture perfectly:  The idea that "It's powerful notion to run a company with the mind-set that virtually every employee is a data analyst." 

Oil companies have always lived and died on BI, says Gary Lensing, VP and CIO for global exploration and production at the $32 billion Hess. "Data drives what we do, always quantifying where that value is." 

"The ability for people on a platform to communicate with people in the home office and work on the same set of data means we can get more production done faster and more accurately," he says. "How you choose to analyze the data and the decisions you make-there's your competitive advantage."

"Engineers and geoscientists and everyone have been taught BI from the start," says Lensing... Give people in any industry access to information along with tools to interpret the past, model the future and imagine different paths between the two, he says, and they can change the trajectory of companies.

There can be powerful systemic from this sort of approach to analytics.  When everyone in an organization is expected to approach problems in an analytical, and fact-driven way, it makes coordination, communication and decision easier, and results in better decisions.  

Erika Morphy of CRM Buyer has an article about the deployment of CRM analytics in the pharmaceutical industry.  My colleague Ted Snyder mentions some of the challenges that deployment of analytics helps these organizations address:

Pharma companies are also applying analytics to more basic, competitive issues, Snyder said. "They want to be able to react quickly to events. Scheduling something like a product launch is never a sure thing, because you can never depend on an approval's timing."

On the other hand, if an already-approved product comes under review by the Food and Drug Administration, pharma companies need to be able to react with as much data on hand as possible, he continued.

In these scenarios, "a lot of data needs to be analyzed very quickly. Then they have to execute on that strategy and enable their sales force appropriately," Snyder explained. Ancillary applications like call planning and and territory realignment come into play as well.

A speedy analytics process has become a competitive advantage for these firms, he said. "Now they can analyze data within days instead of weeks."

It's pretty clear that challenges like the ones that Ted describes aren't unique to the pharmaceutical industry.  To my mind, CRM is one of the best cases for user-friendly analytics.  In many situations, it's not necessary to build sophisticated models, it would be enough just to know which deals have changed status in the week, or to know which customers are your most frequent buyers.  Often that's much more difficult than it should be.  In the context of CRM what's often most important is being able to shift gears quickly and answer questions as they arise from analysis, and this tends to be better addressed by interactive visualization than statistical or predictive analytics.

I'm just back from a busy week at TUCON.  This was the first one we've done where Spotfire has been part of the TIBCO family (the acquisition was announced at the same show last year), and I was really impressed by the event.  Bruce Silver has a good overview of some of the cool things that could be seen at the show (you won't be surprised to know that one of them was Spotfire).

In my mind one of the most exciting things about the show was one that actually flew a bit under the radar.  Given what the news was competing with (the TIBCO messaging appliance announcement, discussion of TIBCO's increasing support for Microsoft, and a number of new product announcements), it's understandable.  However, I think it's the most game changing announcement that the BI industry has seen in sometime.

Ta-da, presenting the first product integrating Spotfire with elements of the TIBCO stack--TIBCO Spotfire Operations Analytics.  It slices, it dices, it does everything but mow the lawn!  Well, and maybe not make you dinner either, but in all seriousness, Operations Analytics has the potential to revolutionize the way that we think about BI, "Pervasive Business Intelligence" in particular, and more generally, how those of us in the information business can deliver insight to our customers. 

The software itself combines Spotfire with TIBCO's continuous event-processing technology, and enables line-of-business professionals to receive pre-specified, data-populated analysis applications in response to business events.  By doing so, Operations Analytics greatly simplifies the task of root-cause analysis, and then, by integrating with other business systems--your BPM queue, for instance--it also makes exception management a lot easier.

Essentially, this is the most concrete realization of in-process analytics around, and the thing I like most about it is that nobody thinks of it as "Business Intelligence" per se.  It's just the application that simplifies their response to a problem in their process--regardless of what that process may be.

Elaine Allen, a professor at Babson college and friend of Spotfire, and her colleague George Recck were recently interviewed by USA Today about their use of Spotfire in helping to build the ultimate fantasy baseball team.  Using Spotfire and other statistical tools, they looked for factors which indicate strong contributors to your fantasy baseball team.

About a year and a half ago, she [Allen] and her students decided to see whether their studies could be applied to an important real-world topic, fantasy baseball.

Using tools developed by the Spotfire division of Tibco Software, Allen and fellow professor George Recck used 21 statistical categories to determine an index value for each player.

The conclusions were published last April and validated some long-held fantasy axioms. For example, consistency is more desirable than performance spikes. Another is that players who score a lot of runs (even if runs are not a category) help create a balanced and successful fantasy team.

However, one of the surprising aspects of the research was a connection between some relatively minor statistics and a player's overall value.

"Walks, doubles, caught stealing and strikeouts had a statistically significant (positive) impact on the index," Allen says.

Not only is it great to have professors like Elaine and George using Spotfire in and out of the classroom, I love seeing the application of analytics to things like fantasy sports.  It's easy to think of it as just a "fun" example, but I think that both wide participation in fantasy sports and the acceptance of analytics over tradition by various professional sports franchises (the current World Series champions, the Boston Red Sox are just one analytically focused team--there are many more) have both played a big role in demonstrating to broad audiences how useful and effective analytics can be.

While no one will argue that the four IT mega vendors (IBM, SAP, Oracle and Microsoft) are an insignifcant part of the Business Intelligence landscape, Mary Hayes Weier's recent article in Intelligent Enterprise "How to Choose Among the Four Bright Lights of BI" definitely overstates the case for their prominence.*

Even if you concede that these vendors are the bright lights of BI--which is debatable--they're definitely not the ones generating heat.  At the moment, hot topics abound in Business Intelligence.  Dynamic visualization is hot.  In-memory analytics, also hot. Real-time Business Intelligence and convergence of BI with other enterprise technologies are also hot topics.  The list goes on, and all of the tools that will ultimately result in the realization of that hottest of topics, Pervasive BI, are themselves hot.  Looking at these topics and technologies, however, one notices a distinct lack of mega-vendors leading the way.  In the cases where they are doing something, it's because they've recently acquired a smaller company that was cutting edge (Microsoft's buying FAST for enterprise search comes to mind).

So sure, maybe they're the leading lights, but they're hardly blazing trails. 

Doug Henschen--the Editor-in-Chief at Intelligent Enterprise--makes some points along these lines, noting that the field is still open for innovation:

[T]his is just the beginning of the journey for the mega vendors in BI, and it's just the beginning for a market that could easily be redefined by developments we can scarcely imagine. The "Bright Lights" article offers a solid analysis of the mega vendor paths forward, but I'd submit that these four companies don't control the destiny of the market or the limits of what you'll be able to do with BI technology in the years ahead.

I agree completely, but I'd take what Doug is saying a step further.  I don't think that IBM, SAP, Oracle or Microsoft control the destiny of what you can you with BI technology now.  They may be talking about moving beyond BI's traditional userbase, but they're not the ones actually doing it. 

* More preceisely, the title overstates the case--the article itself turns out to be an easily digestible comparison of the BI offerings of the mega-vendors.  Interesting, but much less provocative than the title indicates.

We're continuing to work with the Huffington Post on analysis of the primary donation data that I referenced in this post.  Most recently, Sam Stein--keying off the recent dust-up about Obama's "bitter" comments--asked if we could separate the donation for the rural areas of Pennsylvania from the urban and suburban areas around Pittsburg and Philadelphia to see how his donations levels compared to Clinton's in these two distinct areas of the state.

Sen. Barack Obama's political opponents charge that his recent remarks on the economic woes and bitterness of low-income voters put him gravely out of touch with small town Pennsylvanians.

But a review of campaign finance records -- conducted for The Huffington Post by the Spotfire Division of software firm TIBCO -- reveals that it is Obama, not Sen. Hillary Clinton, who has received the majority of donations from these very same Keystone State communities. 

Check out his article here.

We announced the release of version 2.1 of Spotfire at the Gartner BI Summit last week, and got some very positive coverage.  While there are a number of cool new features, the one that people found to be the most interesting are the APIs which we've introduced allowing parts of a Spotfire analysis to be incorporated into mash ups.  What's exciting about this for me is that it's a key step on the road to truly pervasive analytics.

We hear a fair bit about Pervasive, Ubitquious and Embedded BI these days, and not without good reason.  As I mentioned in a previous post, if Business Intelligence is going to grow beyond it's traditional users it's going to have to address the processes of other users directly, rather than in the guise of BI.  By allowing IT organizations to combine analytic applications directly with the tools currently addressing these processes, Spotfire now makes it much easier to do this.

This, to me, is precisely the direction that BI and Analytics need to go if it's going to expand to the other 80% - 90% of the enterprise.  End users who aren't already using BI as such probably won't ever use it.  But that doesn't mean that they can't benefit from the analytic tools and techniques that have blossomed in recent years, they just need those tools to exist in process- or function-specific contexts.

As long as vendors try to control the frame in which BI is delivered, I think that they are unlikely to gain much acceptance beyond their current user communities.

Over at Intelligent Enterprise, Niel Raden has some thoughts kicked off by Gartner's recent report on the future of BI,  "Emerging Technologies Will Drive Self-Service Business  Intelligence," which takes positions that Gartner analyst Kurt Schlegel reitterated and elaborated on in an interview with Doug Henchen and again at the Gartner BI summit in Chicago this last week.

I'm having some problems with a March 20, 2008 article titled "Gartner: Emerging Technologies Will Help Drive Mainstream BI Adoption." This has been the Holy Grail of BI vendors for over a decade — to increase the number of "seats" using their products, widely reported to be about 20 percent of an organization but clearly much less than that. What troubles me the most about this article, or rather, about Gartner's analysis, is the supposition that new technology is going to crack this old chestnut.

Raden goes into some detail about why he thinks each of the five technologies which Gartner identified (Interactive visualization, In-memory analytics, Search integrated with BI, SaaS and SOA) are insufficient to drive broad scale adoption of Business Intelligence.  If Business Intelligence remains as it currently is, he's obviously right--the pool of users who want what traditional BI offers is largly tapped out, and so broader adoption will mean that BI needs to offer something which will appeal to sets of users who, up until now haven't wanted or needed BI.  But it's important to point out that this fact doesn't mean there aren't opportunties for BI-like technologies to improve business processes.  Raden's partner James Taylor points out that mainstream adoption of Business Intelligence "would mean that everyone in an organization - down to the people paid minimum wage at the front line - are making better decisions thanks to the understanding an organization has of its data."  Since we're clearly not there yet, there will be opportunities to bring BI in one form or another to a much broader set of people than the current set of BI users.

As BI reaches past its core users, it will need to adapt to suit a broader set of needs and business processes.  And if you look at the five technologies which Gartner identifed, all of them either make it easier for non-traditional users of Business Intelligence to access and interact with their data (Interactive visualization, In-memory analytics, Integrated Search), or ease the support burden of additional users and use cases on IT organizations (SOA, SaaS).  It seems to me that even if these technologies aren't sufficient to drive increasing BI adoption, they do enable it.

In reality, what will drive increasing BI adoption are the same things that drive all business innovation--things like increasing competition, global challenges, tighter regulatory environments and other issues that businesses across all industries have to face.  The more interesting question is what will broad-scale BI adoption look like, and it's one I'll address in a subsequent post.

In our annual analysis of the 64 teams which enter The Dance, we'd picked Siena as this year's Cinderella, and I was feeling great when the 13 seed beat number 4 seed Vanderbilt (even though we'd picked them to make the Final Four).  Unfortuately they faltered in the second round, and left Villanova, Western Kentucky University, and Davis as the remaining lower bracket challengers.

I'm still holding out hope for Texas and Tennessee, who we picked to appear in the Final Four.

A few of my colleagues and I here at Spotfire are political junkies, and with the primary campaigns running so long, there's been a lot to take in.  One of the conversations we've been having recently touches on the vastly greater number of people participating in the primaries this year, and in particular, the fact that the number of people donating to candidates in the primary is many times greater than it has been in previous cycles.

In order to see what sort of impact this has had, we threw data from the FEC into Spotfire, and took a look.  What we found was pretty interesting.  Some of it's been written up by the Huffington Post today:

But if the fundraising statistics from key battleground states provide any indication of popular support, both Democratic candidates seem well positioned to take on their GOP competitor come November.

According to data compiled by Spotfire Division of software firm TIBCO, Obama and Clinton have raised nearly twice as much as McCain in the traditionally important electoral states through the February filing period.

The interesting insight here is that with the high-volume, small-scale donations enabled by modern campaign architecture, the numbers of people giving money may serve as a good proxy for levels of voter support.  You can take a look at our analysis here, but based on our analysis, we would have been able to call Texas and Ohio for Clinton a good month before the elections, and are currently picking Clinton in a tight win over Obama in Pennsylvania and West Virginia. 

James Kobielus has an interesting post up about complex event processing and what it implies for the user interface:

The problem with the term "complex event processing" is that it seems to imply a complex UI--hence, that CEP's potential user base is limited to rocket scientists, Wall Street quants, IT industry analysts (gasp!), and other folks who are professionally obliged to handle (indeed, embrace) complexity.

But complexity is part of the problem, not the solution, where CEP for I&KM is concerned. We can't stop the world of business from growing more multifaceted. But we can and should filter it all down to the simplest, most meaningful, most actionable experience, targeted specifically to each user, and contextualized precisely to each and every occasion.

Conceptually, James is clearly right--who's going to disagree with simple, meaningful, actionable and perfectly contextualized--but how do we get there?

Most obviously, this is a great opportunity for BI/CEP software to take a page from web 2.0 software and give users the freedom to customize their environments, and the tools to see how other users have done it. End users whose environments deliver precisely what they want will deal much more effectively with information than those who can only use what a developer (no matter how skilled) gives them.

However, even beyond tools for easy interaction and customization of their visual environment, in order to make users really effective--whether the context is CEP or anything else--applications should be driven by business problems, rather than the tools and technologies available to IT professionals. Users are the experts in their domains, and they will know what qualifies as "targeted specifically" and "contextualized precisely."

Ideally, you would have a platform upon which users and/or their colleagues in IT departments could build applications across the full range of business processes that exist in an enterprise. That way you could leverage IT resources and let users take advantage of a common environment, but still address their business processes in the way that best suits them.

Hmm, I wonder where I could get a platform like that...

I have a short piece up over at CIO.com about how some of the difficulties candidates have in selling detailed polices mirror the troubles that can arise when trying to push an analytic initiative forward in the business world, and that the solution there, as here is to focus on the results:

Certainly, the challenges that bedevil candidates in trying to sell analytic policies are the same ones that people trying to implement analytic initiatives in the business world often face: Because numbers convey a sense of certainty, how do you use analytics without appearing to know more than you often do, while still clearly communicating the value of analytical thinking?

In order to get there, people who want to bring analytics into their organization need to understand how to talk about the value of analytics to an audience which may not be interested in the behind-the-curtain details.

Effectively conveying the importance of analytics is a worthy cause, but trying to win over the uninitiated shouldn't be done to the exclusion of thinking about where we're going from here. As I mused in the last bit of the piece on CIO.com:

Politics has been on the forefront of some interesting cultural and business revolutions. Polling, cinematic television advertising and relentless "messaging" have all been part and parcel of successful political campaigns. The analytics tools exist to extend into the policy beyond the politics, and it may not be far off.

We've already stated to see the beginnings of such a change in Health Care with the advent of "Evidence-Based Medicine" and I expect it won't be long before the arrival of other areas where it's simply expected that decisions will be based upon the best available evidence.

As with most things, the public sector will likely lag the private, where we can already see the evidence- based methodology appearing in some novel places. For instance, Fog Creek software makes a bug tracking product which incorporates "Evidence Based Scheduling," a very clever method for producing reliable estimates for software development from developers who work at different rates and have different levels of accuracy in their own forecasting.

As for public policy, we already see this move to some extent in the debate over global warming, with both sides of the discussion arguing over the details of research and what it implies for policy decisions (this debate shows how important it is to have good data and lots of it, if you want to avoid the situation where the same data can be used to tell contradictory stories). I have a feeling that as the tools, and in particular, the data sources continue to develop, we'll see much more of this in the future.

If you follow football at all, you'll know that this weekend is the super bowl. The Giants and Patriots will be playing, and there are a host of fun side events. Some of my colleagues at Spotfire have put together a cool analysis of likely score combinations and given it a cool visual representation.

The other day, a friend sent me a really interesting interview with a Hedge Fund manager (HFM), discussing, among other things, the sub-prime mortgage problems, and what it implied for rigorous analytic trading. There are some very interesting observations about what constitutes expertise, and how effective even the best analysis can be. Here's a key graf:

HFM: So I think in comparison to the overall quality of mortgage origination in the last, call it, 3 or 4 years, ours was really much better. So I think they're happy we did a better job than our competitors--but they're not happy they lost money.

n 1: Is the person who ran that--is he going to get fired?

HFM: He was already fired.

But--to get back to the paradigm shifts--here was a guy who knows the market really, really well, who is a real expert in the nuts and bolts of mortgage lending, and really knew the collateral really well--but he was a true believer, and I think a lot of people were... true believers in the paradigm.

And there were other people at the firm, say, at the middle of last year, who were not mortgage experts, who were saying, you know, "I see the run-up in housing prices in some of these geographies, and I just don't really get it. I go down to Florida and see the forest of cranes, and I just really wonder, who's going to be in all those apartments... I see these commercials on TV, you know, about low-doc, no-doc mortgages--and there is no way, there is no way that this is not going to end badly... This is the perfect example of a bubble--and we should be short, we should be short sub-prime paper."

But the person who was the expert, the person who ran the sub-prime business, who traded sub-prime paper and issued the CDOs, he was a true believer in the paradigm: "In 2003, people said that the credit quality of the average sub-prime mortgage was deteriorating, and now look, those mortgages have performed fine. The sub-prime market works."

And, hey, he was the expert--you defer to the expert.

This touches on something which I've been thinking about recently: As we rely more on predictive tools and analytics, how can we avoid missing the forest for the trees? Nobody doubts the value of expertise, but what about perspective?

What if the non-expert traders in this scenario were able present a convincing data-driven case that the sub-prime mortgage was a bubble? How many tens of millions of dollars could that have saved?