Trends and Outliers

Gift Finder in the NYT

2008-12-12T14:41:00Z

The Bits blog of the New York Times has a nice look at the Spotfire Gift Finder today:

Spotfire, a division of the software maker Tibco, has used its statistics expertise to come up with a tool for sorting potential presents. With the Spotfire Holiday Gift Finder, you can churn through thousands of products, including apparel, electronics, jewelry and tools. The software lets people narrow down their choices based on price and reviews and then points to the appropriate spot on Amazon.com where the product can be purchased.

Have a look!

Happy Holidays

2008-11-26T04:33:00Z

Happy Holidays from everyone at TIBCO. We've put together an application which should simplify your gift shopping this year, have a look!

Well, They Are Addictive

2008-11-04T18:32:00Z

A tounge-in-cheek, yet insightful, article from John Myers in which he refers to Analytics as the "Soft Drugs of Business"

The gray area that has emerged is the class of “soft-drugs” known as analytical business intelligence. To the quants in the organization, analytical applications are viewed as free choice that provides enlightenment to the user and benefit to the organization to “see outside the box” of existing standard operational and financial reports. The business stakeholders or quants fight for the freedom of the analytical applications since they usually provide greater value over the existing IT-sponsored operational or financial reports. To the data governance organization, analytical applications are viewed as something that should be controlled and regulated since they could lead to the “destruction of the youth of the company”

Well worth reading the whole thing.

Building a Fact-Driven Enterprise

2008-11-04T15:41:00Z

I've got an article up at MyCustomer.com which reviews some of the interesting pieces available to organizations who want to compete on analytics:

Because of the importance being placed on analytics, it’s no wonder that the noise from technology vendors can be deafening. The sheer volume of players can complicate the technology selection process and mask the best practices around building and deploying analytics in the enterprise. 'Pervasive analytics', 'prescriptive analytics', 'predictive analytics' and 'business analytics', all have the same stated benefits, so what makes them different? To make the right selection, it's important to understand the various technology options available and be able to separate the wheat from the chaff when it comes to how vendors market themselves.
Instead of focusing on marketing terminology, more light may be shed by breaking down the categories of vendors vying for the analytics pole position. These categories include statistics vendors, vertical application vendors, business intelligence (BI) vendors and visual analytics vendors. In the end, most organisations will likely find that some combination of these approaches will be optimal, as no single approach can solve all needs.

You can read the rest here.

Spotfire 2.2 and Network Analytics

2008-10-29T13:56:00Z

Earlier this week, we announced the release of Spotfire 2.2, the latest update to the Spotfire platform. It's always good to get an update to the platform into the market, and we've made some great strides with Spotfire this year (a topic for another post), but I'm particularly pleased with some of the things we've added in this release.

Spotfire has historically (10+ years) been a leader in in-memory, interactive visualization, and given how much end users like being able to actually understand their data, it's not surprising that we've started to see other vendors adding some data visualization capabilities to their offerings. Not-unrelatedly, we've started to get some questions about whether or not our core historical strengths were enough to continue differentiating ourselves from the rest of the market.

Without getting into the other things that we're doing to marry user-driven analytics with predictive analytics, event-processing and other enterprise technologies, the 2.2 release of the Spotfire platform provides a great response to questions about how Spotfire is different.

The two biggest additions are both new visual analysis tools:

The 3-D Plot allows for the visualization of multiple dimensions on a single plot. While you can add multiple dimensions to a 2-D dot plot with the use of color, shape or size, or by trellising multiple plots, it's not always easy to identify trends within groups, or across different plots.

The 3-D plot addresses some of those challenges, and provides an understandable visual framework for displaying results from statistical techniques such as Principle Components Analysis (a dimension reduction technique for highly multi-variate data).

It's also great for cases, such as the example shown here--measurements from the drill hole of an oil well--where the actual data are measurements made in three dimensions.

Network Analytics, a extensible visualization tool for navigating and analyzing networks, is built entirely using the Spotfire public SDK, and it's something that I'm really excited about.

Wearing my analysis-loving geek hat, I think that analysis of networks is going to be one of, if not the, hottest area of data analysis in the not-too-distant future. It's been used extensively for years in a few areas such as intelligence and other specialized fields, but its value is becoming more and more evident as everything becomes ever-more connected.

For instance, I'm a member of a Harvard-sponsored working group on Food Safety (last meeting detailed here), and it's absolutely critical for the FDA to be able to quickly traverse the immense network of food suppliers when there is an outbreak of food-borne illness, not only to identify the source, but to quickly clear the suppliers whose products aren't at risk.

That's not something that can be readily done with other types of visualization or analysis techniques.

Similarly, social networking sites such as LinkedIn, Facebook, Twitter and others create networks, the analysis of which is interesting to many, and a real business opportunity for folks who would like to advertise to targeted groups of consumers. Such networks are only going to proliferate in the future, and the ability to understand them will be key to decision making across industries and disciplines.

Beyond those two items, there are a number of other improvements to the platform, but it's these two pieces that I'm really excited by.

Best Thing I Read Today

2008-10-22T20:33:00Z

"Analyzing data in aggregate is a crime against humanity."

That's according to Avinash Kaushik, Analytics Evangelist (hey, cool title!) at Google. He goes on to say:

Bold statement, but the reality is that a “monolith” does not come to your website. Your site does not exist for a singular reason either. The core drivers of traffic are magnificently different for each core group of visitors.

So your website’s really a mix of Visitor Sources, Visitor Behavior and your Desired Outcomes.

When you look at all that in aggregate you get nothing. You think Average Time on Site means something. No! You think All Visits and Overall Conversion Rate gives you insights. Nyet! You think understanding Keywords without drilling down to each search engine will be awesome. Non!

If you want to find actionable insights you need to segment your web analytics data.

(emphasis mine)

The only thing that I'd change is that his comments are applicable to all data, not just web analytics data. If you want to find actionable insights, aggregations just won't cut it. You've got to move beyond the cube.

Spreadsheets Don't Cause Problems?

2008-10-20T21:27:00Z

I'd suggest that anyone who thinks that it's not possible to cause all manner of trouble with uncontrolled spreadsheets read this:

A formatting fubar involving an Excel spreadsheet has left Barclays Capital with contracts involving collapsed investment bank Lehman Brothers than it never meant to acquire.

Working to a tight deadline, a junior law associate at Cleary Gottlieb Steen & Hamilton LLP converted an Excel file into a PDF format document. The doc was to be posted on a bankruptcy court's website before a midnight purchase offer deadline on 18 September, just four hours after Barclays sent the spreadsheet to the lawyers. The Excel file contained 1,000 rows of data and 24,000 cells.

Some of these details on various trading contracts were marked as hidden because they were not intended to form part of Barclays' proposed deal. However, this "hidden" distinction was ignored during the reformatting process so that Barclays ended up offering to take on an additional 179 contracts as part of its bankruptcy buyout deal, Finextra reports.

(HT: Andy Hayler)

Harvard Executive Session on Food Safety

2008-10-15T13:20:00Z

I was recently asked to participate in the Harvard University Executive Session on Food Safety--hosted by the Kennedy School of Government--dedicated to enhancing cooperation and data sharing between the various components of industry and the agencies responsible for preventing and responding to outbreaks of food-borne illness. It was attended by senior people from the FDA and State health agencies, as well as leaders from all points in the food-supply chain. I was invited to provide some insight about how analytics might be useful in tracing products back to their origins in the case of outbreaks, and how such outbreaks could be predicted and prevented.
Interestingly, and perhaps unsurprisingly, the challenges aren't predominantly analytic, but related to data integration. Think for a moment about what the FDA needs to go through to trace an outbreak:

From a set of cases, they need to track down where those who are ill ate or bought their food, and from each of those locations, track the implicated food back along its supply chain to its source, looking for points at which multiple cases converge to identify the problem.

If you’ve got the data on who bought what from whom and when, it’s a pretty easy problem. However, the required data don’t conveniently live in someone’s data repository, but are diffused across all points of the food-supply chain. Based on some quick googling, it seems that there are roughly 1 million restaurants in the United States and nearly 200k grocery stores. They are sold to by a vast and complex network of suppliers, distributors, wholesalers, shippers and producers. There is no standard for keeping shipping records, nor standard for describing which items are shipped—you wouldn’t believe how many varieties there are of a single vegetable there are, and how many more names those varieties go by.

The challenge of being able to navigate this data is immense—literally millions of different silos of information, much of it stored only in paper documents such as invoices. Being able to do it under the kind of time pressure the FDA faces when there is an outbreak of food-borne illness is tougher still.
However, it is a tractable problem, and the session yesterday was a step towards a solution, and I’m looking forward to further sessions with the group.

Visit to the Boulder BI Braintrust

2008-10-13T13:36:00Z

On Friday, I visited the Boulder BI Braintrust with Spotfire's Sr. Director of Marketing, Mark Lorion. Mark and I were invited to give the folks in the Braintrust an overview of Spotfire and get feedback from some of the brightest people in Business Intelligence and Data Warehousing.

Though the weather wasn't the perfect Colorado blue sky and fall air that I, being a CO native, bragged to Mark about, the visit with the folks at the Braintrust more than made up for the rain. It was great to have so many smart people together in a single room, and get their feedback on some of the things that we're doing and planning here at Spotfire. One thing that the group found particularly interesting was the on-going integration of Spotfire with TIBCO's event-processing and BPM software, currently sold as Operations Analytics. Richard Hackathorn blogged the meeting, and describes the integration:

Operational BI is seeking advanced analytics that operate upon event streams. The gaps are quite apparent between mainstream BI sitting on top of the enterprise business data versus CEP (like TIBCO) sitting on top of the enterprise business processes. Spotfire can act as an integrating component that bridges those gaps. If Spotfire moves beyond the pixels-on-the-screen, its integration value will be based upon consuming data from and generating data to the BI infrastructure.

It was great to see other people as excited about this as I am. As I've mentioned before, I think that the way that BI becomes pervasive is to embed itself into business processes, and doing analytics on the event stream presents an obvious opportunity for such an integration.

I also recorded a podcast with Claudia Imhoff, which you can find on the Braintrust's podcasts page.

What Does Increased Integration of R Mean?

2008-10-01T15:41:00Z

I was pointed to an interesting post on the growing prevalence of R support in statistical packages.

In terms of OSS, we are seeing wholesale integration of R into such packages as Spotfire and SPSS. SPSS it seems is even offering a menu system to access R routines! I’ve also heard rumors that SAS is demoing an R interface in their SAS/STAT Studio product.
In my opinion, integrating R into each of these packages will have the effect of making statistical code and models portable across packages. This will eventually dilute the value of the packages statistically and make their value being evaluated on ability to manipulate and import data, connect to databases, and how effectively they put together their menu systems.

On one hand, I woudn't say that Spotfire has a "wholesale integration of R," but with the recent addition of S+ to the Spotfire platform, it's clear that our support for R is stronger than ever before.

The point of the original comment stands. But even with the growing stature of R, there's a whole lot of value packaged up in the "ability to manipulate and import data.. and how effectively they put together their menu systems." Offering end-users the ability to easily manipulate their data, and effectively interact with it is the entire value proposition for several software vendors, and a big part of the value for others (including Spotfire). Statistical and predictive analytics are becoming a bigger and more important part of Business Intelligence, but even though they comprise a relatively small part of most BI vendors' offers, BI is still a multi-billion dollar market.

In any case, if R becomes the defacto language for statistical modeling, I like Spotfire's chances of competing with SAS and other BI vendors on quality of user experience!

Quick Reactions

2008-07-23T14:43:00Z

How often have we heard the phrase "We don't want to be reactive, we want to be proactive" with the implication that unless we're able to take action in anticipation of events which will impact our business, we're going to be in trouble?

It may be true that taking pre-emptive action seems like a great thing to do when the information enables one to do so. But as we seem to relearn every few years, past performance is no guarantee of future results, and models which use the past to predict the future are vunerable to changes which make those predictions invalid (just ask anyone at Bear Stearns). Given that, I think it's worth asking what's wrong with developing really good reactions as an alternative (or compliment) to being proactive?

In responding to events once they've happened, we have the advantage of potentially full information on them, and beyond that, we're not locked in by a system built on what we thought would happen, but are free to react to what actually has happened. We're given the freedom to fully analyze and understand the situation.

This needs to be balanced against the fact that if we wait too long to react, it's not much better than not reacting at all, but with the proper tools, I believe that reacting in real time can be even better than committing yourself to moving in one direction and hoping that reality accommodates you.

What makes for tool set that allows for real-time reactions? Though there may be others, in my mind there are two key features of such a tool set:

A responsive, flexible analysis environment. Given that one of the advantages of reactive analysis is that we're not limited by preconceived models, it's important that we not be limited by our tools. The idea is to know what's happened as fast as possible, and if your tools limit the sorts of analysis you can perform, or can't perform them quickly, your reactions aren't as good as they could be.
An infrastructure which supports real-time data capture and distribution. Obviously to do any sort of analysis, appropriate input data is required. If that data needs to go through a lengthy process of being captured, cleansed and transformed before it's available, reacting in real-time is essentially impossible. The data needs to be available when it's fresh and it needs to be easy for analysts to get to.

I'll go into more detail on the details of each of these items in subsequent posts.

Everyone a Data Analyst

2008-06-05T15:50:00Z

I've got an article in DM Review which hits on one of the things about analytics that's often overlooked--the state of mind and organizational culture that their use and popularization encourages:

The guiding principle of businesses with analytic cultures is that they see things as they are. When something sounds too good to be true, they investigate further. They insist on looking closely at available data and challenging assumptions and commonly held biases before leaping to conclusions. This culture of fact-based decision-making is a key to these businesses’ success – and not just success in the cases where they have applied analytics to their business processes, but throughout their organizations.

One of the things that adoption of analytics across the enterprise does is encourage everyone in an organization to think like an analyst. This can have effects a bit like those of the Kaizen process improvement practices that have helped to revolutionize manufacturing organizations, but extended to areas of the organization where the rigor of statistically controlled processes hasn't been found (or possible). Recently CIO magazine focused on how the Oil & Gas industry uses BI, but which featured a quote that I thought captures the idea of an anaytic culture perfectly: The idea that "It's powerful notion to run a company with the mind-set that virtually every employee is a data analyst."

Oil companies have always lived and died on BI, says Gary Lensing, VP and CIO for global exploration and production at the $32 billion Hess. "Data drives what we do, always quantifying where that value is."
"The ability for people on a platform to communicate with people in the home office and work on the same set of data means we can get more production done faster and more accurately," he says. "How you choose to analyze the data and the decisions you make-there's your competitive advantage."
"Engineers and geoscientists and everyone have been taught BI from the start," says Lensing... Give people in any industry access to information along with tools to interpret the past, model the future and imagine different paths between the two, he says, and they can change the trajectory of companies.

There can be powerful systemic from this sort of approach to analytics. When everyone in an organization is expected to approach problems in an analytical, and fact-driven way, it makes coordination, communication and decision easier, and results in better decisions.

CRM Analytics in Pharma

2008-05-19T14:50:00Z

Erika Morphy of CRM Buyer has an article about the deployment of CRM analytics in the pharmaceutical industry. My colleague Ted Snyder mentions some of the challenges that deployment of analytics helps these organizations address:

Pharma companies are also applying analytics to more basic, competitive issues, Snyder said. "They want to be able to react quickly to events. Scheduling something like a product launch is never a sure thing, because you can never depend on an approval's timing."
On the other hand, if an already-approved product comes under review by the Food and Drug Administration, pharma companies need to be able to react with as much data on hand as possible, he continued.
In these scenarios, "a lot of data needs to be analyzed very quickly. Then they have to execute on that strategy and enable their sales force appropriately," Snyder explained. Ancillary applications like call planning and and territory realignment come into play as well.
A speedy analytics process has become a competitive advantage for these firms, he said. "Now they can analyze data within days instead of weeks."

It's pretty clear that challenges like the ones that Ted describes aren't unique to the pharmaceutical industry. To my mind, CRM is one of the best cases for user-friendly analytics. In many situations, it's not necessary to build sophisticated models, it would be enough just to know which deals have changed status in the week, or to know which customers are your most frequent buyers. Often that's much more difficult than it should be. In the context of CRM what's often most important is being able to shift gears quickly and answer questions as they arise from analysis, and this tends to be better addressed by interactive visualization than statistical or predictive analytics.

Back From TUCON

2008-05-13T19:53:00Z

I'm just back from a busy week at TUCON. This was the first one we've done where Spotfire has been part of the TIBCO family (the acquisition was announced at the same show last year), and I was really impressed by the event. Bruce Silver has a good overview of some of the cool things that could be seen at the show (you won't be surprised to know that one of them was Spotfire).

In my mind one of the most exciting things about the show was one that actually flew a bit under the radar. Given what the news was competing with (the TIBCO messaging appliance announcement, discussion of TIBCO's increasing support for Microsoft, and a number of new product announcements), it's understandable. However, I think it's the most game changing announcement that the BI industry has seen in sometime.

Ta-da, presenting the first product integrating Spotfire with elements of the TIBCO stack--TIBCO Spotfire Operations Analytics. It slices, it dices, it does everything but mow the lawn! Well, and maybe not make you dinner either, but in all seriousness, Operations Analytics has the potential to revolutionize the way that we think about BI, "Pervasive Business Intelligence" in particular, and more generally, how those of us in the information business can deliver insight to our customers.

The software itself combines Spotfire with TIBCO's continuous event-processing technology, and enables line-of-business professionals to receive pre-specified, data-populated analysis applications in response to business events. By doing so, Operations Analytics greatly simplifies the task of root-cause analysis, and then, by integrating with other business systems--your BPM queue, for instance--it also makes exception management a lot easier.

Essentially, this is the most concrete realization of in-process analytics around, and the thing I like most about it is that nobody thinks of it as "Business Intelligence" per se. It's just the application that simplifies their response to a problem in their process--regardless of what that process may be.

Fun with Baseball

2008-05-07T23:28:00Z

Elaine Allen, a professor at Babson college and friend of Spotfire, and her colleague George Recck were recently interviewed by USA Today about their use of Spotfire in helping to build the ultimate fantasy baseball team. Using Spotfire and other statistical tools, they looked for factors which indicate strong contributors to your fantasy baseball team.

About a year and a half ago, she [Allen] and her students decided to see whether their studies could be applied to an important real-world topic, fantasy baseball.
Using tools developed by the Spotfire division of Tibco Software, Allen and fellow professor George Recck used 21 statistical categories to determine an index value for each player.
The conclusions were published last April and validated some long-held fantasy axioms. For example, consistency is more desirable than performance spikes. Another is that players who score a lot of runs (even if runs are not a category) help create a balanced and successful fantasy team.
However, one of the surprising aspects of the research was a connection between some relatively minor statistics and a player's overall value.
"Walks, doubles, caught stealing and strikeouts had a statistically significant (positive) impact on the index," Allen says.

Not only is it great to have professors like Elaine and George using Spotfire in and out of the classroom, I love seeing the application of analytics to things like fantasy sports. It's easy to think of it as just a "fun" example, but I think that both wide participation in fantasy sports and the acceptance of analytics over tradition by various professional sports franchises (the current World Series champions, the Boston Red Sox are just one analytically focused team--there are many more) have both played a big role in demonstrating to broad audiences how useful and effective analytics can be.