How to Measure Findability in Enterprise Search Solutions

The results of a recent survey on the findability of information within the enterprise are not encouraging. Roughly half of the responding knowledge workers stated that finding important information was difficult and time-consuming, and that the internal search capabilities provided by their company were "worse" to "much worse" than the equivalent functionality offered to consumers.

Neither of these findings is truly surprising. Corporate Internet sites tend to be directly involved in important and easily measured activities: selling products to new customers (generating revenues) or servicing existing customers (reducing costs). Consequently, Internet search is usually well-funded, staffed and tends to be more successful. Internal search, in contrast, is concerned with productivity—a fuzzier concept that is much harder to measure. Significant investment (and thus success) is therefore harder to achieve.

What is surprising is that roughly half of the survey respondents stated that their company had no formal goal for internal findability. In my view, this is a direct cause for the overall poor results. Companies that don't measure search won't be able to invest appropriately, let alone tune and improve a complicated system with which they likely don't have deep internal experience.

If your organization doesn't measure search, or more precisely, findability, you should start right away. Here are five steps to building a return on investment case for your company’s internal search or information access solution. Keep in mind that no massive investment is required upfront. In fact, one person working a few hours a week can make a difference.

Step #1: Talk to your system administration team

For starters, find out if your organization is saving query logs—hopefully they are. If not, this is the first challenge to overcome. Talk to your system administration team and see if they can help. You don't need to save all logs for all time; just try to get your hands on a day or two of data. That's quite enough to get started.

Step #2: Identify and run a handful of queries

Assuming you have some data to look at, identify a handful (less than 50) of interesting queries. Ideally, you want them to fall into a few different categories: one-word, two-word, multiple words, questions, and a few different business domains.

Run the queries yourself and see what comes up. Look at the first few results and score them on a simple scale (such as 1=incoherent, 5=perfect). If you rate a result poorly, spend a few minutes trying to find out what the better answer might be. Then see if you can infer what’s wrong. 

For example, does the document that answers your question (but doesn’t appear in the results list) contain the terms you put in your query? If it does, you have a relevancy problem; if not, you have some sort of linguistic problem.