IBM researchers have created a search engine that takes a different route from Google, Microsoft Bing and other search services to help users find untagged video, pictures and music content.
SAPIR, which stands for Search in Audio-Visual Content Using Peer-to-peer Information Retrieval and was funded by the European Union, churns through pixels in 100 million images and videos to help users find multimedia content they are looking for. IBM and EU launched a SAPIR demo Website here Sept. 10 for the public to test.
Traditional search engines Google, Bing and Yahoo rely on text metadata associated with each image to help users find content. If there is no metadata associated with the content, those search engines can’t help users find relevant results, Yosi Mass, a scientist at IBM Research-Haifa and project leader for SAPIR, told eWEEK Sept. 10.
SAPIR indexes the content of each image and clip using descriptors such as text, color, layout, shapes, or sounds, to help users find comparable images. For example, SAPIR scans a digitized photograph or the bit streams in an MP3 sound file, even if they haven’t been tagged or indexed with descriptive information; this is because SAPIR automatically indexes and ranks multimedia content users upload to the Web for easy retrieval.
In a demo, Mass showed how a search on the keyword “dolphin” returned dolphin photos of similar colors and shot angles that users had uploaded to Flickr. After clicking the “similar” link on top of one of the photos, eWEEK saw photos that resembled the dolphin photo in color and shape but did not necessarily include dolphins.
A box labeled “use the image in the search” appeared checked above the SAPIR search, so that when a user types in “dolphin” and clicks the search button, they will see more images of dolphins. This search blends dolphin images with similar images that do not include dolphins. The demo was an exercise in helping users conduct a combination text and image search, something that is not possible with current search engines.
Mass also said SAPIR enables users to introduce latitude and longitude coordinates into searches. Do a search for a topic on SAPIR, click “images” and you’ll see boxes for Lat and Long to the left of a button for Google Maps under the site’s search box.
Enter your geographic area of choice and search for photos on any topic. So, in addition to text and image search, users can further narrow down their topics with location services, which also is crucial for travelers accessing SAPIR via Web-enabled phones.
To wit, this YouTube demo shows how a tourist takes a picture of a statue in Madrid, and searches for comparable pictures in SAPIR. The tourist then did a combined search. By adding “Madrid” to the query, he saw images of the statue he just took a picture of, as well as related pictures of the surrounding plaza he was in.
Eventually, this service could be put to use by cities promoting tourism. “Cities that have a large image collection of monuments in their city can give a very nice service to their tourists,” pairing images with historical text information about the attractions users can access from their Web-enabled phones, Mass explained.
IBM has no concrete plans yet to commercialize SAPIR, but there is a clear value in corralling multimedia content. If IBM can harness some of this unstructured data with the SAPIR technology, it might be most useful for the company’s enterprise search portfolio, as well as for consumer service providers looking to provide their customers with better image and video search.