The company’s tech shop @WalmartLabs, was formed from the acquisition last year of start-up Kosmix, which monitored vast streams of social media data to help businesses track sentiment about products and brands. The @WalmartLabs group is now focusing on two main areas: social shopping and mobile commerce, building a variety of applications and technologies. Many of these applications are being built on the deep data technology that was originally developed by Kosmix. The start-up had built a technology infrastructure for tracking massive amounts of social media data called MapUpdate, roughly modeled after Google’s MapReduce technology for tracking massive numbers of web pages.
The @WalmartLabs group under Kosmix-co-founders Venky Harinarayan and Anand Rajaraman—who report to Eduardo Castro-Wright, vice chairman of Walmart Stores and CEO of Walmart Global eCommerce--is working on projects in a kind of venture portfolio model. Some are experimental, with a high risk and high reward, while others are practical projects such as how to improve search on Walmart.com. Eventually, the goal is to make Walmart.com, its social media efforts, offline physical stores and mobile apps all integrated and feeding into one another.
On Facebook @WalmartLabs wants to create an “inherently social” shopping experience, says Harinarayan. One idea the group is testing is a Facebook app called Shopycat through which people could give gifts. The group isn’t talking about it, but the app has the tagline, “Get the gifts they REALLY want.” Gaming is the only category on Facebook that has succeeded so far, says Harinarayan, whose theory is that more and more shopping will move to places like Facebook. “What is the Zynga of shopping on Facebook?” The answer: there isn’t one yet. “Our view is if you look at social, where do you have an inherently social experience? When you buy for or with people. How can we look at Facebook as a platform for that?”
Harinarayan’s group is also developing ways to analyze consumers’ social media interests to predict future purchasing on Walmart.com. While Walmart rival Amazon.com uses past purchases to predict future purchases, that history is not always the best predictor, Harinarayan says. Just because someone buys a television doesn’t mean anything specific about future purchases. (The exceptions are areas like books or music.) “What tends to be a better predictor of interest is what I’m interested in, especially right at that point in time,” Harinarayan says. That can be gleaned from social media, which users could provide on an opt-in basis. Using social media data, search on Walmart.com and product recommendations on Walmart.com’s homepage could be personalized. This could also include personalized alerts for deals or products.
Another area for @WalmartLabs is location-based technology, particularly for Walmart’s massive physical stores. Since Walmart’s stores are so big, it only makes sense to have location-based apps to direct people in the store, help find products. “The physical stores have a footprint which are an advantage,” Harinarayan says. “How do you make use of that to create more social buying in a way that’s consistent with the brand?” Harinarayan says “maybe” next year those types of services will come live to the public.
Finally, @WalmartLabs can use its social media expertise to help understand interest among shoppers at brick-and-mortar stores. The @WalmartLabs group can build social profiles of neighborhoods that help guide the inventory of stores. “With any store location you can look at the social media neighborhood,” Harinarayan says. “In Mountain View (Calif.) folks are more interested in cycling and tech, versus other places it may be fishing. That’s what people are Tweeting about. You can use that to create differentiated stores.”
To understand the deep "fast data" technology that @WalmartLabs has, you have to go to back to Kosmix’s history. Co-founders Harinarayan and Rajaraman have a long ecommerce background, as they were executives at Amazon, coming in through the 1998 acquisition of their start-up Junglee, one of the first shopping search engines. The two were behind the 1999 launch of Amazon’s third-party marketplace. That's notable since the two are now part of Walmart's effort to take on competitor Amazon. In 2000, Harinarayan and Rajaraman left Amazon and started Cambrian Ventures with the backing of Amazon CEO Jeff Bezos and others. The pair invested in a number of early stage start-ups, including Neoteris, which was acquired by Netscreen/Juniper in 2003; and Aster Data, which was acquired by Teradata in 2011.
After deciding they didn’t want to do another fund, Harinarayan and Rajaraman started Kosmix in 2005. The idea was to provide a new way to capture information outside of search engines, focusing on social media sources such as Twitter and Facebook. The “light bulb went off” when Twitter started releasing its open firehose of Tweets for developers, Harinarayan says. While search engines focus on direct intent (e.g. the address for Joe’s Ice Cream) Kosmix focused more on categorizing information. Kosmix used semantic analysis so that, say, if you want to follow social media mentions about a San Francisco Giants pitcher that do not specifically mention “the Giants,” you can still follow get them.
From a data perspective, you can look at the web as a huge file system of web pages, Harinarayan says. But social media is more of a massive stream of constantly flowing data. “The web is very informative but as a valuable consumer experience it’s not easy,” Harinarayan says. “The problem is it’s consumed by humans at the end of the day. There’s only so much you can do with it. We’re categorizing the web. You need to provide a structure around the web. That’s what we created with Kosmix.”
Kosmix helped companies monitor vast amounts of social media data, including livestreams of all activity on a subject, including what is most popular, who is most influential on the topic, images, Tweets, and the like. Companies were interested to monitor how they’re being discussed online and what the trends and patterns are. U.S. intelligence agencies also sought to use the service.
Kosmix’s “social genome” technology, now part of @WalmartLabs, was designed as a new version of the data processing systems that companies like Google had built. Google introduced MapReduce in 2004 to distribute the processing of large amounts of static big data—web pages—but social media is about “fast data,” such as Tweets, Harinarayan says. Every Tweet coming in changes the system. “You can’t do MapReduce computing every time (with every Tweet). You’ll die,” he says. “How do you do it in real-time? We built MapUpdate, or what we call Muppet. We could map a huge amount of data and handle a huge firehose with little latency across millions of entities… We can monitor 100 million (items) at scale. That could be products, stores, anything. It’s the equivalent of MapReduce for fast data.”
This MapUpdate technology is highly scalable, he says, and can manage and track data streams with billions of updates a day, such as literally all the Tweets on Twitter. It can simultaneously handle millions of objects, such as millions of locations, or hundreds of millions of users. Using this technology, @WalmartLabs can do things like keep track of every mention of a particular location, or extract social media data on each person mentioning that location, and how they are connected to other people and topics. “Say you want to know the last check-in (there) and have latency of less than one second,” Harinarayan says. “That’s what we had to build. All data mining, all in real-time. You can keep track of every user on Twitter and what their interests are.”
Harinarayan calls this the “index of social media” just like Google’s index of the web. For developers at @WalmartLabs, they can now quickly pull this information about a topic at a particular point in time to build applications. “They can write a program really easily," Harinarayan says. "We take the complexity and scale away from the programmer.” @WalmartLabs is using this deep data for a variety of its products, from recognizing consumer interests near individual stores to understanding the interests of consumers on Walmart.com.
The @WalmartLabs group is trying to keep to its start-up roots. It occupies the same space that Kosmix did in Mountain View, Calif. The group has about 70 people but is hiring aggressively, trying to compete with the Googles, Facebooks or LinkedIns of Silicon Valley. The group even recently placed a radio ad on the local NPR affiliate KQED. Many in Silicon Valley may not know Walmart as a tech company but they may soon, if the group can make a splash with some of these new products.
