Competition is moving inside stores. A case in point is the Amazon App - an amazing iPhone app that enables users to take pictures of products they like to see if they could find them through Amazon and add them to their wishlist. With this app, Amazon will not only capture demand from consumers’ casual encounters with products they like, but also prey on shopping malls and department stores.

Price arbitrage is becoming much easier. Applications like ShopSavvy enables users to scan product bar codes to compare prices online and in nearby stores. Even without these apps, customers use their smartphones to make online purchases with in-store delivery if they find it more advantageous, only heightening cannibalizing within the same franchise.

Product and service information is becoming more accessible and more transparent. Two popular examples show that retailers’ traditional information advantage is further eroded, with relevant information being delivered in the hands of consumers while they shop. The GoodGuide iPhone app, for instance, delivers ratings about green, healthy and organic products right to your pocket and while you shop. The Yelp iPhone app delivers location-based user reviews and pictures of shops and services.

The advent of the mobile web ushers new challenges and opportunities for retailers. Empowering their sales force with loyalty program offerings and/or some form of price-matching latitude could help retailers tactically meet these challenges in the short run. Retailers should also consider the upside in mobile interactions, from delivering personalized, social or value-added information to sending location-based time-bound coupons to mobile users.

Amazon’s results this week were in sharp contrast with the overall health of the retail sector. As highlighted during the holiday seasons, Amazon ruthlessly grabbed market share in an overall down retail market and turned a respectable profit on its very solid execution.

Amazon’s success builds on

- a strong brand that keeps customer acquisition costs low;

- aggressive pricing that kept the brand at the forefront of many shopping comparison engines; and

- an experience that nurtures superior customer loyalty throughout the entire shopping cycle, from personalized recommendations to customer reviews.

Amazon’s success puts increased pressure on competitors to step up, at a time when most online retailers are in survival rather than investment mode.  Retailers actively should be looking at three crucial areas to grow their top line in this adverse environment.

Build an online shopping experience that fosters discovery

Retailers have to better capture consumers mind before they have actually made their mind on a particular product. Google delivers very precise results when users search for an exact product – bringing competitors one click away from each others, with price becoming the only differentiator. Google is however inefficient at helping users shop a product category, such as a camera or clothes.

Retailers have to rethink how they can leverage their expertise to build a strong funnel from discovery to conversion. Zappos has just released a new way of exploring their catalog that enables customers to intuitively compare and explore shoes and accessories based on colors and shape. Distilled Clothing showcases its collection with an arresting use of videos that keeps the viewer engaged and wanting more.

Expand your reach online and offline

BestBuy has been experimenting for quite some time to expand its reach online and offline. Last year, it started supporting microformats on its website to help build a critical mass of products to tag with Giftag – their social wishlist browser plugin and service. Giftag follows Amazon’s path of helping users share their wish list of products they want to buy, while keeping its finger on the pulse of consumer trends.

With mobile internet becoming pervasive, offline and online shopping are converging. Users are increasingly using their mobile devices to arbitrage between online and offline channels. Multi-channel retailers should however seize on this opportunity to compete with Amazon. Remix – BestBuy’s combined API and affiliate program – does just that. It enables developers to build applications on top of both product catalog, as well as online and offline store inventory.

Use recommendation to build long-lasting customer loyalty

We’ve discussed the opportunities and challenges of online recommendation in numerous earlier posts. The current economic climate only heightens the need for retailers to nurture their customer relationship and improve on customer retention.

As previously highlighted, delivering a personalized online experience remains particularly difficult to achieve, because it requires mastering multiple technologies - user segmentation, content targeting, recommendation, website optimization - as well as aggregating data across online marketing channels and overcoming internal resistance. Those are reasons however why personalization remains a key source of competitive advantage, as it keeps customers engaged and helps strengthen customer loyalty.

“Most popular” lists and 19 high-level categories are hardly up to the task of helping consumers find new apps. Limited discovery is hampering the App Store’s growth and monetization and could have a depressing effect on the ecosystem.

There are however quite a few opportunities for Apple to improve discovery and bring the overall shopping experience on the App Store away from simply being a “port” of the iTunes Musicstore:

- Improve discovery through experience

- Enable free trials – free trials are critical to bring feature-rich and higher priced apps to the ecosystem.

- Add videos and user video reviews – text and screenshots do little justice to an app’s capabilities, design, and overall user experience;

- Improve discovery through richer feedback and needs analysis

- Categories and search are limited tools for demand generation;

- Amazon-style recommendations and discovery needs to be more ubiquitous and prominent on the App Store to enable genuine comparison between apps;

- Blending the feedback process about apps much with the user experience on the iPhone could open up a much more personalized experience

- Improve discovery through finer segmentation and filtering – the free vs. paid segmentation is dragging all paid iPhone apps down in the ringtones pricing range; there are only 14 apps over $100 (out of 10,000) and there are still relatively few games at a price comparable to that of a typical game console.

Is Netflix’s ground-breaking crowdsourcing approach reaching a limit? Netflix has undeniably got a lot of value out of their contest – 10,000s people working and an algorithmic improvement very close to 10% – for only $100K so far. However, the winning method this year – and the likely research directions for next year – have exponential data and computing power requirements – possibly putting the contest out of reach of most casual researchers.

Are “curated” recommendations superior to pure statistics? Curated recommendations have received some hype lately – from Techmeme’s hiring of an editor to the coincidental launch this week of ClerkDogs – a “clerks-in-a-box” online movie recommendation service? A human touch can certainly rebalance or add a forward-looking perspective to recommendations where data sets are ambiguous (such as clickstream), too small, or lack enough historical data.

Consumers have been very receptive and sensitive to price – as expected by retailers. For instance, email promotions and comparison shopping have seen quite a boost in effectively driving sales leads to online retailers.

Online retailers have been very aggressive on pricing, which will inevitably drag down margins. The slide show below shows very vividly the messaging focus on price this year, as opposed to last year’s variety of differentiation strategies.

Techmeme is good at aggregating news overtime but not at breaking them, pointing to several common datamining and recommendation challenges:

- “Cold start” – interestingly digg’s social voting approach hasn’t been able to overcome that challenge either – in both cases, the services cannot anticipate news’ propagation and velocity;

- Context – Techmeme sometimes mixes up headlines and for instance ended up featuring news about Anna Nicole Smith’s hospitalization after she’s already been declared dead;

- Outliers or the “Napoleon Dynamite” problem, as the New York Times dubs it - identifying newsworthy pieces from uncommon sources before they make it into the mainstream is also an issue.

Interestingness and relevance are Techmeme’s other key reasons to bring in a human eye. Techmeme bets that an expert hand can be a better judge than crowdsourced implicit feedback based on clickstream or explicit feedback such as social voting. This approach seems to contradict much of the crowdsourcing mantra, although Techmeme’s case is more about rebalancing than shunning crowdsourcing.

For online retailers, Techmeme’s move to “curated news aggregation” highlights opportunities to blend human input, datamining, and recommendation:

- to add context to a product recommendation – based on usage, audience background, as well as internal needs through promotions;

- to identify new and unexplored relationships between products – for product discovery, up-sell, and outliers.< >< ><-->

The overall mood remains mixed for retailers: consumers’ wallets are thinner and online retailers have to push steep promotions and free shipping to get customers to their sites.

According to ComScore, “more than half (51 percent) of consumers indicated that the level of promotions and discounts is higher this year than last year, while only 12 percent said that there appeared to be fewer, suggesting that retailers are having to be more aggressive in discounting to spur consumer spending.”

ComScore reports that online sales for this year’s Black Friday grew a measly 2% year over year. “Early reports suggest that Black Friday sales in retail stores were slightly better than anticipated in this depressed retail climate, and that performance apparently extended to the online channel, which saw sales on Thanksgiving Day and Black Friday combined increase 2 percent versus year ago,” said comScore chairman, Gian Fulgoni. “It’s probable that on Black Friday consumers responded positively to the very aggressive promotions and discounts being offered in retail stores, so it will be important to see how they respond to similarly attractive deals being offered online on Cyber Monday, the traditional kick-off to the online holiday shopping season.”

Amazon and eBay reported relatively strong sales, according to Reuters, and “PayPal saw almost 34 percent more transactions this Black Friday than a year earlier, eBay said on Saturday. PayPal saw sales rise 26 percent on Black Friday. PayPal said its sales numbers reflected 12 percent of all U.S. e-commerce.”

These gross revenue numbers reflect a fiercely competitive environment, where retailers have offered deep discounts to attract shoppers both online and offline. This year, extensive discounts and free shipping on Cyber Monday should extend Friday’s modest uptick in sales.

According to a Shopzilla survey, “83.7 percent of retailers will have a special promotion for Cyber Monday, up from 72.2 percent last year. The most popular promotions are expected to be specific deals (38.8%), email campaigns (32.7%), and one-day sales (24.5%). Additionally, nearly one-fourth of retailers (22.5%) will offer free shipping on all purchases.”

One additional dynamic possibly at play this year is the sizeable increase of smartphone users with mobile internet access. For the first time, an increase in on-the-spot, comparison shopping between online and brick-and-mortar retailers could have a visible - and detrimental - impact on offline sales.

Mufin has gathered the most attention so far – being a spin off of the Fraunhofer Institut. Mufin relies exclusively on content analysis to make music recommendations. The service analyzes 40 characteristics for each the 4 million songs in its database, including tempo, sound density, and variety of other factors to filter out songs which are similar. This content-based approach uniquely addresses cold start and long tail discovery issues, but feels a bit too “literal” in the way it analyzes music similarities between songs.

The perceptron is an experimental and transparent collaborative filtering service. The service aggregates and filters recommendations made by actual humans (from tinymixtapes.com and epitonic.com), social links between artists and fans (e.g. belonging to the same label or the number of friends on a myspace page), and other social music sharing sources. Interestingly, recommendations sources and the relative weights of the various sources are all public. The service does a great job at mixing familiar and unknown recommendations, thereby making the new stuff all the more interesting. It would be terrific to see a next generation of this service allow users to actually tinker with sources and weights to obtain more personalized recommendations.

The next big sound pushes the music discovery envelope. The service does a great job at crowd-sourcing and uncovering new musical talents, using the music label paradigm. Each listener is allowed to select and sign up to 10 artists at any given time among all the unsigned artists that broadcast their music on the site. This site is a great place for unadulterated creative talent, although a bit of collaborative filtering would enhance the listening experience.

With Black Friday upon us, we are now entering the Holiday season. Many publications have started churning out their gift recommendation lists – from the top vintage geek gift list to the New York Times’ favorite 100 books for the year to Amazon’s top reviewers’ recommendations for the Holidays.

Wouldn’t it be great to have a service that compiles a personalized gift list, based on deep content analysis between products, expert recommendations, social insight and an Etsy-like indie product-sourcing twist to add a few one-of-a-kind gifts?

There are a few early signs of this future yet to come. Semantic Gifts - currently in alpha - mines your friends’ profiles on twitter, facebook, friendfeed, and their personal blog. Giftag and Amazon’s Universal Wish List now provide a mine of user-generated gift lists. BlackFridayAds and BlackFriday now aggregate black friday coupons and special deals accross thousands of online and brick-and-mortar retailers. If you still can’t find the perfect gift, there’s always Paul McCartney’s self-produced album – Electric Arguments.

Kelkoo and other first-generation price comparison engines have a hard time establishing themselves as mainstream destination sites for online shoppers. Online shoppers are much more likely to start their product search on Google than on any of the price comparison engines.

For quite some time, they managed to get around this issue thanks to their SEO and ad arbitrage expertise. However, Google has worked hard to improve search results and capture an increased share of ad revenue. Search results have increasingly pointed more towards manufacturers and online retailers, while paid search prices has gradually made ad arbitrage prohibitive.

Newcomers, however, are showing possible paths for growth, as online retailers still struggle with product search, discovery and recommendation:

- Twenga is combining the largest product index (over 100 million products) with user reviews and coupons to reach the scale needed for a search, SEO and affiliate-based model to work;

- Like.com is turning online window shopping and visual search into a turn on;

- BlackFriday’s annual surge shows the untapped potential of events, such as Thanksgivings.