Ghost’s Engine: How exactly the recommendations work

Edrone uses primarily two recommendation strategies, called Collaborative Filtering and Market Basket Analysis. It’s worth underlining that they are very simple, that’s why we use them as our first choice, to provide AI-based recommendations even for the tiniest shop.

Collaborative Filtering

Collaborative filtering is a widely used method of picking the ideal offer to the client based on his behavioral model. It considers recently viewed products, products added to the cart by customers, and products that were ultimately purchased. Each of these actions should be treated differently, but you will learn why a bit later.

Technically, this strategy simply boils down to predicting the probability of purchase and presenting the offer for which this probability is the highest. There is a good chance that this recommendation method will show premium products, making it a bit like up-selling. Still, it is not a rule, and that is why it is worth considering how the frame can work (what products it can display) for different types of customers.

Functions: Probability, Discovery, Expansion.

Yet, Collaborative Filtering isn’t itself an algorithm. It’s a strategy. CF is fueled with Slope One.

Slope One

Obviously calculating the whole advertising strategy all time, for each visitor, in real time, can be consuming. That’s why Slope One is so handy. Is quite simple, light and easy to maintain and use. However, it performs well and is way better than randomly generated items, or a dull, soulless static offer. What is it all about? Calculating the difference between ratings of products made by users. It goes like this:

		Item A	Item B	Rating Delta
User 1 rating	4	3.5	D1(A,B) = 4 - 3.5 = 0.5
User 2 rating 	4.5	x	D2(A ,B) = 4.5 - x = 0.5

Simple math gives us  X = 4.

We need to create an object called a deviation matrix.

The matrix contains the difference between items co-rated with users. On this basis, the chance for the item to fit the user is calculated.

With one exception:

Not every eCommerce uses ratings, as in a 5-star rating (like on Youtube back in the days) or similar.

Second, item explicit ratings are not the best source of information. Users, when not asked, rate them when they are upset about a product. Sometimes they even rate the product down spitefully, if something else was the problem – they waited for consignment too long, package was broken, etc..

Lack of explicit ratings is not a problem. Simply replace explicit with implicit, assigning rates to specific actions performed on a product. There is no strict rule about how you should rate actions. Usually it goes like this:

• View = 1
• Add to Cart = 3
• Order = 9

However, you can always adjust ratings to your eCommerce dynamics.

• 1, 4, 16 – For shops with expensive items.
• 1, 2, 8 – For shops with rather cheap items.
• 1, 2, 4 – Cheap items, and usually adding to cart = purchase.
• 1, 4, 8 – More expensive items, and users usually do not abandon carts.
• …etc.

…For better and for worse

Slope One is just an example, rather simple. However, it isn’t an exception when it comes to recommendation algorithms. It’s true that AI isn’t free from drawbacks. But! Let’s think about it also as an analog to real life problems. AI solves human problems, and sometimes has similar limitations. The good news is we can also fight them using ideas based, more or less, on a human approach.

Cold Start

Yeah. Starting is not easy. When user visits shop, algorithm has nothing to offer, since have no reference point on which the recommendation is made (no ratings to apply deviation matrix to)

Regarding recommendation systems it’s common in any type of recommender. Even if we are talking about flesh and blood human customer advisors. If a new client enters the shop, what do you recommend him if you have no idea what he is into?

Sparsity

The biggest players on the market sell an enormous number of products, yet, not all of them are equally popular, meaning, not equally often bought. This causes two flaws in the recommendations’ performance.

• Popularity Bias – algorithm favours products that are often rated. On the other hand…
• Long Tail of Products – some products are rare, but at the same time would be an excellent fit for the user. They may not be recommended.

The same problems apply to the brick and mortar shops! The products that generate the biggest “traffic”, the products that are oftenly in the hands of advisors, are the products that first came into mind when it comes to recommend “book for 3-years old fixed on astronomy”.

Scalability

The large number of products mentioned above is also a problem combined with a large number of customers. The dot product of these is enormous, which means that in most eCommerces following this recommendation strategy literally by the book will require computational hecatomb.

In reality? Go into Walmart, and ask a random assistant for a random thing. There is a chance that he or she will know the answer, but probably will redirect you to someone else. Unless you are Doctor Strange, you are unable to process hundreds of thousands items, and pick the right one, keeping in mind all the possibilities, and cases.

However, we’ve learned how to deal with them, or at least to minimize their impact.

Solution?

• For those who are frequent visitors use collaborative filtering with no hesitation (user-based approach).
• For those who are newcomers, content based filtering will be the best option (product-based approach). It’s covered with Basket Analysis.

Market Basket Analysis.

More generally called Affinity Analysis, is even simpler than Collaborative filtering. Basket Analysis comes to creation association rules around the products. When a client buys product A, and product B, there is a chance that another client with product A in the basket, might be also interested in product B.

• If A then B

Simple as that. Affinity can be measured. We use 3 parameters called: Support, Confidence, and Lift.

• Support – Share of transactions supporting our rule, amongst all.
  (Number of baskets containing A & B divided by number of all baskets)
• Confidence – Share of transactions supporting our rule amongst those which fulfil the conditions.
  (Number of baskets containing A & B divided by number of baskets containing A)
• Lift – How often coincidence occurs regarding the mean chance.

Sales techniques on steroids

The last one is most interesting, because it gives us insightful interpretation. Lift takes values around the 1.

If the Lift is greater than 1, it means that randomly picked products appear in the basket less often than in practice. Products are indeed somehow related, complimentary.

• Recommending items with Lift > 1 is applied cross-selling.

If the Lift is lower than 1, it means that randomly picked products appear in the basket more often than in practice. Products are chosen as alternatives.

• After applying the price threshold (price B > price A) we can think about this recommendation as an up-selling.

First, use Intelligence… at first your own!

We said in the beginning this part, that issues with AI can be solved with human approach. As you may have noticed we did it, when we smartly switch between item-based and user-based recommendations, but it’s not everything.

Non-personalized, static recommendations (banners, new collections promotions, manual highlights) will work too. Treat static as warmth up for algorithms, and canvas to learn. At all we need to start from something with. It is you, who knows your clients best. This is why you shouldn’t ever resign from them for the sake of dynamic recommendations. It can solve a lot of problems that may occur, when relying on the Artificial one, and only.

Item based recommendations can be also prepared by yourself, regarding your experience as merchant, and subject matter expert.

Even Neural networks themselves, when created, fill the weights of connections with random numbers. Later in time, after a few – so called – epochs of learning, weights become tuned and do their job. Same is True for Simple Machine Learning techniques… and our brains, isn’t it?

Secondary:

Ask. Engage clients. Lure them into products-browsing paths.Its the best way to amplify performance of each AI system installed in your eCommerce. At the same time is the best panacea for each flaw of recommendation strategy, starting from cold start, on overlearning (overfitting, precisely).

Even the most sophisticated software, network or trick bleaks in the wake of simple questions. Are you searching for men or women Clothes? Or maybe gender doesn’t matter for you? What Your Child is Interested with? Are you searching for something for a gift to someone? Or maybe for yourself?

What are you interested in?