How Amazon Go Works

Amazon Go and Apple Pay have a few things in common: they promised a revolution in shopping, they seemed super futuristic and interesting at their first introduction, and people in Europe or similarly remote places in the world have to wait forever to experience them firsthand.

Personally, I think Amazon Go is the coolest thing in the shopping experience since the invention of friendly cashiers. Here I take a look behind the concept and technology of Amazon Go, which I assure you is quite fascinating. Let's start with what we all love: fast, delicious snacks.

A Snack at Hand at All Times

Amazon says it wants to offer its customers a large variety of food and products in a comparatively small shop - but most of all there is a wide range of freshly prepared meals. The focus clearly lies on snacks that can be prepared quickly or rather healthy convenience foods. These allegedly even come from local suppliers. Each shelf has a number of accurate motion detectors, avoiding empty product trays through central monitoring.

Customers check in to the store via app and QR code scan and take their quick snacks, drinks and other products off the shelf. The technology allows the customer to simply leave the store with the goods. Invoicing is again done via App.

But how does the system recognize in the store which products a customer takes? Amazon itself is happy using nebulous buzzwords, talking about computer vision and deep learning, and merely lets you know that technologies are being used that are similar to self-driving cars. Amazon coined the name Sensor Fusion for the number of their technologies in use.

Cameras and a combination of different sensors provide a learning algorithm that is supposed to recognize which products customers take off the shelf and perhaps put back again. Amazon is likely to rely on a technology similar to face recognition that recognizes objects from a variety of angles and uses models such as RFID to determine the respective position of customer and product. Sensors can also be used to detect the weight of the products on the shelf (or even the weight or size of the customer).

There are reports that in the past the tracking system would get confused when there were two people of similar body shape and stature. Sometimes kids (and occasionally adults) would take items from one shelf and put them back on an incorrect shelf. This also caused problems with the Sensor Fusion technology, since it couldn't understand when the item on the incorrect shelf was later removed by another customer. 

One Concept, Many Application Fields? 

Here is a nice overview of the most discussed features of Amazon Go in social media, created by brandwatch:

In addition to the Just walk out technology (this official term used by Amazon sums it up relatively well) you should not overlook the following aspects:

• The concept is not only suitable for food, but can probably also be transferred to other industries such as fashion, cosmetics, electronics or books.
• By linking to the app, Amazon collects data on offline shopping behavior. This information can in turn be used for online offers.
• Amazon retail plans could hit some existing stores pretty hard. Amazon is planning over 2500 store openings in the coming years around pick-up stores, bookstores and Amazon fresh shops. And with the new technology, these will become a far greater threat to the retail industry than ever before, as Amazon could expand at a fraction of the personnel costs.

A Few Glimpses into the Amazon Patent File

Amazon claims to have developed the concept for a cashless supermarket for four years. Initially, Amazon Go was to open to regular customers in March 2017. However, this deadline could not be met, as the trial phase revealed significant technical problems as soon as more than 20 customers were in the store and using the service at the same time. It was not until January 2018 that the first Amazon Go store on Seattle's 7th Avenue opened to the public.

A patent application, which ReCode discovered some time ago and from which we list two passages with direct quotations below, also offers a little more light into the technology behind Amazon Go. In these, Amazon shows in detail applicable technologies and scenarios:

In addition to cameras, other input devices, such as pressure sensors, infrared sensors, a scale (...) may be utilized with the implementations described herein. For example, a pressure sensor and/or a scale may be used to detect when an item is added and/or picked from inventory locations. Likewise, an infrared sensor may be used to distinguish between a user's hand and inventory items. (...) load cells may be positioned on a surface of the transition area to detect when a user is passing through the transition area.

If the inventory management system cannot determine if the picked item is a bottle of ketchup or a bottle of mustard, the inventory management system may consider past purchase history and/or what items the user has already picked from other inventory locations. (...) if the user historically has only picked/purchased ketchup, that information may be used to confirm that the user has likely picked ketchup from the inventory location.

So, that's about it. What do you think of shopping while a large number of sensors monitor every step you take? Let us know under the article. You can comment, but you can't hide!