The 'Minority Report' shopping experience

Everywhere you go these days -- every store, every urban street, every car park, every ATM -- there are video cameras watching where you go and what you do. Just what are "they" (whoever "they" are) doing with all of that video surveillance?

Consider the Fraunhofer Institute's platform-independent development kit called SHORE, which stands for " Sophisticated High-speed Object Recognition Engine." This software can recognize faces presented full-frontal with 91.5% accuracy, can determine gender with an accuracy of 94.3%, estimate age with a mean absolute error of 6.85 years, and do all of that in realtime.

[ SCIENCE NONFICTION: Will future surveillance include global 'pre-crime' machine spying on everyone? ]

SHORE can also recognize facial expressions ("Happy," "Surprised," "Angry" and "Sad"), identify faces whether recognized or not, and it has a short-term memory for recognized faces to support tracking within a scene.

Some years ago I was in London -- in Harrods to be precise -- going up an escalator, and lining the walls were flat-panel displays showing ads. I started thinking about what could be done by placing a small camera beside each display and making a video of the people passing.

Using SHORE to process the video would enable us to figure out if customers were looking at the ads and which ads they paid the most attention to. This would be based not just on whether they looked at the ads, but for how long and if their eyes tracked the ads. Ideally, we'd also want to detect and measure pupil-size changes, which is known to correlate with interest level.

We could possibly even measure customers' heart and breathing rates visually as they looked at the ads. A friend of mine, Christopher Small, recently told me about a video processing technique called Eulerian Video Magnification he's been working on at Quanta Research Cambridge that amplifies microscopic movements. He's made an online demonstration of this available called Videoscope ... in particular, check out the "face" and "wrist" examples.

So, armed with all of this data about customers and their responses to ads, we'd then know which ads worked best for which demographic dimensions and, if we were processing the video in realtime, we could show ads to each customer individually that were an order of magnitude more effective because they would be tailored to what we determined their unique demographic profile to be. And, over time, using A/B testing, we could continuously refine our targeting ability.

To go further we could use full-blown facial recognition and then, when we identified someone who we had already seen and who we had characterized, we could deliver even more effective ads to them.

And how about analyzing how people navigate around a store? Store layout matters, as I'm sure you've noticed in stores that don't physically work well.

The problem is how to visually track people in a scene, something that has become very easy. Just over three years ago in Gearhead I reviewed a product called Vitamin D which does just that; it can distinguish people from other moving objects even when they are partially obscured!

What's particularly interesting about this product is that it was built using a technology based on a model of how the human brain works, which was developed by Numenta, a company co-founded by Jeff Hawkins, the founder of Palm.

Now, aggregate all of this intelligence about customers captured by all of the camera-augmented displays around the store (both electronic as well as static), along with the analysis of how people move around a store (this could be linked, ultimately, with the systems that recognize individuals to allow detailed behavior and path analysis for each customer visit), and we start to develop astounding insight into customer behavior and preferences in the real world.

Link that big pool of big data to the till data, and we could correlate the entire in-store customer experience (where they went, what ads they saw, what smells we added to the air, what lighting we used, etc.) with revenue potential. What we've done is turn the arts of analyzing and tuning advertising and store layout design into a science.

Of course this data will be very valuable and, in theory, we could share the data on both anonymous and recognized individuals with other businesses, allowing behavior to be tracked across multiple locations or used without prior interaction. This means we'd have to pay attention to protecting the privacy of recognized individuals by generating something like an MD5 hash for their recognition data and only storing that, but then their entire consumer experience could be "optimized" without actually identifying individuals and destroying their privacy.

Yep, welcome to the "Minority Report" shopping experience without the holograms.

So, how long until this happens? Or has it already happened?

Gibbs is probably surveilled in Ventura, Calif. Show yourself to and follow him on Twitter and (@quistuipater) and on Facebook (quistuipater).

Read more about wide area network in Network World's Wide Area Network section.

Join the CSO newsletter!

Error: Please check your email address.

Tags surveillancePhoto / videoapplicationsFraunhofer InstitutesecurityMinority Reportfacial recognitionsoftwarevideoDemographics

More about FacebookHawkinsPalm

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Mark Gibbs

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place