AR-Based Personalized Virtual Skincare Counsellor For The Biggest Skincare and Makeup Brand In Asia

We developed a platform for a Smart Mirror-based kiosk solution with personalized skincare counseling and facial recognition capabilities integrated with a handheld scanner.

Client

Our client is one of the biggest APAC-based cosmetics manufacturers, providing a wide variety of products through retail networks and online shops. They required an AR-based Skincare Counsellor for increasing their revenue from a new series of cosmetic products using personalized skincare counseling for offline and online shoppers.

Industry:

Retail

Team size:

4 people

Project duration:

2019-2020

Services

UX/UI, AWS Cloud architecture design, DevOps, Front-end and Back-end Android development, AR development, API development, CMS development, Quality Assurance

Tech stack:

  • Ruby on Rails
  • Java
  • Android
  • Kotlin
  • ARCore
  • Vue.js
  • Unity
  • PostgreSQL
  • AWS

CHALLENGE

The client contacted SPsoft with an idea of the product features they wanted to develop:

  • Customer identification with a QR-code, barcode, or facial recognition
  • Smooth skin-type assessment using a handheld scanner with facial detection
  • The personalized product offers based on the customer’s skin type
  • Data exchange between a kiosk and a mobile app to enable a seamless online shopping experience based on personal skin parameters gathered during the scanning process
  • Integration between our solution and Watsons app to simplify purchases and customer identification

Value Delivered

We built the required platform from scratch and within the project’s timeframes and budget. It used the most relevant tech stack and solved the main challenges the customer faced. We collected the stakeholder feedback, spruced up the product, and transferred the code to the client.


The process

To help the customer transform their expectations into system requirements, we gathered a small R&D team, which researched the available SaaS solutions. It transpired that SDKs or Software Development Kits for handheld scanners and Augmented Reality screens like Smart Mirror are not freely available to the public.

We built a custom AR-based Skincare Counselor using Smart Mirror. It uses a bespoke API for integration with the most feature-packed handheld skin scanner on the market. We also developed an Android app to enable high market penetration in APAC (>80%).

The platform runs on Android OS and uses AR Core for face recognition. Web UI is built with Vue.js, while PostgreSQL database stores AR Core data, and Core API is written in Ruby on Rails. We chose Kotlin/Java for building components of the kiosk and mobile apps. The product uses a rules-based product recommendation engine based on personal skin measurement results. There is also integration with a CRM system to simplify the customer experience through notifications, such as the need to re-do the skin assessment or refill some cosmetics.

We developed a CMS for the kiosk app, which allows the system admins to add and update various app content, like usage guides and descriptions for new stock items. We have also implemented a QR-code based customer identification. This enabled usage consistency across all devices and participation in a loyalty program. The infrastructure is hosted on AWS to ensure maximum resilience and scalability.

Demo Video

Success stories

One-of-a-Kind Multitenant B2B2C Platform for Salon-Quality Beauty Supplies Distributor

We have delivered a unique multitenant B2B2C platform for the world's largest distributor of salon-quality beauty supplies. Devised to harness the profit-share model's full potential and provide a win-win-win solution for all the stakeholders involved, including the distributor, salons, and our end-consumers.

View Case

Audience Relationship Management Platform For Music Tech Startup

SPsoft provides SaaS engineering services for an audience engagement platform.

View Case

A Multitenant B2B2C Sales Automation Solution for the US Retail Market

Developing a multitenant B2B2C sales automation solution for the US retail market, simplifying and automating the trade process between the manufacturers, vendors, distributors, and customers. Normalizing and integrating the data structure for the stakeholders involved.

View Case
View All Case Studies