AR-Based Personalized Virtual Skincare Counsellor For The Biggest Skincare and Makeup Brand In Asia

The client contacted SPsoft with an idea of the product features they wanted to develop:

• Customer identification with a QR-code, barcode, or facial recognition
• Smooth skin-type assessment using a handheld scanner with facial detection
• The personalized product offers based on the customer’s skin type
• Data exchange between a kiosk and a mobile app to enable a seamless online shopping experience based on personal skin parameters gathered during the scanning process
• Integration between our solution and Watsons app to simplify purchases and customer identification

We built the required platform from scratch and within the project’s timeframes and budget. It used the most relevant tech stack and solved the main challenges the customer faced. We collected the stakeholder feedback, spruced up the product, and transferred the code to the client.

To help the customer transform their expectations into system requirements, we gathered a small R&D team, which researched the available SaaS solutions. It transpired that SDKs or Software Development Kits for handheld scanners and Augmented Reality screens like Smart Mirror are not freely available to the public.

We built a custom AR-based Skincare Counselor using Smart Mirror. It uses a bespoke API for integration with the most feature-packed handheld skin scanner on the market. We also developed an Android app to enable high market penetration in APAC (>80%).

The platform runs on Android OS and uses AR Core for face recognition. Web UI is built with Vue.js, while the PostgreSQL database stores AR Core data, and Core API is written in Ruby on Rails. We chose Kotlin/Java for building components of the kiosk and mobile apps. The product uses a rules-based product recommendation engine based on personal skin measurement results. There is also integration with a CRM system to simplify the customer experience through notifications, such as the need to re-do the skin assessment or refill some cosmetics.

We developed a CMS for the kiosk app, which allows the system admins to add and update various app content, like usage guides and descriptions for new stock items. We have also implemented a QR-code-based customer identification. This enabled usage consistency across all devices and participation in a loyalty program. The infrastructure is hosted on AWS to ensure maximum resilience and scalability.

Demo Video