# Semantic-aware multimodal multilingual deep learning systems for e-commerce

- Date: 2025-07-01
- Who: AI Singapore (AI research and talent-development organisation) · AI Singapore
- Source: https://www.youtube.com/watch?v=9ONmek-BRWE
- sgai: https://sgai.md/videos/v050/
- License: sgai-authored content (summaries, translations, analysis) is CC BY 4.0 — attribute and link to sgai.md. Verbatim source text (Hansard, speeches, transcripts, policy documents) remains © its original rights holders and is reproduced for reference only. Terms: https://github.com/meltflake/sgai/blob/main/DATA-LICENSE.md

## Why it matters

The AISG 100E project deployed a model that improved the win rate from 18% to 56% into Shopee's production line, proving that Singapore's applied AI funding can fill the gap in Southeast Asian minority languages overlooked by global tech giants

## Summary

An AISG 100E project tackles multilingual, multimodal e-commerce settings, addressing two key challenges: scarcity of labelled data for low-resource languages and complex semantic learning.

## Key points

- An AISG 100E project with Shopee replaced manual task flows with fine-tuned LLMs, boosting Taskbot completion and correction rates.
- The upgraded Shopee chatbot incorporates negative feedback into fine-tuning, lifting customer satisfaction and cutting failed interactions.
- NUS trained a Southeast Asia multilingual LLM whose win rate against the GPT baseline jumped from 18% to 50%, and to 56% with on-policy sampling.

## Full text

© AI Singapore — reproduced for reference only.

E-commerce is experiencing exponential growth, playing an increasingly vital role in the global economy. In recent years, deep learning has been increasingly adopted across a wide range of e-commerce applications. However, existing deep learning models struggle to effectively understand and process information when it's presented in multiple languages and different modalities. In collaboration with C, we have developed advanced multilingual and multimodal models, published our findings in leading conferences, and successfully validated our models on C's platform. Previously, shopp relied heavily on manually constructed task flows, which led to high costs and limited coverage. But now, we've transformed customer interactions by harnessing the power of fine-tuned large language models.

Today, Taskbot surpasses every expectation, achieving an impressive completion rate and correction rate. The result, customer interactions are now smarter, faster, and more reliable than ever before. In another application, the shoppy chatbot originally used traditional machine learning, fine-tuning only on good cases while ignoring the failures. Our solution changes that incorporating both positive and negative feedback into LLM fine-tuning. This enhanced approach has driven a significant boost in customer satisfaction and a notable drop in failed interactions. We also trained our own large language model from scratch. Multilingual LLMs for Southeast Asia face unique challenges especially due to limited data for low resource languages. At NTUNC, we're building a model specifically tailored to this region.

Our key innovation, a birectional negative feedback loss enables stable preference alignment even with scarce supervision. This led to a significant improvement. Win rates jumped from 18% to 50% against the GPT baseline. With on policy sampling, we pushed that further to 56%. For shopppee users, this translates directly to a much smarter LLM that truly understands their needs, leading to more helpful and accurate interactions every single time they use the app.