Introduction

As organizations adopt generative AI, information moves through conversations, documents, code, and prompts. Sensitive data such as names and account numbers is embedded in ordinary language. Regular expressions identify predictable formats, but fixed patterns struggle when spelling varies or meaning depends on context.

This paper describes how QueryPie developed sensitive-data detection models for Korean, English, and Japanese: preparing training data, separating lightweight screening from precise extraction, and correcting the character-position errors observed in its initial experiments.

The problem

Recent discussions with enterprise security teams highlighted the need for UEBA to identify unusual AI-agent activity and for dynamic DLP to protect source text sent to external AI services while preserving business context. This paper focuses on DLP. The first step in applying a dynamic protection policy is to identify which exact phrases contain which types of sensitive information.

Rules based on regular expressions can recognize consistently formatted numbers. They are less effective when a person's name appears alongside other data, credentials are embedded in code, or spaces and special characters alter a known format. Analyzing every input with a large language model, meanwhile, increases latency and operating cost. DLP therefore needs both fast risk screening and precise identification of sensitive-data types and source phrases.

Objectives

QueryPie defined two tasks for multilingual business data. The first is to classify the risk that an input contains sensitive information at high throughput. The second is to extract the information type and exact source phrase when precise analysis is required. These tasks have different outputs, so we developed a lightweight classification model and a precision small language model (SLM) for distinct roles.

Many enterprises cannot run GPU infrastructure continuously. Our objective was an architecture in which a lightweight model handles large-scale screening on CPUs and precision analysis is reserved for the inputs that need it, reducing compute demands.

Development

Multilingual training data and sensitive-data types

Starting with Korean data, we prepared English and Japanese examples and included cases where sensitive information appears in code or is altered with spaces and special characters. During cleaning and augmentation, we reduced duplicates and balanced examples across languages. Rather than simply increasing the number of sentences, we focused on the situations the models need to distinguish.

For lightweight classification, the target label indicates whether an input carries sensitive-data risk. For precision extraction, the labels specify both the type and the exact phrase in the source. We consolidated language-specific labels into a shared taxonomy of 21 sensitive-data types. A separate program compared training phrases and their positions with the original text and corrected mismatches.

Multi-stage risk decisions and precision extraction

The risk-decision workflow first checks predictable values with regular expressions and keyword rules. A rule match leads to a block decision; only unmatched inputs proceed to the lightweight ELECTRA-based model, which returns a risk score between 0 and 1. Low-risk inputs pass, high-risk inputs are assessed for protective action, and only ambiguous scores in the gray zone proceed to a third, prompt-based AI decision stage. ELECTRA screens large input volumes quickly on CPUs.

The third-stage decision uses a prompt-based model. Separately, a Gemma-family 12B SLM performs precision extraction of sensitive-data types and source phrases. A program compares the extracted phrases against the original text to calculate their character positions. The model handles contextual and semantic judgment; the program resolves exact string offsets.

Character-position errors in the first experiment

Initially, we asked the precision model to return the sensitive-data type, source phrase, and character offsets in a single response. In evaluation, it sometimes identified the right type but gave incorrect start or end positions or subtly changed the original phrase. Without a reliable match to the source, downstream processing cannot trust the offsets.

The primary issue was the structure of the output task, not simply the amount of training text. Requiring a generative model to produce exact character coordinates in addition to interpreting meaning made results unstable and lowered recall and exact-match rates.

Simplifying output and retraining

We narrowed the precision model's output to the sensitive-data type and the phrase as it appears in the original text. Instead of asking the model to generate offsets, a separate program compares the extracted phrase with the source and computes its position. We updated the target format and retrained and evaluated the candidate models. The model now handles contextual meaning, while code handles string-position verification.

Evaluation results

In internal evaluation, the Korean ELECTRA-based lightweight model recorded approximately 92% accuracy, 97% recall, and a 94% F1 score, with an average response time of about 17.3 ms. We also trained and evaluated models tailored to English and Japanese. ELECTRA serves as a fast screening model for sensitive-data risk across high-volume inputs.

After we changed the output format, the Gemma-family 12B model achieved the highest internal composite score for extracting both sensitive-data types and exact phrases: 92.64 points. On the same precision-extraction task, a Qwen 3.5-family 9B candidate scored 80.75 points and a Qwen 2.5-family 7B candidate scored 74.4 points. The composite score evaluates type classification across 21 sensitive-data categories and source-phrase extraction under our internal criteria.

A compressed Gemma-family 12B candidate also scored 92.64 points in the comparison. Focusing the model output on types and phrases, while assigning position calculation to a program, allowed the model's semantic judgment to contribute more directly to the result. Output-task design proved as important as model selection.

Conclusion

Three choices shaped this multilingual DLP work: training data reflecting business text and altered formats, separate roles for fast risk screening and precise phrase extraction, and programmatic processing of character positions that a generative model can misstate. Together, they improved type-and-phrase extraction and established reliable source-position matching.

The risk-decision architecture screens inputs with rules and a CPU-capable lightweight model, escalating only gray-zone scores for further judgment. In the separate precision-extraction task, Gemma identifies sensitive-data types and source phrases, and code calculates their positions. The next development step is dynamic DLP that connects detection results to organization-specific policies. Planned actions include outbound blocking, on-screen highlighting and masking, user warnings and administrator review, and notifications to security systems.