Configuration Sensitivity of LLM Tabular Data Synthesizers: A Small-Model Memorization Effect and a Reporting Protocol
IEEE Access, 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Basım Tarihi: 2026
- Doi Numarası: 10.1109/access.2026.3732216
- Dergi Adı: IEEE Access
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex, INSPEC, Directory of Open Access Journals
- Anahtar Kelimeler: Benchmarking, configuration sensitivity, in-context learning, large language models, memorization, ranking stability, reproducibility, synthetic tabular data, variance decomposition
- Dokuz Eylül Üniversitesi Adresli: Evet
Özet
Problem. Large language models (LLMs) synthesize tabular data by serializing each row into text, which forces task-irrelevant presentation choices — row serialization, column order, and numeric format — that an ideal generator should ignore. We ask whether comparing LLM tabular synthesizers at a single fixed configuration is safe. Method.We perturb these three axes one-factor-at-a-time for a fine-tuned generator (GReaT) and a few-shot in-context (ICL) generator, with two text-blind generators (CTGAN, TVAE) as a configuration-invariant noise floor. Two arms complement each other: a three-dataset breadth arm (7B in-context model; adult, diabetes, student) tests generalization, and a 15-seed diabetes case study isolates the mechanism. Results. (1) Configuration flips generator rankings on every dataset (66–95% of configuration pairs), and adding two strong recent baselines increases the instability, not removes it. (2) The effect is generator-specific: an ANOVA with a generator×configuration interaction gives a small (6.7%) configuration main effect, and the flipped rankings sit on statistically near-tied generators—so the instability is partly near-equivalence, not a large formatting effect. (3) The in-context sensitivity is largely a verbatim-memorization artifact—the 0.5B model copies 42.9% of its rows, and removing copies at fixed sample size erases its fidelity advantage — receding with model scale (mean 0.5–13.5% at 7B) but tracking schema width (up to 74.4% on 33 columns). Against an independent, memorization-clean reference, averaging three configurations removes the single-configuration disagreement (up to 29%). Contribution.We release two zero-by-construction diagnostics (a verbatim-copy rate and a configuration-spread index), a validated reporting protocol, and all code and data.