Learning Universal Semantic Simplex for Jailbreaking Large Vision-Language Models

Tao Huang, Rui Wang, Xiaofei Liu, Xiaomeng Li, Yi Qin, Tong Xiao, Zhekun Liu, Liping Jing; | 📦Code | 🤗Huggingface (Coming Soon)

• Continuous Paradigm: Introduces Learning Universal Semantic Simplex (LUSS), shifting the jailbreak optimization from point-wise perturbations to continuous adversarial simplices.

• Overfitting Mitigation: Leverages a sharpness-aware penalty and a simplex-based update to capture broad adversarial semantics within a convex hull.

• Superior Improvement: Outperforms state-of-the-art baselines, achieving up to a 60.8% relative improvement in ASR and a 44.7% gain in Malicious Intent Fulfillment Rate.

Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification

Tao Huang, Rui Wang, Xiaofei Liu, Yi Qin, Li Duan, Liping Jing; | 📦Code | 🤗Huggingface

Efficient Framework: A training-free method that quantifies uncertainty using pre-logits features in a single forward pass.

Novel Insight: Layer-wise dynamic analysis reveals that hallucinations correspond to high internal conflict, while OOD failures correspond to high ignorance.

Superior Performance: Outperforms strong baselines on four state-of-the-art LVLMs, improving AUROC by 10.4% and AUPR by 5.3% on average.