Adaptive Conditioning for Robust Pap Smear Cell Semantic Segmentation under Stain and Illumination Variations

Abstract

Pap smear cytology images frequently exhibit substantial appearance variability due to differences in staining procedures and illumination conditions, leading to domain shift that degrades the generalization capability of deep learning-based semantic segmentation models. This study aims to stabilize image appearance prior to binary semantic segmentation (cell versus non-cell) by proposing an adaptive conditioning framework. The proposed framework integrates sample-wise stain normalization, illumination correction, and adaptive local contrast enhancement without modifying the downstream segmentation architecture. Experiments were conducted on the RepoMedUNM dataset consisting of 387 Pap smear images using a fixed U-Net backbone. Segmentation performance was evaluated using Dice Coefficient and Intersection over Union (IoU), while robustness was assessed through controlled stress tests involving brightness/contrast perturbation, color jitter, and illumination gradient simulation. Experimental results indicate a clear operational trade-off: while the baseline U-Net achieves higher absolute metrics on clean test images (Dice 0.9526 vs. 0.9085), the proposed framework provides targeted variance suppression under domain shift. Specifically, the adaptive framework achieves complete chromatic invariance () under color jitter and substantially reduces performance degradation under contrast shifts ( vs. ). However, performance degradation increases under severe illumination gradients (), demonstrating the boundary of sample-wise statistical adaptation under non-uniform spatial shading. These findings highlight adaptive input conditioning as a lightweight and architecture-agnostic strategy for stabilizing segmentation performance against stain and contrast variations in practical cervical cancer screening applications.