Adaptive Neural Network Scaling Guided by Contrastive Loss

Automatically scaling the capacity of artificial neural networks (ANNs) to match task difficulty still lacks a low-cost and robust general solution: most existing methods rely on prediction loss, which yields an unpredictable width-performance landscape and unstable search. Inspired by continuous neurogenesis in the dentate gyrus, we propose to use contrastive loss as a proxy for pattern-separation quality and to let it drive neuron generation or pruning. We propose a one-dimensional “exponential expansion + golden-section refinement” algorithm: network width is doubled while contrastive loss keeps decreasing and is trimmed once the loss rises, taking advantage of the markedly smoother contrastive curve. On eight public benchmarks, the resulting search procedure (Contra) attains a mean prediction error of 8.94%±7.13%, about 25% lower than the same search guided by prediction loss (Pred; 11.91%±11.82%), and reduces run-to-run variance by 40-50% on small, noisy tabular tasks. The contrastive loss-width curve is 54% smoother than that of plain prediction loss, yet overall wall time overhead is negligible because fewer probes are required. These findings demonstrate that contrastive loss offers a stable control signal for capacity, yielding an immediately “plug-and-play” upgrade for width-, channel-, and embedding-dimension search in dynamic networks and neural-architecture search.

Recommended citation: Wang, Y., Zheng, T., Sugino, M., Shimba, K., & Kotani, K. (2026). Adaptive Neural Network Scaling Guided by Contrastive Loss. IEEJ Transactions on Electronics, Information and Systems, 146(4), 366–375. https://doi.org/10.1541/ieejeiss.146.366
Download Paper