Abstract
In the past decade, artificial intelligence has advanced at an unprecedented pace, driven by the dual forces of architectural innovation and data availability. The field has witnessed a shift from specialized, task specific architectures to general-purpose models such as Transformers, Vision Transformers, and, most recently, Large Vision-Language Models (LVLMs). These breakthroughs have redefined the boundaries of multitask and multimodal learning. Yet, they also raise fundamental concerns: the escalating costs of training, the opacity of black-box behaviors, and the growing dependence on synthetic or massive-scale data. This thesis addresses a central question: can we continue to advance AI not by scaling indiscriminately, but by designing architectures and strategies that make smarter use of the data and resources already available? To this end, the work explores alternatives to universal, resource-intensive models, revisiting architectural and data-driven design under constraints of efficiency, explainability, and adaptability. Our contributions are organized into three main directions. In the first part we explore NAS under distribution shifts and with limited data and resources. We introduce DDS-NAS-Bench, the first benchmark designed to study how architectural rankings evolve under input distribution changes. This enables systematic analysis of predictor generalization across datasets. Building on this, we propose GRASP-GCN, a graph-based predictor that incorporates architectural topology to improve ranking accuracy. It addresses a key limitation of conventional NAS methods tied to single datasets while enhancing robustness under distribution shifts. To extend NAS to modern architectures under constrained settings, we propose L SWAG, the first zero-cost proxy tailored to both CNNs and Vision Transformers, enabling reliable architecture evaluation across 14 tasks without training. Furthermore, we present LIBRA-NAS, a framework to strategically combine multiple weak proxies, correcting their biases and enhancing predictive power. Together, these contributions make zero-shot NAS more accurate and practical, reducing computational costs by orders of magnitude while remaining competitive with traditional search strategies. In the second and third parts we focus on architecture design for volumetric and video data. In medical imaging, we propose SSH-UNet, a novel slice-shift mechanism that captures 3D context with the efficiency of 2D CNNs, achieving state-of-the-art results in multi-organ segmentation. We further explore data-centric design with Your Image is My Video, a differentiable augmentation search method that treats static images as temporal sequences, providing more powerful receptive fields and boosting accuracy across diverse vision tasks. Finally, we address video understanding with NAS-Just-Once (VIM-NAS), the first unified framework that searches architectures jointly for images and videos. Leveraging a weight-entangled Vision Transformer supernet, VIM-NAS narrowes down the gap with hand-designed architectures in video action recognition. Overall, this thesis demonstrates that progress in AI does not need to rely solely on scaling model and data size. By combining principled NAS methodologies, zero-shot learning, and domain-inspired design, we show how architectures can be optimized to be not only accurate but also efficient, generalizable, and explainable. This work contributes to a broader vision of AI that is accessible and controllable, pushing the boundaries of what can be achieved with greater insight.