Adapting an existing convolutional neural network architecture to a specific dataset for medical image segmentation remains a challenging task that requires extensive expertise and time to fine-tune the hyperparameters. Hyperparameter optimization approaches that automate the search have been proposed but have mainly focused on optimizing the segmentation performance. However, optimizing the network size is also important to prevent unnecessary and costly computational operations. In this paper, we present a multiobjective adaptive convolutional neural network (AdaResU-Net) for medical image segmentation that is able to automatically adapt to new datasets while minimizing the size of the network. The proposed AdaResU-Net is comprised of a fixed architecture that combines the structure of the state-of-the-art U-Net with a residual learning framework to improve information propagation and promote an efficient training. Then, a multiobjective evolutionary algorithm (MEA) that optimizes both segmentation accuracy and model size is proposed to evolve the AdaResU-Net networks with different hyperparameters. The presented model is tested on two publically available medical image datasets and compared with the U-Net. Results show that the AdaResU-Net achieves better segmentation performance with less than 30% the number of trainable parameters. Additionally, the MEA algorithm generated configurations that are smaller and perform better or equally well than configurations generated with a Bayesian hyperparameter optimization approach.