This work proposes a novel deep learning approach to tackle multi-task optimization problems in multi-user multi-antenna downlink systems. In practice, there is a trade-off between maximizing the weighted sum spectral efficiency (WSSE) and weighted sum energy efficiency (WSEE) in wireless systems. Traditional beamforming algorithms face limitations in jointly addressing multiple optimization tasks, as they heavily rely on task-specific processes aimed at maximizing specific metrics. As a result, the multiple computations to deal with the multi-task problems lead to poor computation and memory efficiency at the base station (BS), which is a challenging aspect to overcome. To address these issues, we present a novel multi-task learning approach that effectively achieves the desired trade-off while reducing the memory burden. We demonstrate the advantages of the proposed scheme that utilizes a single neural network over both existing model-based and data-driven algorithms.