Convolutional neural networks to achieve joint in- version of microwave and ultrasonic data for breast imaging are investigated. The estimated contrast current and scattered field inside the domain of interest are used as inputs. A multi- stream structure is employed to input the data from different modalities. The network outputs the distribution maps of electric and acoustic parameters directly so as to achieve real-time imaging. Apart from the regression task, a multi-task learning strategy is used with a classifier which associates each pixel to a tissue type to yield a segmentation image. Weighted loss is used to assign a higher penalty to pixels in tumors when they are wrongly classified. Comparisons have been carried out between different network structures with the same datasets. The prediction results of the networks are evaluated by Intersection- over-Union for segmentation results and by relative error for reconstructions. The simulations on breast phantoms extracted from a dedicated repository show that with both microwave and ultrasonic data, the network can get a better estimate of the breast structure and small tumors detected. Meanwhile, multi- task learning has improved the regression results and multi- stream input has helped to exploit data from different modalities.