Computer-aided diagnosis systems (CADx) play a major role in the early diagnosis of breast cancer. Extracting the breast region precisely from a mammogram is an essential component of CADx for mammography. The appearance of the pectoral muscle on medio-lateral oblique (MLO) views increases the false positive rate in CADx. Therefore, the pectoral muscle should be identified and removed from the breast region in an MLO image before further analysis. None of the previous pectoral muscle segmentation methods address all breast types based on the breast imaging-reporting and data system tissue density classes. In this paper, we deal with this deficiency by introducing a new simple yet effective method that combines geometric rules with a region growing algorithm to support the segmentation of all types of pectoral muscles (normal, convex, concave, and combinatorial). Experimental segmentation accuracy results were reported for four tissue density classes on 872 MLO images from three publicly available datasets. An average Jaccard index and Dice similarity coefficient of 0.972 ± 0.003 and 0.985 ± 0.001 were obtained, respectively. The mean Hausdorff distance between the contours detected by our method and the ground truth is below 5 mm for all datasets. An average acceptable segmentation rate of ~95% was achieved outperforming several state-of-the-art competing methods. Excellent results were obtained even for the most challenging class of extremely dense breasts.