In this paper, an extended optimal power-flow (OPF) model incorporating a detailed model of a voltage-source con- verter-based–multiterminal high-voltage direct current system (VSC–MTDC) is proposed, hereafter referred to as the mixed ac/dc OPF (M-OPF) model. A cost-benefit analysis approach using the M-OPF model as the calculation engine is proposed to determine the preferred VSC-MTDC alternative to be installed in an existing ac transmission system. In this approach, the op- erational benefits of VSC–MTDC systems are evaluated against their investment costs to derive the benefit-to-cost ratios (BCR) which reflect the cost-effectiveness of the alternatives. A case study has been carried out using a modified Nordic 32-bus system. The results of the study show that VSC–MTDC systems might lead to a reduction in total operation cost, and the reduction of the total system transmission loss depends to a large extent on the VSC–MTDC configuration. The results from sensitivity analyses show that if the VSC loss could be reduced to a third of the original level, the total benefit from the system would be increased by about 70%. A suggestion for the placement and configuration of a VSC–MTDC system is made based on calculated BCRs.