In this thesis the properties of unbound nuclei like 12,13Li are studiedby extending the multistep shell model to the complex energy plane. Thenuclei 12Li and 13Li are described starting from the one-particle states in 10Liand two-particle states in 11Li. The ground state of 12Li is found to be anantibound state. No bound or antibound state is found in 13Li. It is alsoshown that the odd proton plays a minor role in these nuclei.This framework is also applied to study the recently proposed spin-alignedproton-neutron pair coupling scheme. For this the MSM is extended to theproton-neutron space as well as the isospin space. In the model, a nonorthogonalbasis is introduced, which allows us to identify simultaneously theroles played by all configurations. The four-particle, six-particle and eightparticlenuclei in the heaviest N = Z region are evaluated using the MSMbasis within the space spanned by the single 0g9/2 hole shell.A novel Monte Carlo representation in the complex energy plane is alsodeveloped for studying nuclear excitations in the continuum. The calculationson realistic potentials show a stable performance and high accuracy in the oneparticleand two-particle cases. This will provide a convenient tool to studyopen systems with many nucleons in the continuum.