The optimal growth of perturbations to transiently growing streaks is studied in Poiseuille flow. Basic flows are generated by direct numerical simulation giving primary optimal spanwise periodic vortices of finite amplitude as the initial condition. They evolve into finite amplitude primary transiently growing streaks. Linear secondary optimal energy growth supported by these primary flows are computed using an adjoint technique which takes into full account the unsteadiness of the basic flows. Qualitative differences between primary and secondary optimal growths are found only when the primary streaks are locally unstable. For locally stable primary streaks, the secondary optimal growth has the same scalings with Reynolds number R as the primary optimal growth and the maximum growth is attained by streamwise uniform vortices, suggesting that the primary and secondary optimal growth are based on the same physical mechanisms. When the primary streaks are locally unstable the secondary optimal growth of unstable wavenumbers scale differently with R and the maximum growth is attained for streamwise nonuniform sinuous perturbations, indicating the prevalence of the inflectional instability mechanism.