Human-robot collaboration (HRC) is critical to today's tendency towards high-flexible assembly in manufacturing. Human action recognition, as one of the core prerequisites for HRC, enables industrial robots to understand human intentions and to execute planning adaptively. However, existing deep learning-based action recognition methods rely heavily on a huge amount of annotation data, which may not be effective or realistic in practice. Therefore, a transfer learning-enabled action recognition approach is proposed in this research to facilitate robot reactive control in HRC assembly. Meanwhile, a decision-making mechanism for robotic planning is introduced as well. Lastly, the proposed approach is evaluated in an aircraft bracket assembly scenario to reveal its significance.