In robotics, there has been a growing interest in expressing actions as a combination of meaningful subparts commonly called motion primitives. Primitives are analogous to words in a language. Similar to words put together according to the rules of language in a sentence, primitives arranged with certain rules make an action. In this paper we investigate modeling and recognition of arm manipulation actions at different levels of complexity using primitives. Primitives are detected automatically in a sequential manner. Here, we assume no prior knowledge on primitives, but look for correlating segments across various sequences. All actions are then modeled within a single hidden Markov models whose structure is learned incrementally as new data is observed. We also generate an action grammar based on these primitives and thus link signals to symbols.