Transport Mode Detection (TMD) systems play a pivotal role in facilitating applications in transport, urban planning, and more. Exploiting the advancements in smartphone sensing capabilities, TMD systems have evolved for mobile applications with local classification on smartphones as a common approach. Yet, local approaches relying on centralized training raise privacy concerns due to the transmission of sensitive data (e.g., GPS logs) over the Internet. In this paper, we propose FogFLeeT, a novel Federated Transfer Learning (FTL) framework for TMD, addressing both privacy and performance concerns. Our approach relies on Federated Learning (FL) to train a global model on various datasets from different cities while employing transfer learning to adapt the global model to the specific characteristics of individual smartphones and cities. FogFLeeT relies on an architecture that integrates edge, fog, and cloud layers, with dedicated fog nodes for each city to simplify cross-silo federated learning. Experimental results demonstrate the effectiveness of the FogFLeeT framework in higher TMD accuracy by up to 20% than its comparable centralized approach. Furthermore, it outperforms the FL solutions reported in the literature with at least an 8% increase in accuracy. In this work, we also highlight the importance of sufficient training data for distributed training and discuss the impact of smartphone sensor qualities on the performance of TMD systems. Our work contributes to advancing TMD systems by providing an adaptive and privacy-preserving solution suitable for deployment in diverse urban environments and across various geographical locations.