It is envisioned that the future electric grid will be underpinned by a vast number of smart inverters linking renewables at the grid edge. These inverters' dynamics are typically characterized as impedances, which are crucial for ensuring grid stability and resiliency. However, the physical implementation of these inverters may vary widely and may be kept confidential. Existing analytical impedance models require a complete and precise understanding of system parameters. They can hardly capture the complete electrical behavior when the inverters are performing complex functions. Online impedance measurements for many inverters across multiple operating points are impractical. To address these issues, we present the InvNet, a machine learning framework capable of characterizing inverter impedance patterns across a wide operation range, even with limited impedance data. Leveraging transfer learning, the InvNet can extrapolate from physics-based models to real-world ones and from one inverter to another with the same control framework but different control parameters with very limited data. This framework demonstrates machine learning as a powerful tool for modeling and analyzing black-box characteristics of grid-tied inverter systems that cannot be accurately described by traditional analytical methods, such as inverters under model-predictive control. Comprehensive evaluations were conducted to verify the effectiveness of the InvNet.