The rise of human-water research in hydrology has increased the need for data on human behavior and decision-making. To address this demand, hydrologists have embraced methods from the social sciences, such as surveys, interviews, and agent-based modeling (ABM). However, collecting human data is expensive and time-consuming. Therefore, some social scientists have started evaluating whether Large Language Models (LLMs) could serve as a tool for generating human-like data for surveys and social simulations. This approach have the potential to transform human-water research by making access to human data faster and more affordable. Yet, human-water researchers should be cautious about adopting this method. The method has faced criticism in the social sciences, and similar caveats would apply to human-water research. While LLMs can provide responses based on different demographic personas, they fail to replicate human complexity and diversity. They also pose challenges to scientific rigor due to limited transparency and the risks of hallucinations. Most importantly, LLM-generated data fails to accurately represent marginalized groups, and undermines efforts to make human–water research more participatory, inclusive, and transformative.