Thermal response tests are used to estimate the thermal properties of the ground and the borehole heat exchanger being tested. They are thus important for the design of borehole thermal energy storages and ground source heat pump systems. In this study, a theoretical framework is proposed in order to investigate if noise on the heat rate leads to a bias in the parameter estimation. Under the sole assumption of a linear time-invariant system and the use of the sum of squared errors as cost function, it is shown analytically that estimates are in fact biased when the heat rate is corrupted by noise. To understand how large this bias can be, a Monte-Carlo study is performed. It includes more than 126,000 simulations with different noises, thermal parameters and heat rate profiles. Negative biases as high as -0.44 W/(m K) (11%) and -11.10(-3) m K/W (4.1%) are observed for the thermal conductivity and borehole thermal resistance estimates, respectively. In addition, the parameter estimation is stochastic due to randomness of measurement noises. This cannot be ignored since only one thermal response test is performed, in general. Population of estimates with 95% confidence intervals as large as 1.0 W/(m K) (25%) and 24.10(-3) m K/W (9.4%) appear in this study. Although the bias and confidence intervals are not significant in all simulated cases, they cannot be generally disregarded and one should therefore be mindful of this potential issue when analyzing thermal response tests. An observed trend is that the confidence intervals and bias are higher for higher parameter values, with a particular dependency on thermal conductivity. To reduce the bias and spread of the estimates, having larger heat rate per meter appears to be a good strategy. Having a higher sampling frequency and/or longer tests might also help, but only in reducing the spread of the estimates, not the bias.