An intrusion detection system (IDS) is a security measure that can help system administrators in enterprise environments detect attacks made against computer networks. In order to be a good enterprise security measure, the IDS solution should be effective when it comes to making system operators aware of on-going cyber-attacks. However, it is difficult and costly to evaluate the effectiveness of IDSs by experiments or observations. This paper describes the result of an alternative approach to studying this topic. The effectiveness of 24 different IDS solution scenarios pertaining to remote arbitrary code exploits is evaluated by 165 domain experts. The respondents’ answers were then combined according to Cooke’s classical method, in which respondents are weighted based on how well they perform on a set of test questions. Results show that the single most important factor is whether either a host-based IDS, or a network-based IDS is in place. Assuming that either one or the other is in place, the most important course of action is to tune the IDS to its environment. The results also show that an updated signature database influences the effectiveness of the IDS less than if the vulnerability that is being exploited is well-known and is possible to patch or not.