With the growing complexity of memory types, organizations, and placement, efficient use of memory systems remains a key objective to processing data-rich workloads. Heterogeneous memories including HBM, conventional DRAM, and persistent memory, both locally and network-attached, exhibit a wide range of latencies and bandwidths. The delivered performance to an application may vary widely depending on workload and interference from competing clients. Evaluating the impact on applications to these emerging memory systems challenges traditional simulation techniques. In this work, we describe VLD-sim, an FPGA-accelerated simulator designed to evaluate application performance in the presence of varying non-deterministic latency. VLD-sim implements a statistical approach in which memory system access latency is non-deterministic, as would occur when request traffic is generated from a large collection of possibly unrelated threads and compute nodes. VLD-sim runs on a Multi-Processor System on Chip with hard CPU plus configurable logic to enable fast evaluation of workloads or of individual applications. We evaluate VLD-sim with CPU-only and near memory accelerator-enabled applications and compare against an idealized fixed latency baseline. Our findings reveal and quantify performance impact on applications due to non-deterministic latency. With high flexibility and and fast execution time, VLD-sim enables system level evaluation of a large memory architecture design space.