According to the general theory of relativity, time can flow at different rates depending on the configuration of massive objects, affecting the temporal order of events. Combined with quantum theory, this gravitational effect can result in events with an indefinite temporal order when a massive object is prepared in a suitable quantum state. This was argued to lead to a theory-independent test of the nonclassical order of events through the violation of Bell-type inequalities for temporal order. Here we show that the theory independence of this protocol is problematic: one of the auxiliary assumptions in the above approach turns out to be essential, while it is explicitly theory dependent. To illustrate this problem, we construct a complete scenario where accelerating particles interacting with optical cavities result in a violation of temporal Bell inequalities. Due to the equivalence principle, the same problem arises when one considers the gravitational case, and thus theory-dependent additional assumptions are needed behind Bell inequalities for temporal order to interpret a violation of the final inequalities as a signature of indefinite temporal order.