The control of physical systems is increasingly being done by resorting to networks to transmit information from sensors to controllers and from controllers to actuators. Unfortunately, this reliance on networks also brings new security vulnerabilities for control systems. We study the extent to which an adversary can attack a physical system by tampering with the temporal characteristics of the network, leading to time-varying delays and more importantly by changing the order in which packets are delivered. We show that such attack can destabilize a system if the controller was not designed to be robust with respect to an adversarial scheduling of messages. Although one can always store delayed messages in a buffer so as to present them to the control algorithm in the order they were sent and with a constant delay, such design is overly conservative. Instead, we design a controller that makes the best possible use of the received packets in a minimax sense. The proposed design has the same worst case performance as a controller based on a buffer but has better performance whenever there is no attack or the attacker does not play the optimal attack strategy.