Internet Research Needs Better Models: Why Models Matter

Why Models Matter:

• S. Gorinsky and H. Vin, Extended Analysis of Binary Adjustment Algorithms, Technical Report TR2002-39, Department of Computer Sciences, The University of Texas at Austin, August 2002.
  "It is still common to use Chiu-Jain model for comparison of binary adjustment algorithms. This paper argues against such practice... Chiu-Jain model is not suitable for trustworthy conclusions about properties of an adjustment algorithm."

• W. Willinger, R. Govindan, S. Jamin, V. Paxson, and S. Shenker. Scaling Phenomena in the Internet: Critically Examining Criticality, Proc. Nat. Acad. Sci., 99, suppl. 1, pp. 2573--2580, February 19, 2002.
  "We bring to bear a simple validation framework that aims at testing whether a proposed model is merely evocative, in that it can reproduce the phenomenon of interest but does not necessarily capture and incorporate the true underlying cause, or indeed explanatory, in that it also captures the causal mechanisms."

• Y. Joo, V. Ribeiro, A. Feldmann, A. C. Gilbert, and W. Willinger, TCP/IP traffic dynamics and network performance: A lesson in workload modeling, flow control, and trace-driven simulations. CCR, April 2001.
  "Depending on the underlying assumptions about the inherent nature of the dynamics of network traffic, very different conclusions can be derived for a number of well-studied and apparently well-understood problems in the area of performance evaluation."

• Reiner Ludwig, Almudena Konrad, Anthony D. Joseph, Randy H. Katz, Optimizing the End-to-End Performance of Reliable Flows over Wireless Links, Proceedings of ACM MOBICOM 1999.
  "An unrealistic error model of the wireless channel can completely change the results of a performance analysis."

• Kathleen Nichols, An Opinionated View of the Current State of IP Differentiated Services, September 1999. In particular, Part II: Issues in Evaluating Differentiated Services.
  "Beware... sweeping conclusions based on very simple traffic models and topologies."

• John Heidemann, Nirupama Bulusu, Jeremy Elson, Chalermek Intanagonwiwat, Kun-chan Lan, Ya Xu, Wei Ye, Deborah Estrin, and Ramesh Govindan, Effects of Detail in Wireless Network Simulation SCS Multiconference on Distributed Simulation, January 2001.
  "Too little detail can produce simulations that are misleading or incorrect, but adding detail ... can distract from the research problem at hand."

Dangers:

• B. Conrad and M. Mitzenmacher, Power Laws for Monkeys Typing Randomly: The Case of Unequal Probabilities, IEEE Trans. on Inf. Theory.
  "Miller's result serves as warning: just because one finds a compelling mechanism to explain a power law does not mean that there are not other, perhaps simplier, explanations."