Internet Research Needs Better Models: Why Models Matter

Why Models Matter:

• S. Gorinsky and H. Vin, Extended Analysis of Binary Adjustment Algorithms, Technical Report TR2002-39, Department of Computer Sciences, The University of Texas at Austin, August 2002.
  "It is still common to use Chiu-Jain model for comparison of binary adjustment algorithms. This paper argues against such practice... Chiu-Jain model is not suitable for trustworthy conclusions about properties of an adjustment algorithm."

• W. Willinger, R. Govindan, S. Jamin, V. Paxson, and S. Shenker. Scaling Phenomena in the Internet: Critically Examining Criticality, Proc. Nat. Acad. Sci., 99, suppl. 1, pp. 2573--2580, February 19, 2002.
  "We bring to bear a simple validation framework that aims at testing whether a proposed model is merely evocative, in that it can reproduce the phenomenon of interest but does not necessarily capture and incorporate the true underlying cause, or indeed explanatory, in that it also captures the causal mechanisms."

• Y. Joo, V. Ribeiro, A. Feldmann, A. C. Gilbert, and W. Willinger, TCP/IP traffic dynamics and network performance: A lesson in workload modeling, flow control, and trace-driven simulations. CCR, April 2001.
  "Depending on the underlying assumptions about the inherent nature of the dynamics of network traffic, very different conclusions can be derived for a number of well-studied and apparently well-understood problems in the area of performance evaluation."

• Reiner Ludwig, Almudena Konrad, Anthony D. Joseph, Randy H. Katz, Optimizing the End-to-End Performance of Reliable Flows over Wireless Links, Proceedings of ACM MOBICOM 1999.
  "An unrealistic error model of the wireless channel can completely change the results of a performance analysis."

• Kathleen Nichols, An Opinionated View of the Current State of IP Differentiated Services, September 1999. In particular, Part II: Issues in Evaluating Differentiated Services.
  "Beware... sweeping conclusions based on very simple traffic models and topologies."

• John Heidemann, Nirupama Bulusu, Jeremy Elson, Chalermek Intanagonwiwat, Kun-chan Lan, Ya Xu, Wei Ye, Deborah Estrin, and Ramesh Govindan, Effects of Detail in Wireless Network Simulation SCS Multiconference on Distributed Simulation, January 2001.
  "Too little detail can produce simulations that are misleading or incorrect, but adding detail ... can distract from the research problem at hand."

Dangers:

• B. Conrad and M. Mitzenmacher, Power Laws for Monkeys Typing Randomly: The Case of Unequal Probabilities, IEEE Trans. on Inf. Theory.
  "Miller's result serves as warning: just because one finds a compelling mechanism to explain a power law does not mean that there are not other, perhaps simplier, explanations."

Dangers in other Domains:

• S. Labaton, The Reckoning: Agency's '04 Rule Let Banks Pile Up New Debt, October 2, 2008, New York Times.
  "The agency ... decided to rely on the firms’ own computer models for determining the riskiness of investments, essentially outsourcing the job of monitoring risk to the banks themselves."
  The Audio Slide Show is particularly interesting.

• S. Lohr, In Modeling Risk, the Human Factor Was Left Out, November 4, 2008, New York Times.
  "Today’s economic turmoil, it seems, is an implicit indictment of the arcane field of financial engineering — a blend of mathematics, statistics and computing. Its practitioners devised not only the exotic, mortgage-backed securities that proved so troublesome, but also the mathematical models of risk that suggested these securities were safe."
  • S. Lohr, Wall Street’s Math Wizards Forgot a Few Variables September 12, 2009, New York Times.
    "In the aftermath of the great meltdown of 2008, Wall Street’s quants have been cast as the financial engineers of profit-driven innovation run amok.... But the real failure, according to finance experts and economists, was in the quants’ mathematical models of risk that suggested the arcane stuff was safe.... The risk models proved myopic, they say, because they were too simple-minded. They focused mainly on figures like the expected returns and the default risk of financial instruments. What they didn’t sufficiently take into account was human behavior, specifically the potential for widespread panic."