Dependency Analysis in Distributed Systems using Fault Injection: Application to Problem Determination in an e-commerce Environment

Distributed networked applications that are being deployed in enterprise settings, increasingly rely on a large number of heterogeneous hardware and software components for providing end-to-end services. In such settings, the issue of problem diagnosis becomes vitally important, in order to minimize system outages and improve system availability. This motivates interest in dependency characterization among the different components in distributed application environments. A promising approach for obtaining dynamic dependency information is the Active Dependency Discovery technique in which a dependency graph of e-commerce transactions on hardware and software components in the system is built by individually “perturbing” the system components during a testing phase and collecting measurements corresponding to the external behavior of the system. In this paper, we propose using fault injection as the perturbation tool for dynamic dependency discovery and problem determination. We describe a method for characterizing dependencies of transactions on the system resources in a typical e-commerce environment, and show how it can aid in problem diagnosis. The method is applied to an application server middleware platform, running end-user activity composed of TPC-W transactions. Representative fault models for such an environment, that can be used to construct the fault injection campaign, are also presented.