Matchmaking: An extensible framework for distributed resource management

Federated distributed systems present new challenges to resource management. Conventional resource managers are based on a relatively static resource model and a centralized allocator that assigns resources to customers. Distributed environments, particularly those built to support high-throughput computing (HTC), are often characterized by distributed management and distributed ownership. Distributed management introduces resource heterogeneity: Not only the set of available resources, but even the set of resource types is constantly changing. Distributed ownership introduces policy heterogeneity: Each resource may have its own idiosyncratic allocation policy. To address these problems, we designed and implemented the Matchmaking resource management framework. Customers and resources are all described by classified advertisements (classads) written in a simple but powerful formal language that describes their attributes and allocation policies. A Matchmaker server uses a policy-independent matching operation to discover pairings. It notifies the parties to the match, which use a separate, bilateral claiming protocol to confirm the allocation. The resulting framework is robust, scalable and flexible, and can evolve with changing resources. Matchmaking is the core of the Condor High Throughput Computing System developed at the University of Wisconsin — Madison. Condor is a production-quality system used by scientists and engineers at sites around the world. Condor derives much of its flexibility, robustness and efficiency from the matchmaking architecture. We describe the use of matchmaking in Condor, presenting several examples that illustrate its flexibility and expressiveness. This revised version was published online in July 2006 with corrections to the Cover Date.