Middleware Components and Middleware Consortia
From EGI Knowledge Base
Currently, a variety of Middleware components are deployed in the EU e-Infrastructure. They are the result of several years of European and international competitive efforts aimed at satisfying the needs of a large number of user communities with complementary requirements and dimensions ranging from teams of few individuals to very large international collaborations with thousands of researchers. They all adhere to a general service oriented approach aimed at compliance with the evolving Web Services and the Open Grid Forum standards. They are largely part of three more or less complete solutions: gLite , ARC and UNICORE which provide the foundation services for the current EU e-Infrastructure. Other EU middleware projects (GridWay, pGrade, AssesGrid, GRIA...) in Europe, funded by the European Commission and by National funds, have provided high level services which complement the basic services provided by these 3 EU stacks. The recent efforts in particular by the OMII Europe project have already improved their interoperability. Pragmatically (after 8 years of successful developments) these 3 EU stacks with the mentioned enrichments mainly from other EU initiatives provide a very large fraction of the services in use in the EU e-Infrastructure together with some components from the US-based Globus and VDT. These 3 European stacks thus constitute the basis for the creation of the open source Unified Middleware Distribution (UMD) that the future European Grid Initiative (EGI) will make available to the national Resource Providers as a key integral part of its offer and business model. The availability of certified grid services that can be easily downloaded from a common UMD repository, together with the set of common procedures, policies and rules that EGI will establish, will make easy for the research teams to access and share all type of IT resource and data made available by their national resource centers and funded at national level. The EGI Middleware function needs to continue to guarantee the current level of quality of the deployed services to the European scientific user communities during the transition and in the initial years of consolidation of the new EGI organization. During the transition to the new sustainable European organization embodied by EGI, the middleware currently represented by these stacks and other identified services needs to continue to be supported, maintained and further developed, in particular in view of emerging standards, and, in some parts, completed and hardened from their current stage to fully satisfy the operational quality requirements. The requirements will be established by a new EGI body including representatives of VOs, Operations and Middleware development, called EGI Technical Coordination Board –TCB) which represents a straightforward evolution of current best practices of EGEE, DEISA and other national experiences. The maintenance, development and evolution towards standards for these 3 EU stacks and related middleware projects is currently funded by national Institutions or Consortia and co-funded by the EU Commission via competitive bids.
The Advanced Resource Connector (ARC) is developed by the NorduGrid collaboration and associated projects since 2001. It features a decentralized architecture, leading to high efficiency, low maintenance costs and robust performance. It is highly portable and is available for all major Linux flavours. This in turn allows a decentralized deployment of ARC in over 60 sites, with over 20.000 cores. In particular, ARC is adopted by the NDGF (Nordic DataGrid Facility) to support the world’s only distributed heterogeneous Tier1 center. Currently, the next generation of ARC is under development, which minimizes dependencies on third-party components, improves extensibility, interoperability and allows portability to non-Linux platforms.
gLite services are the result of a truly pan-European development effort made by the EDG-EGEE project series started in 2001 and co-financed by EC via competitive bids. They are currently deployed in about 250 sites distributed in all EU countries and provide a general, unified and robust access service to ~50.000 compute nodes (largely commodity clusters with some HPC systems) and to very large (>15 PB) distributed storage systems. The gLite middleware consists of an integrated set of components compliant with open standards and covering all the aspects of the Grid infrastructure. It is developed for the Scientific Linux environment, but extensive effort is recently provided to make it operating system independent. The gLite environment is tightly coupled with the ETICS build and test system (also funded by the EC), which provides an automated environment for the integration and validation of new components and their new versions.
UNICORE has a traditional HPC background (since 10 years) and is open source since 2004 (http://www.unicore.eu). It is used in HPC-related infrastructures like DEISA (serving a similar amount of CPUs as in EGEE but concentrated on a few powerful supercomputers) and in the future PRACE (European PetaFlop/s Supercomputers), but also in non-HPC-focused NGIs like D-Grid and some Swiss SwiNG projects. UNICORE is characterized by its open, extensible, lean, and interoperable Web services architecture which supports many open standards, providing a seamless, secure and intuitive access to Grid resources. UNICORE comes with a strong focus on workflow capabilities, security, application support and ease of installation and configuration.
