From EGI Knowledge Base
- To see the earlier version of this page, please go to EGI Functions (old)
This area is dedicated to the definition, discussion and refinement of the functions to be fulfilled in EGI, where by "EGI" we mean the central EGI Organisation (EGI.org) plus the NGIs.
The new (as of 13 May 2009) draft of the Deliverable 3.2: Final Definition of the EGI Functions is available here
An earlier version of this deliverable is available here
These functions presuppose the general Business Model described elsewhere in this site.
 User Community Services
The overviews conducted by the EGI_DS prep team show that the user community that already makes use of the grid is very large and diverse. This community expects to continue to use the current infrastructure and to drive its evolution. It is also growing quickly and diversifying even further, generating new requirements. As the e-Infrastructure becomes more generalised, the EGI will become a point of reference for international collaborations as envisioned in the ESFRI Roadmap:
- Europe has a long-standing tradition of excellence in research and innovation, and European teams continue to lead progress in many fields of science and technology. However our centres of excellence often fail to reach critical mass in the absence of adequate networking and cooperation. There is therefore a need to bring resources together and build a research and innovation area equivalent to the “common market” for goods and services.
Within this ecosystem, the set of services commonly referred to as application support, training, and technical dissemination need to evolve to fit an EGI model which is more inclusive yet more "customisable" than in current models. We will refer to these as User Community Services (UCS).
The needs of the scientific communities cannot be assumed a priori to be homogeneous and well known. Each community has its specific applications, its specific training needs, and its specific communication venues. Hence the extended support services must be articulated according to well defined sets of requirements by each scientific community. This leads to the concept of Specialised Support Centres (SSCs). The role of EGI.org in this context is to provide central coordination and representation to the various communities and VOs via the extended services offered by the SSCs.
 Listening to the Users
Please see User Requirements
 User Community Services: Tasks
Please see UCS Tasks.
 Operations and Security
The operations and security function includes those EGI tasks needed to ensure optimal functionality of the pan-European infrastructure and the overall seamless effective interoperation of national and regional Grids.
This Chapter provides a thorough description of activities and the related resource demand. A summary of the Chapter is available in EGI_DS Deliverable 4.4 the EGI Blueprint.
Operations and security activities are composed of EGI.org tasks and of international NGI tasks, which are complementary and equally important. The relationship between EGI.org tasks and NGI international tasks is illustrated in the following graph:
EGI operations and security tasks (blue rectangle) as a sum of NGI international tasks and EGI.org tasks.
The operation of the pan-European Grid infrastructure relies on a number of principles.
These are the foundation of the set of operational and security tasks identified in this Section:
- Reliability of Grid services and SLAs: notwithstanding the different and evolving needs of application communities and NGIs, a key component of the EGI vision is the provision of a large-scale, production Grid infrastructure – built on NGIs that interoperate seamlessly at many levels, offering reliable and predictable services to a wide range of applications, ranging from “mission critical” to prototyping and research. It is understood that it will be a long and continuous process to reach this, with additional NGIs and/or application communities joining at different times, with varying needs and different levels of “maturity”. In addition, sites of widely varying size, complexity and stage of maturity must be taken into account. The EGI shall negotiate the minimal size and set of functions for an NGI to participate in a wider context, including the associated Service Level Agreements. This includes the agreement and follow-up of the associated certification processes. In some cases, these requirements may be more stringent than those used within a given NGI. That is, only a subset of sites participating within an NGI may satisfy the wider requirements at the EGI level.
- Multi-level operation model: highly centralized models – e.g. for monitoring – have been shown to be both intrusive and non-scalable. This suggests a move to a multi-level operations model. Whilst building on the positive experience of today’s production Grids, these concerns must nevertheless be taken into account as part of the EGI/NGI architecture. This includes designing and deploying for low-cost-of entry and ownership, whilst maintaining sufficient flexibility to meet the requirements of the application clusters. The EGI shall foster agreement on the definition of the key operations infrastructure, its establishment and delivery. Such functions are preferably located at one or more NGIs (to offer both resilience and scalability).
- EGI, NGI and ROC: The NGIs participation to the operation of the European grid infrastructure requires set of services to be operated in a coherent way. Currently, within EGEE, this is guaranteed by the Regional Operations Centres (ROCs), that either span over several countries (NGIs) or are serving one country only. The NGIs must assure that the services are operated, either at the NGI level or through associating into ROC equivalents.
- A secure environment: Security is essential for establishing trust in the Grid infrastructure. It spans a wide range of topics, from low level computer forensics through middleware security to the highest level policies negotiated between networks and institutions. It ranges from immediate incident response to adapting to advances in technology which may be years from deployment. Furthermore, security will be vary between NGIs, and will certainly differ between different types of Grid middleware. The challenge is to build on the security expertise of the NGIs, to foster collaboration, coordination and best practice, to ensure that the whole is not just as strong as the weakest link. The continuing development of international standards, for example in OGF, will be essential for interoperability. EGI.org, the NGIs and the resource centers have responsibilities to ensure a secure operating environment for users, VOs and sites.
- Planning, coordination and gathering of new requirements: EGI.org operations represent a “thin layer” mainly responsible for operations planning and coordination of efforts by the various NGIs and other parties. Also, EGI.org operations staff works towards a smooth evolution of tools and operational procedures according to the new requirements gathered.
- Cooperation: EGI.org and NGIs cooperate for a sustainable and effective operation of the e-Infrastructure and to tackle problems of common interest such as: implementation rules for robust services, security best practices, middleware security issues, steering of new developments, intervention procedures, incident response, escalation procedures and so forth.
- Federation, interoperability and data aggregation: EGI must federate a variety of operational aspects – some of which are implemented by NGIs and/or component sites. Consistency of security procedures, user support, incident tracking, monitoring and accounting must be ensured. EGI ensures interoperability of operational tools/infrastructures for security, monitoring, support, accounting, etc. For scalability reasons, operational data such as monitoring information, availability statistics and accounting records – collected by the NGIs need to be aggregated at the EGI.org level for SLA monitoring in full respect of the relevant national legal constraints.
The operations and security model adopted in EGI is distributed and needs to satisfy various requirements:
- scalability and interoperability
- we expect the level of complexity of the EGI Grid infrastructure to gradually increase due the growing number of resource centres involved, of user communities supported and, in the transition phase, to an increasing complexity of the middleware to be deployed and supported. Scalability and interoperability of operations need to be guaranteed under such conditions.
- availability and reliability
- operations need to be structured in a way which eases the delivery of a production-quality e-Infrastructure.
- responsibility of daily operations and of ensuring high availability of services need to be distributed to NGI’s and to the resource centres themselves. This is also achieved through increasing automation, the improvement of Grid operational tools and the establishment of bilateral SLAs.
- autonomy of NGI’s
- the operational model needs to be sufficiently flexible to allow the NGI to fully conform to EGI policies and procedure and to satisfy specific requirements and activities in the country.
Distribution is one possible approach to both ensure smooth transition and to address the above-mentioned requirements. For example, reporting of usage will need to be collated and trouble tickets may well also traverse several helpdesks. Interworking relies on common standards and/or specifications for interoperation between NGI’s. For example, exchange of information between Grid domains is necessary to support functionalities such as resource discovery and accounting; protocols for this need to follow common guidelines. To this end, collaboration from the NGIs is important to jointly define specifications, policies, best practices, and in general, to share operational responsibilities. In what follows EGI Operations and security tasks and the related manpower effort are described.
 Tasks and services
Given the operational described above, the main operational and security tasks of EGI.org are the coordination of NGI activities, definition of procedures, policies, specifications and standards for interoperation, and the operation of central data aggregation services and user-support services such as the helpdesk. The added value of the EGI.org tasks is to grant the seamless and efficient integration of the National Grids, providing coordination, procedures, repositories etc. Note that the current EGEE III model is already partially distributing responsibilities to regions and analyzing the operation tools and processes to see where additional distribution can increase efficiency. EGI.org tasks are defined to be mandatory if deemed necessary to ensure interoperation of operational tools and to implement a consistent operational model across countries. We assume mandatory tasks to be already provided in year one of EGI. The list of tasks for EGI.org and NGI is provided below, while a summary table of EGI.org activities and corresponding effort is provided in Table 2. Tasks are described in general and abstract terms, however they rely on the current operational model developed in the framework of the current and past EGEE projects. Activities are divided into five categories:
 Middleware Development and Support
The general goal of EGI is to realize a large-scale, production Grid infrastructure for the sharing of IT resources and data – built on National Grids that interoperate seamlessly at many levels, offering reliable services to a wide range of applications, ranging from “mission critical” to prototyping and research” will be reached by distributing very different responsibilities between the various players. The EGI specific technical objective, to be achieved by a coordinated effort of the EGI.org central organization and the National Grid Initiatives, is to take care, on behalf of all the stakeholders, of the procurement, certification, deployment, and operation of the software services ( i.e. of the software infrastructure) and to set up the organizational rules, policies and procedures which will provide the required standard access and sharing mechanisms for all sort of IT resources and data which currently are and will continue to be made available to the researchers by national resource providers. These today are mostly constituted by public or semi-public Resource Centers of very different size and dimensions which will continue to be completely funded at national level. In the Blueprint we have discussed the need for the establishment of a middleware function in EGI which needs to remain in full control of the software infrastructure which constitutes one of its key services offered to all the stakeholders. The Blueprint proposes to implement such function, at least in the first EGI years, to avoid the risk of disruption of the current services used daily by thousands of researchers, with a limited manpower in EGI.org, and without requiring any further immediate mandatory contribution from the NGI’s. The maintenance and support of the Middleware components used in the current e-Infrastructure should continue to be co-funded at a level of 50% from the EC , without excluding contributions from other interested partners and sustained for the remaining part by the Middleware Consortia and other development teams, that have agreed to move toward a Unified Middleware Distribution (UMD), under the steering of EGI. In future, once the EGI e-Infrastructure will be well established, the maintenance and support of the legacy widely adopted services will be progressively covered by the NGIs through the payment of service charges while the necessary innovation and the new developments, as defined by the need of the user communities and operations, will continue to be co-funded by EC through competitive calls. In this Chapter more details are provided, in separate sub-chapters about:
- Middleware Components and Middleware Consortia
- Guidelines for UMD
- Role of the EGI.org Middleware Unit
- Components and Services proposed
 External Liaison Functions
A small team within this function will execute the dissemination activities of the EGI.org. The team will focus on content production and coordinating activities. Technical and specific services will preferably be bought from third party. The objectives of the dissemination activities of EGI.org are:
- to ensure the visibility and inform about EGI among decision makers, funding bodies, research communities, industry partners and other grid initiatives in Europe and worldwide
- to inform the user communities and NGIs
- to arrange activities in collaboration with the NGIs
- to create and maintain excellent PR/media relations
- to coordinate publishing of activity and management reports
- to organise events such as EGI conferences and user forums
The dissemination activities need to be effective and well targeted. For EGI the dissemination activities at large must be executed both by EGI.org and the NGIs with a clear division of responsibilities. The EGI.org will typically be in charge of tasks requiring coordination between NGIs. In general terms, the EGI.org will deal with the common actions of the EGI while the NGIs are responsible for the EGI dissemination in their local and regional areas. It is important to note that in order to achieve good results the dissemination team needs to act in close collaboration with the user-oriented, grid operational and technical activities of EGI. The dissemination team of the EGI.org will serve as a horizontal link between the stakeholders (NGIs) and existing user communities, and has therefore a central role in maintaining the information flow to these parties. A dynamic and up to date website is a key element in maximizing the visibility, providing support to users and stakeholders and informing about the EGI. It is therefore a clear need for a professional and dedicated web editor. The dissemination team of the EGI.org will support and coordinate the publication work of the EGI. Press releases and Newsletters of the work and key achievements will be published and widely distributed in order to increase the visibility of the EGI. NGIs have to contribute by providing material to paper and electronic publications. The EGI.org will also be in charge of organisation of annual events and conferences, similar to e.g. EGEE User Forum and DEISA Symposium. These events not only increase the visibility and inform existing users but also aim to enlarge the user community. Also presence in other major events in Europe and outside will be coordinated and organised by the dissemination team of EGI.org, whereas NGIs are responsible of the EGI presence in local and regional events. The presence can be for example, a presentation, where the dissemination team would support in finding the right experts. Further in practical arrangements, such as drafting of abstracts, or planning for an exhibition booth. The core EGI.org dissemination team will be small at the beginning, but can be augmented by a rotation of 1-2 colleagues from the NGIs. The NGIs will further be requested to provide a contact person for the dissemination activities within their organisations.
 Industry take up
It has been identified that sustainability of EGI would benefit from a persistent activity aimed at increasing participation of the private sector in the European Grid Infrastructure, bringing additional competences and financial resources to the initiative. As a publicly funded infrastructure aimed for research, the usage policies will be determined not only by EU policies but primarily by national law and policies. The usage policies can be expected to be comparable to those of other similar research infrastructures, such as the GÉANT network and thus dedicated to research usage. Business usage is then limited, and primarily in form of research collaborations with European and national research institutes, universities and other educational institutions. The EGI.org management must develop a business model for the grid infrastructure although it by essential qualities will be limited in its commercial potential. The general interest and potential use by industry can be very different. The most likely forms are:
- use of the EGI infrastructure in R&D (collaboration with the publicly funded research community).
- the EGI infrastructure as “state-of the-art”/”best practice” for industry.
- industry use of the EGI infrastructure for testing and learning.
- industrial projects with occasional exceptional requirements (critical computing on demand)
Preferably EGI.org would initiate a work together with the stakeholders, to elaborate on policies for access for industrial research projects in the pre-competitive domain and industrial production projects accessing innovative technologies or deploying innovative strategies. The NGIs are expected to work along similar lines on a national level. Following the recommendations of the e-Infrastructure Reflection Group (e-IRG) Task Force on Sustainable e-Infrastructures, industry has to be seen as both a potential user and a service provider. Today it is possible to identify an emerging business based on the major European grid technologies. EGI.org should act positively towards such initiatives and establish policies allowing emerging companies and other initiatives a fair competition in providing services for the EGI.
 Other External Relations
External relations are defined as relations with organisations and initiatives outside of the EGI and of direct relevance for the EGI in terms of collaboration or interoperation. Examples of such organisations and initiatives are:
- Grids outside Europe
- Commercial grids (e.g. cloud computing efforts)
- Large-scale international research collaborations (e.g. the EIROForum organisations and WLCG)
- Networking organisations (e.g. NRENs, DANTE, TERENA)
- Policy and standard shaping bodies (e.g. e-IRG, ESFRI, OGF)
The EGI.org management and specifically the Director should be in charge of External Relations. This responsibility should primarily be focused on:
- establishment of formal relations when necessary
- promotion of common understanding on policies in scope of grid interoperation
- influence on policy and Standards shaping activities
- networking and enlargement of the EGI “sphere of influence”
The operational aspects in interoperation with other grids are handled by the EGI.org Grid Operations function. The activity is not actively pursuing standardisation work but handles the relations of EGI.org with organisations such as OGF, e-IRG and OASIS. EGI.org should consider membership in organisations like OGF and OASIS if it is found beneficial for EGI. The work could include coordination and reporting of participation in different Standards working groups and interfacing with the technical teams doing the actual standardisation. To maximise the outcome of the external relations activity the EGI.org management should encourage bridging to external organisations and initiatives through the NGIs.
One of the obviously necessary functions for EGI is the management of EGI.org. The following description contains a rough sketch of the management levels and (within text-boxes) assumptions how the functions should be funded.
 Assumptions about EGI.org's financing structure
The first assumption is on the general budgetary structure:
- (a) Central management
- (b) Projects
- (c) Service provisioning
are accounted in separate cost centres. Cost centres are an efficient way of presenting activities of EGI.org in a transparent way to the EGI Council.
The most important formal characteristic is that only the EGI Council is able to transfer resources from one cost centre into another one.
The second assumption is on the budget sources. The income is provided by three different streams:
- membership fees from NGIs according to an EGI-Key, which is decided by the EGI Council,
- income from project grants
- service charges to be paid by those NGIs who get specific services from EGI
 EGI council
The top level management layer in EGI.org is the EGI Council. The NGIs own EGI.org and voice their views on all EGI matters through the EGI Council.
The EGI Council may install committees, which elaborate recommendations to the EGI Council for specific topics.
It may furthermore elect an Executive; details will be determined later.
The EGI.org Management has to provide legwork services to the EGI Council and its committees and it is assumed that arising costs are covered from the EGI “general central services” budget.
 EGI.org director and heads of units
The EGI.org Director, who works full time, provides the organizational interface to the EGI Council, to political bodies (EU etc.) and
to several EGI committees on one side and to the Heads of the EGI.org Units on the other side.
The EGI Director has to direct the group of unit heads.
For all internal and external activities the EGI.org Director has one person who will assist him with handling his work.
Within the unit heads the functions of a Central Technical Officer (CTO), a Central Operational Officer (COO) and a Central Administration Officer (CAO) are implemented. The administration also covers efforts for the public relations and contains positions in the administrative and legal services.
The EGI.org Director needs a secretariat and must have some staff which prepares policy developments, the representation on European level and the legwork function for the EGI Council.
It is assumed that the positions of the EGI.org Director, the CAO and the administration and PR group are paid through the EGI.org “general central services” budget which relies (only) on the NGI’s membership fees.
EGI.org should be positioned in a flexible way as far as EGI.org units are concerned.
It seems that only three units should have a permanent basis: the Operational Unit, the Development Unit and the Administration Unit with the COO, the CTO and the CAO as head of the respective unit.
Projects may, based on EGI.org’s findings, be embedded in these units or they may be organized as a separate project oriented unit within EGI.org, but always embedded in the organization’s structure.
This project management layer should, if possible, be paid by project grants and their complements, mostly resources organized by the NGIs.
The following graph summarizes the features of the EGI.org management structure: