Study to support the Interim Evaluation (and Panel) - European ...

Technical report January 2008

Study to support the Interim Evaluation (and Panel) of the ICT Policy Support Programme under the CIP

Study to support the interim evaluation of ICT-PSP – DG INFSO Technical report

Contents 1. Database analysis........................................................................ 4 2. Key stakeholders involved .........................................................15

Key stakeholders among large companies.............................................................. 15 Key stakeholders from former programmes .......................................................... 16

3. Network analysis ...................................................................... 18

Method .................................................................................................................... 18 Findings ................................................................................................................... 19

4. Mapping applications and participations, and regional analysis 29 5. Online survey of stakeholders: technical report ........................ 38 6. Interviews with National Contacts Points...................................51 7. Interview with project coordinators.......................................... 74 8. ICT event 2008..........................................................................111 9. Case studies ............................................................................. 113

PEPPOL ................................................................................................................. 113 epSOS .................................................................................................................... 121 DTV4All ................................................................................................................. 131 T-SENIORITY .......................................................................................................140 ECRN ..................................................................................................................... 147 NESIS .................................................................................................................... 154 CALLIOPE ............................................................................................................. 164

© EURÉVAL / RAMBØLL - 10/02/2009 – page 2

Study to support the interim evaluation of ICT-PSP - DG INFSO Technical report

Euréval and Rambøll Management have been awarded the assignment to study the ICT policy support programme under the Competitiveness and Innovation Programme (CIP) in view of supporting the interim evaluation undertaken under the supervision of Mr. Gérard POGOREL (chairman of the panel) This is the technical report joined to the final report on the study. The main acronyms used in this document are: AC ............................................................................................................. Associated countries CIP .................................................................... Competitiveness and Innovation Programme CP1............................................................................................................ 1st Call for Proposals ERDF ..........................................................................European Regional Development Fund ICT .................................................................... Information & Communication Technologies ICT PSP .................................................................................. ICT Policy Support Programme ICT RTD ....................................................................ICT Research & Technical Development MS ..................................................................................................................... Member States NCP ………………………………………………………………………………….…… National Contact Points NMS...........................................................................................................New Member States NGOs/NPOs........................... Non-Governmental Organisations/Non-Profit Organisations R&D .................................................................................................. Research & Development SME .........................................................................................Small and Medium Enterprises

Eureval- 10/02/2009 page 3/171


1. Database analysis

Note: Several data sets, all provided by DG INFSO, have been used for this analysis: •

A database of abstracts and contacts;

•

A database compiling the results of the projects’ evaluation phase;

•

An ad hoc database which gathers the type-related information that was available in the application forms.

We merged and cleaned (double entries, heterogeneous typos, etc.) the three databases. Though there might still be some inconsistent data, we believe that the final database is now largely reliable. It must be taken into consideration that all calculations have been made on existing information, and that some projects are provisional, as most contracts have not yet been signed. Our integrated database is available as an Excel file with this report.

Distribution of proposals and selected projects The first call for proposals of the ICT-PSP opened in May 2007 and closed in September 2007. Negotiations started at the beginning of 2008, and are still in progress with selected consortia and applicants. 87 proposals were received with a total proposed budget of € 153.9 million1. “…Quality standards were maintained, despite the expected and planned low level of competition for certain Objectives in the call”2. The table below is based on the ICT-PSP First Call – Evaluation report and the ICT-PSP Draft implementation plan. It shows proposals by themes, objectives and sub-objectives. As the contract negotiations are still on-going, the number of proposals actually selected and contracted is not yet known: • •

35 proposals were judged eligible (‘above threshold’); Among them, 22 were finally selected, 11 were added to a ‘reserve list’ and 2 were ‘low budget’.

The total indicative budget for the call was € 54 million3. Themes and objectives

Nb of projects

Efficient and interoperable eGovernment services: Enabling EU-wide Public eProcurement (pilot A) Towards pan-European Recognition of electronic IDs (eIDs) (pilot A) Innovative solutions for Inclusive and Efficient eGovernment (pilot B) Mutual recognition and interoperability of electronic documents Accessible and inclusive eGovernement Services Combined delivery of social services Experience-sharing and Consensus-building in the Uptake of innovative eGovernment services (TN)

1 1 1 1 -

One proposal was withdrawn at the request of the applicant. See ‘External Monitoring of the ICT-PSP 1st Call, Independent Monitoring Panel’s Report’. Data used for the following tables are also extracted from this report. 3 CIP ICT-PSP Evaluation programme for 1st call. 1

2

Eureval- 10/02/2009 page 4/171


Themes and objectives

Nb of projects

Promoting local and regional eParticipation Stimulating measurement of impact and user satisfaction of eGovernment Brokering pan-European eGovernment solutions and services online ICT for accessibility, ageing and social integration Accessible digital audiovisual (AV) systems (pilot A) ICT for ageing well (pilot B)

1 1 0 1 -

Independent living4

5

Chronic disease5 Experience-sharing and consensus-building in ICT for inclusion (TN) Best use of ICT to support active ageing at work Best use of ICT for social integration and cultural diversity Making inclusion of mainstream consideration in innovation in ICT ICT for sustainable and interoperable health services EU-wide implementation of eHealth services to support continuity of care: patient’s summary and ePrescription (pilot A) Experience-sharing and consensus-building in eHealth (TN) Creating a European coordination network for eHealth interoperability implementation Facilitating a wider implementation of FRID applications in the healthcare sector Other themes and horizontal actions Sharing experience on ICT initiatives for SMEs (TN) To improve SMEs’ business environment through partnership-building and the development of business ecosystems To improve the capacity of business and in particular SMEs to benefit from ICT-based innovations in their products and services Supporting sustainable growth (TN) To promote ICT solutions aimed at improving energy efficiency at home, in the work place as well as in business processes To promote ICT solutions aimed at enhancing European operational capacities for monitoring and reporting environmental impact and threats Intelligent cars awareness action (TN) Privacy protection infrastructure (TN)

0 1 0 1

1 0 1 2 1 1 1 1

Table 1: Themes and objectives in the 1st call for proposals

Note: columns at the right merge two lines when sub-objectives were merged at the decision stage.

“To improve the quality of life of elderly people and their families, by substantially prolonging the time they can choose to live independently at home and manage their day-to-day activities, and supporting their social interactions, notably for people with cognitive problems and mild dementia”. 5 “To improve chronic disease management and provide better personalised healthcare services, contribute to the continuity of care and improve the quality of life. Elderly people are indeed often affected by multiple, parallel chronic diseases and, given cognitive difficulties, require support with medication management”. 4

Eureval- 10/02/2009 page 5/171


The indicative budget was respected, with a provisional budget of EUR 53,979,4016. Types of project (pilot A, B, TN) and budgets are not evenly distributed (see Table 2 and Table 3): • • • • •

Pilot A Pilot B TN

2 out of 3 pilot A projects are entered under the eGovernment theme; as a consequence, Theme 1 accounts for 45% of the call’s budget (EUR 24.3 M); Theme3 (eHealth) is almost exclusively about one pilot A project (Smart Open Services – S.O.S.); 6 out of 8 pilot B projects deal with eInclusion. Objective 2.2 (ICT for ageing well) has 5 of them, making it the best-funded objective within the 1st call (EUR 13.2 M, 25% of total budget); 7 out of 11 thematic networks concern ‘other themes’ (Theme 4: SME sharing experience, sustainable growth, intelligent cars, privacy protection). Only SME sharing experience has a significant budget (EUR 1.4 M); Finally, 4 objectives (1.1, 1.2, 2.2 and 3.1) account for more than 80% of the call’s total budget, with more than EUR 10 M each7. By comparison, the average budget for the other objectives is EUR 1.1 M. Theme 1: eGovernment

Theme 2: eInclusion 2 2 2

Theme 3: eHealth -

Theme 4: Others 1

6 1

TOTAL -

1

7

3 8 11

Table 2: Distribution of projects by type and theme

Objective 1.1 Objective 1.2 Objective 1.3 Objective 1.4 Objective 2.1 Objective 2.2 Objective 2.3 Objective 3.1 Objective 3.2 Objective 4.1 Objective 4.2 Objective 4.3 Objective 4.4 TOTAL

Theme 1: eGovernment Theme 2: eInclusion Theme 3: eHealth Theme 4: Others TOTAL 9 800 000 € - € - € - € 9 800 000 € 10 000 000 € - € - € - € 10 000 000 € 3 500 000 € - € - € - € 3 500 000 € 1 000 250 € - € - € - € 1 000 250 € - € 1 462 651 € - € - € 1 462 651 € - € 13 240 000 € - € - € 13 240 000 € - € 494 000 € - € - € 494 000 € - € - € 11 000 000 € - € 11 000 000 € - € - € 500 000 € - € 500 000 € - € - € - € 1 416 500 € 1 416 500 € - € - € - € 910 000 € 910 000 € - € - € - € 334 000 € 334 000 € - € - € - € 322 000 € 322 000 € 24 300 250 € 15 196 651 € 11 500 000 € 2 982 500 € 53 979 401 €

Table 3: Distribution of selected projects (scenario budget)

6 7

This budget is expected to change once negotiations are over. PEPPOL (Objective 1) is EUR 9.8 M. Eureval- 10/02/2009 page 6/171


Number of stakeholders 1,000 organisations applied once or several times in the 1st Call for Proposals: • • •

128 out of 1,000 organisations applied in at least two different projects; 55% of ‘multi-applicants’ were selected, and 35% of ‘mono-applicants’; However, among the 13 organisations which applied 4 times or more, only 3 were actually selected more than once (Centro Nazionale per l'Informatica nella Pubblica Amministrazione [4], Katholieke Universiteit Leuven [3], Regione Emilia Romagna[2]).

Among ‘multi-applicants’ which have been selected at least once8: • • •

The share of academic organisations is significantly higher than average (32% vs. 15%); The share of governments & public authorities is significantly lower than average (25% vs. 30%); idem for interest groups (6% vs. 14%); The share of private companies is a bit higher but remains stable (27% vs. 25%), though SMEs represent only 47% of private companies (64% of selected private companies are SMEs).

Overall 373 organisations were selected (37% of the applicants). More than half of the participants are involved in a thematic network (see Figure 3 below).

Figure 1: Distribution of participants by type of project9.

25 organisations were selected for more than one project: • •

One was selected for 4 projects (Centro Nazionale per l'Informatica nella Pubblica Amministrazione) and two other for 3 projects (Katholieke Universiteit Leuven and Fundació Privada Centre Tic i Salut). The 23 other ‘multi-project’ organisations were selected for 2 projects.

Finally: •

8 9

2 organisations were selected for 2 pilot A projects: the Centro Nazionale per l'Informatica nella Pubblica Amministrazione and the Regione Lombardia;

We applied a typology only to the 373 selected organisations. The total number is superior to 373, as some organisations were selected for several projects. Eureval- 10/02/2009 page 7/171


• •

one organisation was selected for 2 pilot B projects: the Fundació Privada Centre Tic i Salut; one organisation was selected for 3 thematic networks: the Katholieke Universiteit Leuven, and 8 other organisations for 2 thematic networks.

Two thirds of the ‘multi-selected organisations’ are public. Among them, one third is composed of governments or public authorities and another third of academic organisations (the rest is composed of diverse organisations, see below for typology).

Composition of projects Figure 410 shows that the average number of applicants per project is 18. Selected projects include from 8 to 30 organisations. As regards the national origin of the organisations, the projects have different profiles11: • •

On average there are 9 countries represented in the selected projects; On average, the most heavily represented country has 5 organisations in a project (~25% of an average project);

As regards the distribution of national origin of the organisations, three types can be mentioned: • • •

35 30

30

25 22 20 18 15 13 10 8 5

One third of the selected projects show a very 0 strong predominance of one country (e.g. Den4Dek, 12 eGos ); 4 of these projects are thematic networks; Figure 2: Number of One other third of the selected projects have a very applicants by project even distribution (e.g. CLEAR, Co-llab, (box-plot) CommonWell); they are mostly pilot B projects; The last third is composed of projects in which a small group of countries [3-4] is predominant, the rest being composed of one or two organisations per country (e.g. PrivacyOS, SOS). Two pilot A projects and 5 thematic networks comprise this group of 7.

Types of stakeholder Note: Two typologies have been used here. The first one is based on the information provided by the stakeholders in the application form (see Table 4 next page). The second is a ‘revised’, more precise typology that we have developed (see Table 5 next page). Though the second typology may include some mistakes (we tried to guess the type for each selected organisation, based on available information), it is more precise. Moreover the concepts used are more consistent: for instance, most small NGOs or public bodies have claimed in the application forms to be SMEs, which is not in line with the EC definition13. According to the application forms, 50% of the applicants were SMEs though only 25% were ‘Commercial’. The following elements can be highlighted from the application forms’ typology:

10 Figure 1 is a box-plot: Circles show minimal and maximal value, bars show the first quartile (under which 25% of the projects lie) and the second quartile (above which 25% of the projects lie), and the cross shows the average. 11 Distribution is based on coefficient of variation by project. 12 For Calliope, inclusion in this group is due to the large number of European interest groups located in Belgium in its composition. 13 See The new SME definition, User guide and model declaration, DG Enterprise http://ec.europa.eu/enterprise/enterprise_policy/sme_definition/sme_user_guide.pdf

Eureval- 10/02/2009 page 8/171


•

Public bodies accounted for half of the applicants, and slightly more (53%) of the selected organisations;

•

Small organisations (‘SME’ here) and ‘commercial organisations’ represent a smaller proportion of the selected organisations than of the applicants. Only 33% of small organisations and 26% of commercial organisations have been selected (Public bodies: 39%).

Budget-wise, the differences are quite important: • •

Public bodies obtained 2/3 of the available funding; Commercial organisations obtained 15%.

Our revised typology gives some additional insights on selected organisations: •

More than half of the organisations have a national scope, and one third has a local scope;

•

12% of the organisations have a European scope; half of them are interest groups (e.g. ‘Pharmaceutical group of the European Union’ or ‘Union européenne des médecins spécialistes’);

•

Half of the organisations involved are public, one quarter private, and 18% NPOs or NGOs see Figures 5 & 6 below);

•

30% of the organisations are governments and public authorities; there are slightly more national (55%) than local (41%) authorities.

•

Among the 94 private companies involved, 2 out of 3 are SMEs;

•

Academic organisations (15%) and interest groups (14%) also represent a significant share of the organisations.

Figure 3: Typology of participants

Eureval- 10/02/2009 page 9/171


Figure 4: Typology of participants (2)

Types (total number) SME --Public Body Commercial Standardisation Other --No answer

Number

Types (retained) SME --Public Body Commercial Standardisation Other --No answer

Number

Share (when answered) 463

50%

455 234 14 259

49% 25% 2% 28%

75

7%

Share (when answered) 152

44%

183 62 8 121

52% 18% 2% 35%

24

6%

Table 4: Typology of organisations (according to application forms): all applicants and selected organisations

Eureval- 10/02/2009 page 10/171


Types (total number) Local National European Worldwide --Public Private companies Other/NI NPOs-NGOs --Interest Groups Ad hoc consortiums/groupings Academic organisations Governments & Public authorities Private companies Others ---Among authorities Local Authorities National authorities Other authorities ---Among companies SME Other companies

Number

Share (when answered) 120 203 46 7

32% 54% 12% 2%

195 95 19 67

52% 25% 5% 18%

53 29 56 112 126

14% 8% 15% 30% 0% 34%

46 62 4

41% 55% 4%

61 34

64% 36%

Table 5: Typology of participants (revised typology)

Origin of stakeholders Applicants came from 37 countries, including 10 non-EU countries. Organisations from 32 countries were selected, including 5 non-EU countries (Norway, 16 organisations; Iceland, 8 org.; Switzerland, Croatia and Macedonia, 1 org.). There is at least one selected organisation in each EU country (1 in Malta and 1 in Cyprus). The selection rate is above average for some countries, especially Norway (67%), Slovenia, Denmark and Belgium (55%)14. 6 countries make up 57% of the organisations: Italy (15%), Spain (12%), Belgium (9%), Germany (8%), Great-Britain (7%) and France (6%). Belgium’s rank is very high because most European-wide organisations are based in Brussels. In the share of total selected projects, Italian organisations are still first, as they are present in 18 out of 22 projects. They are followed by Spain, Germany and Great-Britain. Three other countries are also present in at least half of the projects (Belgium, France and Netherlands, see Figure 7 below). When considered by regional origin, some large differences appear (see Figure 7 below):

14

•

184 selected organisations (49%) come from the ‘Big 5’ (Germany, France, GreatBritain, Italy, Spain);

•

121 (33%) come from the ten smaller countries of EU15;

•

42 come from the new member States. These organisations are moreover mainly represented in thematic networks (29/42); only 4 NMS organisations are included in pilot A projects. Slovenia is the first NMS with 10 applications selected.

Only countries for which there have been more than 10 selected organisations have been considered here. Eureval- 10/02/2009 page 11/171


Others; 26

Countries (involvement in projects)

New MS; 42

Italy (18/22) Big 5; 184

Spain (17/22) Germany, United Kingdom (14/22)

EU 15-5; 121

Belgium (13/22) France, Netherlands (11/22)

Figure 5: Origin of participants and involvement in projects

Targeted countries When considering a national breakdown of budget requested and obtained, however, the picture is somewhat different. Note: for analysis purposes, the ‘obtained grant’ is the addition of the grants requested by the organisations when they have been retained for a project. Though these grants are very likely to be negotiated before contracting, this is the best available approximation in our case (final grants are not yet available at the time of this study).15 First, budgetary analysis shows that organisations from the Big 5 obtained half of the available funding, the other EU15 countries, more than a third, and that NMS obtained less than 15% of the funding. This is overall consistent with the calculation organisation-wise. However, while the four main countries are still ahead of the ranking, the overall hierarchy is still quite different. In some countries (e.g. BE, SI), organisations have been massively joining thematic networks, for which grants are very low. Figure 6 below shows that: • •

all countries did not invest similarly in the Call for proposals; all countries did not benefit from a similar return on investment.

Countries above the line have received more than average when compared to their request. Spain and Norway, and to a lesser extent Austria, Belgium and Denmark, are in this situation. Countries below the line have received less in average when compared to their request. Italy and Greece, especially, are in this situation. The very low requests excepted, a correlation between requested and obtained is overall respected. Germany, United Kingdom, France, Sweden, Netherlands, Portugal, Finland and the Czech Republic are in this situation. This analysis also shows that the Czech Republic ranks first among new Member States in terms of funding, above Slovenia which is mostly involved in thematic networks. The Italian case is also interesting. Though Italian organisations have requested almost twice the funding requested by Spain (EUR 28M vs. 15.6), Italy barely obtained more (EUR 8M vs. 7.7).

15

As a consequence, the total ‘obtained budget’ here is EUR 61M instead of EUR 54M. Eureval- 10/02/2009 page 12/171


Requested vs. obtained grant (M€) 12

10

8

ES

IT

DE

6

AT

NO 4

BE

DK

FR

SE NL

PT

2

UK

FI CZ

EL

0 0

5

10

15

20

25

30

Figure 6: Linear regression, requested (x) and obtained (y) grant

As a last point, it should be noted that application to a European Call for proposal comes for a cost (in FP6, it was estimated between EUR 10 to 30 K€ for a project). As a consequence, it can be considered that organisations from countries below the line (especially Italian and Greek organisations) had to invest more to obtain 1 euro than organisations above the line. This can be alternatively measured through the ‘level of effort’ of organisations, i.e. how many did they have to request to obtain one euro. Overall, organisations had to request 2.5 € to obtain 1. Among the countries which benefited most from the 1st Call, the following countries are: • • •

above average: Greece (3.9), Italy (3.5), France (3.0) and Netherlands (2.9); around average: Sweden and UK (2.7), Ireland, Finland and Germany (2.4); below average: Spain and Austria (2), Denmark and Belgium (1.7).

Among new Member States, some countries have particularly suffered of the application process: Bulgaria (12.3), Lithuania (11.4), Poland (7.3) and Romania (4.3). Other new Member States have been more modest in their requests and have been rewarded accordingly. Several new Member States have applied for funding in large projects (Pilot A and B), but were only retained in the end in Thematic networks. Overall, organisations from new Member States had to ask more to obtain funding than organisations from other European countries (3.3 in NMS vs. 2.7 in the Big 5 and 2.6 in the remaining countries).

Preliminary conclusions: first findings and hypotheses These first findings will have to be completed with the survey results in order to be meaningful in the context of the evaluation. The following elements should however be highlighted: •

80% of the 1st call for proposal budget is allocated to 4 objectives (out of 13); special attention should be paid to these;

•

Though only one quarter of the projects were selected, one third of the applicants are involved in the selected projects: a significant share of the organisations (12%), Eureval- 10/02/2009 page 13/171


especially academic organisations (30%), are present in several projects. This may give a hint on the central role of some organisations, especially academic; some others may have multiplied applications in order to increase their chances of being selected; •

Some projects show a very strong country predominance, especially thematic networks, while pilot B projects are the most evenly distributed; relevance of these distribution patterns should be checked in the evaluation;

•

Few organisations (23) have been selected for more than one project; they are mainly governmental and/or academic organisations. Contrary to what is observed in R&D Framework programmes, there might be few networking at global level (few organisations were selected in several projects); however, there might be links at sector or national/regional/local level;

•

Moreover, almost one quarter of the organisations (interest groups and ad hoc consortiums or groupings) are actually organisations specialised in networking;

•

Half of the selected organisations come from the Big 5 member States. More precisely, those countries are in average present in 15 (out of 22) projects.

Eureval- 10/02/2009 page 14/171


2. Key stakeholders involved Key stakeholders among large companies Method In order to assess whether the most innovative companies were involved in the ICT-PSP process, the ranking produced by Boston Consulting Group for Business Week (The World's 50 Most Innovative Companies16) has been used and compared to the database of applicants to the programme. This ranking is by no means a definitive approach to innovative companies, but the method tends to follow representations by actors of corporate innovation. The method is the following: Executives were asked to vote for the most pioneering companies in 2007, and this vote was weighted with financial indicators (on a 80/20 basis).

Results Within the 1st Call for Proposals, 5 out of the ‘50 most innovative companies’ applied to the programme: • • • • •

Nokia Group (#10); Hewlett-Packard (Italiana, HP Limited#15); IBM (Euro-coordination, IBM Italia, IBM Research Gmbh, #12); Microsoft EMEA (#5); Siemens (#38).

Out of these 5 organisations, 2 are European and 3 American. IBM is actually the most involved large innovative company in the ICT-PSP. These organisations were not all retained. After the application stage, the remaining organisations are: • • • •

16

Nokia; IBM (Euro-coordination and Research Gmbh); Hewlett-Packard (HP Ltd.); Microsoft EMEA

http://bwnt.businessweek.com/interactive_reports/innovative_companies/index.asp Eureval- 10/02/2009 page 15/171


Key stakeholders from former programmes Method used Our hypothesis was that a number of organisations which applied to the ICT-PSP had participated in former programmes such as the 6th Framework Programme for Research and Development (FP6). By comparing the ICT-PSP database of applicants and the FP6 database of beneficiaries, the objective was to highlight which organisations crucial to the FP6 projects were also present in the ICT-PSP. Crucial organisations have been defined as organisations which participated in a large number of projects (more than 30 projects, i.e. ~1% of organisations) and which coordinated several projects (more than 5 projects, i.e. 10% of coordinating organisations). It should be remembered that this comparison cannot be totally accurate, as the names of organisations can be written differently (in English or in the local language, the legal/formal name or a shorter version, precisions on a section of the organisation or not, etc.). Moreover, different methods have been used on the two bases (e.g. the ICT-PSP distinguishes the different Fraunhofer Institutes, while the FP6 database considers that it is only one organisation). Different filters have however been used to increase the accuracy of the analysis (use of legal name, of short name, of country). Specific organisations such as universities and regions, and all aforementioned crucial organisations, have been treated manually.

Results FP6 organisations in ICT-PSP Almost one third (31%, 320 org.) of the 1,016 organisations which applied to the ICT-PSP had benefited before from the FP6. This rate does not change significantly when we consider only retained applicants (32%, 120 org.), or ‘multi-beneficiaries’ of the ICT-PSP (35%), which means that these stakeholders, although experienced, did not benefit from a higher success rate in applying to the ICT-PSP. More than half of the FP6 organisations (181) which applied to the ICT-PSP had participated in 3 or more projects in FP6. This is a very high proportion, when all the FP6 beneficiaries are considered (91% have been included in only one or two projects). Finally, the 62 organisations which have participated in at least 3 FP6 projects represent 16% of the 376 retained organisations in the ICT-PSP. Participation of the ‘crucial’ organisations Of the 28 ‘crucial’ organisations in FP6, 13 have applied to the ICT-PSP, including: • • •

Public research organisations and institutes such as the German Fraunhofer Institutes, the French CNRS, the Italian CNR and the Norwegian SINTEF (but not the French INRIA and CEA or the Dutch IMEC); Academic organisations such as the Universitat Politècnica de Catalunya, Vienna University of Technology, the Universitaet Karlsruhe (but not the Politecnico di Milano); Large companies (or their research branches) such as Siemens, Telefónica, Atos Origin or Philips (but neither France Telecom, Thales nor SAP).

Of the FP6 organisations whose project has been turned down, the largest are the 5 Fraunhofer institutes and the CNRS (two backbones of the European FP6 network). They are followed by several academic organisations (Vienna University of Technology, the Universitat Politècnica de Catalunya, and the University of Karlsruhe) and research organisations (ICCS). Companies such as Telecom Italia, Siemens and Philips have also been rejected.

Eureval- 10/02/2009 page 16/171


In the end, the following ‘crucial’ organisations were retained in the ICT-PSP 1st call for proposals: • •

ATOS Origin and Telefónica (large companies); Stiftelsen Sintef (Research Centre).

Other major organisations in FP6 (>30 projects), but less crucial to the FP6 network, include: • • •

Katholieke Universiteit Leuven and Universidad Politécnica De Madrid (Academic organisations); Valtion teknillinen tutkimuskeskus (Finnish Research Centre); Nokia Group (large company).

Main themes and types of projects concerned Organisations which participated to FP6 projects have tended to favour Thematic networks (66% of the FP6 organisations vs. 57% of overall organisations) and Theme 4 (others, 43% vs. 31%). Of the 14 FP6 organisations which participate in a Pilot A project, 11 are actually in the STORK project (secure identity) and account for more than one third of the 29 partners. The distribution of these organisations in Pilot B is more even (they are present in the 8 Pilot B), but they are again concentrated in fewer projects, especially DTV4ALL (6 out of 8 partners), which can be considered as a ‘FP6 project’ here. Other significant concentrations include NEXES (5 out of 12), T-Seniority (6 out of 17) and Commonwell (4 out of 12). Share of funding Funding requested and received by organisations is not yet available at the time of this analysis.

Conclusion The following facts are noteworthy for the analysis: • • •

One third of the retained organisations in the ICT-PSP has participated or still participates in FP6 projects. Half of them (~60 org.) can be considered as active, well networked organisations (3 or more projects); FP6 participants are not evenly spread out. They are most strongly present in one Pilot A (out of 3) and in half of the Pilot B projects (4/8); Of the 28 ‘crucial’ organisations for FP6, 13 applied and only three have been retained; altogether, 7 central organisations in FP6 are still present in the ICT-PSP, out of 376 beneficiaries.

Eureval- 10/02/2009 page 17/171


3. Network analysis Method The network analysis has been performed with Cytoscape 2.6.1. and Network analyser17 on a constituency of 1,000 organisations which applied to the ICT-PSP’s 1st Call for proposal, among which 378 were retained for at least one project. The main data source is the internal EC database, which has been cleaned by Euréval for analysis purposes. Attributes such as the typology of retained organisations have been added by the Euréval team. As it is a small constituency (when compared for instance to FP6’s), care is needed to interpret the results obtained. Especially, some statistical indicators may not be relevant here to show the linkages in the network; we have therefore decided to rely only on the most reliable ones in our case, such as the shortest paths. The results have been supervised by our expert, Marko Grobelnik, to ensure their accuracy. Several hypotheses had to be formulated before the analysis was performed: •

First, we considered that each link in a project represented a channel of collaboration, knowledge exchange and information spillovers18;

•

Second, we considered that projects were ‘circle-shaped’ networks, i.e. that all organisations in the networks are linked to each other. Empirical analysis would show that this is rarely the case, and that in reality networks would vary between ‘star-shaped networks’ (one organisation is at the centre of all communications in the project) and ‘circle-shaped’ ones. However, we assumed that along the project, relations will strengthen between all partners;

•

Third, and as a consequence, we have considered that relations created within not retained projects between organisations retained in other projects were an evidence of extra-collaborations between organisation;

•

Fourth, in order to find the hubs, we identified organisations which had been strongly active in FP6 before (organisations with more than 3 projects in FP6). Though FP6 and ICT-PSP are very different programmes, we considered that organisations active in FP6 had 1/ experience in EU-wide programmes, 2/ international networks (and research networks) to bring to the projects.

http://cytoscape.org/ and http://med.bioinf.mpi-inf.mpg.de/netanalyzer/ This is the definition given in the CESPRI report: Networks of Innovation in Information Society: Development and Deployment in Europe (2006), from which several hypotheses have been drawn. 17

18

Eureval- 10/02/2009 page 18/171


Findings All organisations The first step was to consider all organisations involved (retained or not) in the first Call for proposals (CP1). Out of 1,000 organisations, one giant component includes 874 of them (87%), and 75 out of 87 projects. 7 of these 12 disconnected projects have no connections with other projects, while one component is made of 3 projects and another of 2 projects. The average path length of this network is 3.7, i.e. 3.7 steps are in average necessary to go from one organisation to any other in the main component.

Retained organisations (with not retained links) The second step consisted in considering only the 373 organisations retained with their links in retained and not retained projects. They are all connected in one giant component (100%). The average path length of this network is a bit lower, at 3.5. Analysis in this network shows the presence of groups of organisations which applied together for not retained projects but which were retained for another project. For instance: • •

tGov4inclusion: Regione Toscana (IT), Università di Firenze (IT), Institut für Rundfunktechnik Gmbh (DE)and Universitat Ramon Llull (ES); NECI: iCentres Association (BG), Inclusion Alliance Europe GEIE (BG), Ministry of Communications and Information Technology (RO);

Some other organisations have applied several times together: • •

Three times: o Regione Toscana (IT) and Università di Siena (IT); Twice: o Fachhochschule Vorarlberg (AT), Helsingin kauppakorkeakoulu (FI), University of Szeged (HU)and Stichting Living Lab (NL); o Service Public Fédéral Technologie de l'Information et de la Communication (BE), Government To You (EL) and Centro Nazionale per l'Informatica nella Pubblica Amministrazione (IT); o Università di Siena (IT), Regione Toscana (IT) and ALTEC SA (EL); o City of Issy-les Moulineaux (FR) and the Politech Institute (BE); o Inclusion Alliance Europe GEIE (BG) and Ministry of Communications and Information Technology (RO); o Comune di Bologna (IT) and City of Ghent (BE).

We draw the assumption that these partners are used to work together. A strong share of them has been active in FP6 before.

Most connected organisations We selected in the former network the organisations with a low clustering coefficient, i.e. which are connecting different retained or not retained projects19. Our hypothesis is that these organisations are the most connected among research and deployment networks. The obtained network is composed of 70 connected organisations. 19 Organisations which are in only one project have a clustering coefficient of 1, which means that they make all potential connections with their neighbours (this is due to the circle shape which was given to the projects).

Eureval- 10/02/2009 page 19/171


Retained projects only The third step consisted in considering only retained organisations and their links in retained projects only. This network is more fragmented, with 7 components (6 of them corresponding to 6 different projects: 3 Thematic networks and 3 Pilot B projects). The main component contains 287 organisations (77%). The statistics displayed in Table 6 below characterise the network of retained projects. These values do not compare favourably with FP6’s20. For instance, the average path length in FP6 is 2.6, but the small size of the ICT-PSP network is a strong factor here. Features

Information

Number of nodes

373

Number of edges

3882

Network density21

0.055

Size of the largest component

287 nodes (77%)

Longest shortest path (network diameter)22

6

Average path length

3.3

Table 6: Features of the network

Regarding the disconnected components, it is noteworthy that the thematic networks concerned were filled under the theme ‘Others’ (e.g. the iCars Network); this can explain why they are not connected. However, the presence of 3 Pilot B projects (all filled under the eInclusion theme) should be specially highlighted. We draw the hypothesis that the selection process has excluded some projects which were creating more linkage between organisations through some key stakeholders, e.g.: • • •

VTT (which applied 4 times) for the iCars Network; Regione Toscana (which applied 3 times) for the T-Seniority Project; Brunel University and the Institut für Rundfunktechnik GmbH for DTV4All.

The FP6 networks is used here as a benchmark of a very integrated network. Network density is the proportion of ties in a network relative to the total number possible. 22 The network diameter is the largest distance between two nodes in the main component. 20 21

Eureval- 10/02/2009 page 20/171


S.O.S.

eGovMoNet

Figure 7: Network of retained organisations. In blue, thematic networks; in green, Pilot B and in red, Pilot A. Font and node sizes are functions of the number of retained projects by organisation.

The obtained network shows some other interesting motifs: • •

Two projects are ensuring the connectivity of the network: S.O.S. and eGovMoNet; Two groups of two strongly interlinked projects appear: Den4Dek/Co-LLAB (top of the diagram) and S.OS./CALLIOPE (bottom of the diagram).

This network is otherwise quite fragmented, as shown in Table 7 below. Deleting 5 nodes doubles the number of components from 7 to 14.

Number of nodes deleted

Number of components

0

7

1

9

2

11

...

...

5

14

Table 7: Fragmentation of the network

These 5 bridging organisations are: • • •

Centro Nazionale per l'Informatica nella Pubblica Amministrazione (IT); Fundació Privada Centre Tic i Salut (ES); Háskóli Íslands / University of Iceland (IS); Eureval- 10/02/2009 page 21/171


• •

empirica Gesellschaft für Kommunikations - und Technologieforschung GmbH (DE); Regione Lombardia (IT).

Two denser groups appear when these 5 organisations are removed: • •

The remains of the giant component: Bridge IT, CO-LLAB, DEN4DEK, NESIS, PEP-NET, PrivacyOS, STORK, which are linked together by the eGovMoNet project23; A second component made of the S.O.S. and CALLIOPE partners.

STORK and CALLIOPE excepted, these are all thematic networks. All the Pilot B projects are on the other hand very dependent on a small set of organisations to be linked to the network.

Network by themes

Figure 8: Network of retained organisations. eGovernment is in red, eHealth in blue, eInclusion in orange and ‘Others’ in green. Sizes of names are function of the situation of organisations as being Core, Central or not (see chapter below)

Considering the networks by theme (see Figure 8) shows very different situations in terms of connectivity: • •

The eGovernment theme is very connected within itself and with other themes, especially thanks to the eGovMonet project. The eHealth theme is also strongly interlinked, but this time the thematic network does not add much linkage to the other themes.

Seven organisations in eGovMoNet are essential to the linkage: Danish Technological Institute, Government To You, Service Public Fédéral Technologie de l'Information et de la Communication (BE), Univerza v Mariboru, Katholieke Universiteit Leuven, Technicka univerzita v Kosiciach, Regione Emilia Romagna

23

Eureval- 10/02/2009 page 22/171


•

Finally, the eInclusion theme is very fragmented, and relies on 3 organisations only to link to other themes.

The role of thematic networks can be specifically highlighted here: • • •

In the first case, one thematic network connects almost all thematic actors and provides a link towards other organisations and themes; o TN concerned: eGovMoNet In the second case, thematic networks allow to extend the community of organisations involved in the theme, but do not provide external links; o TN concerned: Calliope, PEP-Net In the third case, thematic networks do not even provide extra-connectivity within the theme. o TN concerned: Bridge IT

Hubs and central organisations in the network The network has been split in three categories: • • •

Core organisations (25): these are the organisations with a low clustering coefficient, which means that they are linking several organisations together (hubs); Central organisations (45): these are the retained organisations with a low clustering coefficient in the network made of all applicants which are not core organisations; Peripheral organisations (303): these are the other organisations in the network.

These organisations have specific features: CORE org. (7%)

CENTRAL org. (12%)

OTHER org. (81%)

Pilot A: 60%

Pilot A: 9%

Pilot A: 15%

Pilot B: 16%

Pilot B: 49%

Pilot B: 28%

TN: 88%

TN: 42%

TN: 57%

eHealth: 52%

eHealth: 2%

eHealth: 9%

eGovernment: 40%

eGovernment: 40%

eGovernment: 32%

eInclusion: 12%

eInclusion: 38%

eInclusion: 26%

Academic org.: 28%

Academic org.: 33%

Academic org.: 11%

Govt. and public auth.: 36%



SME: 8%

SME: 13%

SME: 16%

Large companies: 8%

Large companies: 16%

Large companies: 8%

17 different countries



DE, IT: 3, AT, BE, DK, NL: 2

ES: 9, IT: 8, BE: 5, DE, EL: 4

IT: 47, ES: 39, BE: 25, DE 23...

FP6 active: 24%

FP6 active: 40%

FP6 active: 12%

Grant obtained: EUR 9.8M (16%)

Grant obtained: EUR 7.1M (12%)

Grant obtained: EUR 36M (72%)

Table 8: Features of the core, central and other organisations

Within the main component of this network (57/70 nodes, see Figure 9 below), the average path length drops to 2.78, though the network diameter keeps at 6 steps.

Eureval- 10/02/2009 page 23/171


Figure 9: The main component of the core and central organisations of the network.

Table 8 and Figure 10 show that core and central organisations in the ICT-PSP CP1 have quite specific features: • • • • • • •

They are more likely to be academic organisations; Pilot A public authorities are core organisations in the network, while Pilot B public authorities are more peripheral; SMEs and large companies tend to be either central or peripheral organisations in the network; They are more evenly distributed across Europe than the whole constituency of retained organisations, though in less countries; Core organisations are 7% of all participants, but they received 20% of the available funding: these organisations are ‘big players’; 22 out of 25 hubs participate in a pilot A or B project and in a thematic network, which gives them connexion that the pilot projects miss, especially pilot B (see before). Half hubs are involved in eHealth, which seems to be a lot more connected than the other themes at stake.

Eureval- 10/02/2009 page 24/171


Figure 10: Position of types of organisations and themes in the network

Hubs and central organisations in projects All projects include at least one core or one central organisation: • • •

Thematic networks tend to classify in High concentration or Low concentration; Pilot B projects tend to classify in Average or Low concentration; Pilot A do not tend to classify in any.

Types

Projects

High concentration of CORE and CENTRAL organisations

SOS, CALLIOPE, eGovMoNet, DEN4DEK, COLLAB

AVERAGE concentration CENTRAL organisations

T-Seniority, NEXES, ECRN, DTV4ALL, STORK, PEP-Net

of

CORE

and

Low concentration of CORE and CENTRAL organisations

CLEAR, PEPPOL, Dreaming, Commonwell, ICT21EE, iCars Network, Privacy OS, NESIS, Net-Share, eGOS, Bridge IT

Table 9: Concentration of core and central organisations in retained projects

Mapping of Core and Central organisations (see note on cartography) shows that these organisations are evenly distributed in regions with many retained organisations: there are no ‘orphan’ regions on this behalf. Eureval- 10/02/2009 page 25/171


It should be noted also that though Pilot B projects are poorly connected to the ICT-PSP networks, they mostly benefit of several central organisations in their partnerships (6/8).

Repartition of relations Figure 11 below shows that core and central organisations in the network have very few relations at regional and national level. Core organisations tend to have even more international relations than the other groups. These figures are mostly consistent with the shares seen in research networks such as FP624.

Figure 11: Repartition of relations on a geographical basis

As a matter of comparison, Figure 12 below shows the relations of all organisations depending on the type of project they are in. Organisations involved in Pilot B projects have a significantly higher rate of relations at regional (10%) and national (13%) levels. These rates are consistent with those observed in former deployment networks (eTen, eContent, see foot note 24). Following the logic of the CESPRI report, Pilot B projects have some of the necessary features to ensure potential regional deployment.

24 See the CESPRI report, op. cit. Regional and national links together account for 10% of the relations of research hubs, while in deployment networks, this rate climbs to 21%.

Eureval- 10/02/2009 page 26/171


Figure 12: Repartition of relations on a geographical basis by type of project

Gatekeepers According to the CESPRI report, gatekeepers are “organisations that link [the research and deployment networks] together [and therefore] are in a unique position to speed up the process of innovation and technology diffusion [...]”25. We have considered that ‘FP6 active organisations’26 were potentially in that position in the ICT-PSP network. Only 4 out of 22 projects do not benefit from the presence of such an organisation (Commonwell, NESIS, PEPPOL, Dreaming). Four others have only one such organisation in their partnership.

“A key role in fostering complementarities between research and deployment networks is played by gatekeepers, i.e. organisations that link the two networks together and, by doing so, allow others to access information and capabilities developed in other networks and contexts. [...] The gatekeepers sit between the two networks: they are positioned in both networks. [...] We hypothesize that gatekeepers are in a unique position to speed up the process of innovation and technology diffusion, since they work as bridge between the two different networks.” CESPRI report, op. cit., p. 17. 26 FP6 active organisations are