Big Data, Open Data and Data Development

30 downloads 1619 Views 6MB Size Report
Smart Innovation Set coordinated by. Dimitri Uzunidis. Volume 3. Big Data, Open Data and Data Development. Jean-Louis Mo
Big Data, Open Data and Data Development



Smart Innovation Set coordinated by Dimitri Uzunidis

Volume 3

Big Data, Open Data and Data Development

Jean-Louis Monino Soraya Sedkaoui

First published 2016 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address: ISTE Ltd 27-37 St George’s Road London SW19 4EU UK

John Wiley & Sons, Inc. 111 River Street Hoboken, NJ 07030 USA

www.iste.co.uk

www.wiley.com

© ISTE Ltd 2016 The rights of Jean-Louis Monino and Soraya Sedkaoui to be identified as the authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988. Library of Congress Control Number: 2016931678 British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library ISBN 978-1-84821-880-2

Contents

Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xi

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xix

Chapter 1. The Big Data Revolution . . . . . . . . . . . . . . . . . .

1

1.1. Understanding the Big Data universe . . . . . . 1.2. What changes have occurred in data analysis? 1.3. From Big Data to Smart Data: making data warehouses intelligent . . . . . . . . . . . . . . . . . . 1.4. High-quality information extraction and the emergence of a new profession: data scientists . . . 1.5. Conclusion . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

2 8

. . . . . . . . . .

12

. . . . . . . . . . . . . . . . . . . .

16 21

Chapter 2. Open Data: A New Challenge . . . . . . . . . . . . . . .

23

2.1. Why Open Data? . . . . . . . . . . . . 2.2. A universe of open and reusable data . 2.3. Open Data and the Big Data universe . 2.4. Data development and reuse . . . . . . 2.5. Conclusion . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

23 28 33 38 41

vi

Big Data, Open Data and Data Development

Chapter 3. Data Development Mechanisms . . . . . . . . . . . . 3.1. How do we develop data? . . . . 3.2. Data governance: a key factor for data valorization . . . . . . . . . . . . . . 3.3. CI: protection and valuation of digital assets . . . . . . . . . . . . . . . . 3.4. Techniques of data analysis: data mining/text mining . . . . . . . . . . . . 3.5. Conclusion . . . . . . . . . . . . . .

43

. . . . . . . . . . . . . . . . .

44

. . . . . . . . . . . . . . . . .

54

. . . . . . . . . . . . . . . . .

60

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65 72

Chapter 4. Creating Value from Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

73

4.1. Transforming the mass of data into innovation opportunities . . . . . . . . . . . . . 4.2. Creation of value and analysis of open databases . . . . . . . . . . . . . . . . . . . 4.3. Value creation of business assets in web data . . . . . . . . . . . . . . . . . . . . . 4.4. Transformation of data into information or “DataViz” . . . . . . . . . . . . . . . . . . . 4.5. Conclusion . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

74

. . . . . . . . . . . . .

82

. . . . . . . . . . . . .

87

. . . . . . . . . . . . . . . . . . . . . . . . . .

94 100

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

101

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

109

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

121

Acknowledgements

This book is the product of several years of research devoted to data processing, statistics and econometrics in the TRIS (traitement et recherche de l’information et de la statistique) laboratory. It is the fruit of several projects carried out within the framework of research and development for several startups within the Languedoc-Roussillon region and large private and public groups. I would like to thank all of the members of the RRI (réseau de recherche sur l’innovation), and more particularly, Dimitri Uzunidis, its president, for his attentive and careful reading of the first version, and who encouraged us to publish this book. Thanks also to M. Bernard Marques, who had the difficult task of proofreading the manuscript and who had many important notes to help with the understanding of the book. I would also like to thank my teacher and friend Jean Matouk, who was the cause of this publication, thank you for his encouragement and unfailing support over the years. Many thanks to all the researchers at the laboratory for their help and support and most especially to Soraya Sedkaoui; without her this book would never have seen the light of day. Thanks to all those who have supported me through difficult times and who have transformed an individual intellectual adventure into a

viii

Big Data, Open Data and Data Development

collective one, in particular Alain Iozzino, director of the startup E-prospects, with whom we have carried out many research and development projects over the years. Finally, I must express my special gratitude to those dear to me, my family, and most of all to my wife, who has had to put up with my moods over the last few years. Jean-Louis MONINO This book was a work of adaptation, updating and rewriting in order to adapt all of the work of the TRIS laboratory. Its creation was fed by exchanges and discussions with my teacher Jean-Louis Monino, without whom this book would never have seen the light of day. I am infinitely grateful to him for having included me in this adventure. It would not have been possible to produce this book without my family who have always encouraged and supported me throughout all my ideas and projects, no matter how far away they have sometimes been. Special mention must go to my mother, there are no words to express how important she is and how much she has done in making me what I am today. Finally, I would like to thank Hans-Werner Gottinger, Mohamed Kasmi and Mustapha Ghachem for their unfailing support and for the interest that they have always shown in what I am doing. Soraya SEDKAOUI

Foreword

The world has become a digitalized place, and technological advancements have multiplied the ways of accessing, processing and disseminating data. Today, new technologies have reached a point of maturity. Data is available to everyone throughout the planet. In 2014, the number of Internet users in the world was 2.9 billion, which is 41% of the world population. The thirst for knowledge can be perceived in the drive to seize this wealth of data. There is a need to inquire, inform and develop data on a massive scale. The boom in networking technologies – including the advent of the Internet, social networks and cloud computing (digital factories) – has greatly increased the volume of data available. As individuals, we create, consume and use digital information: each second, more than 3.4 million emails are sent throughout the world. That is the equivalent of 107,000 billion emails per year, with over 14,600 per person per year, although more than 70% of them are junk mail. Millions of links are shared on social networks, such as Facebook, with over 2.46 million shares every minute. The average time spent on the Internet is over 4.8 hours per day on a computer and 2.1 hours on a cellphone. The new immaterial substance of “data” is produced in real-time. It arrives in a continuous stream flowing from a variety of generally heterogeneous sources. This shared pool of all kinds of data (audio, video, files, photos, etc.) is the site of new activities aimed at analyzing the enormous mass of information. It thus becomes necessary to adapt and develop new approaches, methods, forms of knowledge and ways of working, all of which involve new paradigms and stakes as a new

x

Big Data, Open Data and Data Development

ordering system of knowledge must be created and put into place. For most companies, it is difficult to manage this massive amount of data. The greatest challenge is interpreting it. This is especially a challenge for those companies that have to use and implement this massive volume of data, since it requires a specific kind of infrastructure for the creation, storage, treatment, analysis and recovery of the same. The greatest challenge resides in “developing” the available data in terms of quality, diversity and access speed. Alain IOZZINIO E-PROSPECTS Manager January 2016

Key Concepts

Before launching into the main text of this book, we have found it pertinent to recall the definitions of some key concepts. Needless to say, the following list is not exhaustive: – Big Data: The term Big Data is used when the amount of data that an organization has to manage reaches a critical volume that requires new technological approaches in terms of storage, processing, and usage. Volume, speed, and variety are usually the three criteria used to qualify a database as “Big Data”. – Cloud computing: This term designates a set of processes that use computational and/or storage capacities from remote servers connected through a network, usually the Internet. This model allows access to the network on demand. Resources are shared and computational power is configured according to requirements. – Competitive intelligence: It is the set of coordinated information gathering, processing and dissemination activities useful for economic actors. According to the Marte Report, competitive intelligence can be defined as the set of coordinated information research, processing and dissemination actions aimed at exploiting it for the purpose of economic actors. This diverse set of actions is carried out legally with all data protection guarantees necessary to preserve the company’s assets, with the highest regard to quality, deadlines and cost. Useful information is needed at the company or partnership’s different decision-making levels in order to design and put into place strategies and techniques coherently aimed at achieving company-

xii

Big Data, Open Data and Data Development

defined objectives and improving its position in the competitive environment in which it operates. These kind of actions take place in an uninterrupted cycle that generates a shared vision of company objectives. – Data: This term comprises facts, observations and raw information. Data itself has little meaning if it is not processed. – Data analysis: This is a class of statistical methods that makes it possible to process a very large volume of data and identify the most interesting aspects of its structure. Some methods help to extract relations between different sets of data and thus draw statistical information that makes it possible describe the most important information contained in the data in the most succinct manner possible. Other techniques make it possible to group data in order to identify its common denominators clearly, and thereby understand them better. – Data governance: It constitutes a framework of quality control for management and key information resource protection within a company. Its mission is to ensure that the data is managed in accordance with the company’s values and convictions, to oversee its quality and to put mechanisms into place that monitor and maintain that quality. Data governance includes data management, oversight, quality evaluation, coherence, integrity and IT resource security within a company. – Data journalism: The term designates a new form of journalism based on data analysis and (often) on its visual representation. The journalist uses databases as his or her sources and deduces knowledge, meaningful relationships or intuitions from them that would not be accessible through traditional research methods. Even when the article itself stands as the main component of the work, illustrating ideas through graphs, diagrams, maps, etc., is becoming more important day by day. – Data mining: Also referred to as knowledge discovery from data, is intended for the extraction of knowledge from large amounts of data using automatic or semi-automatic methods. Data mining uses algorithms drawn from disciplines as diverse as statistics, artificial intelligence and computer science in order to develop models from

Key Concepts

xiii

data; that is, in order to find interesting structures or recurrent themes according to criteria determined beforehand and to extract the largest possible amount of knowledge useful to companies. It groups together all technologies capable of analyzing database information in order to find useful information and possible significant and useful relationships within the data. – Data reuse: This practice consists of taking a dataset in order to visualize it, merge it to other datasets, use it in an application, modify it, correct it, comment it, etc. – Data science: It is a new discipline that combines elements of mathematics, statistics, computer science and data visualization. The objective is to extract information from data sources. In this sense, data science is devoted to database exploration and analysis. This discipline has recently received much attention due to the growing interest in Big Data. – Data visualization: Also known as “data viz”, it deals with data visualization technology, methods and tools. It can take the form of graphs, pie-charts, diagrams, mappers, timelines or even original graphic representations. Presenting data through illustrations makes it easier to read and understand. – data.gouv.fr: The French government’s official website for public data, which was launched on December 5th 2011 by Mission Etalab. In December 2013, data.gouv.fr was transformed deeply through a change in both the site’s structure and its philosophy. It has, without doubt, become a collaborative platform oriented towards the community, which has resulted in better reuse of public data. – Dataset: Structured and documented collection of data on which reusers rely. – Etalab: This is a project proposed in the November 2010 Riester Report and put into place in 2011 which is responsible for implementing the French government’s open data policy, as well as for establishing an almanac of French public data: data.gouv.fr. – Hadoop: Big Data software infrastructure that includes a storage system and a distributed processing tool.

xiv

Big Data, Open Data and Data Development

– Information: It consists of interpreted data and has discernible meaning. It describes and answers questions like “who?”, “what?”, “when?” and “how many?”. – Innovation: It is recognized as a source of growth and competitiveness. The Oslo Manual distinguishes between four types of innovation: - Product innovation: Introduction of a new product. This definition includes significant improvements to technical conditions, components or materials, embedded software, user friendliness or other functional characteristics. - Process innovation: Establishing a new production or distribution method, or significantly improving an existing one. This notion involves significant changes in techniques, material and/or software. - Marketing innovations: Establishing a new marketing method involving significant changes in a product’s design, conditioning, placement, promotion or pricing. - Organizational innovation: Establishing a new organizational method in practices, workplace organization or company public relations. – Interoperability: This term designates the capacity of a product or system with well-known interfaces to function in sync with other existing or future products or systems, without access or execution restrictions. – Knowledge: It is a type of know-how that makes it possible to transform information into instructions. Knowledge can either be obtained through transmission from those who possess it, or by extraction from experience. – Linked Open Data (LOD): This term designates a web-approach proposed by supporters of the “Semantic Web”, which describes all data in a way such that computers can scan it, and which links to it by describing its relationships, or by making it easier for the data to be related. Open public data is arranged in a “Semantic Web” format,

Key Concepts

xv

such that its items have a unique identifier and datasets are linked together by those identifiers. – Open innovation: It is defined as increased use of information and knowledge sources external to the company, as well as the multiplication of marketing channels for intangible assets with the purpose of accelerating innovation. – Open knowledge foundation network: A British non-profit association that advocates for open data. It has most famously developed CKAN (open source data portal software), a powerful data management system that makes data accessible. – Open data: This term refers to the principle according to which public data (gathered, maintained and used by government bodies) should be made available to be accessed and reused by citizens and companies. – Semantic Web: This term designates a set of technologies seeking to make all web resources available, understandable and usable by software programs and agents by using a metadata system. Machines will be able to process, link and combine a certain amount of data automatically. The semantic web is a set of standards developed and promoted by W3C in order to allow the representation and manipulation of knowledge by web tools (browsers, search engines, or dedicated agents). Among the most important, we can cite: - RDF: a conceptual model that makes it possible to describe any dataset in the form of a graph in order to create knowledge bases; - RDF Schema: language that makes it possible to create vocabularies, a set of terms used to describe things; - OWL: A language that makes it possible to create ontologies and more complex vocabularies that serve as support for logical processing (interfaces, automatic classification, etc.); - SPARQL: A query language for obtaining information from RDF graphs. – Semi-structured information: It is worth noting that the boundary between structured information and unstructured information is rather fuzzy, and that it is not always easy to classify a given document into

xvi

Big Data, Open Data and Data Development

one category or the other. In such a case, one is no doubt dealing with semi-structured information. – Smart Data: The flood of data encountered by ordinary users and economic actors will bring about changes in behavior, as well as the development of new services and value creation. This data must be processed and developed in order to become “Smart Data”. Smart Data is the result of analysis and interpretation of raw data, which makes it possible to effectively draw value from it. It is, therefore, important to know how to work with the existing data in order to create value. – Structured information: It can be found, for example, in databases or in programming languages. It can thus be recognized by the fact that it is arranged in a way such that it can be processed automatically and efficiently by a computer, but not necessarily by a human. According to Alain Garnier, the author of the book Unstructured Information in Companies, “information is structured when it is presentable, systematic, and calculable”. Some examples include forms, bills, pay slips, text documents, etc. – Text mining: This is a technique that makes it possible to automate processing of large volumes of text content to extract the main tendencies and statistically assess the different subjects they deal with. – Tim Berners-Lee: He is the co-inventor of the Semantic Web. He is very active and engaged in data.gov.uk. In particular, he has defined a five star ranking system to measure the Semantic Web openness level for putting a dataset online. – Unstructured information: Unlike structured information, unstructured information constitutes the set of information for which it is impossible to find a predefined structure. It is always intended for humans, and is therefore composed mainly of text and multimedia documents, like letters, books, reports, video and image collections, patents, satellite images, service offers, resumes, calls for tenders, etc. The list is long. – Web 1.0: This term refers to the part of the Internet that makes it possible to access sites composed of web pages connected by

Key Concepts

xvii

hyperlinks. This Web was created at the beginning of the 1990s. It creates a relationship between an edited site that publishes content or services and Internet users who visit it and who surf from site to site. – Web 2.0: This term designates the set of techniques, functions and uses of the World Wide Web that have followed the original format of the Web. It concerns, in particular, interfaces that allow users with little technical training to appropriate new Web functions. Internet users can contribute to information exchanges and interact (share, exchange, etc.) in a simple manner. – Web 3.0: (also known as the Semantic Web). This is a network that allows machines to understand semantics, which is to say the meaning of information published online. It expands the network of Web pages understandable by humans by adding metadata that is understandable by a machine and that creates links between content and different pages, which in turns allows automatic agents to access the Web in a more intelligent manner and to carry out some tasks in the place of users.

Introduction

Today data comes from everywhere: GPS tracking, smartphones, social networks where we can share files, videos and photos, as well as online client transactions made possible through the intermediary of credit cards. Of the 65 million people in France, 83% are Internet users and 42% (or 28 million) are on Facebook. More than 72 million telephones are active, and the French people spend a daily average of 4 hours online. Mobile phone users spend over 58 minutes online, and 86% of the population is on a social network. The French people spend over 1.5 hours per day on social networks. Developing this massive amount of data and making access to it possible is known as “Big Data”. This intangible data comes in a constant stream and its processing is especially challenging in terms of knowledge extraction. This is why new automatic information extraction methods are put into place such as, for example, “data mining” and “text mining”. These are the sorts of processes behind radical transformations in the economy, marketing and even politics. The amount of available data will increase greatly with the appearance of new connected objects on the market that are going to be used more and more. Some objects we use in our daily lives are already connected: cars, television sets and even some household appliances. These objects have (or will one day have) a chip designed to collect and transfer data

xx

Big Data, Open Data and Data Development

to their users through a computer, a tablet or a smartphone. More importantly, these objects will also be able to communicate with one another! We will be able to control equipment in our homes and in our car by simply logging onto our smartphone or some other device. This phenomenon is known as the “Internet of Things”. The American economist, Jeremy Rifkin, predicted the development of a new society of wealth and abundance brought about by technology, especially by the Internet of Things and 3D printing. New technologies would modify socio-economic relationships to the point of significantly reducing profits for capitalist enterprises. In the world of the Internet, the advent of the zero marginal cost society has already taken place. As information has become dematerialized and as it has become possible to reproduce and distribute it with near-zero marginal costs, radical changes have come about in these industries’ business models. Box I.1. Zero marginal cost society

This phenomenon has attracted the interest of operational decisionmakers (marketing managers, finance chiefs, etc.) seeking to benefit from the immense potential involved in analyzing data hosted by companies in real-time. In order to meet the Big Data challenge, measures must be taken, including incorporating tools that make more restrictive data processing possible and actors capable of analyzing that data. This will only be possible if people become more aware of the benefits of “data development”. When databases are organized, reorganized and processed by statistical methods or econometric modeling, they become knowledge. For a company, it is essential to have access to more and more data about the environment in which it operates. This will make it possible to scrutinize not classes of behavior, but individual cases. This explains why this revolution has brought with it the emergence of socalled “start-up” companies whose objective is to process the data known as Big Data automatically. We certainly find ourselves in front of one of the elements of what some people are calling the “new industrial revolution”. The Internet and the digital and connected objects have opened up new horizons in a wide array of fields.

Introduction

xxi

Data access makes it possible to enrich quantitative and qualitative analyses. Client contacts can be analyzed through data collected by a call center. This kind of product can also be offered in a limited quantity, as does e-prospects. It is necessary to develop that data by exploring the content of emails and voice calls, and to match that information with browsing activities on the company website. Beyond that, it is also possible to study messages exchanged on social networks (Facebook, Twitter, LinkedIn, etc.) in order to identify new trends or to identify the products that are being most talked about. Box I.2. A data access example

Contact and appointment buyers • •

Professionals Request contacts and appointments in their sectors

• •

Demand Purchase

Choice between various providers Possibility of buying individually Different selling/purchasing models Neutral and legitimate information on the provider

• • •

Demand Supply Production

Contact and appointment providers • • • •

Call centers Web agencies Telemarketing companies Leads professionals

E-prospects1 has developed an innovative service based on a community platform that makes it possible to buy and sell contacts and meetings through a range of products and countries. Research and development with the University of Montpellier’s TRIS laboratory2. The program is supported by the LanguedocRoussillon region and by the “Transferts L-R” program3.

Example I.1. The startup E-PROSPECTS

1 http://www.e-prospects.biz. 2 TRIS stands for “Information and Statistics Development and Research” in French. 3 Transfert LR stands for “Regional Languedoc-Roussillon Innovation Agency” in French.

xxii

Big Data, Open Data and Data Development

In order to get the full potential out of data, it must be available to all interested parties with no additional obstacles and at reasonably accessible costs. If data is open to users [MAT 14], other specialized data processing companies can be created. This activity will meet the needs of users without them having to develop models and equations themselves. Open Data, beyond its economic and innovative potential, involves a philosophical or ethical choice4. Data describes collective human behavior, and therefore, belongs to those whose behaviors it measures. The cultivation of these phenomena depends on the availability of data that can be communicated easily. The Internet Age has detonated a boom in information research. Companies are flooded by the wealth of data that results from simple Internet browsing. In other words, they are forced to purchase pertinent information to develop high added value strategies that allows them to succeed in the face of incessant changes in their business environment. Industrial strategies now rely strongly on the capacity of companies to access strategic information to better navigate their environment. This information can, thus, become the source of knew knowledge (knowledge pyramid). The process of gathering, processing and interpreting information is not limited to defining ideas, but also consists of materializing them in order to ensure improved knowledge production that leads to innovation. Competitive intelligence allows each company to optimize its service offerings in qualitative and quantitative terms, as well as to optimize its production technology.

4 To speak just of the economic advantages of open data, a study carried out for the European Commission estimated that the total market for public sector information in 2008 in the EU reached 28 billion euros. According to this study, the global economic advantages resulting from more open public sector data would represent about 40 billion euros per year for the EU [VIC 11]. For the entire EU economy, the total direct and indirect gains from ISPs (Internet Service Providers) and applications based on that data would be in the order of 140 billion euros per year.

Introduction

EXAMPLE (enlarged)

xxiii

In the last few years, several initiatives to verify the authenticity of diplomas have been launched. In this regard, it is worth noting that University of Montpellier and, in particular, its research laboratory TRIS have been pioneers in the fight against diploma falsification since 2011.

Example I.2. Information processing C2i certificate 5 6 security and massive processing by QRCode

Beyond the advent of ICT and of increased data production, dissemination and processing speeds, another element has recently become critically important: time. The importance of time carries with it a notion of information circulation speed. This prompts companies to rethink their strategies beyond the challenges involved in processing large volumes of data. The value of a given piece of data increases in time and depends on the variety of uses it is given. In this sense, companies must possess the capacity to absorb the entirety of data available, which allows them to assimilate and reproduce knowledge. This capacity requires specific skills that make

5 C2i: IT and Internet Certification Established in 2004 by the French National Ministry of Education and Research. One of the websites devoted to the certification: http://www.portices.fr 6 A QR code is a type of two dimensional barcode (or datamatrix code) composed of black squares aranged in a larger squeare with a white background. The distribution of these points defines the information contained in the code. QR (an abbreviation for quick response) means that the code’s content can be quickly decoded when read by a scanner.

xxiv

Big Data, Open Data and Data Development

it possible to use that knowledge. Training “data scientists” is, therefore, indispensable in order to be able to identify useful approaches for new opportunities, or for internal data exploitation, and in order to quantify their benefits in terms of innovation and competitiveness. However, Big Data is just a single element in the new set of technical tools known as “data science”. Data scientists have the task of extracting knowledge from company data. They hold a strategic function within the firm, and to that end, must be in command of the necessary tools. They must also be able to learn on the go and increase their understanding of data mining regularly, as the volume of data requires increasing skills and techniques. E-prospects’s research and development project, carried out in collaboration with TRIS, has been developed by using Statistica’s data mining.

Example I.3. Data mining and Statistica software

When confronted with this multiplicity of data, companies are driven to apply sophisticated processing techniques. In fact, technical competence in data processing is today a genuine strategic and useful stake for companies’ competitive differentiation [BUG 11]. Processing this mass of data plays a key role for tomorrow’s society because it can be applied in fields as varied as science, marketing, customer services, sustainable development, transportation, health and even education.

Introduction

xxv

Big Data groups together both processing, collection, storage and even visualization of these large volumes of data. This data, thus, becomes the fuel of the digital economy. It is the indispensable raw material of one of the new century’s most important activities: data intelligence. This book shows that the main challenges for Big Data revolve around data integration and development within companies. It explores data development processes within a context of strong competition. More specifically, this book’s research brings together several different fields (Big Data, Open Data, data processing, innovation, competitive intelligence, etc.). Its interdisciplinary nature allows it to contribute considerable value to research on the development of data pools in general. I.1. The power of data Companies are very conscious of the importance of knowledge and even more so of the way it is “managed”, enriched and capitalized. Beyond all (financial, technical and other) factors, the knowledge that a company has access to is an important survival tool, whether it is market knowledge, or legal, technological and regulatory information. Knowledge is an extension of information to which value has been added because it is underpinned by an intention. It is a specificallyhuman phenomenon that concerns thought in rational, active and contextual frameworks. It represents an acquisition of information translated by a human element and which requires analysis and synthesis in order to assimilate, integrate, criticize and admit new knowledge. Information corresponds to data that has been registered, classified, organized, connected and interpreted within the framework of a specific study. Exploiting collected data requires: sorting, verification, processing, analysis and synthesis. It is through this process that raw

xxvi

Big Data, Open Data and Data Development

data collected during research is transformed into information. Data processing provides information that can be accessed in decisionmaking moments. Lesca [LES 08] explores the problems behind interpreting data to transform it into strategic information or knowledge. Interpretation systems, which are at the heart of competitive intelligence, are defined as “meaning attribution systems”, since they assign meaning to information that companies receive, manipulate and store [BAU 98]. According to Taylor [TAY 80], the value of information begins with data, which takes on value throughout its evolution until it achieves its objective and specifies an action to take during a decision. Information is a message with a higher level of meaning. It is raw data that a subject in turns transforms into knowledge through a cognitive or intellectual operation. This implies that in the information cycle, collected data must be optimized in order to immediately identify needs and address them as soon as possible. This will, in turn enhance interactions between a diversity of actors (decision-makers, analysts, consultants, technicians, etc.) within a group dynamic favoring knowledge complementarity, one which would be aimed at improving the understanding, situational analyses and information production necessary for action. Indeed, “operational knowledge production quality depends on the human element’s interpretation and analysis skills when it is located in a collective problem solving environment” [BUI 06, BUI 07]. Everyone produces data, sometimes consciously, sometimes unconsciously: humans, machines, connected objects and companies. The data we produce, as well as other data we accumulate, constitutes a constant source of knowledge. Data is, therefore, a form of wealth, and exploiting it results in an essential competitive advantage for an ever-tougher market. The need to exploit, analyze and visualize a vast amount of data confirms the well-known hierarchical model: “data, information and

Introduction

xxvii

knowledge”. This model is often cited in the literature concerning information and knowledge management. Several studies show that the first mention of the hierarchy of knowledge dates back to T.S. Elliot’s 1934 poem “The Rock”. The poem contains the following verses: – where is the wisdom we have lost in knowledge; – where is the knowledge we have lost in information. In a more recent context, several authors have cited “From Data to Wisdom” [ACK 89] as being the source of the knowledge hierarchy. Indeed, the hierarchic model highlights three terms: “data”, “information” and “knowledge”7. The relationship between the three terms can be represented in the following figure, where knowledge is found at the top to highlight the fact that large amounts of data are necessary in order to achieve knowledge.

Knowledge

Information Information

Knowledge

Data

Data

Figure I.1. Relationship between data, information and knowledge [MON 06]

7 This hierarchic model was taken up by Monino and Lucato [MON 06] in order to demonstrate the importance of information in the business intelligence process (see also [MON 13a]).

xxviii

Big Data, Open Data and Data Development

Wisdom accumulation (intelligence) is not truth. Wisdom emerges when the fundamental elements giving rise to a body of knowledge are understood. For Elliot, wisdom, is hence, the last phase of his poem. In our presentation of the concept of competitive intelligence, we have made decision-making the equivalent of wisdom (see Figure I.2).

AWARENESS

FINANCIAL INTELLIGENCE

OBSERVATION

DECISIONMAKING

DATA

INFORMATION

Define search criteria

Collect and process

Search for the data

Validate and spread information

KNOWLEDGE

Analyze the information and integrate it into the decision-making process

Orientate Share Change Improve

Figure I.2. The hierarchic model: data, information, 8 and knowledge [MON 06]

The boom in the number of available data sources, mostly coming from the Internet, coupled with the amount of data managed within these sources, has made it indispensable to develop systems capable of extracting knowledge from the data that would otherwise be hidden by its complexity. This complexity is mostly due to the diversity, dispersion and great volume of data.

8 Jean Louis Monino, is a professor and director of the TRIS laboratory at the University of Montpellier. Gilles Lucato is the technological council in charge of “digital data acquisition, processing and visualization” at Transfert LR, the Languedoc-Roussillon Regional Innovation Agency.

Introduction

xxxix

How wever, the voolume will coontinue to inccrease at an exponential e rate in yearss to come. Itt will, therefoore, become necessary foor companiess to addresss challenges to position thhemselves on n the markett, retain markket share annd maintain a competitivve advantage.. The greatesst challenge hhas to do with w the capaacity to comppare the largeest possible amount a of daata in orderr to extract thhe best decisiion-making elements. e

Data conccerning RV parking inn the Bouchesdu-Rhône department waas compared with tourist data in ordder to make RV V travelers’ journeys j easierr. In Renness, it was possiblle to developp an app makingg handicappped persons’ trips easieer. The same idea was taken t up by the city of Moontpellier. Website: http://www w.handimap.orgg/

Example I.4 4. An example from France’s s Bouches-du u-Rhône Administrattive Departme ent and from th he city of Monttpellier

Expploited data generates g billlions of dollars, accordiing to differeent reports. See, for example, e thee McKinsey y Institute’s report, whiich recentlyy showed thhat making “Open Dataa” available to the pubblic would allow the Unnited States to save 230 billion dollaars by 2020 by t develop innovative i services aimeed at reduciing allowinng startups to useless energy spennding.

xxxx

Big Data, Open O Data and Data D Development

Another exam mple that recentlyy made headlines: thhe arrival of thee American coompany “Netflix” in France. It sells “videeo on demand” in tthe form of film ms and TV series, competing w with classic TV operators (e.g. Orange) at a lower price. How ddoes it decide what content to provide, ggoing as far as to makke personalizedd offers? By making a progressivelyy enriched anaalysis of our purchasees and consumptionn habits that ultimateely defines our ttastes with a high ddegree of precision. Example I.5. Netflix

McKinsey & Company C is a consulting c firm advising top management m perrsonnel that was founnded 1926 in Chicago C by Jam mes Oscar McK Kinsey, a well-kknown professor at thhe University of o Chicago. McKinsey set up the t Global Insttitute a few blocks aw way from the White W House in Washington DC with the aaim of studying maccroeconomic teendencies. Som me of their reesearch is published, especially in thhe McKinsey Quarterly Q journaal. Box I.3. I McKinsey & Company

The challeenge with Big Data has to do withh studying thhe large voolume of available a daata and con nstructing models m capaable of

Introduction

xxxi

producing analyses compatible with the businesses’ requirements. This data can in turn be used as explaining variables in the “models” on which a variety of users rely to make predictions or, more precisely, to draw relations based on past events that may serve to make future projections. If data is collected and stored it is because it holds great commercial value for those who possess it. It allows to target services and products to a more and more precise set of customers as determined by the data. It is indeed at the heart of “intelligence” in the most general sense of the word. I.2. The rise of buzzwords related to “data” (Big, Open, Viz) In recent years, economic articles that do not make some mention or other of terms related to “data” (such as “Big Data”, “Open Data”, or “data mining”) when talking about companies have become few and far between. This tendency has led some scholars to believe that these concepts are emerging as new challenges that companies can profit from. We will chart the emergence of the data boom, especially of Big Data and Open Data, by studying search tendencies on Google (Google Trends).

Figure I.3. Web searches on “Big Data” and “Open Data” 2010–13 according to Google Trends. For a color version of the figure, see www.iste.co.uk/monino/data.zip

xxxii

Big Data, Open Data and Data Development

But what exactly is “data”? Data is a collection of facts, observations or even raw elements that are not organized or processed in any particular manner, and that contain no discernible meaning. Data is the basic unit recording a thing, an event, an activity or a transaction. A single piece of data is insignificant on its own. Data does not become significant until it is interpreted and thus comes to bear meaning. The way in which we “see” and analyze data has progressively changed since the dawn of the digital age. This change is due to the latest technology (Smart, 4G, Clouds, etc.), the advent of the Internet, and a variety of data processing and exploration software, since the potential to exploit and analyze this large amount of data has never been as promising and lucrative as it is today. In order to process more and more data, and to advance in the direction of real-time analysis, it is necessary to raise awareness about this new universe composed of data and whose processing is a genuine asset. In general terms, the technological evolution related to information processing capacities through its entire transformation chain drives us to explore the current interest in terms related to “data”9. One of the most important terms related to the data family is of course “Big Data”, a phenomenon that interests companies as much as scholars. It establishes constraints to which companies must adapt: a continuous flow of data, a much faster circulation of data, even more complex methods. Big Data represents a challenge for companies that want to develop strategies and decision-making processes through analysis and transformation of that data into strategic information. In

9 Buzzwords in the data family are those words currently “in style”, but they remain generally poorly understood, even if they may appear to make people who use them sound smarter. As such, “Big Data”, “Open Data”, “data visualization”, “data scientist”, “data mining” and other similar terms are new concepts that have replaced the word “data” as companies’ main focus. These fields help drive companies’ activities in “data mining”, and are radically changing paradigms that may lead companies to define product offerings according to available data.

Introduction

xxxiii

these new orders of magnitude, data gathering, storage, research, sharing, analysis and visualization must be redefined. The growing production of data, generated by the development of information and communication technologies (ICTs), requires increased data openness, as well as useful sharing that would transform it into a powerful force capable of changing the social and economic world. “Open Data”, another term assigned to the data craze, became popular despite the novelty of the practice, thanks to its capacity to generate both social and economic value. This phenomenon has attracted much attention in the last few years, and one of the Open Data initiative’s main objectives is to make public data accessible in order to promote innovation. Moreover, a number of professional contexts require reading and understanding very diverse sets of data: market studies, dashboards, financial statistics, projections, statistical tests, indicators, etc. To this end, a variety of techniques aimed at solving a variety of problems is used in the professional world, from client service management to maintenance and prevention, including fraud detection and even website optimization. These techniques are known today as “data mining”. The concept of data mining has gained traction as a marketing management tool since it is expected to bring to light information that can prove useful for decision-making in conditions of uncertainty. Data mining is often considered a “mix of statistics, artificial intelligence, and database exploration” [PRE 97]. Due to its growing use, this field is considered an ever more important domain (even in statistics), where important theoretical progress has taken place. However, in order to explore this data, human intelligence coupling mastery of statistical methods and analysis tools is becoming gradually more important in the business world: the term “data scientist” has become more and more common. In order to understand data and take the first steps towards more efficient processing, however, data scientists must embrace Big Data.

xxxiv

Big Data, Open Data and Data Development

The term “data science” was invented by William Cleveland in a programmatic article [CLE 14]. Data science constitutes a new discipline that combines elements of mathematics, statistics, computer science and data visualization. “Data visualization” refers to the art of representing data in a visual manner. This may take the form of graphs, diagrams, mappers, photos, etc. Presenting data visually makes it easier to understand. Beyond illustration, it is possible to go as far as presenting data in the form of animations or videos. These terms surrounding the data craze are ultimately not just buzzwords, they are also terms that confer more value to data. This is why we can now combine enormous volumes of data, which leads to deeper and more creative insight. “Big Data”, however, is not just big because of the volume of data it deals with. The word “big” also refers to the importance and impact involved in the processing of this data, since, with Open Data, the amount will progressively increase, which means that “Big Data” helps us “think big”. Therefore, the universe surrounding data is a movement in which France, with all its assets, must participate from now on. But, in order to meet the challenge, it is important for all actors to come together around this great movement and transform the risks associated with Big Data into opportunities. I.3. Developing a culture of openness and data sharing The growing production of data, generated by the development of information and communication technologies (ICTs) requires increased data openness as well as useful sharing that transforms it into a powerful force capable of changing the social and economic world. Big Data has become an important element in companies’ top management decision-making processes. They must learn to work together to innovate with their customers, taking into account their

Introduction

xxxv

behavior and remaining attentive to their needs, so as to create the products and services of tomorrow, which will allow companies to produce benefits to achieve their objectives. This forces us to rethink the value of data in a world that is advancing towards constant digital interoperability, sharing and collaboration. Open Data has spread throughout the world due to its capacity to generate both social and economic values. In this regard, several countries are actively contributing to the evolution of data by allowing access and reuse of government data. The term “Open Data”, which means that all data should be freely available, was first used at the end of the 2000s, originally in the US, and then in the UK, since the two countries were the first to commit themselves to the process. In 2007, the city of Washington DC was the first to make its data openly available on an online platform. In January 2009, President Obama signed a transparency initiative that led to the establishment of the Internet portal data.gov in 2009. It was, thus, at the beginning of 2010, that data marketplaces emerged in the US, such as Data-market, Factual, Infochimps, Kasabi, and Windows Azure Data Marketplace. The President of the United States went far in advocating Open Data. It is not just a question of making data available openly in the interest of transparency. According to him, “entrepreneurs can rely on open data to create jobs by solving problems faced by Americans that the government cannot solve”. Box I.4. Open Data and job creation

We can say that the principle of Open Data dates back to the Declaration of the Rights of Man and of the Citizen, which states: “society has the right to call for an account of the administration of every public agent”. This obviously requires complete transparency on behalf of the government. This imperative, whose realization has

xxxvi

Big Data, Open Data and Data Development

unfortunately always been imperfect, was formalized in French history through the Right of Access to Public Records Law, in 1794, through the creation of a large-scale statistics and information government body [INS 46] and through the establishment of a public body to diffuse the same right.

Map of gridded population estimates

INSEE, France’s National Institute for Statistics and Economic Studies, undertook a classification and sectorization of the French population. Two grids are made available, either with 1 km or 200-meter sectors. On the left, we see the map of France. It displays data corresponding to population estimates on a 1 km by 1 km grid. Website: http://www.insee.fr/

Example I.6. INSEE and sectorization

The French government’s Open Data initiative was materialized in the launching of the data.gouv.fr website in 201110. Furthermore, on 27 January 2011, the city of Paris launched the platform “Paris Data”.

10 The law of July 17 1978 proposed the idea of generalized open public data and granted access to public information, thus setting the foundation for the information society. The right to reuse public information was created by the 2003 PSI European directive, which was translated in French law in 2005, and whose revision was approved by the EU Council on April 2013.

Introduction

Central value

xxxvii

In the same regard, and based on INSEE and Autour.com’s data, the TRIS laboratory developed an algorithm referencing the region of l’Hérault, which is called “Snail Algorithm”. It makes it possible to know the location or situation of the item queried in reference to an address, which is represented with a reference point that constitutes the search item. Work realized by J.L. Monino for the startup Autour.com and supported by Alter-Incub in Montpellier. http://www.alterincub.coop/

Example I.7. A startup population tracking app

Open Data and data sharing are the best ways for both the government and companies to organize, communicate and bring about the world of collective intelligence11. Open Data’s culture is founded on the availability of data that can be communicated easily. This makes it possible to generate knowledge through transformation effects where data is supplied or made available for innovative applications.

11 Handbook on public open data and data sharing, September 2013.

xxxviii

Big Data, Open Data and Data Development

The city of Rennes was the first to genuinely work on the idea of making Open Data a vector for public service improvement. France’s Open Data website: https://www.data.gouv.fr/fr/ The Paris Data website: http://opendata.paris.fr/page/ home/

Example I.8. Open Data in the city of Rennes

It is also a means for constructing working relationships with actors who care about the public good in order to prolong public action by designing new services that favor all parties involved. The objective is to favor or make it easier to reuse and reinterpret data in the most automated way possible. This has great potential, in terms of activities that create added value through data processing. In France, INSEE supplies a massive amount of economic, demographic and sociological data (complimented by data from the OECD, Eurostat, the IMF and INED). This data is not quite raw, since it has been processed slightly. This data can, in turn, be used to explain variables in the “models” that a variety of users rely on to make predictions, draw relations or make comparisons. In order to be actually reusable, and in order for Open Data to give its full effect, it must be further processed. The entities that may

Introduction

xxxix

benefit from this data do not possess the statistical capacities to link some datasets with others and draw operative conclusions. Because of the costs involved, not all companies have access to the statistical means required to identify correlations or causal links within the data, or to “locate” a piece of data with respect to the rest through “data analysis”. The collection of these econometric techniques constitutes a specific kind of knowledge that governments and large public entities may possess, but that is unviable for most of the companies. And yet, without econometrics, there is little competitive intelligence to be had. “Processed” means that the data should be freely accessible to all (which is not always the case) so that it can become “information”. Today, in order to process data, companies must often hire specialized companies, consulting firms, to gather, process and draw operative conclusions from data. This call to subcontractors, inevitably, has a cost, and it does not always lead to satisfactory results. The subcontractor does not always treat the data with the same care that the company itself would apply if it were transforming the data into information on its own. Administrators sometimes resort to external companies, either because they lack the technical resources to process data internally, or because the specialized companies ultimately offer a lower price for processing than it would cost to do it internally. Data processing algorithms that users can access and that process data according to their requirements are available. Currently, data processing chains forms a sort of “ecosystem” around “data”: – producers and transmitters of data: entities that produce and, at most, make their data available openly by making it accessible to all; – data users who possess the necessary processing tools as well as the human elements required to deliver and comment information resulting from open data processing;

xl

Big Data, Open Data and Data Development

– lastly, the end users who benefit from this information, and thus, develop intelligence about their economic, political, social or even cultural “environment”. Big Data, in the present context, represents an all-new opportunity for creating “online” companies that do not sell data processing services, but which provide algorithms and rent them out for use by the final users, the third actor in the abovementioned ecosystem. There is an enormous potential for data-processing algorithm-development favoring all the users. When an internal statistic research service in charge of processing data is created, a license is purchased for a fixed or undetermined duration, and it becomes necessary to assign someone to the job of processing. If companies hire a consulting service that goes out to gather, process and draw information and knowledge from the data, they are forced to always purchase their information, even if in a limited degree. Ultimately, the experts will always choose data to be processed (or not), select available data or data necessary to choose in order to ideally arrive at operative conclusions. French startups like Qunb and Data Publica have adopted the same strategy as American startups like Infochimps and DataMarket, which rely on data drawn from Open Data for processing and sharing. For its part, Data Publica12 offers the possibility of preparing datasets according to clients’ requirements. Public data can be a genuine source of energy for startups in any domain. With these algorithm startups, as with many other sectors, there is a transition from purchasing software to purchasing a usage license for the software. This is comparable to the move towards car or bikesharing systems in transportation.

12 The website, Data Publica, was launched in March 2011, and was created by startups (Araok, Nexedi, and Talend). It is an almanac aiming to reference French public data.

Introduction

xli

In January 2014, Data Publica launched its product “C-Radar”, the first predictive selling service in France, catering to B2B companies. It allows organizations to find new prospects, partnerships and suppliers thanks to an up-to-date search engine of the most recent data, which uses unique machine learning algorithms to map and segment a company’s market and ecosystem to identify new business opportunities in targeted sectors. Data Publica’s website: http://www.datapublica.com/ C-RADAR’s website: http://www.c-radar.com/ Example I.9. Data Publica and C-RADAR

To give a more precise example in the field of economics, a startup could offer a sales forecast model. Imagine a company that sells a given product or a category of products, and that seeks to predict its future sales. The startup develops a series of explaining variables to define its stock (growth rate, revenue growth, price index, etc.) and it performs one or several linear regressions to relate the quantity of product sold by the company to those explaining variables. The final user only needs to introduce its own sales series and specify or let the algorithm specify the explaining sales variables. The user, thus, obtains a relationship between those explaining variables and its sales directly, which allows it to predict future performance with precision. There is neither the need to go out and look for external data, nor to update the “model” – already developed by the

xlii

Big Data, Open Data and Data Development

startup and made available to users willing to pay. There can be as many startups as there are models or groups of models. It is also possible to imagine, based on the coloration model [MON 10] that a client seeks to know how to “color” its PR communications. The startup can have a search engine of all texts that of interest to the firm, and establish a global coloration analysis system on all those texts, possibly based on words provided by the client itself. The coloration model makes it possible to relate exogenous data on an underlying asset’s color variation, and thus, inquire if the information’s color variation coincides with stock market fluctuations. The results show that this model predicts market developments in a statistically significant manner. This model examines if the coloration of exogenous information involves a variation in the direction of an identical action. The model relating stock market data and informative coloration is based on Markov chains. The model has been applied to the financial market field, although it is clear that it can be extended to other fields as well. Coloration of an exogenous event can also be interesting for politics, in order to examine the impact of a speech on a politician’s popularity. Box I.5. Monino-Boya’s coloration model

It is easy to see why there is that much potential for new automatic Big Data processing companies. We certainly find ourselves in front of one of the components of the industrial revolution inaugurated by the digital age.

1 The Big Data Revolution

The amount of data generated by people, Internet-connected devices and companies is growing at an exponential rate. Financial institutions, companies and health service providers generate large quantities of data through their interactions with suppliers, patients, customers and employees. Beyond those interactions, large volumes of data are created through Internet searches, social networks, GPS systems and stock market transactions. This widespread production of data has resulted in the “data revolution” or the Age of Big Data. The term “Big Data” is used to describe a universe of very large sets of data composed of a variety of elements. This gives way to a new generation of information technology designed to make available the increased processing speeds necessary to analyze and extract value from large sets of data, employing – of course – specialized materials and software. The phenomenon of Big Data not only refers to the explosion in the volume of data produced, which was made possible by the development of information storage and dissemination capacities on all sorts of platforms, but the term also refers to a second phenomenon, which involves newfound data processing capabilities. In general terms, the concept of Big Data describes the current state of affairs in the world, in which there is a constant question of how to manage lumps of data in a better way, and how to make sense of the massive volume of data produced daily.

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

2

Big Data, Open Data and Data Development

Data sources are multiplying: smartphones, tablets, social networks, web services and so on. Once these intelligent objects are connected to the Internet, they can feed data into enormous databases and communicate with other objects and humans [PRI 02]. This data must be processed and developed in order to become “intelligent” or “smart”. Intelligence, which can be brought out by using analysis techniques, can provide essential information that top management will require in order to determine strategies, boost operational performance and manage risks. To this end, “data scientists” must pool their strengths in order to face the challenges of analyzing and processing large pools of data, gaining clarity and precision. Data scientists must make data “speak” by using statistical techniques and specialized software designed to organize, synthesize and translate the information that companies need to facilitate their individual decision-making processes. 1.1. Understanding the Big Data universe The IT craze that has swept through our society has reached a new level of maturity. When we analyze this tendency, we cannot help being overwhelmed by the transformations that it has produced across all sectors. This massive wave developed very quickly and has resulted in new applications. Information and communication technologies (ICTs) and the advent of the Internet have triggered an explosion in the flow of information (Big Data). The world has become digital, and technological advances have multiplied points of access to data. But, what exactly is Big Data? The concept really took off with the publication of three important reports from the McKinsey Institute: – Clouds, Big Data, and Smart Assets: Ten Tech-Enabled Business Trends to Watch [BUG 10]; – Are You Ready for the Era of “Big Data”? [BRO 11]; – Big Data: The Next Frontier for Innovation, Competition and Productivity [MAN 11].

The Big Data Revolution

3

“Big Data” describes: “a series of data, types of data, and tools to respond quickly to the growing amount of data that companies process throughout the world1”. The amount of data gathered, stored and processed by a wide range of companies has increased exponentially. This has partially benefited from an explosion in the amount of data resulting from web transactions, social media and bots. The growth of available data in terms of quantity, diversity, access speed and value has been enormous, giving way to the “four Vs”: “Volume”, “Variety”, “Velocity” and “Value”2, that are used to define the term Big Data: – Volume: the advent of the Internet, with the wave of transformations in social media it has produced; data from device sensors; and an explosion of e-commerce all mean that industries are inundated with data that can be extremely valuable. All these new devices produce more and more data, and in turns, enrich the volume of existing data; – Variety: with the rise of Internet and Wi-Fi networks, smartphones, connected objects and social networks, more and more diverse data is produced. This data comes from different sources and varies in nature (SMSs, Tweets, social networks, messaging platforms, etc.); – Velocity: the speed at which data is produced, made available, and interpreted in real-time. The possibility of processing data in realtime represents a field of particular interest, since it allows companies to obtain results like personalized advertisements on websites, considering our purchase history, etc.; – Value: the objective of companies is to benefit from data, especially by making sense out of it. The challenges of Big Data are related to the volume of data, its variety, the speed at which it is processed, and its value. Some 1 http://www.ssc.upenn.edu/~fdiebold/papers/paper112/Diebold_Big_Data.pdf. 2 In this sense, Viktor Mayer-Schönberger has underscored the difficulty in quantifying the value of data in his book, Big data, A Revolution That Will Transform How We Live, Work and Think.

4

Big Data, Open Data and Data Development

scholars add another three “Vs”, namely3: “Variability”, “Veracity”, and “Visualization”. The first V refers to data whose meaning evolves constantly. The second qualifies the result of the data’s use, since even though there is a general consensus about the potential value of Big Data, data has almost no value at all if it is not accurate. This, particularly, is the case for programs that involve automatic decision-making, or for data feeding into unmonitored machine learning algorithms. The last V, which touches on one of the greatest challenges of Big Data, has to do with the way in which the results of data processing (information) are presented in order to ensure superior clarity. The expression “Big Data” represents a market in and of itself. Gilles Grapinet, deputy CEO of Atos notes that “with Big Data, organizations’ data has become a strategic asset. A giant source of unexpected resources has been discovered.” This enormous quantity of data is a valuable asset in our information society. Big Data, therefore, represents a large discipline that is not limited to the technological aspect of things. During recent years, the concept has sparked growing interest from actors in the information management systems sector. The concept of the “four Vs” or even that of the “seven Vs” opens up new avenues for consideration and research, but they do not provide a clear definition of the phenomenon. The sum of these “Vs” gives way to new perspectives for new product creation through improved risk management and enhanced client targeting. Actions aimed at anticipating and reducing subscription suspensions or at making customers more loyal can also be envisioned. The increase in the volume of data, processing speed and data diversity all present new challenges to companies and affect their decision-making processes. And yet, companies that produce, manage and analyze vast sets of data on a daily basis now commonly use terms such as terabyte, petabyte, exabyte and zettabyte.

3 http://dataconomy.com/seven-vs-big-data/.

The Big Data Revolution

Name

Symbol

1 byte

5

Value 8 bits

1

1 kilobyte

KB

103 bytes

1,000

1 megabyte

MB

106 bytes

1,000,000

1 gigabyte

GB

109 bytes

1,000,000,000

1 terabyte

TB

1012 bytes

1,000,000,000,000

1 petabyte

PB

1015 bytes

1,000,000,000,000,000

1 exabyte

EB

1018 bytes

1,000,000,000,000,000,000

1 zettabyte

ZB

1021 bytes

1,000,000,000,000,000,000,000

1 yottabyte

YB

1024 bytes

1,000,000,000,000,000,000,000,000

Table 1.1. Data units of measurement

In 1999, Wal-Mart (one of the US’s most important retail chains) had a database of 1,000 terabytes (that is, 1,000,000 gigabytes of data). By 2012, this quantity had grown to over 2.5 petabytes (2.5 million gigabytes) of data. Box 1.1. Wal-Mart’s database

The Big Data phenomenon has rendered classical data processing methods antiquated, and now stands as an opportunity in the business world, especially for companies that know how to use it. Amazon.com, a leader in product suggestions processed through Big Data, analyzes purchase histories, product listings, and commentaries posted on its site. Box 1.2. Amazon and Big Data

6

Big Data, Ope en Data and Da ata Developmen nt

There are several s methhods with wh hich a compaany can creatte value froom its data assets. Dataa can be useed to improvve understannding of cuustomers’ needs and adappt products accordingly. a Companies can use daata to monittor and conttrol performance of keyy functions oon their website, identtify factors that contrib bute to the gaps observved and diiscover neceessary correective meassures, or find fi new w ways of opptimizing thee existing maanagement sy ystems. Some comppanies combbine data to predict p custoomers’ behavvior and thhus take the necessary n meeasures. Several other usses allow com mpanies too better naviggate their envvironment. To show how to use a large volume of o data for ppractical purposes, we will taake the example off Wal-Mart, whhich was able to identify a strong correlation between purchhases of baby diappers and beeer by analyzing sales s receipts. T This realtime correllation system ooperating on a constaant flow of data allowed Wal-Mart to t increase its sales by simply plaacing diapers aand beer closer togetther on their sheelves. Ex xample 1.1. A sales receipts s analysis by Wal-Mart

Anoth her illustrative example is Am mazon’s autom matic book suuggestion servvice for custom mers, which iss based on thee books that th he clients havee already purchhased or consu ulted. Amazon develops a proofile for each of its customerrs based on his or her previo ous searches annd purchases, aand then compares it to thhe content of other availaable books, whiich allows the ccompany to su uggest books according to clients’ tastes. Example 1.2. Boo ok suggestions for Amazon customers

The Big Da ata Revolution

7

he company Nike N provides iits As anothher example, th clients with w a complete “ecosystem” focused fo on healtth and welll-being. The data sources used (see, fo for example,, Nike+ and Fu uelBand) providde a personalizeed service for f clients and d allow them to t monitor theeir physical activity. In exchange, e the company gainns access too real-time intim mate data on itts clients’ habiits and usess, which it can n, in turns, usee to improve iits product offerings. o Exam mple 1.3. An ecosystem e pro ovided by Nike e

Big Data has, thherefore, traansformed co ompanies in all sectors, as mic well ass their operattions and waays of acting.. We can also add dynam real-tim me analyses,, thanks to the speed at which teechniques aand softwarre make it poossible to obbtain results. Analyses brring to light tthe changees in client behavior b andd reveal new w needs. They also makee it possiblle to predictt needs that do not even n exist yet, which enables strategiic decision-m making. Big Data allowss companiess to measuree different asspects of daaily life andd to find corrrelations betw ween these different d meaasures, all w with the aim m of finding relations thhat companiees themselvees might nevver have im magined. It opens up thhe possibilitty of examining a markket compossed of millioons of clientss and to see them not as a vague maass, but rathher as indivviduals with specific tasstes and valuues. It enables compannies to have a statistical base b for iden ntifying tenddencies throuugh data annalysis tools. Thee rise of Big Data D reflectss the growing g awareness of the “poweer” behind data, and of o the need to t enhance gathering, g exxploitation aand mpanies. As it enables more efficieent sharingg processes within com decision-making procedures, p g gaining acceess to a larrge volume of informaation and to t the toolss necessary to process it can alloow compannies to attaiin a better strategic s possition. Data thus becom mes compannies’ new strrategic asset, no matter th heir sector.

8

Big Data, Open Data and Data Development

1.2. What changes have occurred in data analysis? Companies have always needed to analyze data in order to have a precise understanding of their situation and to predict their future business moves. Data analysis, when it is not preceded by the word “Big”, refers to the development and sharing of useful and effective models. For the most part, it uses a variety of methods from different research fields, like statistics, data mining, visual analysis, etc. It caters to a wide range of applications, including data summarization, classification, prediction, correlation, etc. In the 1970s and 1980s, computers could process information, but they were too large and too costly. Only large firms could hope to analyze data with them. Edgar F. Codd and Hubert Tardieu were the first to work on data organization by designing database management systems (DBMSs), in particular relational databases. Data processing and analysis, in the present day, are brought together under the notion of “Business Intelligence”, due especially to computers’ increased processing capabilities. A fundamental requirement for successful data analysis is to have access to semantically rich data that links together pertinent information elements for objective analysis. However, the situation has changed with Big Data because data now comes from several sources of very different kinds and in different forms (structured, unstructured). This leads us to say that new data processing tools are now necessary, as are methods capable of combining thousands of datasets. In the Big Data universe, companies seek to unlock the potential of data in order to generate value. They are also impatient to find new ways to process that data and make more intelligent decisions, which will result in better client service, improved process efficiency and better strategic results. In the literature, the concept of Big Data is defined in terms of the theory of the “four Vs” or of the “seven Vs”. The exponential speed at

The Big Data Revolution

9

which data is generated, as well as the multiplicity of sources that generate it in different formats (digital, text, images, etc.), are characteristic of this phenomenon: “Big Data refers to volume, variety, and velocity of data – structured or unstructured – that is transmitted across networks in transformation processes and across storage devices until it becomes knowledge that is useful for companies”, (Gartner Research Firm).4 The following image shows one perspective on the massiveness of data and on its growing evolution through different interconnected technologies. This volume of data is available today due to storage capacities that have increased while their cost has correspondingly diminished.

Figure 1.1. Diversity of data sources

This large collection of data is often created in real-time and its quick processing provides knowledge to managers that was previously inaccessible. At the same time, it allows them to optimize their decision-making processes. Data is, therefore, transformed into a plan of action to be put into place, into decisions to be taken and into new markets to explore. 4 American consultant and IT research specialist.

10

Big Data, Open Data and Data Development

There are essentially three types of challenges surrounding Big Data: – massive data storage management, in the order of hundreds of terabytes or of petabytes, which go beyond the current limits of classic relational databases in terms of data storage and management; – unstructured data management (which often constitutes the largest portion of data in Big Data scenarios). In other words, how to organize text, videos, images, etc.; – analysis of this massive data, both for reporting and for advanced predictive modeling, but also for its deployment. In current usage, the term “Big Data” does not refer exclusively to vast sets of data. It also involves data analysis and value extraction operating on large volumes of data. The expression “Big Data” thus refers to the technologies, processes and techniques that enable organizations to create, manipulate and manage data on a large scale [HOP 11], as well as to extract new knowledge in order to create new economic value. The large volume of data collected, stored and disseminated through different processing technologies is currently transforming priorities and developing new analysis tools, which are in line with changes in companies’ operations and which will transform the business landscape. At the same time, new analytic techniques make it possible to examine the datasets. Processing them will play a crucial role, and will allow companies to gain a competitive advantage. The process has to do with gathering data, cleaning it up, and organizing it in different databases. Next, the data is compared in order to be organized and uploaded to software capable of analyzing it (to find correlations within it). The data from the different sources is then combined to be transformed into a technical diagram. In other words, data warehouses, created for the analysis, are examined in order to present the results in different forms (visualization).

The Big Data Revolution

11

The present day software tools make it possible to process and assimilate this massive volume of data quite quickly. Understanding the technological dimension of things is nevertheless fundamental because it makes it possible to understand its limits and potentialities, as well as to identify the most relevant actions to take. With exponential increase in the volume of data, companies attempt to use available analysis tools to find out how to extract value from their gathered data. A study carried out by [MAC 12] showed that companies that adopted advanced data analysis tools attain more productivity and better profit margins than their competitors. In fact, technical competence in data processing is now a genuine strategic asset for companies’ competitive differentiation [BUG 11]. Thanks to new Big Data methods and tools, it has become possible to work on large volumes of data. The result is an advantage stemming from the possibility of bringing to light correlations in new data. Interpreting this large volume of data is the greatest challenge facing Big Data, since information resulting from it can be the basis for new knowledge that brings about development opportunities. Technology now offers a perspective on data as structured and therefore, static. Technological limits, in terms of performance and storage, reduce the scope of possible analysis to sets of explicit data. Most solutions provide the couple: “storage and processing”. It is worth noting that growth in the volume of data has been accompanied by a reduction in the price of storage. Currently, one of the innovations that make it possible to share and store large volumes of data is “Cloud Computing”. The “Cloud” allows access to shared computing resources through an on-demand telecommunication network or self-service modules. The cloud transforms storage infrastructure and computing power into services through the intermediary of companies that possess servers and rent out their capacities. This approach makes it possible to share costs and to provide greater data storage and processing flexibility for users.

12 2

Big Data, Open Data and Data D Developme ent

Storage in 1956: IBM 3055 RAMAC5M MB of disk drivve storage; 500 disks; 24 inchees; access timee: 10 characterss per second.

Storage in 2013: 1 petabytte; 22 million IOP PS; Latency perriod in microseconnds (ȝ= 10-6 seeconds)

Exa ample 1.4. Th he development of storage capacities c

1..3. From Biig Data to Smart S Data a: making data d wareh houses in ntelligent a held strategic vaalue, but the scale of tthe data Data has always avvailable and processing capacities today have resulted in a new caategory of assets. a We find f ourselves at the beeginning off a long joourney wheree, with the right r princip ples and guiidelines, we will be abble to gatherr, measure and a analyze more and more m data tto make beetter decisionns, individuaally or collecttively. The massivve flow of daata, or “Big Data”, D whichh is generatedd by the Innternet, sociaal media, cloud c compu uting, etc. is i developinng very quuickly. This prompts coompanies to rethink theiir strategies and go beeyond the diffficulties invvolved in processing largee volumes off data. It will soon beecome possible to orgaanize and trransform daata into

The Big Data Revolution

13

information, which will, in turn, be transformed into knowledge useful for cognitive or intellectual operations. However, attaining the complete potential of data depends on the way in which it is presented. It must be used and reused in different ways, without its value being diminished. This requires making data available in the right form and at the right time to any party interested in exploiting and adding value to it. “Data Is the New Oil” [ROT 12]. It is the indispensable raw material of one of the new century’s most important activities: data intelligence. However, it is important to be prudent in our predictions because a lot of data is not yet “the right data”. There is, therefore, an underlying difficulty behind Big Data, since more data is not necessarily better data. It is possible to obtain better results by making better use of available data. Twitter has become a popular source for Big Data exploitation, but working with Twitter data represents a huge methodological challenge that is seldom taken up. Box 1.3. A methodological challenge: Twitter

When researchers encounter a set of data, they need to understand not only the limits of the available set of data, but also the limits of the questions that it can respond to, as well as the range of possible appropriate interpretations. However, it is imperative for those combinations to be made rigorously and with methodological transparency. This leads us to say that it is not so much a question of size, but rather of what can be done with a given set of data. After all, the objective of gathering and analyzing that data is not simply to attain knowledge, but also to act. In the field of marketing, for example, Big Data processing must employ the necessary tools to determine the appropriate type and level of action it recommended from attracting and retaining each client at the lowest possible cost and for negotiating the continuous relationship with an optimal profit level.

14

Big Data, Open Data and Data Development

The McKinsey Global Institute’s July 2013 report estimates that Big Data applications could result in a $30 million increase in retail sales due to productivity gains, increased transparency, and more sophisticated targeted marketing, advertisements and sales. Box 1.4. McKinsey Global Institute’s 2013 report

But how can we obtain or make progress with such benefits by using Big data? How can companies bring together and combine data from disparate sources to achieve projected gains? What role can the data analysis play in what amounts to an IT challenge? What changes are required in order for data analysis to become a more practical discipline? These questions refer to some of Big Data’s greatest challenges, and they represent the difficulties that make it a “Big Challenge”. The greatest objective for Big Data involves intelligent database management aimed at identifying and extracting pertinent information allowing companies or users to establish strategies that actually address identified needs. Intelligent data makes it possible to go from raw (structured or unstructured) data coming from internal or external sources to strategic information. The ultimate goal is not only to collect, combine or process all data, but also to increase its value and efficiency. This means that we must evolve from “Big” data to “Smart” data, since the effectiveness of companies’ strategies now depends on the quality of data5. Data quality refers to its adequacy for its envisioned use in operations, processes, decision-making and planning. Data quality, in this regard, also has an impact on product lifecycle analysis. Data quality is highly important because it represents a key source of value for companies.

5 In its flexibility approaches, companies seek operational efficiency. Exploiting highquality data makes it possible to optimize participation and interaction of different staff members in the decision-making process. In its analysis and presentation of data, companies must ensure quality because its impact is important. The general idea is to manage data as a company asset, just like products, employees and clients.

The Big Data Revolution

15

Data quality is important for monitoring and evaluating progress towards objectives. It is all the more important when it relates to reliable and accurate information gathered through company data management systems. Having access to accurate information makes it possible to: – demonstrate responsibility and good governance; – provide decision-makers with the information necessary to plan, allocate resources and elaborate strategies; – monitor progress towards the attainment of established goals and objectives. Indeed, companies must not rely on the size of their data – it is not useful unless it is applied in an intelligent manner. Therefore, the volume of data is of little importance, since internal data must be combined with external data in order for a company to obtain the most out of its data. What is truly necessary are excellent analytic skills, a capacity to understand and manipulate large sets of data, and the capacity to interpret and apply the results. The challenge is to consider the data’s use, rather than its quantity. This could become the most profitable way of extracting the value of data from the massive sources available. The evolution from “Big Data” to “Smart Data” represents a new awareness of the importance of data processing. It is at this level that the figure of the “data scientist” appears. They are well-trained in computer science, mathematics and statistics, which they combine with good knowledge of the business world. They must be able to analyze a phenomenon from all possible angles in order to draw profit from the company’s data assets. In this way, the term “variety” involves several different issues. First of all, data – especially in an industrial environment – can be presented in several different ways, such as texts, functions, curves, images and graphs, or a combination of these elements. On the other hand, this data shows great variety, which often reflects the complexity of the studied phenomenon. It is, therefore, important to

16

Big Data, Open Data and Data Development

be open about the structure of the observations’ content in order to draw the right conclusions from it. Processing large volumes of data by enlisting people from the “new” profession of data scientist is a major focus for companies that have placed themselves at the center of the flood of data requiring specialized processing. The development of information technology and computation tools makes storage of large databases possible, as well as processing and analysis of very large sets of data. More recently, improvements in software and their interfaces, both for statisticians and non-specialized users, have made it much simpler to apply these methods. This evolution, as well as the popularization of new algorithmic techniques (neural networks) and graphing tools, have led to the development and commercialization of software that brings together a subset of statistical and algorithmic methods, and which is known as “data mining”. In this regard, data mining refers to the search for pertinent information that may be helpful for decision-making and planning. It employs statistical machine learning techniques that can handle the specificity of large to very large volumes of data. However, in the Big Data universe, the main objective is still related to a much more important “V”, namely value-extraction. “The consensus today is to place the data scientist at the intersection of three fields of expertise: computer science, statistics, and mathematics, “Business Knowledge” [ABI 13]. In relation to new data structures, the statistician-turned-data scientist revisits basic notions to focus on tools and methods that can lead to useful applications compatible with new information systems. 1.4. High-quality information extraction and the emergence of a new profession: data scientists The massive amount of data currently produced in real-time requires analysis, processing and exploration. In order to face the Big Data challenge, companies must have the capacity to make their

The Big Data Revolution

17

constant flow of data “speak”. This has resulted in the emergence of employment prospects in new careers that attract increasingly more attention. We are of course talking about “data scientists” and “data analysts”. Data scientists examine large volumes of data from which they, in turn, identify the main tendencies in order to help companies make better decisions. Data scientists analyze data from different sources and examine information from all possible angles; it is therefore useful to understand the importance of their contributions to companies. For Simon Rogers6, “a data expert is above all capable of bringing data and analysis together, and to make his or her work accessible. Theirs is a translation job: by translating data into a form that can be understood by more people, data experts improve the world a little bit”. Moreover, John Foreman, as chief data scientist at MailChimp, confirms, “If by data scientist you mean someone who can create a data summary or aggregate, or model a task specifically assigned in advance, it’s not surprising that the job can be paid at $30 an hour”. On that same note, DJ Patil, data expert for LinkedIn, the professional social network, explains that “the role of data scientists requires striking a balance between technical data skills and a capacity to tell that data’s story”. This is a perspective shared by Hilary Mason, science director at Bitly, who describes the ideal candidate for this atypical job: “A data scientist is a rare hybrid between a developer, a statistician, and a careful analyst of data and human behavior”. Data scientists therefore have a digital-age job that is as related to finance as it is to banking, insurance, marketing and human resources. The job of data scientist is a profession born out of data science. It is a new discipline that brings together elements from different fields including mathematics, statistics, computer science and data

6 The Twitter data journalist.

18

Big Data, Open Data and Data Development

visualization and modeling. In fact, data science extracts knowledge both from companies’ internal and external data7. Since 2010, demand for this new career profile has increased dramatically, as is shown in Figure 1.2.

Figure 1.2. The importance of data scientists. Source: http://www.indeed.com/jobtrends/Data-scientist.html

The increase in data scientist demand is fed by the success of companies like Google, Facebook, LinkedIn and Amazon, which have invested in data science only in order to use their databases in a creative manner. Once the data is organized, a company can focus on understanding its meaning and implications instead of wasting time managing it. The need to analyze and use enormous amounts of data more efficiently drives companies towards data science in the hope of unlocking the power of Big Data. A data scientist must have a general grasp of business and be capable of analyzing data in order to extract knowledge by using computer science applications.

7 http://www.jobillico.com/blog/metier-avenir-scientifique-des-donnees/.

The Big Data Revolution

19

Data scientists are not only highly-trained computer scientists, they are also innovative thinkers capable of gleaning new perspectives on general tendencies in the data available. A data analyst analyzes data from a variety of sources and examines its different tabs in order to attain a general understanding of the phenomenon it describes and enable a company to develop competitive improvements. A data scientist’s interpretations enable top management personnel to take advantage of relevant information and thereby obtain excellent results. Google’s main economist, Hal Varian, confirms: “The most popular job in the next ten years will be that of statistician: being able to take data, understand it, process it, extract information from it, and visualize and communicate it”. A study by the McKinsey Global Institute estimates that by 2018, there will be a deficit of 140,000 to 190,000 people with analytic skills in the US, as well as 1.5 million managers capable of using Big Data analytics in order to make enhanced decisions. The recruitment firm Robert Half published a list of 6 golden jobs, including data scientist, for 2014 and 2015. The six “golden jobs” according to Robert Half 2014-2015: – Financial Comptroller (Corporate Finance) – General Accountant (Accountancy) – Compliance Officer (Banking) – Data Scientist (Insurance) – Compensation and Benefits Manager (Human Resources) – Technical Director (IT) Box 1.5. The six jobs of the future according to R. Half

Data science occupies the central position in companies’ priorities. As an example, Yahoo devoted a significant amount of resources to data science. After witnessing Google’s use of “MapReduce” to analyze enormous quantities of data, companies realized they had similar needs. The result was “Hadoop”, which today is one of the most important tools in a data scientist’s toolbox.

20 0

Big Data, Open Data and Data D Developme ent

“Hadoo op” is a specifiic kind of openn source databasse architecturee, which maakes it possiblee to process alll kinds of data in mass (includiing unstructureed data) based on a non-relational mode (data ( is not prrocessed accordiing to its internnal relations). Hadoop constitu utes a sort off library from m which multi-th hread computaations can be carried out. “MapR Reduce” is a massive multti-thread program mming model designed to process very larrge volumes off data. MapRedduce is a Googlee Corporporatioon product. Prrograms that ad dopt this moddel are autom matically threadeed together annd run on coomputer clusterss. E Example 1.5. Two T examples s of open sourrce data

The best data d scientissts will also understand the latest bbusiness treends and bee able to compare c a company’s c d data to induustry or coompetitor inddicators in order o to correectly diagnoose the situattion and obbtain useful informationn. An “analyst” or dataa scientist eemploys diifferent availlable methodds to interpreet data in rellation to a ddecisionm making contexxt. Their rolee requires a variety v of skiills, includingg: – technicall and statisticcs training; – general teechnology annd IT-savvin ness; – familiaritty with the field in wh hich the anaalyzed data will be appplied. Data has always playyed an impo ortant role and contribbuted to coompanies’ operational o and strateg gic planningg. Howeverr, it is neecessary forr each com mpany to deefine the seettings that help it unnderstand thee enormous heaps of avaailable data the t most. Buut in the Biig Data uniiverse, it is also imporrtant to openn up new rresearch annd analysis perspectivess, which req quires highlyy skilled peersonnel

The Big Data Revolution

21

capable of exploiting and processing sets of data and adding value to them. 1.5. Conclusion Big Data is gradually becoming an inevitable concept that revolutionizes the way in which many companies do business, since the scope of their business goes beyond the boundaries of their specific sectors and belongs to a globalized world. Smartphones, tablets, clouds, GPS, the Web, etc. – these are the new tools of a trade whose goal is to refine a certain raw material: data. Data, the new strategic asset, will without a doubt, influence global strategies throughout. As a consequence, the data processed by technological platforms has become fundamental in overhauling decision-making processes. It becomes necessary to clean up, analyze and compare (structured or unstructured) data produced by companies, governments and social networks to develop new uses. Big Data is the general term used to describe the exponential increase in the volume of data, which has been accompanied, of course, by growth in the capacity to transfer, store and analyze that data. However, storing large amounts of data is one thing; processing it is another. Big Data does not amount to processing more data; it is rather a question of developing it. Analytic capacities capable of developing new applications and uses of the massive amount of data available are crucial. This is the work of data scientists who possess both the technical skills required for data analysis and the capacity to understand the strategic stakes involved in the analysis. But data quality is of utmost importance: well-prepared, wellclassified and integrated data that allows companies to benefit completely from its analysis. Without this preparatory phase, analytic processing will not produce performance data, which companies need in order to become more competitive. The challenges of Big Data are therefore related to the volume of data, its variety, the velocity in which it is processed and its value.

2 Open Data: A New Challenge

The world of data is becoming more competitive every day, as reflected in terms of volume, variety and value. This is why we now speak about Big Data. Data is a key asset for value creation, as well as an element that favors and promotes innovation, growth and development. With the digital revolution, data has taken on a central role in the economy. However, attaining the full potential of data depends on the way in which it is presented. It must be used and reused in different ways without diminishing its value. This means making data available in the right form and at the right time to any party seeking to exploit it and add value to it. Open Data adds a new dimension to the analysis of data warehouse, and gives way to new forms of innovation. Sharing and opening up data means making essential data available online so that it can enhance many decision-makers’ analysis; it also means making it possible for people to save time, or for them to make more informed decisions in all sorts of sectors. It therefore means creating large sets of reference data shared by all actors and that encourage the development of several high added value services. 2.1. Why Open Data? Open Data is private or public digital data. It is produced by collective bodies or (possibly outsourced) public services. It is disseminated in a manner structured according to a given method and

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

24

Big Data, Open Data and Data Development

with an open license that guarantees free access to it, as well as the possibility for anyone to reuse it without technical, legal or financial restrictions. Open Data is comprised of a number of sources and types of data: – public data or information coming from the public sector. This includes all data collected by public organisms at all levels; – data from scientific research, in particular, from publicly funded research; – private sector data, which can be made public with the right incentives and privacy protections. Open Data therefore means making this data available for access, exploitation and reuse by any interested actor (companies, scientists, etc.). The philosophy behind this movement is, at its foundations, truly centered on the citizen. In this sense, free access to data contributes to an enhancement of democratic institutions from a citizen’s point of view. This event must help to enrich democratic debate, stimulate public life, and contribute to renewing public services. Open Data is a political process whose message is built around transparency in innovation and in the development of public action. It is worth noting that Open Data has been used since the 1970s, mostly through statistical data processing and modeling that made data freely available and that made it possible to communicate and transfer it [BUC 98]. It is possible to summarize the history of Open Data development in the following timeline. After several years, France’s Open Data and data sharing strategy was materialized in the online platform https://www.data.gouv.fr in 2011. On 27 January 2011, Paris became one of the first cities to evaluate the question of Open Data by launching “Paris Data”, an online platform making it possible to disseminate public data and maps. Rennes has also played an important role by being the first to genuinely work in favor of the idea of making Open Data a vector for public service improvement: by developing public data, uses and

Open O Data: A Ne ew Challenge

25

services in order to be able to impro ove civil soociety throuugh transpaarency in pubblic data. 1966

2002 2003

Adoptiion Adoption de laofloifree sur le Informatio Access law libre accèsonà l’information Achievement putting Instaurat ion de la of mise en ligne legal information online juridiques des infeformations

Europe: Europe : Government Réutilisation des d data reuse données publiques (November) (Novembre)

UK: ity islargely largely UK:data dataaccessibil accessibi ility Royaume-Uni : L’acccessibilité des came into into fo orce orce came

20005

2007 2008 2009 2010 2011

Create an Créer un niveau opening level d’ouverture du of government gouvernement (election / (Election/Obama) Obama)

données est largement enntrée en vigueur

Data.gov du project Lancement projet launch (March) Data Gov(Mars)

Legislation allow wing data et la Législation permettant l’a accessibilité accessibiliity réutilisation de l’innformation

2013

Ettalab talab:: Créer create feeding Et et and alimenter les G open data dGouvernment données publiques ouvertes D Data.gov.fr (December) Data.gov.fr (Décembre)

Canada Canada :prProjet Project:: Data.gc data.gcc.ca c.ca Open da Ouverturreatades (March h)

données (M Mars)

Kenya: Kenya : Openn Pen data Data Initiativee initiative (Juillet)

Open n data: Rennes Opeen Data Rennes Par Porris rtaidata Parisportal data

Open government t law La loi Open Governm ment

Frrance F France U EtaUSA ats-Unis O Others Auutres

Op pening of government Ouverture du Directivedirec duction witth the position of three pillars: gouv vernement avec la position dees trois tra ansparency, participation and a pilierrs : La transparence, la particiipation collaboration

UK: datta.gov.uk Royaume-U ni : Projet :project data.gov.uk Acts anddu sttatutory site Ouverture Site : Acts andopening statutory

et la collaboration

Figure 2.1.. Open Data: history h

Figure 2.2. Open Da ata platform grrowth in Francce. Sourrce: [COU 14]

26

Big Data, Open Data and Data Development

Open Data also means the public sector gives up its role as the gatekeeper of data and replaces it instead with the role of a data and information provider, which leads to a realignment of power dynamics among the private and public sectors. This opening will make it possible to establish relationships between cities, companies and citizens around data by encouraging transparency, the development of new applications, and individual participation in data enrichment. The definition of Open Data involves the following, more specific, elements: – availability and access: data must be available as a whole, and it must be accessible in a way that is comfortable and modifiable; – reuse and redistribution: data must be made available in ways that make its reuse and redistribution possible, including the possibility of combining it with other sets of data; – universal participation: everyone must be able to use, reuse, and disseminate data. Data compared, analyzed and understood also goes through an incremental cycle that stretches from creation to use [MER 07]. This means that the value of data is not inherent to it, but rather that it is a product of its aggregation, cross-referencing, analysis, and reuse. The Internet and smart objects have therefore constituted a data “universe” that plays an important role in the value creation process. The importance of this role has developed around the fluidity of online exchanges. Data often has more value as an exchange and influence mechanism than it does isolated in company data storage1.

1 Ed Parsons – Geospatial Technologist for Europe, Middle East and Africa at Google, at the seminar “Innovation and development in the geographic information sector” organized on October 5, 2009 at Grande Arche de la Défense by the Conseil National de l’Information Géographique (CNIG, National Council for Geographic Information), AFIGÉO, and Institut des Sciences et des Techniques de l’Équipement pour le Développement (ISTED, Institute for Equipment Development Science and Technique).

Open Data: A New Challenge

27

The emergence of (public or private) open data reuse, which has been encouraged by new applications and uses, has therefore revealed a value chain based on this data. As the raw data is made available for re-exploitation, it becomes possible to create new services. Companies like Amazon and Netflix have, as previously mentioned, taken advantage of the great number of users on their sites by developing consumer preference models, which in turns, allow them to make highly personalized purchase suggestions based on their clients’ tastes. Open Data coming from both, public and private sources, implies greater and better quality access to data. Quality criteria can be thought in terms of comprehensiveness, coherence, and precision. Additionally, we may consider temporal and territorial variables, as well as that of interoperability through open formats. These principles are based on important values: accessibility, autonomy, sharing and freedom [MER 14]. Tim Berners-Lee observes on the subject of Open Data that “If we share data online – public data, scientific data, citizens’ data, whatever – then other people will be able to develop marvelous creations from that data that we could never even have imagined”. In 2010, the same author provided a set of criteria to grade data quality on a scale from zero to five stars. We can say from this table that Open Data must have three characteristics: – technical: raw data must be exploitable in an automatic manner, and made available in open-source formats as much as possible; – legal: licenses must clarify the rights and obligations of the owners and of people who reuse data. They must be as open as possible; – economic: few or no royalties (since they may constitute obstacles for reuse), marginal cost pricing.

28

Big Data, Open Data and Data Development

Stage

Description

‫ڎ‬

Making data available online in any format without licensing restrictions.

Users can see, print, and store the data, as well as manually select it on a system.

Publishing the data on a structured format (e.g. an excel file instead of a scanned format)

Data can be processed automatically, visualized, and transformed into other formats.

Using an open source format (e.g. CSV instead of Excel).

Data can be manipulated independently of its format and of any given software.

Using URIs to identify elements in order to click on them.

Data can be linked, marked, and reused.

Linking data to other’s data in order to contextualize it.

Data patterns can be automatically identified and it is possible to dynamically uncover complementary data related to the original data.

‫ڎڎ‬

‫ڎڎڎ‬

‫ڎڎڎڎ‬

‫ڎڎڎڎ‬ ‫ڎ‬

Benefits

Table 2.1. Open data in five stages. Source: http://lab.linkeddata.deri.ie/2010/star-scheme-by-example

Data increases gradually with one of the elements structuring the digital world in the present hour. In order for data to generate knowledge, it must be available for all to use online. It must be available to all those users who can combine and mix it in order to create value. In order to benefit fully from Open Data, it is necessary to place data within a context that creates new knowledge and offers powerful services and applications. 2.2. A universe of open and reusable data Today within the context of public sector transparency and modernization efforts, Open Data, alongside Big Data, are quickly

Open Data: A New Challenge

29

evolving. The more open data becomes, the more it can be used, reused, reoriented, and combined with other data in the interest of all. Insofar as our economy and our society are structured around knowledge, data is a key asset for value creation, as well as an element that favors and promotes innovation, growth and development. Transparency, participation and collaboration are the main challenges for the integration of different economic actors within the paradigm of Open Data. The Open Data reuse process involves two types of actors: producers and reusers: – Public actors: regional and local governments, as well as administrative public bodies, produce and receive a considerable amount of data. – Companies: because of their nature, companies use data in a variety of internal processes aimed at achieving their goals and strategies. Companies, thereby, produce data (as per reports) that they can relate to other data coming from their external environment (competitors, clients, providers, web, etc.). – Scientific research: scientists are by nature data collectors and producers. The scientific world is increasingly governed by the imperative to publish: scientists’ recognition is tied to the results they publish and make available to be evaluated by the scientific community. – Individuals: each individual can collect data and enrich it with smart tools (smartphones, tablets, GPS, etc.). Data reuse is, of course, geared for the most established economic actors, especially large companies in the IT sector, which can use public data to improve their production processes. Since it contains an important innovative potential, public data is an especially important ingredient in startup development for young innovative companies in the digital economy.

30 0

Big Data, Open Data and Data D Developme ent

Data journalissm is an exampple of the potenttial importance off public data reuuse in democraatic citizenship. n has played an important role in the The Guardian UK, and The New York Tim mes decided in 22009 to base of people, institutions, annd make its datab events dating back to 1913 available a in the form of n France, the innformation site O OWNI Open Data. In presents itselff as a “digital thhink tank” with the goal of becom ming an “innovaative journalism m lab”. Exam mple 2.1. Data a journalism

If companiies whose gooal it is to ex xploit data made availablee by the goovernment begin b to emerge, e com mpanies, sccientists andd even inndividuals miight follow suit. s At the time t being, few f companiies have thhemselves em mbraced the Open Data movement m by making thheir own daata freely avvailable. Nevvertheless, th hey producee data in theeir daily acctivities, and they often consume c exteernal data which w allows them to annalyze the economy or interact with h their partnners. For thhem, the quuestion of Open O Data depends moree on commeercial or innnovation oppportunities than on trannsparency co oncerns, evenn though thaat would also allow them m to improve their imagee. However, Open Dataa goes beyo ond the scoope of pubblic and sccientific dataa. A recent study by the t McKinsey Global IInstitute suuggested thatt opening upp corporate an nd governmeent data wouuld have thhe potential to t unlock appproximately $3.2 trillionn in the valuue added peer year in seeven differennt sectors off the world economy. W With the hiigh-speed deevelopment of o mobile Intternet, technology now m makes it poossible to proocess large am mounts of daata. Open Datta can geneerate severaal advantagges for com mpanies, inncluding: – acceleratiing product developmen nt and stimullating innovaation by crreating a dynnamic ecosysstem of partn nerships while improvinng client saatisfaction; m trend ds. – better undderstanding market

Open Data: A New Challenge

31

For example, in the health sector, France’s National Health Insurance Fund (CNAM) possesses a very large stock of data consisting of millions of its patients’ health forms, the information of which is stored by CNAM. This data is not yet available. CNAM has resisted requests for it to open up its data by appealing to medical privacy. But, in fact, medical privacy can easily be ensured by encoding patient identities, which is a technology that already exists. Doctors’ anonymity can also be guaranteed, even though it could be useful to expose doctors with abusive practices, not so much to punish them, but rather to drive them to modify their behavior, which could very well prove crucial in balancing France’s social security budget. If data is made available to honest users [MAT 14], which is to say excluding ill-intentioned “insurance companies”, and more so “unscrupulous journalists”, health policies could be greatly improved and social security budgets much better managed. However, the data “mine” is also fed increasingly by another very specific category of data, which is often collected without our knowledge by a variety of organisms: personal data. Our marital status, age, income bracket, tastes, frequent or sporadic purchases, the reason for those purchases – these constitute a growing part of Big Data, but a largely hidden one, which allows great profit for a select few users. Big Data comes mainly from: – the Web: newspapers, social networks, e-commerce, indexing, document, photo and video storage, linked data, etc.; – the Internet and connected objects: sensor networks, call logs; – science: genomics, astronomy, sub-atomic physics; – commercial data (e.g. transaction histories in a supermarket chain);

32

Big Data, Open Data and Data Development

– personal data (e.g. medical files); – public (open) data2. Information management has become a crucial factor for companies’ success. Efficiently processing (public and private) data in order to transform it into information is the key. Using a combination of methods and technologies, companies can attain and benefit from a competitive advantage. It is true that the information (which is originally just data) becomes a strategic asset and a form of power for those who possess it. However, it must be understood as a performance element within companies, since its value is generated through sharing and exchange. Open Data tends to support this ideal of power based on possession, but it draws closer to an ideal based on sharing. The more open data becomes, the more it can be used, reused, reoriented and combined with other data in the interest of all. Different business models can be imagined for best exploiting Open Data. Since the exploitable data generates millions of dollars according to several studies, Open Data has become a planetary movement. It is a movement that seeks open access and exploitation of data for all and by all. Sharing and opening up data means making essential data available online so that it can enhance many decision-makers’ analysis, save time or enable more informed decision-making in all sorts of sectors. Open Data should be understood as an opportunity and as a long term process for the different actors involved (administrators, companies, scientists, citizens).

2 The intersection of Open Data and Big Data, which involves massive amounts of data, is an example of how data can create value in terms of usefulness. Making data open means making it available online, which involves a new concept, “linked data”. This concept is a growing movement for companies because they must ensure their data is in a format that can be read by machines. This allows users to create and combine datasets and make their own interpretations of them. The data web consists of presenting that structured data online and linking it, which will increase its visibility and its capacity to be reused.

Open Data: A New Challenge

33

Open Data represents a major source of information for all actors, which is why it should be considered a tool for rationalizing companies’ public actions. Open Data is also a way of renewing public action implementation methods, and of developing links between (and with) governments, companies and citizens, since transparency is not the only goal of Open Data. 2.3. Open Data and the Big Data universe The Web is the largest and most dynamic reference point for data in the world. It is the ideal resource to best exploit data and transform it into information. Data on the Web is very diverse and has a huge volume (content and format). The most important asset of large volumes of data has to do with the fact that they make it possible to apply knowledge and create considerable value. Combined with advanced analysis methods, they can provide new explanations for several phenomena. There are two ways to transform data into a valuable contribution to a company: – transforming data into information is one of the stages of data value production, which is exploited in order to obtain useful information and to successfully carry out company strategies. This automatically involves database information in company decisionmaking processes; – transforming data into products or processes adds value to companies. This is produced when data analysis must be implemented in the physical world. The data revolution is interesting because it enables the development of competitive advantage. Big Data stands out today as a genuine ecosystem since it spans all sectors. Large companies and startups must collaborate. Interactions between experts, scientists, startups and large companies will promote the development of new energy leading to new knowledge and new projects.

34

Big Data, Open Data and Data Development

The government, therefore, has a crucial role to play by making public data available and modifying regulations so as to facilitate data access and use in order to create value. Nevertheless, when governments make raw data openly available, they must ensure that everyone can understand it. Easy access to data for all parties offers several advantages, both for those who own data and those who use it, including: – rediscovering other uses and applications (new data development) thanks to the use, reuse, linking, cross-referencing and combination of data with other forms (of data) coming from different sources. These new uses can be derived well after the data has been collected and used; – promoting innovation by ensuring ease of access to data, since if it is difficult to access the data, it is impossible to evaluate its full potential or to exploit and re-exploit it; – gaining feedback through Open Data as a means to compliment internal analyses with ideas coming from external sources; – increasing transparency, which will reinforce public control over government actions, ensures an increase in the reliability of scientific results, and results in newfound trust in companies’ credibility; – benefiting from the effects of web-networks by combining several elements, since the more data becomes open, the more the whole data ecosystem gains value. The main principle of the Open Data philosophy is free access to data, which will allow its use and reuse. Reusers will cross-reference it in order to provide new information that better responds to companies’ expectations. Open Data will make it possible to: – create new products or services; – gain access to aggregated and updated information;

Open Data: A New Challenge

35

– enhance companies’ images; – analyze and make sense out of data; – commercially exploit and develop the new information drawn from this data; – obtain financial returns. The biggest question for a company is no longer deciding if it should launch new products, but rather taking advantage of available (structured or unstructured) data and to know how to adapt different key success factors to its environment. These principles are not limited to internal data (client data, etc.). Instead, they also include all external data surrounding the company (cities, universities, etc.) that can allow it to increase the value created by a given piece of data. Optimizing production processes, fine-tuning client knowledge, improving its reputation, rationalizing supply costs, promoting research, etc. The possibilities are endless in the data revolution. In this way, it is necessary to first identify innovation pools and new economic models in order to go out and seek improved economic performance and attract value. The objective of an Open Data policy is to encourage creativity, stimulate innovation, and promote data reuse of both private and public data by relying on collective intelligence, as well as on scientists’ and companies’ will to create new knowledge capable of generating information. The nature of data allows it to be used, reused, linked and recombined in several different ways without diminishing its value. Open Data supports the emergence and success of great data potential and its main effect resides in the variety of its sources. Although the volume of data increases as do processing and exploitation speeds, different sets of Open Data compete to find new ways of developing data, which come to complement existing ones.

36 6

Big Data, Open Data and Data D Developme ent

pen Data repreesents a new way of “Op und derstanding innformation annd data gov vernance. We cannot imagine the info ormation system m other than thhrough a systtem extending beyond the coompany. Parrticipating in Oppen Data and B Big Data meaans concentratiing both the vvalue of Big g Data producced by the company itseelf, and on exxogenous data coming from m the compaany and its ppartners’ natu ural ecosysteem”, J.M. Lazard, “Op penDataSoft” foounder. Example 2.2. 2 Open Data a and governa ance

In 2011, France emulaated the Ameerican platfoorm data.govv, which was launchedd in 2009, too create its own public data dissem mination syystem, managged by Etalaab, with the aim of acceelerating movvement. Buut availablee data alreaady possessses great ecconomic vaalue for innnovative enntrepreneurs. However, governments are not thhe only orrganizations to go into Open Data. Etalab’s miission is to support puublic data opening o andd sharing, in n order to facilitate reeuse by coompanies andd citizens. In 2012 annd 2013, Etaalab organizeed a series of innovativee project annd service crreation conteests, aimed att encouraginng public datta reuse. Thhe initiativee was know wn as “Dataconnexions””, and it soought to reecognize thee best dataa applicatio ons, servicees, and intteractive viisualizations reusing puublic data: six startupss received awards. Ettalab also contributes c t shine a light on thee best data reuses, to esspecially by promoting p thhem within government. g In France, public p Openn Data and daata sharing arre free servicces, free too be reused and a availablee in open so ource formatts. This is allready a reesource used by hundredds of startupss that develoop new addeed value seervices. Evenn if public suupport for in nnovation in Big Data is present inn France thannks to the opening of pub blic data and financial inccentives foor startups and SMEss, few dataa developm ment strategies are im mplemented.

Open Data: A New Challenge

37

Once useful data has been surveyed and its quality criteria have been analyzed, it will be possible to organize data and transform it into information. Data generates information, and in that sense it requires a fairly strict method: – precisely defining, on a case by case basis, what it is necessary to retain. Prepare to store raw data and processed data; – conserve as much instrumental information as possible in order to save the memory of the processed data’s individual quality (observation periods, interruptions, etc.); – determine the computational constitution of the observation log beforehand (and not after the fact), and allow for future use of the data by introducing additional fields for selecting and identifying observations; – establish synthetic charts for quick access to data and keep them up-to-date. Box 2.1. Proposal for action

The volume of data is very large: it is necessary to gather it, store it, index it to make it accessible, and edit it for others to access. Companies manage data deposits and free access to that data gives them the capacity to fine-tune it, interpret it, find tendencies in it, and identify their characteristics. Data exploration is, therefore, a key stake for companies: knowing how to process data to obtain better performance. The data revolution will necessarily go through Big and Open Data. It is, therefore, essential to take an interest in these subjects and in their dissemination. The two concepts can transform the business world, the government sphere and civil society. Big Data gives us the power to understand, analyze and ultimately change the world in which we live. Open Data hopes for that power to be shared and for the world in which we live becomes more democratic. Big Data has to do with methods of gathering and processing a very large volume of data in real-time; it holds enormous wealth. Open Data represents a mode of access to information. Open Data is also a basic trend in data access: unlike most digital information, it is made available to the public or professionals free of cost. Open public data makes it possible to respond to an economic, scientific, environmental or social demand for innovation.

38

Big Data, Open Data and Data Development

In this regard, the foregoing reflections surrounding Open Data account for two needs on the part of data reusers: – more raw data updated in real-time, and, of course, processed; – contextualizing documents that make it possible to understand how and why a given set of data has been constructed. Open Data represents a major source of information for all actors, which is why it should be considered a tool for rationalizing companies and public actions. It is also a way of renewing public action implementation methods, and of developing links between (and with) governments, companies and citizens, since transparency is not the only goal of Open Data. Big Data and Open Data are closely related, but they are not identical to one another. Open Data provides a perspective that can make Big Data more and more democratic. Big Data is defined by its size. It is a term used to describe very large sets of data. But these are subjective judgments which depend on technology: the volumes of data available today might not seem quite as large in a few years, when data and IT analysis evolve. Creating value at the different stages of the data value chain will be at the core of the coming together of Big Data and Open Data. Big Data and Open Data represent information deposits that have yet been underexploited. 2.4. Data development and reuse The Web is entering a new phase of its existence, and one of the properties that distinguishes it is the quantitative and qualitative jump of the data available in it. Sources of data become more diversified: government agencies, companies and individuals. In the near future, objects will publish, share and circulate more and more data online. For the last few years, several voices have spoken out in favor of a freer flow of data.

Open Data: A New Challenge

39

Speed and ease of access to data are, therefore, crucial in a world where the quality of available information increases constantly. We can distinguish three main categories of online data according to the source of the data. We propose the following definitions: – raw data produced by public entities (demographic statistics, etc.): it is data coming from government agencies, local governments, and national statistic institutes; – raw data produced by private or public companies (catalogues, directories, reports, etc.): it can be sold or made available free of charge (e.g. Amazon’s book database); – raw data produced by individuals (age, comments,etc.): it is personal data as such and is protected by privacy laws and regulations. However, some data produced by users belongs to the service where it is hosted, as is stated in the terms and conditions. The relationship between different types of (statistical, scientific, administrative, geographic, and Web) data makes it possible to create ecosystems by integrating a large volumes of data coming from a variety of different datasets. Open data reuse plays an important role in terms of its capacity to add value. With the digital revolution and the advent of the Internet, which improve massive data production and processing, public Open Data and data sharing become a powerful tool for: – reinforcing trust among individuals due to greater transparency of public sector activities; – allowing for new forms of coproduction with civil society and supporting social innovation (like the Handimap project, which made it possible due to data from the cities of Rennes and Montpellier to develop an application for calculating routes for physically disabled persons); – improving administrative operations (as is evidenced by the very strong use of public data by the government itself); – improving the efficiency of public action by developing new modes of organization and new work processes (like automobile

40

Big Data, Open Data and Data Development

accident follow-up by highway safety services, which makes it possible to improve roadworks and construction); – supporting economic dynamism by creating new resources for innovation and growth. The process of information understanding is often considered to be an ascending progression going from data, to information, to knowledge, and finally to wisdom. In the same way, Open Data is gaining importance and becoming more functional as it is gathered, structured and disseminated. Data assembled in a relevant manner thus reconstructs a world of information and structures a world of digital data. Opening up data does not only involve improvement in responsibility and trust, but also, in the case of entrepreneurs, in innovation through application development, most often in the form of “mash-ups”, or digital products created from already existing elements. After identifying, gathering and classifying data, a processing phase ensues in which cleaned data ready to be used for analytic purposes is produced. This processing phase is important because it makes it possible to classify the data into datasets, which provides the maximum amount of information. Available data growth in terms of quality and value is the modern response to the always growing need for information, both by individuals and organizations. More and more, governments throughout the world define and implement Open Data strategies, with the aim of capitalizing on the phenomenon’s three pillars: transparency, participation and collaboration. Open Data advocates hope that this type of change will reinforce democracy and improve the impact of government activities through increased transparency, participation and collaboration. They thus consider that this will allow for greater efficiency through information infrastructure allowing better reuse of data. They are also motivated by Open Data’s potential to produce new innovations through its use. The Open Data culture is founded on the availability of data and is oriented towards communication: this makes

Open Data: A New Challenge

41

it possible to generate knowledge through transformation effects in which data is provided or made available for innovative applications. France is one of the few countries in the world to have an organization like the French National Institute for Statistics and Economic Research (Institut national de la statistique et des études économiques, INSEE), which produces a vast amount of data. We therefore have great potential in terms of activities that create added value through data processing. The advantage is, in fact, both private and public. States can benefit from allowing the fruits of these new activities to flourish and expand: not only do they create value directly, they also provide services for society at large. The current debate on Open Data evokes public entities and companies’ reflection of whether to make their data available for the purpose of generating business or image benefits (transparency, information, communication, etc.). In the context of Open Data, data can circulate freely to favor innovation research or simply to achieve productivity gains. 2.5. Conclusion Indeed, with the progress of the digital revolution, Open Data presents a common reference point, favoring the economy and society, but also research and knowledge production. The Open Data movement seeks to promote knowledge and innovation due to information sharing and cross-sector collaboration. This movement is made possible by information and communication technologies that enable exchanges of an almost unlimited amount of data (Big Data). By opening its data to the world, an organization allows for it to be reused for different purposes. The results of this secondary reuse can, in turn, be shared with the community, which creates a multiplying effect. The phenomenon of Open Data, which opens up digital government data to the public, developed very quickly in the US before taking hold in Germany, France and several other countries.

42

Big Data, Open Data and Data Development

For Tim Berners-Lee, the phenomenon of Open Data should determine the future of Web developments. Moreover, with the development of all kinds of data, the evolution of data quantities should generate high added value, both in terms of usage and in terms of decision-making and development enhancement, thanks to improved synergy in actors’ actions. Furthermore, one of the main advantages associated with Open Data is that it promotes the development of a culture centered on information sharing and on cross-sector collaboration. As a crosssector principle, Open Data can generate benefits for the economic, cultural and social spheres. The possible benefits that stem from such opening of data can therefore be exponential.

3 Data Development Mechanisms

Since the advent of IT and the internet, the amount of data stored in digital form has been growing rapidly. An increasing number of data silos are created across the world. Individuals are putting more and more publicly available data on the web. Many companies collect information on their clients and their respective behaviour. As such, many industrial and commercial processes are being controlled by computers. The results of medical tests are also being retained for analysis. The increase in data produced by companies, individuals, scientists and public officials, coupled with the development of IT tools, offers new analytical perspectives. Thus, currently, not only is the quantity of digitally stored data much larger, but the type of data is also very varied. The data analyzed is no longer necessarily structured in the same way as in previous analysis, but can now be in the form of text, images, multimedia content, digital traces, connected objects, etc. Faced with this volume and diversification, it is essential to develop techniques to make the best use of all of these stocks in order to extract the maximum amount of information. There are several techniques, such as “data mining”, which are not new but respond to the principles of descriptive and predictive methods. This is an integral part of data analysis, especially when it is substantial.

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

44

Big Data, Open Data and Data Development

Companies are increasingly aware of the potential benefits of Big Data in terms of developing new products and services, transforming their business model, improving customer knowledge or exploring new areas of development. To make the most out of Big Data, the issue is not limited to the “simple” technical issues of collection, storage and processing speed. The use of Big Data requires rethinking the process of collecting, processing and the management of data. It is necessary to put in place governance that enables the classification of data and the sorting of analytical priorities. Every company has a deposit value through their data; Big Data is a generator of value as an analysis of these data. It is the “analysis” that will be applied to data which will justify Big Data, not the collection of data itself. We must therefore allow companies access to the “information” which will be likely to generate the most value. But the process whereby information acquires value is that of “competitive intelligence”. It becomes increasingly useful once the company has a large amount of information stored on a database. This is the best way for an entire company to get hold of Big Data in order to create value. If data development is itself a new strategic challenge, then governance is a required tool for success. 3.1. How do we develop data? The amount of data (structured and unstructured) which originates from diverse sources and is produced in real-time leads us to suggesting that there is a “Malthusian law of data”. Malthus noted that the quantity of food only increased arithmetically, whilst the number of humans increased in a geometric progression [MAL 08]. Similarly, Pool [POO 84] notes that the supply of data increases exponentially whilst the amount of data consumed increases, at best, in a linear manner.

Data Development Mechanisms

45

This explosion of data volumes will gradually increase as the Internet of Things develops. Human beings are relayed in their “data production” by a multitude of sensors, whether at home or in their means of transport. In parallel, the production of data outside of the traditional boundaries of a company (and their information systems) is growing exponentially, particularly via social media. Data produced are relatively unstructured (photos, videos, etc.). The data market is, therefore, in a situation where the quantity offered is much higher than the quantity demanded. This phenomenon is essentially due to the fact that our mental capacities and our available time to deal with the information are limited. The increasing digitization of our activities, the ever-increasing ability to store digital data and the subsequent accumulation of all kinds of information generates a new industry that focuses on the analysis of large volumes of data. Big data has been qualified as “the next frontier for innovation, competition and productivity” [MAN 11]. It offers businesses unprecedented opportunities for increased revenues, performance and knowledge related to their business, to the market and clients. Data from social media is a perfect example. Using technology to capture data from Facebook, Linkedin, Twitter, etc., the company can refine brand management, find new distribution channels and strengthen the customer relationship. The accumulation of data through diverse information systems has a potential value that companies are not always aware of. Even if they do not necessarily understand how to use them themselves, they have resources that they do not yet value and this data and its usage is a source of capital for these businesses. The challenge of valuing data is working out, in an intelligent manner, which of them are usable. Solutions and methods have been put in place by companies such as Google, Amazon and others for whom “Big Data” really makes sense. These organizations are characterized

46

Big Data, Open Data and Data Development

by their strong capacity of innovation and particularly their “data science”. It is crucial to highlight the mapping of data by companies and their subsequent capacity to extract value from them. We need to define a “trade priority”, put into place solutions to respond and see if this system is labelled Big Data or not. Using these resources is therefore the biggest challenge, or at least in the context of the phenomenon of Big Data, the challenge is the awareness and prevention of intersections of numerous data, and not an eventual generation of value. Companies must be aware of data wealth that they have internally available, and how to exploit them to extract value before wanting to expand them through external data. The question is no longer about identifying data that needs to be stored, but what can we do with this data? The capturing and mixing of this data becomes highly strategic in order to reveal new knowledge. The analysis of large volumes of data can bring about some clear insights that would allow for a competitive advantage. But this requires tools to process data in large volumes, at high speeds and in various formats. This calls for an efficient and profitable infrastructure: “cloud computing”. The cloud becomes a reality in the lives of companies. It provides a method that supports the volume of data and advanced analytical applications that are capable of bringing added value to the business. The development of cloud computing is linked closely to the phenomenon of Big Data. Hence, we must know how to elevate these technologies to the scale of billions of information units. It is also about developing a culture of data, its operational logic and its visual interpretation. The challenge today is to develop technologies to visualize massive data flows: “data visualisation”. Visualisation is a way for operational decision-makers to best understand and use the results of the analysis. The visual restitution achieves the necessary level of abstraction needed to achieve an understanding of the “bulk data” and to give it meaning. Correlations between data allow for the extraction of new knowledge.

Data a Development Mechanisms

47

ocessing large datasets requirees Pro a mastery m of calculations in ordder to achieve resultts in real tim me. he major actors such as Googlle, Th Am mazon, Yahooo, etc., develoop “daata centres” in which they havve speecific spaces for f storing largge am mounts of daata (webpagees, queeries, customerr lists, items fo for salle, messages, ettc.) in raw form m, unfformatted annd without a relational structurre. Thanks to thhe am mount of data noow available, thhe mo odels of data annalysis are able to pro ovide more accuurate results.

Exam mple 3.1. Data a centres or diigital factoriess

Thee developmeent of increeasingly quicck and pow werful tools is pushingg decision-m makers to seeek “decision support” thhat is rapid aand efficiennt, and visual advice in any a geograph hical area, annd hence on tthe web. Inn our societyy of informaation and com mmunicationn, statistics aare playingg an increasiingly importtant role. Th hey are omniipresent. Eveery day, paarticularly thhrough the media, m we arre all exposeed to a weaalth of information from m quantitativve studies: opinion o pollss, “barometerrs” of the popularity p of politicians, satisfaction n surveys annd indicators of nationaal accounts. In addition a to many m functionns, modern liife requires the t reading aand understtanding of the t most divverse statistical data: maarket researcch, dashboards, financial statistics, forecasts, statistical teests, indicatoors, t base of the social and a econom mic informatiion etc. Staatistics are the system. They are,, nowadays, extremely affordable thanks to tthe tremendous develoopments of databases, d no otably via thhe Internet, or

48

Big Data, Open Data and Data Development

easily buildable through spreadsheets integrated with the most current software. Amongst the available techniques are: – descriptive techniques: they seek to highlight existing information that is hidden by the volume of data (customer fragmentation and research of product association on receipts). Thus, these techniques help reduce, summarize and synthesize data such as: - factor analysis: projection of data in graphical form to obtain a visualization of the overall connections (reconciliations and oppositions) between the various data; - automatic classification (clustering–segmentation): Association of homogeneous data groups to highlight a segmentation of certain individuals in some classes; - research of association (analysis of receipts): this involves spotting dependencies between the objects or individuals observed. – predictive techniques: predictive analysis allows for very accurate projections to identify new opportunities (or threats) and thus anticipate appropriate responses to the situation. They aim to extrapolate new information from existing sources (this is the case of scoring). These are statistical methods which analyse the relation between one or several variables depending on independent variables, such as: - classification/discrimination: qualitative;

the

dependent

variable

is

- discriminate analysis/logistic regression: find allocation rules from individuals to their groups; - decision trees: they allow dividing the individuals in a population into classes; we begin by selecting the best variable to separate individuals in each class into sub-populations called nodes, depending on the targeted variable; - neural networks: from the field of artificial intelligence, this is a set of interconnected nodes that have values. They can be understood by adjusting the weight of each node until you find an optimal solution to achieve the fixed number of iterations; - forecast: the variable is continuous or quantitative;

Data Development Mechanisms

49

- linear regression (simple and multiple): it allows for modeling the variation of a dependent variable with regards to one or more independent variables, and thus, to predict the evolution of the first in relation to the last; - general linear model, it generalizes the linear regression with continuous explanatory variables; The following diagram summarizes the principle of this Contractors

Providers

Deposit of projects Helping module in drafting of specifications book

Data processing

Data processing

Automatic selection (Search engine) Email automatic sending Classify module: Companies’ classification support

Responses processing Manual selection

Sending negative response

Sending positive responses with contact details of the contractors and the specifications drafted by 123 PRESTA

Tenderin g

Drafting of specifications book

Select 4 providers maximum

Data archiving

Sending negative response

Data archiving

Final selection of providers by the contractors

Provider choice

----- Project monitoring ----Communication interface Management / monitoring the current project Promotion of projects carried

Classification by data analyzing. Around a group. Plan 1, 2 Presentation of individuals

It is important to highlight in this conclusion that nowadays there are very few qualified and up to date directories specialized in TIC. http://www.123presta.com/ The objective remaining the same, as to becoming the leader of intermediation in France and publishing the figures of a TIC observatory at a national level. The forecast for these two coming years are quite modest with regards to the domestic and international market potential. There are today just over 10,000 providers that can reference themselves to our directory. Research and development work carried out by TRIS laboratory and supported by Transfert-LR.

Example 3.2. An example of data processing for the startup “123PRESTA” in 2010. For a color version of the figure, see www.iste.co.uk/monino/data.zip

50

Big Data, Open Data and Data Development

Achieved

Forecasting by the neuron method. Achieved

Forecast

As part of an agreed exchange between the laboratory and the European distribution giant, they tested two methods to obtain mass forecasts: – the neuronal method; – the classic method of forecasting.

Achieved

Forecast

Example 3.3. Forecasting of time series using neural networks, the turnover of large retail stores. For a color version of the figure, see www.iste.co.uk/monino/data.zip

These techniques are called upon to develop themselves in order to improve data processing. Thus the software necessary for this processing must be able to detect interesting information: the capacity to combine more data will reinforce the interconnectivity in the company, which increases responsiveness. Another aspect of the Big Data revolution, which reinforces the power of mathematical formulas

Data Development Mechanisms

51

to help explain data, is “algorithms”. Beyond data collection and storage, algorithmic intelligence is indispensable to make sense of data. Algorithms are used to establish correlative models, ordering, sorting, prioritizing data and making them intelligible through correlation or prediction models. Companies can now understand complex phenomena and share these analyses to increase their collective intelligence. But it should be noted that every company must define its own way of extrapolating data. To do this they need the help of someone often called a “data scientist”. They must be capable of analyzing enormous quantities of data to find correlations. However, these correlations must be applicable, cost effective and achievable. The necessary qualities are therefore: understanding the use of data, creativity in interpreting the data and confirming correlations and an understanding of the company in order to master the use of data or find models which improve its profitability1. The “Big Data” phenomenon, therefore, raises several questions relating to what technological developments it may create and what value these may have. It requires the development of new interdisciplinary training programs, resulting in an operational expertise particularly suited to the field of “Big Data”, in IT interface, statistics and mathematics as well as a strategic vision to design new services and products and the deployment of advanced decisionmaking systems. Another important factor in the process of using data is linked to the “quality” of data. Data, the basic elements of the process, constitute a vital and historical asset for the company. The quality of the information processed by Big Data is directly linked to the quality of the datasets entered, whether they originate from within or outside the company. The quality of data is an important factor in the 1 A statistician becomes a data prospector (data scientist or data analyst) whenever the volume to be used is particularly large, and the analysis must be done by focusing on research efficiency and company profitability. The emergence of “data science” in companies and an understanding of business data is a strategic asset that can become a powerful vector of differentiation and performance. It requires making sense of massive data deposits and bringing out the added value.

52

Big Data, Open Data and Data Development

phenomenon of Big Data because dependency on unreliable data can lead to bad decisions, and therefore, bad choices. The Stock Exchange, chance, deterministic chaos.

Monthly Returns from 1926 to 2005

Linear (LogRS1)

Linear (LogRS2)

Graph convergence of the kurtosis bootstrap

Since 1985, “deterministic chaos” has been featured in stock market studies. Numerous studies have tried to confirm it’s pertinence in face of the random progression, which is also the basis of the theory of low efficiency for FAMA, as well as for the various models claiming to represent the functioning of the markets. We have used French data collected over a long period of time, including more than 13,000 datapoints (1965-02), to try to extract the daily returns, starting from the Hurst calculation.

Example 3.4. Chaos, exponents of Hurst and Bootstrap. An example applied to the Paris Stock Exchange [MAT 05]

“Data” is no longer structured and relational; it is unstructured and heterogeneous content (reviews, videos, images, sounds, sensor data, etc.). On the other hand, recognizing images or objects, defining variables, solving semantic subtleties and crossing different data sources all comes from an unstructured analysis.

Data Development Mechanisms

53

Thus “open data” is another source of data; it is not another amount of data that is added to Big Data but rather reliable data than can be referenced. The volume is not a real problem: the challenge of Big Data is identifying intelligent data and being able to interpret it to improve the competitiveness and performance of the company. The goal of interoperability is to mix a great amount of data from diverse sources. As such, an “Open Data” approach represents a new business model in which the company is producing data. In a context of crisis and accelerating cycles, companies must continuously optimize their productivity and their operational efficiency. Poor quality data (customer, product, supplier, structure, etc.) will have a direct impact on the competitiveness, efficiency and responsiveness of the company. To transform data into value it is necessary to invest in technologies: mastering the techniques, the methods and tools will fill this function. But technology alone will not settle the issue. Collecting data without prior justification, without defining a specific strategy, may be much less profitable than expected. The absence of a unified and controlled management can also have serious consequences on losing control of operational risks. This therefore requires genuine political governance beyond simply the collection and processing operations. The aim is to help decision-making in a context where “information”, which is at the heart of “competitive intelligence”, has become a major strategic asset for companies. The use of masses of collected data requires sorting, testing, processing, analysis and synthesis. It is through this process that raw data collected during the search will be transformed into “strategic information”. The interpretation of the collected data through analysis and synthesis will represent the level of success of the “competitive intelligence” process. Data development process, which is used to generate highly valuable information for companies and aims to help them take advantage of “Big Data”, consists of collecting, processing and managing large volumes of data to identify what they can bring to the

54

Big Data, Open Data and Data Development

company as well as the measurement and monitoring of their use, consumer consumption patterns, their storage and their governance. The latter includes the preceding steps and establishes processes and good operational practices as and when required. 3.2. Data governance: a key factor for data valorization It is difficult for companies to best utilize the ever-increasing volume of various data. Taking advantage of Big Data, which is often summarized in four, or sometimes seven, words (4V and 7V respectively), appears to be a major challenge. Moreover, automatically generated data often requires new analytical techniques, representing an additional challenge. According to the available statistics, 80% of data in companies is unstructured. In addition, documents in plain text, videos, images, etc. add to the types of data. Textual data, log data, mobile data, videos, etc., have disrupted traditional processing technologies because they are not structured data. The new challenge is the processing of these new data formats. A portion of these resources are stocked in databases, developed in order to be managed and available for use. Once again we are faced with the problem of the quality of data. In this regard, how can companies know that the available data is reliable and suitable for use in order to extract information (see the pyramid of knowledge)? Treating erroneous and unreliable data in the same way as other data will likely distort the analysis, and therefore, the subsequent decisions. This risk related to the quality of data is at the origin of a new concept, “Smart Data”. We must, therefore, recognize the importance of searching for the right data (from the warehouse), that is reliable and useful, which will allow for obtaining information and creating value, rather than only looking at the processing. In these circumstances the company has to find a solution that facilitates the processing of this volume of data and its availability for

Data Development Mechanisms

55

very different analytical purposes. Moreover, the substantive discussions on the control of a company’s information holdings leads them to adopt data governance principles. For a company to be able to value its data as a corporate asset it needs to change its culture and the way data is managed. This means mixing the business’s data, which is already available and structured, with those coming from Big Data (meaning nonstructured). Data governance is a set of functional domains that allow for managing, integrating, analysing and governing data. Data governance is defined as “the capacity to mix reliable and timely data to increase analytics on a large scale, regardless of data source, the environment or the role of the users”2. This is a quality control framework aiming to evaluate, manage, operate, optimize, control, monitor, maintain and protect corporate data. Based on this foundational concept, users can easily explore and analyze mixed and validated data from an entire company. Once the relevant data has been obtained, it must be formatted properly to be analyzed. Data ready for analysis must comply with all the relevant rules of governance, notably in being reliable, consistent, timely and properly mixed. The introduction of analysis tools based on massive volumes entails: – access to existing data, but for uses that are not necessarily listed. These uses may lead to violations of security policies, especially in regards to regulatory constraints (protection of privacy, health data, banking data etc); – the imports from social networks, the Web or new data that has not yet been listed in the company. These new data need a policy review, taking into account their particular sensitivity; – the results of the analysis are in themselves new data to protect.

2 According to Pentaho Business Analytics, available at: www.pentaho.com/5.0.

56

Big Data, Open Data and Data Development

Security and governance become critically important. In March 2014, Gartnet published a report [GAR 14] which highlighted that: “the information security managers should not manage the security of Big Data in isolation but need rules that cover all data silos to prevent this management turning into chaos”. It recommends evaluating the implementations of data protection with respect to the safety rules around databases, unstructured data, cloud storage and Big Data. The multiplicity and disparity of sources creates challenges for data collection, processing and storage. Multiple uses exist: from targeted marketing (scoring, rebound and behavioural analysis) to more technical uses (log analysis) through to the piloting activity, particularly the transformation phase. Google, Amazon and Facebook have demonstrated the power of data to improve the customer relationship, sell more and stand out from competitors. Box 3.1. Challenges of multiplicities of sources

The introduction of data analysis solutions requires both a review of existing policies to integrate new uses of data and an extension of policies to incorporate issues specific to new data. This requires that the lifecycles of the data collected are optimized so that the needs are met instantly and treated properly. This highlights the interaction between multiple actors (policy makers, analysts, consultants, technicians, etc.) in a group dynamic that allows for a combination of knowledge: better understanding, better analysis of the situation and production of information that is necessary for taking action. To ensure relevant collection and before moving on to analyzing data, it is essential to define for what analytical needs this data will be researched and collected, and with what techniques and tools. This requires defining an interdepartmental responsibility across the company. The transformation of data information into knowledge is the result of a collective process centered on the shared success of

Data Development Mechanisms

57

problem-solving. It should be noted that this principle refers to the notion of collective intelligence [DUP 04]. The concept of knowledge makes use of collective processes, community, apprehended at the organisational and managerial level and in line with the strategy of the company. With the sheer volume of data that is accumulating in most organizations, it is not always easy to find the right data or even convert them in order to extract a useful analytical view. Efficient data governance is essential to preserve the quality of data, because its volume and its complexity are growing as much as there are interactions within a company (internal environment) and between the company and its external environment (customers, suppliers, competitors, etc.). The first step in building a governance framework is identifying the extent of the requirement by comparing the existing practices and the target set for good governance: – manage all data and associated data; – manage data consumption process; – manage data lifecycle in a cross-domain view of the company; – improve the quality of data. Data governance involves a set of people, processes and technology to ensure the quality and value of a company’s data. The technological part of governance combines data quality, integration and data management. The success of data governance strategy represents various technical challenges. Firstly, companies need to integrate different data sources and implement measures of quality to improve their functionalities. They should then establish and improve the quality of their data. Collaborative work is finally required for the different teams to work together on data quality issues. In other words, it seems necessary to steer the managerial and organizational practices to develop a growing organization based in

58

Big Data, Open Data and Data Development

creativity and the use of knowledge. This brings us to a very important concept in the business world, which is “competitive intelligence” (CI) as a method of data governance that goes beyond a mere mastery of strategic information. CI helps change the representations that company shareholders have of their environment. In this sense, the strategic and collective dimension of CI lies in the triptych: ownership, interpretation and action [SAI 04, MOI 09]. CI will indeed help anticipate new opportunities for growth that the company can seize on by modifying its activities. To make the company more innovative, we must adapt the strategy by controlling the most decisive information. CI analyzes information, anticipates developments and makes decisions accordingly, thus developing information security, data protection, the anticipation of risks and the control of its information system in and beyond its networks. The set of fields that constitute competitive intelligence, such as knowledge management, information protection and lobbying, can be grouped into the overall concept of strategic intelligence. CI is a mode of governance whose purpose is the control and protection of strategic and relevant information for any economic actor. It can be: – offensive when it collects, analyzes and disseminates useful information to economic actors; – defensive when protecting strategic information for the company from malicious acts of negligence, both internal and external. With the advent of the new knowledge-based economy, industrial issues in companies have become more complex. In this light, CI is not confined to the management of data flows, but is fundamentally interested in their interpretation and use in creating knowledge. The pervasiveness of the Big Data phenomenon complicates the control of information that may lead to the extraction of knowledge and information. In contrast, CI practices facilitate the measuring, management and increase of the value of data, which in turns, influences the decision-making process to obtain the best possible results through the global sharing of knowledge.

Data a Development Mechanisms

59

Presentattion of the CI concept in nine instructioonal videos. These insstructional sources are a put online byy AUNEGE E3 and are available on Canal-U4, PORTICE ES5and FUN6.

Ex xample 3.5. Sh hort videos prresenting CI

Ourr age of inforrmation is chharacterized by an exponnential increaase in dataa production and processsing, but also o by a massive increase in the speeed of data trransmission as a well as th he speed of access a to storred data. We W probably cannot fightt Big Data, but b if we waant to extracct a real proofit from it, it is essentiial to masterr it through a strategy thhat creates value from m data. It is only o through h a governannce of reliabble data thhat businessses’ data caan become a strategic asset, whiich ultimattely brings a competitivve advantag ge and increeased value to compannies. The value v creation of data is onee of eight digiital trends highhlighted by thhe Microosoft Digital Trrends study in 2013. To mak ke sense of dataa and “generatte value”” from it, it is obvious that theyy must be goveerned. Box 3.2. Vallue creation fro om data

3 AUNE EGE: Computting Universityy in Economics and Managgement. Webssite: http://ww ww.aunege.org//. 4 Canal-U: The computting channel off higher educatio on. An online station s for studeents h nal-u.tv/. and teachhers. Website: https://www.can 5 PORT TICES: Passee Opérationnel pour Réusssir avec les Technologies de l’Informaation et de laa Communicatiion dans l’Ensseignement Suppérieur – projject undertakken by the Minister for Educatiion and Researcch: http://www..portices.fr. 6 FUN: Computing University U of France. F Websitte: http://www.france-universitenumeriquue.fr/.

60

Big Data, Open Data and Data Development

The main objective of data governance is to ensure that the use of Big Data meets a formal strategy aiming to obtain accurate results: it ensures that all the relevant questions are asked for the best use of Big Data. The governance of large volumes of data therefore remains built into companies. This requires establishing a hierarchy of data as well as ensuring their protection, which enforces general compliance throughout the company. 3.3. CI: protection and valuation of digital assets Today, a company’s success is inextricably linked to its control of information. Access to information, its analysis, dissemination and presentation are all elements that determine success. Information is now instantly and readily available to all. It has become an economic asset and represents a valued product generated by sharing and exchange. Depending on its value, this information can become sensitive or even strategic; it is therefore linked to the notion of secrecy because it represents a key factor that affects the whole economy. This exponential increase in the amount of available data creates new opportunities for usage of this information. The value of all of this data is nullified if there is no an existing process of transforming available information into analysis. As an overview, the factors that assisted the rise of Big Data and which continue to develop it are [SYL 15]: – the automation of data exchange; – a storage revolution; – the advent of new data science; – the progress of data visualisation; – new opportunities for monetization.

Data Development Mechanisms

61

We will see just as many applications grow in strength over the coming years provided that businesses adapt their operations to this development. To do so, they must be innovative, using and mixing data from different connected objects constantly. This requires them to transform data (available in different forms, often unorganized and collected through various channels) into information, and then into knowledge. Unfortunately, the necessary information is currently hidden in a huge mass of data. The exponential growth of data coupled with the use of algorithms has contributed to the emergence of the concept of “Big Data”. CI practices are not able to ignore this phenomenon or the revolution affecting their processing. The success of a CI strategy, thus, relies on the capacity of businesses to manage and use the mountains of data that they have, in other words “data mining”. This operation attaches even more value to information, which must be protected. The use of Big Data allows data to be mixed as to obtain a precise mapping. This should provide quality information that allows a company to evaluate alternatives, in order to better decide its behaviour and ensure the safe management of its holdings as part of an approach based on collective intelligence. Now companies must face – in real time – a significant increase of available data that could influence the process of decision-making. Several events help situate the concept of CI, and thus shed some light on its origins and evolution by identifying the conditions that led to its emergence and development. The methods and tools of CI help to validate collected data (from different reliable sources) into coherent information tailor-made to the company’s profile and needs. To manage such volumes of data and information, it is absolutely essential to have sorting and selection methods that are both pragmatic and effective. Competitive intelligence is a set of steps, from surveillance to researching strategic information. These steps are implemented by companies in order to monitor their environment,

62

Big Data, Open Data and Data Development

increase their competitiveness and manage the risks associated with their activity.

Figure 3.1. A model of economic intelligence [MON 12]

As an example, a distribution group can rely on geolocalization data of a mobile provider in order to identify the most promising zones. Here, we are at the crossroads of competitive intelligence (strengthening the competitive advantage) and marketing. A company involved in the “B to C” could analyse large masses of comments produced by internet users on social media networks in connection with its brand. This is at the crossroads of competitive intelligence (surveillance of the environment) with communications. Box 3.3. Data for geolocalization

The CI process will allow companies to be both reactive in adapting to change and proactive by having a pre-emptive attitude to understand the dynamics of their environment better. Setting up a CI approach centered on Big Data at the heart of a business is an effective response to the challenges of an increasingly complex and unpredictable globalized world. In this context, it is necessary to

Data Development Mechanisms

63

control information flows from the whole company before taking a strategic decision. The development of a CI approach at the heart of a company can only be carried through with everyone’s participation. The confrontation of differing levels of responsibility, highlighted by a strategic line, is the best way to help the decision-maker to make the correct decision at the right time. Estimating the quality and reliability of information and determining its usefulness for the company is undoubtedly the most important part of CI. To widen its competitive advantage, the company must be able to create an asymmetry of information to its advantage. Information is becoming a strategic issue, not only for state security but equally to defend a country’s overall competitiveness. The significance of information as a raw material for the business world was only recently discovered: economic information is unlike any commodity in our society. CI aims for a better control of information in order to help the decision-making process. Even seemingly derisory information can, after processing and crosschecking, have an economic value. Information is thus an economic and even historical good for the whole company. Moreover, from the point of view of the amount of data, CI provides a global vision of the company’s environment and monitors the whole informational sphere (markets, technologies, etc.) and allows for absorbing strategic information from the available mass. This has become one of the growing activities around the world; it can be perceived as both: – an informational approach which includes a set of operations or processes through which collected information becomes usable and worthwhile. This approach also aims to harmonize research, processing, distribution and protection in-line with the requirements in that context and with the actors involved. The management of these actions is the purpose of the monitoring activity, which is essential to the competitive intelligence process;

64

Big Data, Open Data and Data Development

– mediation between the different actors: - the decision-makers: involved in this process at different levels, most often solicited through several roles, and functions, - the monitors: responsible for providing useful information in the best available conditions, speed, and cost, according to an information request explicitly or implicitly formulated. CI can be considered as a new managerial practice for the corporate strategy, enabling it to improve its competitiveness. It constitutes a strategic response to the questions and concerns of decision-makers as well as improving the strategic position of the company and its environment. According to Salmon et al. [SAL 97] “giving decision makers the necessary information to define their medium and long-term goals in a better manner”. CI produces an “info action” enabling pro-activity and interactivity [BUL 02]. This principle of dynamic use of information takes place within a coherent structure and is based on the phenomenon of collective intelligence. According to Martre7 [MAT 94], CI can be defined as set of coordinated actions on research, processing, and delivery of useful information to economic actors. These various actions are carried out legally with all the necessary protection guaranteed for preserving the company’s holdings quickly and cost-effectively. Useful information is that which is needed by different levels of the company or community to develop a coherent strategy, and tactics that are necessary for the company to achieve its objectives and improve its position in a competitive environment.

7 In France, many politicians have expressed interest in this concept and without a doubt Edith Cresson was the first to realize France’s weakness in this field. Edith Cresson was the first person in France to become interested in CI, whilst she was Minister for Foreign Trade with Japan and the USA in 1988. During the 1980s, as Minister, Edith became aware of France’s weakness in CI. In 1991, she was part of the working group of the General Planning Commission that would, three years later, produce the Martre report: “Competitive Intelligence and Business Strategies”. In 1998, Edith Cression published “Innovate or fail” in which the European Commissioner for Research and Technology calls on France to innovate at all costs and in all areas. She created “the beginning of a culture in France”.

Data Development Mechanisms

65

These actions, at the heart of the company, are organized around an uninterrupted cycle which generates a shared vision of corporate goals. CI stems from a strategic vision and promotes interactions between different business, community, territory, national, transnational and state actors. This definition helps create a global vision of the environment in which CI must emerge. The coordination of these actions offers the company a view on its different relations with its environment through constant monitoring of the behaviour of its competitors and the realities of the market. The analysis of a larger amount of data in real time is likely to improve and accelerate decisions in multiple sectors, from finance to health, both including research. The new analytical power is seen as an opportunity to invent and explore new methods which are able to detect correlations in the available data. Everything that relates to data or CI can be qualified as “Big Data” and it seems that this craze of excitement will soon reach its peak of enthusiasm with “cloud computing”, which saw existing offers renamed and entire organisations move to “cloud” technology overnight. 3.4. Techniques of data analysis: data mining/text mining Nowadays, we have a lot more data to manage and process: business transactions, scientific data, satellite images, text documents, text reports and multimedia channels. Due to this large increase in data, the design and implementation of efficient tools which allow users to access “information” that it considers “relevant” is increasingly necessary as this information allows for a better understanding of one’s environment, keeping up-to-date with the market and to focus its strategy better. The diversity and growing volumes of data that rapidly converge in a business makes accessing and processing them very difficult. These reservoirs of knowledge must be explored in order to understand their meaning, to identify the relationships between the data and to have models explaining their behaviour. Faced with huge amounts of data, we have created new requirements that allow us to make the best managerial choices. These requirements represent the automatic

66

Big Data, Open Data and Data Development

summary of the information from stored data, in order to discover new knowledge. In order to meet these needs, a set of structures, approaches and tools – some new and some pre-existing – were grouped under the name “data mining”. The birth of data mining, a term initially used by statisticians, is essentially due to the convergence of two factors: – the exponential increase in businesses of data related to their activities (customer data, inventories, manufacturing and accounting) and that contain vital strategic information that can assist decisionmaking; – very rapid advances in hardware and software. The aim of data mining8 is thus the extraction of value from data in corporate information systems. Processing a multitude of data held within the organization in order to achieve better decision-making is therefore often seen as a major challenge for companies. Data mining represents a set of discoveries of new structures in large data sets which involve statistical methods, artificial intelligence and database management. It is therefore a field at the intersection of statistics and information technology, with the aim of discovering structures in large data sets. Data mining as a process (as automated as possible) goes from taking basic information in a “data warehouse” to the decision-making forum, bringing “added informational value” to each step up until the automatic release of actions based on the synthesized information. Data mining is the process of automatically extracting predictive information from large databases. According to the Gartner Group, this process can be repetitive or interactive depending on the objectives. We can say that the main task of data mining is to automatically extract useful information from data and make 8 In this case, “data mining” is seen as the tool for carrying out this mission. Also referred to as “data exploration”, “data sorting”, “data foraging”, “extraction of knowledge from data” (ECD) or “knowledge discovery from data” (KDD).

Data Development Mechanisms

67

it available to decision makers. As a process, it is relevant here to note that data mining refers to both tools and a highly developed computer technology. Essentially, we must remove the role of the human who must be totally involved in each phase of the process. The different phases of the process are therefore tied to: – understanding the task: this first phase is essential in understanding the objectives and requirements of the task, in order to integrate them into the data mining project and to outline a plan to achieve them; – understanding the data: this involves collecting and becoming familiar with the available data. It also involves identifying at the earliest possible opportunity problems of data quality and develop the initial institutions and detect the first sets and hypothesis to be analyzed; – preparation of data: this phase is comprised of all the steps necessary to build datasets that will be used by the model(s). These steps are often performed several times based on the proposed model and the results of analysis already carried out. It involves extracting, transforming, formatting, cleaning and storing data appropriately. Data preparation constitutes about 60-70% of the work; – modeling: this is where modeling methodologies of statistics come into play. Models are often validated and built with the help of business analysts and quantitative method experts, called “data scientists”. There are in most cases several ways of modeling the same problem of data mining and several techniques for adjusting a model to data; – evaluation of a model: at this stage, one or several models are built. It must be clear that the results are deemed satisfactory and are coherent, notably in relation to their targets; – the use of the model: the development of the model is not the end of the data mining process. Once information has been extracted from the data, they still need to be organized and presented so as to make them usable for the recipient. This can be as simple as providing

68

Big Data, Open Data and Data Development

a descriptive summary of data or as complex as implementing a comprehensive data mining process for the final user. In any case, it is always important that the user understands the limits of data and analysis and that their decisions are made accordingly. Data mining is a technology that creates value and extracts information and knowledge from data. The development of this technology is the result of an increase in digital data which, relative to their abundance, is underexploited without the proper tools and expertise. This technology is based on a variety of techniques (artificial intelligence, statistics, information theories, databases, etc.) that require diverse skills at a high level. The success of this discipline has grown with the size of the databases. We can say that with the rise of the phenomenon of Big Data we have now entered a phase of mastery of the discipline. The challenge of Big Data for companies now is not so much the capacity for analysis but rather two issues that tend to be ignored [GAU 12]: – data collection methods must remain known and controlled to ensure that data mining analysis does not produce any counterproductive effects for the company; – the analysis of large amounts of data must not be done at the expense of their quality. Not all of them have the same purpose and do not add value to the company. The types of models that can be discovered depend on the tasks of data mining used. There are two types of data mining tasks: descriptive tasks that describe the general properties of existing data, and predictive tasks to make forecasts on available data. The data mining features and the variety of areas of knowledge explored are: – description: the importance of this task is to allow the analyst to interpret the results, either from a data mining model or an algorithm, in the most efficient and transparent manner. Thus the results of the data mining model should describe clear characteristics that can lead to an interpretation and an intuitive explanation;

Data Development Mechanisms

69

– estimation: the main interest of this task is to arrange the results in order to retain only the most valuable information; this technique is mainly used in marketing in order to offer deals to the best prospective clients; – segmentation: this consists of, for example, allocating customers into homogeneous groups which should then be addressed by specific means adapted to the characteristics and needs of each group; – classification: this concerns aggregating data or observations of groups with similar objects. This separates the group of data to form homogeneous subgroups. These are called clusters, which are classes in which their data is similar to each other, and, by definition, different from other groups; – prediction: the results of the prediction are unknown, which differentiates it from estimation; – association: this function of data mining allows for discovering which variables go together and what rules will help quantify the relationships between two or more variables. However, the heterogeneity of data sources and their characteristics means that data mining alone is not enough. With the evolution of the Web (the transition towards the Semantic Web) there has been an explosion of textual data; this is unstructured data which raises several possibilities for companies who cannot ignore their existence and their impact on the ecosystem (suppliers, competitors, and customers). The Web is another factor that justifies the extraction of knowledge from texts. Indeed, with the Web, unstructured data (such as text) has become the predominant type of online data. In this context, useful information is not only found in quantitative numerical data but also in texts. The tools for accessing and collecting textual data must be equally capable of operating from the Web on HTML documents as on databases, either bibliographical or textual. This is the analysis of text, or “text mining”, which is a technique of extracting knowledge from unstructured documents or texts by using different computer algorithms.

70

Big Data, Open Data and Data Development

This means modeling linguistic theories using computers and statistical techniques. The use of information technology to automate the synthesis of texts is not particularly recent. Hans Peter Luhn, a researcher at IBM, and the real creator of the term business intelligence in 1958, published a report titled: “The Automatic Creation of Literature Abstracts” [LUH 58]. This exciting study is directly available from the IBM website. Text mining requires firstly recognition of the words or phrases, then to identify their meaning and their relationships to then be able to process, interpret, and select a text. The selection criterion is divided into two types: – novelty: this consists of discovering relationships, notably the implications that were not explicit (indirect or between two distant elements in the text); – similarity or contradiction: in relation to another text or in response to a specific question, this consists of discovering texts that best match a set of descriptors in the original application. Text mining generates information on the content of a particular document; this information will then be added to the document thus improving it. The main applications of text mining is recounting surveys and data analysis projects for which certain responses come in an unstructured or textual form (for example electronic messages, comments, suggestions in a satisfaction survey with open questions, the description of medical symptoms by patients of practitioners, claims, etc.) which are best integrated into the overall analysis. In the background is the data mining process of Statistica, on the left is the database and on the right the different methods in the sphere of data mining.

Example 3.6. A base of e-prospects with Statistica as an example of data processing

Data Development Mechanisms

71

These techniques are often used to produce predictive models to automatically classify texts, for example to direct emails to the most appropriate recipient or to distinguish between “spam” and important messages automatically. Text mining thus facilitates: – automatic classification of the documents; – preparation of a document overview without reading it; – supply to databases; – ensure monitoring on documents. Two approaches can be considered: – the statistical approach: produces information on the number of uses of a word in a document; – the semantic approach: relies on an external element which is the repository, which can be static (keywords) or dynamic; it implements logic (i.e. information that is deduced from the repository). Text mining is a technique which automates the processing of large volumes of text content to extract the key trends and to statistically identify the different topics that arise. Techniques of text mining are mainly used for data already available in digital format. Online text mining can be used to analyze the content of incoming emails or comments made on forums and social media. The demand for different types of “data mining” can only increase. If this demand develops, it will steer research in the field of “data mining” into digital, textual, images, etc., and on the development of viable systems. This is an essential consideration that companies will have to develop. The procedural methods of mining and of updating models – with a view of automating the decisions and decisionmaking – must be designed in conjunction with data storage systems in order to ensure the interest and usefulness of these systems for the company.

72

Big Data, Open Data and Data Development

3.5. Conclusion The Big Data phenomenon has changed data management procedures because it introduces new issues concerning the volume, transfer speed and type of data being managed. The development of the latest technologies such as smartphones, tablets, etc., provides quick access to tools; the Internet becomes a space for data processing thanks to broadband. As previously mentioned, this phenomenon is characterized by the size or volume of data. The speed and type of data are also to be considered. Concerning the type, Big Data is often attached to unstructured content (web content, client comments, etc.) which presents a challenge for conventional storage and computing environments. In terms of speed, we can handle the speed at which the information is created. Thanks to various new technologies, it is now possible to analyse and use large masses of data from website log files, analysis of opinions on social networks, video streaming and environmental sensors. Most of the data collected in the distributed file systems are unstructured, such as text, images or video. Some data is relatively simple (for example calculating simple numbers etc.) where others require more complex algorithms that must be developed specifically to operate efficiently in the distributed file system. The quality of data affect the merits and appropriateness of strategic decisions in the company, hence the need to consider the “information value chain”. All the actors in the information value chain are strongly aware of the identification – and especially the reporting – of quality defects; the solution is a communication campaign and tools for catching any quality defects of data. The data governance approach is a pragmatic approach that formalizes and distributes data management responsibilities throughout the value chain.

4 Creating Value from Data Processing

Today, as long-term economic growth depends on the accumulation of knowledge and the ability to introduce new products, processes, services, and business and organizational models, business competitiveness is determined by a company’s ability to organize a space that is beneficial to science, technology and innovation. With the phenomenon of Big Data, data is now central to a company’s decision-making process, and is a new precious tool for directing their strategic vision. In a changing world, data – increasingly large, rich, and varied – represents a major asset for businesses provided they can capture, combine, and analyze it. Data can be seen as an asset if it has the potential to create value for companies. These can predict possible outcomes by studying both structured data and the continually expanding mass of unstructured data from different sources, and then by sharing those results to discover any relevant correlations. By publishing reusable data, public and private organizations are thus supporting the development of new services as well as the content within them. The availability of more and more “open data” is important. Open Data is a broad movement of opening up data, initiated in the USA during the “Freedom Act” which, like Google Maps, Amazon or eBay has revolutionized how we consume data. This openness will lead to new applications and promote innovation with new tools for the value-creation of these data.

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

74

Big Data, Open Data and Data Development

Faced with the numerous challenges of opening up data, French and European public cultural institutions initiated some early publications using the latest models in “web semantics” or “data web”. Data is “the new global currency” and the Web is the currency exchange office. Companies have understood that the right decisions are made by the proper management of their data. The integration of Web Semantic technologies in data management systems is one of the approaches that bring new perspectives. The processing and understanding of increasing amounts of data; and their transformation into relevant information is one of the challenges posed by the Internet and the Web. But the design of information and its display or presentation is a growing concern for web experts. A smart web is one that represents an intermediary of digital tools associated with a digital display interface. This allows for graphical representation that facilitates decision-making. This is where there is the need for “data visualization” which is optimal and suited to the processing of Big Data. “Data visualization” revolutionizes the use of data and offers a simple and efficient way to access information from the mass of data produced. 4.1. Transforming the mass of data into innovation opportunities With the emergence of the information society from the 80s, we are witnessing the transition from the material economy to the virtual economy. The change reduces the importance of raw materials and physical presence and leads instead to increasingly immaterial products. This new “postindustrial” economy is based on intelligence, knowledge, and innovation, and is materialized through the development of these services. This gradual transformation of society organized around production, the circulation and exchange of knowledge simultaneously affects the modes of production and consumption, sources of growth, and competitiveness, modes of organization and management of companies, the skills building process and acquisitions of new qualifications for human capital. This confirms that knowledge is the most crucial

Creating Value from Data Processing

75

resource for any company; it is a strategic resource. By observing the knowledge pyramid we find that “data” is the important pillar of knowledge. Data represents an infrastructure resource, an “input”, which – in theory – can be used by an unlimited amount of users to generate numerous applications. In this context, whilst the borders of economic life are evolving, the most powerful companies are most likely the ones who successfully managed to take advantage of the available amounts of data. The ability to use this data can bring value to the economic activity. The proliferation of data created by individuals, companies, and public authorities will support new uses and gains in productivity. The scale of the Big Data phenomenon can, therefore, be seen as an economic value in itself. Companies that are highly innovative are most likely to rely on large data analysis and data mining; this was confirmed by the latest analysis of the world’s most innovative companies, published in 2014 by the Boston Consulting Group (BCG). From 1–10

From 11–20

From 21–30

From 31–40

From 41–50

31

Proctor & Gamble

41

Fast Retailing

1

Apple

11

HewlettPackard

2

Google

12

General Electric

22

3m

32

Fiat

42

Walmart

3

Samsung

13

Intel

23

Lenovo Group

33

Airbus

43

Tata Consultancy Services

4

Microsoft

14

Cisco Systems

24

Nike

34

Boeing

44

Nestlé

45

Bayer

5 6 7 8

IBM

15

Amazon Tesla Motors Toyota Motors

Siemens

21

Volkswagen

25

Daimler

35

Xiaomi Technology

36

Yahoo

46

Starbucks

16

Coca-Cola

26

General Motors

17

LG

27

Shell

37

Hitachi

47

Tencent Holdings

18

BMW

28

Audi

38

McDonald’s

48

BASF

9

Facebook

19

Ford Motor

29

Philips

39

Oracle

49

Unilever

10

Sony

20

Dell

30

Soft Bank

40

Salesforce.com

50

Huawei Technologies

Table 4.1. The 50 most innovative companies in 2014 Source: 2014, BCG Global Innovator Survey

76

Big Data, Open Data and Data Development

The success of large companies such as Amazon, Google, Facebook, etc. proves the emergence of a fourth factor in the development of today’s hyper-connected world. Aside from raw materials, labor and capital, data has undoubtedly become an essential element in gaining a competitive advantage. But their production into mass online data and in different digital forms indicates a new era of Big Data, just like the new technologies which enable their analysis. For network companies in particular, the use of data allows for the unprecedented optimization of business (ability to prevent network failures, service interruptions, cancellations of subscriptions, etc.) the development of “smart services” and the creation of value from data from third parties. On this last point, as an example, the three largest telecommunications operators in the United Kingdom have formed a joint company called “Weve” to sell anonymous data from their clients (purchase data, geolocalisation data, internet data etc.) to advertisers.

50

Behavior=Belief

Technology (software)

40 Technology (IT services) Insurance

30

Telecommunications Financial services Chemicals

Energy

Pharmaceuticals, biotechnology and healthcare Technology (hardware)

20

Consumer products Retail Industrial products and processes Automotive

10 10

20

30

40

50

Figure 4.1. Companies actively targeting Big Data in their innovation programs (over the next 3 – 5 years). Source: BCG (2014)

60

Creating Value from Data Processing

77

In the same light and in the medical field, a study by the McKinsey Global Institute suggests that if the US health care sector knew how to process the vast amounts of electronic data that it has access to (from clinical trials and experiments), then a better management of this data would improve the system’s efficiency and save them more than $300 billion annually. According to BCG, 53% of companies confirm that Big Data will impact their ability to innovate over the next 3 to 5 years. The graph below shows companies actively targeting Big Data in their innovation programs. BCG has also found that business leaders who process large amounts of data generate an income of over 12% more than other companies that neither experiment nor gain value from Big Data. Companies analyzing and processing mass data (internal/external, structured/unstructured) are much more likely to be innovative. In order for businesses to take advantage from the value created by Big Data, they must: – examine data: since the company possesses a large amount of data from various sources, it must understand the relationship between them. At this stage the key attributes in the analysis must be sufficiently standardized to generate valid results; – process the data: once data has been well documented and is of a sufficiency quantity, what remains is to draw out the key information. This requires comparing them in order to obtain results that will guide the company in their strategy. At this stage, the company must analyze data using a system with processing power that allows for extracting value; – transform data: after having processed data, that company must use a tool that is capable of handling different data sources, including unstructured data, and to transform them into useful and readable information.

78

Big Data, Open Data and Data Development

Examine

Processing

Transform

Structured data Cross

Unstructured data

Figure 4.2 Massive data processing and results visualization

Big Data allows for recovering data hitherto unknown but useful for inspiring new ideas that can lead to new discoveries and thus fuel the cycle of research an innovation. The applications of Big Data are numerous and are a main factor in strengthening the capacity for innovation within companies by playing the two dynamics of exploration and processing. Innovation will certainly stem from combinations and processes that were not originally thought of. For example, the correlation that Google identified between a handful of search terms and the flu is the result of 450 million mathematical models. In the same spirit, the United Nations has developed a program anticipating epidemics and economic downturns through keywords used on Twitter. The Danish company “Vestas Wind Systems”, one of the most influential wind turbine manufacturers in the world, uses “IBM Big Data Analytics” and “IBM Systems” solutions to decide the location of wind turbines within a few hours by crossing varied data such as weather data, satellite images, etc. The exploration of large amounts of data enables the launch of new products and services, new processes, and even new business models. By making data speak each company will have access to a better understanding of the context and its environment.

Creating Value from Data a Processing

79

In the t automottive sector, cars are in ncreasingly equipped e w with sensorss and softwarre that allow w them to anaalyze their ennvironment aand act acccordingly. Thhe car can be customized by integraating and usiing data; annd becomingg more conneected, even without w a drivver.

h been testedd The Google Car has on American roadds since 20100. The software usedd was perfectedd by machine m learniing tools, usingg dataa collected from m hundreds oof thou usands of kilom meters of testing.

Example 4.1. 4 The Goog gle car

Thee collection and use off data on connected c products or on consum mer behaviorr can improove the opeerational funnctioning off a companny and foreccast any marrket develop pments. “Lokkad”, a Frennch start-upp in softwaree publishing,, has develop ped an algoriithm that hellps retailerrs to optimize their dailyy inventories by analyzinng receipts, aand sales hiistory. In terms t of “Urrban Data”, there are ass many citiees as there aare modelss. Many larrge-scale urrban innovaation projectts have maade makingg “smart” infrastructuress and integrating sensors and increasiing networkk capacity thheir main puurpose. The Big Data tecchnologies ccan help citties meet theeir most presssing challeng ges. As prooof, the cities currently mostt advanced in this area in Europe E (Helsinkki, Berlin,, Barcelona, Bordeaux, and Rennes) R have all a been able to t implement aan originaal strategy takiing into accouunt their historrical, demograpphic, and urbaan qualitiees. Box 4.1. Metropolise es

80 0

Big Data, Open Data and Data D Developme ent

The term “smaart” used here iss not restricted to thhe concept of intellectual com mmon sense. Sm mart cities are sustaainable because they respect the envvironment and ddevelop opportunities for f global grow wth through innovaation. Accordinng to Giffinger, a professor from thhe Vienna Univerrsity of Technollogy, 6 criteria are neccessary for recoognizing a smart city: – transparent and a participatorry government; m off – sustainable management resource allocaation; – social cohesiion and quality of life; – openness to learning l from nnew citizens; – labor markett for perpetual opportunities; – safe and ecollogical transporrt. Example e 4.2. Smart City C - Montpelliier

The “smartt city” is equually deployed thanks too Open Data and the im mplementatioon of an ecosystem of program mmers and mobile appplication developers. m transsport of Thee company managing Greeater Manchester, “Transport for Greeater Manchester”, uses the W Windows Azu ure Platform too host collectedd public dataa. It is now poossible in real time to kno ow not only the t location off public tran nsport vehicles but also the nuumber of avaailable places on the mosst used rou utes1. Example 4.3. An n application on o a transport company

1 http://www.micr h rosoft.com/casesstudies/Case_Stu udy_Detail.aspxx?casestudyid=71000003 034.

Creating Value from Data Processing

81

Open Data can promote innovation processes, open firstly between businesses and public enterprises which hold usable data, and secondly start-ups or SME suppliers of innovative technologies. This represents an opportunity for radically new solutions for all stakeholders. Big Data is therefore part of “open innovation” approach that encourages organizations to collaborate with other actors outside of the traditional company boundaries, on either strategic or unexpected topics, in what is deemed “open innovation outside-in”. It also requires reflecting on the value creation from projects and produced data, in which the company has no plans to capitalize directly into its core activity of “open innovation inside-out”. Beyond the huge volume, it is the diversity of data sources that gives Big Data its full scope. But the development of new products and services and their adaptation to new uses are facilitated by the mixing of large data sets. Big Data marks a major turning point in the use of data and is a powerful vehicle for growth and profitability. A comprehensive understanding of a company’s data, its potential and the processing methods can be a new vector for performance. All companies are overwhelmed with data but do not often know how to use them appropriately. For data to lead to innovative opportunities it is necessary to have significant computing and storage capabilities. Storing data online makes them available without temporal or spatial constraints, which is the first technological process that is essential to Big Data processing. Another essential factor, which also allows explain data, is the power of formulas and calculations. Data stream and analysis processing tools develop their power every day (such as Cloud). The Big Data revolution that extends the potential of “Machine Learning” is an opportunity for change. Big Data allows for an understanding of the heterogeneity of data sets, and adjusting the analysis based on data in real time to enable a more appropriate response. The disruptive nature of the innovation introduced by the dissemination of Big Data into an increasingly broad range of industrial fields opens up important opportunities for creating value

82

Big Data, Open Data and Data Development

and differentiation. Sectors such as insurance, retail, automotive manufacturing or even energy have experienced significant changes in their value chain through the influence of Big Data. There are several values (monetary value, usage value, reusage value, etc.) and depends on those who use data. 4.2. Creation of value and analysis of open databases The OECD said in a 2013 report [OEC 13] that the convergence of technological and socioeconomic trends related to the development of internet usage and the decreasing management costs of dematerialized data leads to the creation of a large amount of data that can be used to create new products, new wealth, and new value to business. The OECD identifies five sectors driving value creation: – improved – the

research and development;

creation of new products based on data;

– optimization

of the manufacturing process;

– optimization

of targeted marketing;

– improved

managerial approaches.

It therefore seems evident that data must be considered as an intangible but essential asset to the company. The increasing production of data, generated by the development of information and communications technology (ICT), requires an increased openness as well as a useful division which can be harnessed into changing the economic and social world. The birth of Open Data did not come from chance; rather it is completely linked to the Internet and its mutations. The development of digital technology and the potential uses for data calls for the acceleration of this movement; more data must be shared freely, in open formats, with freedom for reuse. The opening up of data, or the phenomenon of “Big Data”, has spread throughout the world despite being new thanks to its ability to generate both an economic and social value. For this, various state

Creating Value from Data Processing

83

actors are actively involved in this development by allowing access to and reuse of their data. Open Data signifies that the public sector relinquishes its role as guardian of data and replaces it with a new role, of provider of information and data, leading to a realignment of power dynamics between the public and private sectors. This phenomenon has attracted considerable attention in recent years; one of the central principles of the open data initiative is to make public data available in order to promote innovation from these data. The question that arises is: how can we create value or profit from simply reusing or transforming these open data? Data deposits are a raw material that revolves around a cycle consisting of suppliers that emit data to consumers, who – through a transformation process – will produce information and therefore value. But to generate value, these data must be available to everyone. In 2007, the working group “Open Government Data” defined the eight principles of data accessibility; data is considered open if it is: – complete; – primary; – appropriate; – accessible; – usable; – non-discriminatory; – not

privately owned;

– free

of royalties.

To advance Open Data we must recognize this variety of conditions for opening data. We, therefore, potentially have very large volumes of data concerned, with very diverse characteristics that can be mixed with other types of data – from social networks for example – to enhance applications or make new ones. This is possible through data analysis and processing. The relevant data mixing is the basis for all applications based on Open Data.

84

Big Data, Open Data and Data Development

The civil society initiative “Where does my money go?” The site, developed by the Open Knowledge Foundation, allows for visualizing the areas of British public spending.

Example 4.4. “OpenKnowledge Foundation”

In a similar vein and in order to best use Open Data, the Open Data portal data.gouv.fr in France provides more than 350,000 data sets through the “Etalab Mission Open License”. The released data are very varied and can generate interesting applications.

Home

Map

Tools for Urban Mobility

Access to the map

Help

contact

about

By combining data and mixing them with specific points of interest (bus stops, etc.) the city was able to propose a route calculation service for those with reduced mobility. This gave birth to handimap.org, accessible online and on mobile. This approach allowed for using data that had been privately reserved at the lowest cost, in turns, helping enhance the national image of this city as an innovative territory.

Example 4.5. A route calculation service for people with reduced mobility

Creating Value from Data a Processing

85

In March M 2010, the city of Rennes R made some of itts data publiccly availabble on the platform p datta.rennes-metropole.fr. From F this daata stemmeed 60 servicees on fixed and a mobile In nternet providers on them mes as varieed as transpoort, accessibility, environment, and cuulture.

Transport of people

Parisian region

Traffic Management

Informati on to users

Imp proved

Open Data

Concerns

General public

Move arround the tra ain easily Website General public

Improved

Quality Optimize

Personal

Subscriptions

Material

Traffic Management

Ticket

In n order to improve thhe in nformation provided to itts cu ustomers, SNCF F has developeed an n application thhat allows for thhe crreation of value througgh reeusing open daata, through thhe ap pplication “Trannquilien” starteed in n 2012. By using u predictivve models, m this Appp tells customerrs which w trains have h the most av vailable seats. The predictionns from “Tranquilieen” are updateed in n real time by ussing informatioon su upplied by paassengers. Thiis im mproves the seervice and helpps prredict the impacct of new eventts on n the network.

Exam mple 4.6. The SNCF S and the e reuse of data a

By the end of 2011, Orange hhad launched a weebsite “Where ddo you really livee?” in associatioon with OWNII, Fing, annd Everydatalab. This site invittes o the land theey users to test out live on by using u tests wiith publicly availlable data. It w was hugely succeessful, receivinng more than 20,,000 users in leess than 2 monthss. The generaliization of thhis practice will undoubtedly u leaad to an accelerrated release of data sets annd an aim to improve their quality. Example 4.7 7. Orange and the site “Whe ere do you really live?”

86

Big Data, Open Data and Data Development

The term Open Data refers to public or private data that can be reused. The concept of opening up data refers to the “general public” dimension, this is to say that all people can consult a dataset in a clear and simple manner. This is free data that generates profit through reuse. However we must first define a theoretical framework that explains the impact of data availability. This is what G. Sawyer and M. de Vries presented to the “European Space Agency” based on empirical modeling [SAW 12]. This modeling identifies three phases during which the opening of data can vary: – a “sowing phase”: opening data is a cost to the administration; – a “growing phase”: the number of re-users grows, public services gain efficiency and the reuses begin to generate profits for the company; – a “harvesting phase”: the effects of data opening are felt on employment and public finance activities.

Figure 4.3. The 3 phases of opening up data. Source: [SAW 12] for the European Space Agency. For a color version of the figure, see www.iste.co.uk/monino/data.zip

Creating Value from Data Processing

87

The reuse of data will help to enrich these data and mix it with other existing data, and thus, offer new products and services. Open Data allows for creating a partner ecosystem. Opening this data is a big step towards co-production of new activities for companies. Data becomes information, and then knowledge, which can be a source for innovation by mixing it with other data. As such the challenge for companies is to manage data value chains. Open Data has thus quickly become a growing sector, attracting many companies and startups whose mission is to process, sort and analyze data to give it meaning and render it usable. The new connected objects from the Web 3.0, the “quantified self” (watches, bracelets, smart cars, etc.) will produce data to be shared with those who wish to enrich it or aggregate it. Combined with different technologies, these data are also an important vector for innovation, enabling the creation of new services. Initiating Open Data is a selection of raw data making easily available and reusable: public data, scientists, citizens, rising companies, etc. Improving the analysis, products, and services are the main aims of companies who use and reuse open data. The decision to initiate an Open Data approach must be taken with regards to the potential benefits it can bring to a company, working closely with the objectives of its digital, and innovation strategies. Current initiatives for opening up data, such as data.gouv.fr, represent an important step in the implementation of the web of data. But, there is still much to be done before reaching the famous “five stars” defined by Tim Berners-Lee: structured, identified data in a non-proprietal format, and linked together semantically. 4.3. Value creation of business assets in web data Data, structured or unstructured, which has grown exponentially in volume and variety within companies holds the most promising value when it interacts with the gigantic mass of data from the “Web”. Additionally, if we add the phenomenon of “Big Data” which we have already have spoken about, to the phenomenon of “Open Data” –

88

Big Data, Open Data and Data Development

which makes data accessible – and even “Linked Data” – which allows for connected data from different sources – then the possibilities are even larger. The Web promises to be a goldmine for companies. The Web has become the preferred location for data research, and it has put many types of data online that are potentially useful for a company’s monitoring system. The Web has revolutionized the way of capturing, accessing, and integrating data in order to use it. Indeed, a large amount of data available online contains potentially useful information for decision-making. The growing volume and variety of information poses real obstacles to effectively accessing this goldmine. When we use a search engine it does not understand our request. It finds the pages which feature out keywords but does not guarantee a real response. Various data research and collection techniques on the Web have been proposed to build Tools that refine the search for relevant results. There is a need to build a space for exchange of data between machines, allowing access to large volumes of data, and providing the means to manage it. In order to understand the concept and value of data, it is important to consider the existing data exchange and reuse mechanisms on the Web. We therefore need to reach a “smart” web, where data is no longer stored but “understood” by computers to provide users with what they are really seeking. The result is a global data space that we call the “Web of Data” or “Semantic Web” which Tim Berners-Lee started using between 1991-01. The concept was again taken up and developed into the “World Wide Web Consortium (W3C)”. The semantic Web provides a new platform for a more intelligent management of content through its ability to manipulate resources based on their semantics. A machine can understand the amount of data available on the Web and thus provide a more consistent service, so long as we empower it with certain “intelligence”. In fact, the integration of the Semantic Web is not a new idea but is in fact born with the Web [FAL 01, MOT 01]. The Semantic Web is the starting point for the development of smart web services.

Creating Value from Data Processing

89

The vision of the Semantic Web proposed by Tim Berners-Lee [BER 01] can be summarized as follows: the development of the Web was first made possible by the development of standards such as TCP/IP (transmission and routing bits across a network) or http:// and HTML (transport and rendering of Web pages and hyperlink texts). The first generation of the Internet consisted essentially of handwritten pages. The second generation offered automatically generated content. On the basis that these two architectures are dedicated solely to human–human or human–machine interactions, the Semantic Web initiators defend the idea that the next Web generation should promote access to resources so that they are automatically processed by software agents, in other words: machines. In order to offer the automatic processing capacity, the Semantic Web adds to the data a layer of metadata and makes them usable by computers. These metadata provide unambiguous semantics to automize the processing. The architecture of the Semantic Web is based on a pyramid of languages proposed by Tim Berners-Lee to represent knowledge on the web which meets the criteria of standardization, interoperability, and flexibility. The following points introduce the main function of each layer in the architecture of the Semantic Web: – XML: it is now considered a standard for the transmission of data on the web; – Resource Description Framework also known as the RDF layer: this represents the metadata for Web resources; – the ontology layer: it is based on a common formalization, and specifies the semantics of metadata provided in the Semantic Web; – the logiclayer: it is based on inference rules that allows for intelligent reasoning performed by software agents. The origins of this Web are based in the research efforts by the community of Semantic Web researchers and especially on the project

90

Big Data, Open Data and Data Development

Linking Open Data2 (LOD) from W3C, started in January 2007. The original objective of this project, which generated an active and still expanding community, was to start the data Web by identifying sets of existing and accessible data under open licenses and to convert them into RDF in line with the principles of connected data and to publish them on the Web [HEA 07, Chapter 3]. In the universe of the Semantic Web, it is not enough simply to put data on the web, it is especially important to create links between them so that this web of data can be explored. Linked Data is a subproject of the Semantic Web, which helps find related data. For Tim Berners-Lee, related data is a very important change, for which he sets out four basic principles: – use of URIs, “Uniform Resource Identifier” instead of a URL to identify what we want to make available on the Web as data resources; – use of HTTP addresses to locate them and access their content; – provide useful information about the resource (URI) when consulted; – include links to other Uris related to the resources used in order to improve the discovery of useful information. Data binding allows for the browsing and sharing of information between different sites. Connecting a database to the Web is establishing a link between a web server and a database management system. This link leads to processing techniques that are associated with the computing world and bound to the “client/server” framework, putting at risk web oriented programming languages, communication protocols, data manipulation languages etc. Large amounts of structured data have been published, notably in the framework of the “Linking Open Data” project3.

2 http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData. 3 http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData.

Creating Value from Data Processing

91

The interest of linked data is to identify and typify the relationships between entities, using web standards that are favorable to be read by machines and that reveal interconnections that would be impossible to produce through solely human intervention. Linked data refers to a set of best practices to implement to publish and link structured data on the Web. Linked data are a set of design principles for the sharing of machine-readable data for use by public administrations, businesses and citizens [ECI 12]. The concept of linked data corresponds to a more technological vision of data related to the Semantic Web. It consists of a collection of data sets published using Semantic Web languages and links between data sets, allowing for multiple queries on data sets from one individual search. Many tools are beginning to appear for data storage, for visualizing data in varied ways, offering visualization interfaces with maps.

Time

Figure 4.4. Evolution of Linked Open Data. For a color version of the figure, see www.iste.co.uk/monino/data.zip

The developments of semantic technologies and the evolution towards the Web of data have led to changes in the representation of

92

Big Data, Open Data and Data Development

terminologies and formats of metadata, and to new opportunities for machines to reason on the basis of terminologies. These metadata databases are increasingly large and can be used more remotely and directly via web services. Technologies of the web of data demonstrate the immense prospect of pooling, sharing and utilization of metadata on the web. Some metadata are already implemented. The algorithms “Page Rank” and “Trust Rank” by Google use metadata concerning web structure. In addition, the search engine “Bing” from Microsoft is beginning to interpret and contextualize requests in natural language through the use of the technology “Powerset”. Facebook has also introduced the “Open Graph Protocol”, a technology based on the Semantic Web which allows third party sites to interact with the social Networking site by the receiving and sending of information to Facebook. This new protocol is based on the RDF syntax and aims to “sustain” the social interactions between visited sites and the Facebook profile of an Internet user4. The website “e-retailer Best Buy” was one of the first to enhance its search capabilities with the opportunities presented by the Semantic Web, via the integration of the RDX syntax with officials’ blogs representing each of its stores. The aim is to make the essential data of its stores more visible (contacts, opening hours, product pricing) by enabling search engines to better display them. In companies, the applications that stem from the Semantic Web are currently mainly represented through software research Tools and data structuring. These include “Exalead” and its “Cloud View” Platform which collects, analyzes and organizes large volumes of data (e-mails, social networks, blogs, etc.) whether from fixed terminals, mobile systems or the web. Its objective is to enable companies to scale their information needs to their specific issues and areas of activity.

4 http://www.commentcamarche.net/faq/29732-web-semantique-quelles-applicationsaujourd-hui.

Creating Value from Data Processing

93

It is worth noting that the Semantic Web is an approach to the management of corporate knowledge in which data is given a clear definition to allow computers and users to work in cooperation. The Semantic Web is not only used in intelligent searches of information, but is also studied in research on knowledge engineering, knowledge representation systems, automatic processing of natural language, learning, intelligent agents, automated reasoning, etc.

Figure 4.5. Web of the future Source: N. Spivack, “The Future of the Net”, 2004, available at http://novaspivack.typepad.com/nova_spivacks_weblog/ 2004/04/new_version_of_.html

The expressions “Semantic Web” or “Web of Data” refer to a vision of the future Web as a vast space of exchange of resources supporting the collaboration between humans and machines, in order to use large volumes of data and the various available services on the web more efficiently. This new web generation reflects the twopronged dream of its creator Tim Berners-Lee. On one side the web becomes a very powerful means of cooperation between human beings, and on the other side it is also a means of cooperation for “machines ” (computers)5.

5 Tim Berners-Lee, in an interview with UNESCO in 2000.

94

Big Data, Open Data and Data Development

4.4. Transformation of data into information or “DataViz” Nowadays the different uses of the Internet, social networks, Smartphones, and other connected objects result in the production of billions of data that can be stored, analyzed and used. The advent of High performance processing technologies can enable a systematic analysis in real time of this enormous potential of data. This is however increasingly complex to interpret and it becomes increasingly pertinent for companies to invest in this approach which generate value and income. First a team of “Data Scientists” must identify the right data sets, have access to these data sets, develop a mathematical model to address this, and finally start generating ideas. Once this work is completed, the team delivers the results to “top management” in a report or spreadsheet, which can be hard to understand. In fact, the results are not just the numbers of the report they are also the decisions and hypotheses that were issues throughout the previous work. These results require a debate in which, currently, many top managers would not be able to participate because they do not have access to the necessary information in an understandable format. Therefore, leads can carry on strategic decisions for the company without even realizing what they may have missed. This is why Big Data must be accompanied by a visualization of the processed results to better succeed. In addition to the collection, processing, and analysis phases, which represent a cycle of new information and which must be accompanied by Big Data to most effectively create value from data deposits, what other phase is the most important? The answer to this question lies in the “visualization of data” which is the most transparent, intuitive, and contextual manner in which to use data beyond simple numbers. The need for data integration for the success of Big Data analytics, is therefore, undoubtedly the answer. Companies have an incentive to move towards analytical solutions and must complete the data value chain from the capturing of data to

Creating Value from Data Processing

95

its presentation and dissemination. Data visualization provides a common language for executive directors, data scientists and top managers, and thus, allows them to have a conversation on data. These three groups usually have “different business languages”, and data visualization will replace these differences with a visual dialogue that everyone will understand. William Playfair confirms this idea by noting that: “instead of calculations and proportions, the surest way to hit to touch the spirit is to speak to the eyes”. Visual data analysis, is thus, based on three essential elements: – data visualization; – data analysis; – data management. “Data visualization” can be used both as an exploratory model to find patterns and extract knowledge from processed data, and as an explanatory model to clarify and illuminate relationships between data. Through the visualization of data, companies can take advantage of the real value of Big Data by accelerating the understanding of data and allow leaders to take quick and decisive action on business opportunities. As the volume and variety of data increases, the visualization of data becomes increasingly important to stimulate a collaborative dialogue. When faced with an enormous amount of data, visual grouping can bring together points of measurement that can help decision makers understand the relationships between data, and thus, make effective decisions. A recruitment team can identify and justify the objectives of the recruitment for the sales department by using a network graph from a professional networking site. This type of data visualization immediately relays the candidate’s position in the industry, based on previous business relations. With this information, the recruitment team can act with more determination and hire the best sales executives. Box 4.2. Data visualization

96

Big Data, Open Data and Data Development

Information presented in a visual format is also more likely to grab people’s attention. As reported in Bistro media, people Retweet infographics much more than articles or images6. If a company must reply to a question, or has an article to deliver, a well designed data visual can make the message more compelling and also boost their ideas to a much wider audience. Data visualization must be regarded as the result of a carefully considered process, which understands the capture of data quality to allow for good analysis results, and the effective communication of these results throughout the process. In order to process Big Data more efficiently we must have visualization functions at every stage of the analytical process, namely: – collection and preparation of data stage: the combination of various data sources is a key step in the data analysis process. A company must be able to visualize this process with a tool that allows for verifying that the data assembly mode accurately reflects the significance of data sources. The more a company’s data source is varied, the more it needs visualization to know precisely what data it has available and to help it understand how they can help solve the initial problem; – modeling stage: visualization is extremely important in modeling, notably because in most cases the model must be adjusted according to the different issues; – deployment stage: many tool sets only allow for visualization during the deployment stage. Visualization here plays a crucial role: the analysis is embedded in an application where it generates value for a variety of users and provides them with the information necessary for their work. Data visualization is a technique for exploring and analyzing data. It is an excellent solution to address the growing complexity of available data. Data visualization is a tool to visually represent data, in

6 Media Bistro, All Twitter, August 15, 2012.http://www.mediabistro.com/alltwitter/ infographics-on-twitter_b26840.

Creating Value from Data Processing

97

an illustrated form, that makes it more readable and understandable. As such, there are three broad categories of data visualization7: – visualization of “fixed” data: static infographics, representing the targeted data (illustrations, typography effects, or photographs, “photoviz”). This category has the advantage of being completely adapted to the context of social network use, including those with high potential to go viral such as Facebook, Twitter, Google Plus, FlickR, etc.; – visualization of “animated” data: screenwriting and storytelling can rely on videos, thus, diverting the animation codes to servicing the information; – visualization of “interactive” data: possibility for the user to “play” with these and help making decisions. If the aim is to make a decision, we are more likely to use an interactive “data visualization” which enables the display of large amounts of information with specific details on the numbers or nuances. The explosion of computing power currently allows for undertaking more pieces of information. To extract important information from such volumes of data, companies must have visualization tools in order to interactively explore and analyze data flows. So, data visualization cannot be processed without addressing the interaction. Visualization aims to produce actionable visual representations for analyzing and understanding complex data. But interactivity is a key point of visualization to better understand what is observed. This interactivity is all the more important when the volume of data to explore is significant. An increase in the volume of data to process and display, however, constitutes a natural halt to this interactivity and demands much more consistent calculation efforts. The amount of data that must be visualized is a direct result of the exponential growth of available computing power.

7 http://www.jouve.com/fr/la-data-visualisation-dataviz-dans-tous-ses-etats.

98 8

Big Data, Open Data and Data D Developme ent

Once the analysis a is finnished, the reesults must be b clearly preesented. Thhis is to enabble decision-making by removing thhe complex ppart and beeing as clearr as possible about the isssues and chhallenges. Thhe many caalculated fiellds representt the consideerable volum mes of data thhat must firrst be storedd then processed through h an interacttive visual aanalysis. Developmentss in IT have enabled a laarge capacityy for data coollection us power over visuaalization annd generatioon but alsso enormou techniques. Knowledge K abbout visual perceptions p i humans ccan help in prrovide practical solutionss to some ideentified probblems. Humaans have thhe ability to visualize higghly develop ped informaation and whhich can pllay a major roole in their cognitive c pro ocesses.

The “Tag Cloud” is a visual representation of the mosst used website. keywords (taags) on a w Generally, thee words are diisplayed in a larger foont depending oon how often they are used. u

Example e 4.8. Clouds of o texts or worrds

The increaase in compputing poweer and of thhe quality oof GUIs (ggraphical useer interface) has allowed d for explorring and devveloping innteractive tecchniques thaat make bestt use of the available daata. The innteraction in real time ennables easy navigation within w the daata area annd thus a bettter understaanding of thee structure annd characteriistics of thhis space; thiis is what chharacterizes the t exploratoory phase annd helps im mprove our knowledge. Thereafter the user can formulatte more effficient queriies in the seaarch system.. It is at the level of inteeraction thhat we can im mprove the coollaboration of visualizattion techniquues, and daata research and a processinng techniquees.

Creating Value from Data Processing

The contribution of the internet sector to the GDP of G8 countries

Inactive map of cyberthreats produced by Kapersky

Visualization animated in space: it is an interactive moving image so it can be seen from different sides while positioning itself to best present the readable information on the graph.

Example 4.9. Three-dimensional visualization. For a color version of the figure, see www.iste.co.uk/monino/data.zip

Bipartite graph: this type of chart can be used to represent classification.

Example 4.10. Bipartite e-graph

99

100

Big Data, Open Data and Data Development

Interactivity is a key point of visualization. To better understand what is being observed, the company must be able to quickly change its view, in other words direct its course of action, and access as quickly as possible clear representations of the fields to be analyzed. Data visualization, in light of the amount of data to be displayed, is thus an important analytical tool. It is becoming increasingly clear that this is an essential aspect of the effective communication of results. 4.5. Conclusion The considerable increase in the volume and diversity of digital data generated, coupled with Big Data technologies, offer significant opportunities for value creation. This value cannot be reduced to simply what we can solve or improve, but rather it knows what the new potential discoveries are that may arise from cross-exchanges and correlations. To create value from these huge sources, we need algorithms, for which you need access to some outstanding talent. Thus, “You have to create intelligence from the bytes that are circulating” (G. Recourcé, Scientific Director of: EVERCONTACT.). Every day huge amounts of data circulate around the Internet. We constantly produce them ourselves each time we make a connection or transaction. This data has a value, and this value increases as processing and visualization tools are developed, which allows for exploring knowledge from the correct reading of these flows. Companies that manage to correctly read the available data receive pertinent information and all the elements necessary for innovating.

Conclusion

In our world of economic globalization, economic exchanges, and the rapid and profound transformation of science and technology, modern companies must anticipate changes in its environment in order to adapt and remain competitive. As large data sets are currently available from a wealth of different sources, companies are looking to use these resources to promote innovation, customer loyalty and increase operational efficiency. At the same time they are contested for their end use, which requires a greater capacity for processing, analyzing and managing the growing amount of data, equally called the “Big Data” boom. The revolutionary element of “Big Data”, is indeed, linked to the empowerment of the processes of production and exchange of massive, continuous, and ever faster data. Businesses collect vast amounts of data through interconnected tools, social networks, e-mails, surveys, web forms, and other data collection mechanisms. Each time that someone searches on Google, uses a mobile App, tweets, or posts a comment on Facebook, it creates a data stream. By a simple navigation online, a user leaves various traces, which can be retrieved in real time. It is up to the company to determine which might represent a competitive advantage. These digital traces must be explored through interrelations and correlations, using specialized tools, in order to offer quality products and services, and to implement appropriate development strategies. So

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

102

Big Data, Open Data and Data Development

we can add the “value” that these data represent for a company to the other “V’s” that make up Big Data (volume, variety, velocity, etc.). The subject of the “business value” of data highlights several contextual and complex questions, and is dependant on our interpretation (processing, analysis, visualization, etc.) and the importance that we give to data. This is one of the biggest challenges of Big Data, “value creation from data” (data development). Companies must create value from data flows and the available content in real time. They must consider how to integrate digital into their strategy. This consists of transforming complex, often unstructured data from different sources into “information”. But the analysis of Big Data refers to the tools and methods used to transform the massive amounts of data into strategic information. The challenge, therefore, requires the ability to mix the maximum amount of data from internal and external sources in order to extract the best elements for decision-making. To create value through the large amounts of available data, especially since the development of the “Web”, it is necessary to: – develop and create a favorable culture of data internally, because data analysis is part of every business decision; – work in collaboration between the different stages of value creation from data: synergy; – master the data (structured, unstructured, etc.), which involves using IT Tools for analysis and processing, which help collect, store, manage, analyze, predict, visualize, and model the data. This requires technical skills to help develop new algorithms for the management and processing of data, particularly given its speed, volume, and variability; – master the analysis techniques and know how to choose the most appropriate one in order to generate a profit. This requires human skills, “Data Scientists” or “Big Data analysts” who can make data speak. This necessitates mastering several disciplines (statistics, mathematics, computer science) and an understanding of the business application of “Big Data”;

Conclusion

103

– ensure the quality of data because the technology is not sufficient; it is thus necessary to define a preliminary logic and a specific data management strategy. Data governance involves a set of people, processes and technology to ensure the quality and value of a company’s information. Effective data governance is essential to preserve the quality of data as well as its adaptability and scalability, allowing for a full extraction of the value of data; – strengthen “Open Data” initiatives or the provision of data, both publicly and privately, so that they are usable at any time; – include in the data processing a knowledge extraction process, “data mining” or “text mining”, so that they have meaning. The raw data itself may not be relevant, and it is therefore, important to modify them sot hey become useful and understandable; – make the best visual presentation of data; “Business Intelligence” tools should provide a set of features for enhancing data. In order to understand data and the information they provide, it is important to illustrate them with diagrams and graphs in order to present clear and eloquent information. Big Data – the processing and analysis of very large amounts of data – should therefore revolutionize the work of companies. But for French companies, this approach is particularly weak. According to a study published by the consultancy firm E&Y, French companies are notably behind in this area. For two thirds of them, Big Data is “an interesting concept, but too vague to constitute an opportunity for growth”. However, some sectors are particularly active in the collection and processing of data on a large scale: this is particularly the case in retail, telecommunications, media, and tech companies, which are the most developed in using customer data, according to the firm. In this light, France must work quickly to be at the forefront of data developement. Data developement represents the seventh priority of the report “one principle and seven ambitions” of the “Innovation 2030” commission chaired by Anne Lavergeon. The report indicates that Big Data is one of the main points to develop in the coming years. According to the same report, the use of mass data that governments

104

Big Data, Open Data and Data Development

and businesses currently have would be a vessel for significant competitive gains. The report provides a series of guidelines, such as: – making data publicly available so that it can be used. France has already embarked on this aim through the project “Etalab”. But the report wishes to accelerate the movement according to the British model, to allow start-ups to “create eco-systems in France through certain uses for commercial purposes”; – the creation of five data use licenses, stemming from employment centers, social security, national education, higher education as well as help to the enhancement of national heritage. This is to strengthen the collaboration between public and private actors; – the dedication of a right to experimentation, under the auspices of “an observatory of data”. This involves evaluating the effectiveness of certain data processing techniques, before considering a possible legislative framework; – the creation of a dedicated technology resource center, to remove the entry barriers and reduce the time-to-market of new companies; – strengthening the export capacity of SMEs, to avoid being isolated in the French market whilst encouraging the authorities to intervene further in this sector. Data has become a gold mine; they represent raw materials like oil in the 20th century. For geological reasons, oil is concentrated in certain parts of the world, whereas data is generated by users worldwide, transported on the Web, and accumulated and analyzed through specialist techniques and tools. The volume of data generated and processed by companies, is therefore, continually increasing. This observation is not new. But in the era of Internet and the proliferation of connected devices, the analysis of these deposits has become complex. Due to the constantly increasing volume of electronically available data, the design and implementation of effective tools to focus on

Conclusion

105

only relevant information is becoming an absolute necessity. We also understand not only the use of expert systems but also the use of data analysis methods which aim to discover structures and relationships between facts using basic data and appropriate mathematical techniques. It is about analyzing these data in order to find the relevant information to help make decisions. For example, when we search or order a book from Amazon, the website’s algorithm suggests other items that have been bought by thousands of users before us. Netflix goes even further: the collected data is used to decide which films to buy the move rights to and which new programs to develop. Google Maps offers an optimized route based on traffic conditions in real time while Facebook offers us new friends based on existing connections. All these proposals are made thanks to the analysis of a huge amount of data. In the USA, the use of facial recognition cameras at the entrance of nightclubs has been transformed into a service. This is “ScenTap”, a mobile App that lets you know in real time the number of men and women present. The directors manage their facilities in real time and customers can decide which venues they prefer to visit. In France, the SNCF has focused on developing its business for the Internet and smartphones in order to control the information produced and disseminated on the Internet and to prevent Google from becoming the one that manages customer relations. This is what happened with Booking.com, on which all hotels are still dependent today. Data are currently produced daily and in real time from telephones, credit cards, computers, sensors and so on. It is not just about the quantity and speed of production of these data, the real revolution lies in what can be created by combining and analyzing these flows. Big Data processing requires an investment in computing architecture to store, manage, analyze, and visualize an enormous amount of data.

106

Big Data, Open Data and Data Development

This specific technology is a means to enrich the analysis and display the strength of available data. We must have the right processing method to best analyze data. But it also involves investing in the human skills that help precisely analyze these quantities and to elaborate algorithms and relevant models. In other words, the company needs data scientists to explain these deposits. It is indeed important to select the most relevant data, and thus, include in the analysis the concept of quality and not only quantity: this is where the “governance strategy” comes in.

Decision

DATA GOVERNANCE

We can thus see that Big Data involves a number of components that work together to provide a rich ecosystem and to ensure even more powerful analyses. Such an ecosystem can be presented as in Figure C.1.

Figure C.1. Data governance model in the age of data revolution developed by Monino and Sedkaoui

Big Data opens up new possibilities of knowledge and provides another form of value creation for businesses. Today companies have

Conclusion

107

large amounts of data. These reservoirs of knowledge must be explored in order to understand their meaning and to identify relationships between data and the models explaining their behavior. The interest of Big Data is to take advantage of the data produced by all stakeholders (companies, individuals, etc.) that becomes a strategic asset, a value creation tool, and gives birth to a new organizational paradigm with opportunities for innovation.

Bibliography

[ABI 13] ABITEBOUL J. et al., “Turbulent momentum transport in core tokamak plasmas and penetration of scrape-off layer flows”, Plasma Physics and Controlled Fusion, vol. 55, no. 7, p. 074001, 2013. [ACK 89] ACKOFF R.L., “From data to wisdom”, Journal of Applied Systems Analysis, vol. 15, pp. 3–9, 1989. [AGE 10] AGERON B., MARIE-LYNEGOURY, SPALANZANI A., Knowledge Management appliqué aux problématiques de développement durable dans la Supply Chain, Cahier de recherche n° 2010–03 E5.Version 1, CNRS, 2010. [AGR 83] AGRELL B., HULDT, Clio Goes Spying: Eight Essays on the History of Intelligence, Lund Studies in International History, Scandinavian University Books, Malmö, 1983. [AGU 67] AGUILAR F.J., Scanning Business Environment, Macmillan Co., New York, 1967. [ALB 62] ALBAUM G., “A new approach to the information function in marketing”, Communication, vol. 16, no. 2, pp. 163–176, 1962. [ALB 02] ALBERT P., BERNASCONI M., GAYNOR L., Les incubateurs: émergence d’une nouvelle industrie, available at: www. ladocumentationfrancaise.fr/var/storage/rapports-publics/024000373.pdf, April 2002. [ALM 12] ALMEIDA F., “Web 2.0 technologies and social networking security fears in enterprises”, International Journal of Advanced Computer Science and Applications, vol. 3, no. 2, pp. 152–156, 2012.

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

110

Big Data, Open Data and Data Development

[AMA 02] AMABILE T.M., “Motiver la créativité dans les organisations: faites ce que vous aimez et aimez ce que vous faites”, in GETZ I. (ed.), Créativité organisationnelle, Vuibert, Paris, 2002. [BAU 91] BAUMARD P., Stratégie et surveillance des environnements concurrentiels, Masson, Paris, 1991. [BAU 92] BAUMARD P., Guerre économique et communauté d’intelligence, La Revue Politique et Parlementaire, 1992. [BAU 98] BAUMARD P., BENVENUTI J.A., Compétitivité et systèmes d’information, de l’outil d’analyse au management stratégique, InterEditions, Paris, 1998. [BAU 99] BAUMARD P., “La guerre du savoir a commencé”, L’expansion Management Review, no. 92, pp. 60–69, March 1999. [BEN 15a] BENKARAACHE T., BENABDELOUAHED R., “L’émergence du Big Data dans le secteur bancaire: Cas des banques marocaines”, Actes du Congrès VSST 2015, Granada, Spain, 2015. [BEN 15b] BENKARAACHE T., MONINO J.L., SALAM G., “Intégration du Big Data dans le Supply Chain: vers une supply chain ‘intelligente’, Actes du congrès AIM 2015, Rabat, Maroc, 2015. [BER 00] BERRY M.J.A., LINOFF G.S., Mastering Data Mining, Wiley, New York, 2000. [BER 01] BERNERS-LEE T., HENDLER J., LASSILA O., “The semantic web”, Scientific American, vol. 284, no. 5, pp. 28–37, May 2001. [BOY 08] BOYA-GUIERDEL C., Modélisation de la relation entre les informations exogènes médiatiques et le marché des actions, Phd Thesis, University of Montpellier, France, 2008. [BOY 09] BOYA-GUIERDEL C., “Analyse des informations exogènes secondaires et mesure d’impact sur le marché action”, Revue internationale d’intelligence économique, vol. 1, pp. 237–253, 2009. [BOY 10] BOYA C., MONINO J.-L., “The impact of exogenous information on stock value through the coloration concept: a test model”, Journal of Innovation Economics, no. 6, pp. 163–180, 2010. [BRO 11] BROWN B., MANYIKA J., CHUI M., “Are you ready for the era of ‘big data’”, McKinsey Quarterly, vol. 4, pp. 24–35, October 2011.

Bibliography

111

[BUG 10] BUGHIN J., MANYIKA J., CHUI M., “Clouds, big data, and smart assets: ten tech-enabled business trends to watch”, McKinsey Quarterly, vol. 56, no. 1, pp. 75–86, 2010. [BUG 11] BUGHIN J., CHUI M., “The rise of the networked enterprise: Web 2.0 finds its payday”, McKinsey on Business Technology, vol. 22, pp. 1–9, 2011. [BUL 02] BULINGE F., Pour une culture de l’information dans les PMO: un modèle incrémental d’intelligence économique, PhD Thesis, University of Toulon, December 2002. [BUL 05] BULINGE F., AGOTINELLI S., “L’analyse d’information: d’un modèle individuel à une culture collective”, Actes du Collogue ParisPekin, 2005. [BUL 06] BULINGE F., “Le cycle du renseignement: analyse critique d’un modèle empirique”, Marketing & Communication, vol. 3, no. 3, pp. 36– 52, 2006. [BUL 07] BULINGE F., “Un modèle d’analyse collective en situation: la warroom”, Marketing & Communication, vol. 7, no. 4, pp. 50–69, 2007. [BUT 09] BUTLER MARTIN RESEARCH, “The business value of business intelligence: a review of benefits, costs and risks”, Business Value Series, no. 3, pp. 4–7, 2009. [CAV 04] CAVAGNA E., DUMAZERT J.P., MONINO J.L., “Construction d’un indice de satisfaction: une réfexion autour de la question de la mesure”, Revue Française de Marketing, no. 197, pp. 61–77, 2004. [CHE 95] CHESNAIS F., La montée des firmes-réseaux, Alternatives Economiques, January 1995. [CHI 12] CHIGNARD S., Open data: comprendre l’ouverture des données publiques, Éditions Fyp, 2012. [CHU 09] CHUI M., MILLER A., ROBERTS R.P., “Six ways to make Web 2.0 work: McKinsey Quarterly, pp. 1–7, available at: http://www.mckinsey.com/ insights/business_technology/six_ways_to_make_web_20_work, 2009. [CHU 13] CHUI M., “The social economy: unlocking value and productivity through social technologies”, McKinsey Quarterly, vol. 153, 2013. [CLE 14] CLEVELAND W., Data science: an action plan for expanding the technical areas of the field of statistics, available at: www.stat.purdue.edu/~wsc/papers/datascience.pdf, 2014.

112

Big Data, Open Data and Data Development

[COD 02] CODY W.F., KREULEN J.T., KRISHNA V. et al., “The integration of business intelligence and knowledge management”, IBM Systems Journal, vol. 41, no. 4, pp. 1–3, 2002. [COR 97] CORTES C., PREGIBON D., Mega-monitoring, University of Washington/Microsoft Summer Research Institute on Data Mining, 6–11 July 1997. [COU 14] COUDERCY L., DELOBELLE G. et al., Les données cartographiques dans une démarche Open Data, available at: slideplayer.fr/slide/2740238, 2014. [CRO 03] CROZET Y., MUSSO P., Réseau, services, territoires. Horizon 2020, Editions de l’Aube, 2003. [DAV 06] DAVID A., L’impact des TIC: logistique, transport, relation de service, organisation, La Documentation française, Paris, 2006. [DUP 04] DUPUIS A., “Gouvernance, ajustement mutuel, communautés humaines et intelligence collective”, Cahier de recherche 2003–04, CERGO, 2004. [DUP 07] DUPUY G., GENEAU DE LAMARLIERE I., Nouvelles échelles des firmes et des réseaux. Un défi pour l’aménagement, L’Harmattan, 2007. [EDE 99] EDELSTEIN H.A., Introduction to Data Mining and Knowledge Discovery, 3rd ed., Two Crows Corp, Potomac, MD, 1999. [ELI 34] ELIOT T.S., “The rock”, Faber & Faber, 1934. [ESC 76] ESCARPIT R., Théorie générale de l’information et de la communication, Classiques Hachette, Paris, 1976. [EUR 12a] EUROPEAN COMMISSION, Open data: an engine for innovation, growth and transperant governance, available at: eur-lex.europa.eu, 2012. [EUR 12b] EUROPEAN COMMISSION, EC ISA case study: how linked data is transforming e-government, 2012. [FAB 95] FARBEY B., TARGETT D., LAND F., “Evaluating business information systems: reflections on an empirical study”, Revue Information Systems Journal, vol. 5, no. 4, p. 242, 1995. [FAL 01] FALQUET G., MOTTAZ JIANG C.-L., “Navigation hypertexte dans une ontologie multi-point de vue”, Actes de la conférence NîmesTIC 2001, Nîmes, France, 2001.

Bibliography

113

[FAY 96] FAYYAD U.M., PIATETSKY-SHAPIRO G., SMYTH P. et al., Advances in Knowledge Discovery & Data Mining, MIT Press, Cambridge, MA, 1996. [FIO 10] FIORETTI M., SANT’ANNA S., Open Data Open Society, Dynamics of Institutions and Markets in Europe, available at: http://www.lem.sssup.it/WPLem/odos/odos.html, 2010. [FLI 08] FLICI A., A systematic evaluation of business intelligence tools: an investigation into potential tangible/intangible value, PhD Thesis, Brunel University, 2008. [FOR 00] FORAY D., L’économie de la connaissance, La Découverte, Paris, 2000. [FRA 97] FRANCO J.-M., Le Data Warehouse et le Data Mining, Eyrolles, Paris, 1997. [GAR 99] GARDARIN G., Internet/intranet et bases de données, Eyrolles, Paris, 1999. [GAR 14] GARTNER, Big data needs a data-centric security focus, available at: www.gartner.com/document/2691917, 2014. [GAU 12] GAULTER D., Le Big data: un cadean empoisonné pour les data miners? Available at: www.decideo.fr, 2012. [GET 07] GETZ I., ROBINSON A.G., LEROY D., Vos idées changent tout!, Editions d’Organisation, Paris, 2007. [GOË 12] GOËTA S., Open data: qu’ouvre-t-on avec les données publiques?, Thesis, CELSA, 2012. [GRO 11] GROUPE DE TRAVAIL SUR LES DONNÉES OUVERTES (GTDO), Rapport sur l’ouverture des données de la Ville de Montréal: un capital numérique générateur d’innovation et de participation, available at: http://www1.ville.montreal.qc.ca/banque311/webfm_send/1453, 2011. [GUY 99] GUYOT B., “Le système d’information comme objet de recherche”, in DELCAMBRE P. (ed.), Communication organisationnelles, objets, pratiques, dispositifs, Presses universitaires de Rennes, 1999. [HAN 00] HAN J., KAMBER M., Data Mining: Concepts and Techniques, Morgan-Kaufman, New York, 2000. [HAR 90] HARBULOT C., “Techniques offensives et guerre économique,” Étude no. 131, Centre de Prospective et d’Évaluation/Aditech, Paris, 1990.

114

Big Data, Open Data and Data Development

[HAS 01] HASTIE T., TIBSHIRANI R., FRIEDMAN J.H., The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer, New York, 2001. [HEA 07] HEATH T., BIZER C., Web de données: Méthodes et outils pour les données liées, Pearson, 2007. [HOP 11] HOPKINS B., EVELSON B., “Expand your digital horizon with big data”, available at: http://www.asterdata. com/newsletter-images/30– 04–2012/resources/Forrester_Expand_Your_ Digital_Horiz.pdf?lf1=970 73534d99767367906a1885620, 2011. [JAC 00] JACOB R., TURCOT S., La PME apprennante: information, connaissance, interaction, intelligence, Développement Economique Canada, 2000. [JAK 09] JAKOBIAK F., L’intelligence économique: techniques et outils, 2nd ed., d’Organisation, Groupe Eyrolles, France, 2009. [JAR 94] JARROSSON, Décider ou ne pas décider: réflexions sur les processus de la décision, Maxima, Paris, 1994. [JUL 03] JULIEN P.A., VAGHELY I., CARRIER C., PME et contrôle de l’information: le rôle du troisième homme, Colloque d’Agadir, 2003. [JUL 04] JUILLET A., “Du renseignement à l’intelligence économique”, La Revue Défense Nationale et Sécurité Collective, no. 12, pp. 7–20, 2004. [KEF 04] KEFI H., KALIKA M., Evaluation des systèmes d’information: une perspective organisationnelle, Economica, 2004. [LAC 11] LACOMBE R., BERTIN P.-H., VAUGLIN F. et al., Pour une politique ambitieuse des données publiques, available at: http://www. ladocumentationfrancaise.fr/var/storage/rapportspublics/114000407/0000. pdf, 2011. [LEF 98] LEFÉBURE R., VENTURI G., Le data mining, Eyrolles, Paris, 1998. [LEG 13] LE GUEN F., Analyse et visualisation des données, Pearson, France, 2013. [LES 86] LESCA H., Système d’information pour le management stratégique de l’entreprise, McGraw-Hill, Paris, 1986. [LES 08] LESCA H., “Gouvernance d’une organisation: prévoir ou anticiper?”, La Revue des sciences de gestion Direction et Gestion, nos. 231–232, pp. 11–18, February 2008.

Bibliography

115

[LUH 58] LUHN H.P., “The automatic creation of literature abstracts”, IBM Journal of Research and Development, vol. 2, no. 2, p. 159, 1958. [MAL 08] MALTHUS T.R., An Essay on the Principle of Population: as it Affects the Future Improvement of Society, Lawbook Exchange Ltd., 2008. [MAN 11] MANYIKA J., CHUI M., BROWN B. et al., “Big Data: the next frontier for innovation, competition, and productivity”, McKinsey Quarterly, 2011. [MAR 94] MARTRE H., Intelligence économique et stratégie des entreprises, La Documentation française, 1994. [MAT 05] MATOUK J., MONINO J.L., “Le marché de Paris a la mémoire courte”, Revue d’Economie Financière, vol. 81, pp. 1–25, 2005. [MAT 14] MATOUK J., “La revue de la MTRL”, Mutuelle et Santé, no. 84, 2014. [MAY 13] MAYER-SCHONBERGER V., CUKIER K., Big Data: A Revolution That Will Transform How We Live, Work and Think, Dolan/Houghton Mifflin Harcourt Publishing Company, London, 2013. [MCA 06] MCAFEE A.P., “Enterprise 2.0: the dawn of emergent collaboration”, MIT Sloan Management Review, vol. 47, no. 3, pp. 20–29, 2006. [MCM 04] MCMAHON A., LIU Y., CROSSLAND R. et al., “A best practice advice system to support automotive engineering analysis processes”, Engineering with Computers, vol. 19, no. 4, pp. 271–283, 2004. [MCN 14] MCNULTY E., Big data: the seven Vs, available at: http://dataconomy.com/seven-vs-big-data/, 2014. [MER 14] MERCIER D., Données ouvertes: qualité et visualisation, available at: http://dianemercier.quebec/donnees-ouvertes-qualite-et-visualisation/, October 2014. [MOI 10] MOINET N., “Mythes et réalités de l’intelligence économique, available at: www.ege.fr/download/SeminaireEGENicolasMoinet 26avtil2010.pdf, 2010. [MON 06] MONINO J.L., LUCATO G., Manager l’information: comment faire aujourd’hui?, CCI Montpellier, 2006. [MON 11] MONINO J.-L., BOYA C., La coloration de l’information dans l’efficience semi forte, Innovations, no. 3, pp.147–157, March 2011.

116

Big Data, Open Data and Data Development

[MON 12] MONINO J.-L., “Un modèle hiérarchique pour l’Intelligence économique,” Colloque Intelligence économique compétitivité et attractivité des territoires, Montpellier France, 2012. [MON 13] MONINO J.L., “L’information au cœur de l’intelligence économique stratégique, in Marché et organisations, Harmattan, February 2013. [MON 14] MONINO L.L., SEDKAOUI S., MATOUK J., “Big data, éthique des données, et entreprises”, Les Cahiers du CEDIMES, vol. 8, no. 2, pp. 57– 67, 2014. [OEC 13] OECD, Exploring data-driven innovation as a new source of growth: mapping the policy issues raised by “BigData”, Report, OECD Digital Economy Papers, No. 222, OECD Publishing, 2013. [PEU 10] PEUGEOT V., Les enjeux publics économiques et citoyens de l’ouverture des données: l’expérience britannique, available at: http://laborange.academia.edu/Val%C3%A9riePeugeot/ Papers/343143/ Les_enjeux_publics_economiques_et_citoyens_de_louverture_des_donne es_lexperience_britannique, 2010. [PIM 08] PIMOR Y., FENDER M., Logistique: production, distribution, soutien, Dunod, Paris, 2008. [POO 84] POOL I.S., Communications Flows: A Census in the United States and Japan, Elsevier Science, New York, 1984. [ROT 12] ROTELLA P., Is data the new oil?, available at: www.forbes.com/sites/perryrotella/2012/04/02/is-data-the-new-oil/, 2012. [SAÏ 04] SAÏD A., Rôle de l’intelligence économique dans la stratégie de l’entreprise, PhD Thesis, Mention Sciences de Gestion, 2004. [SAL 97] SALMON R., LINARÈS Y., L’intelligence compétitive, une combinaison subtile pour gagner ensemble, Economica, Paris, 1997. [SAW 12] SAWYER G., DEVRIES M., About GMES and data: geese and golden eggs, a study on the economc benefits of a free and open data policy for sentinel satellite data, Report, European Association of Remote Sensing Companies, Brussels, 2012. [SYL] SYLVESTRE G., Big data: une valeur stratégique et indispensable demain, available at: www.master–iesc–angers.com, 2015.

Bibliography

117

[TAY 80] TAYLOR R.S., “Value-added aspects of the information process”, Communicating Information: Proceedings of the 43rd ASIS Annual Meeting, Anaheim, CA, vol. 17, 5–10 October, 1980. [THE 02] THEPAUT Y., Pouvoir, Information, Economie, Economica, Paris, 2002. [TUF 05] TUFFÉRY S., Data Mining et statistique décisionnelle, Technip, Paris, 2005. [UNE 00] UNESCO, Tim Berrers-Lee: the web’s brainchild, available at: unesco.unesco.org/images/0012/001203/120395E.pdf, 2000. [VIC 11] VICKERY G., Review of recent studies on PSI re-use and related market developments, available at: https://ec.europa.eu/digital-agenda/ en/news/review-recent-studies-psi-reuse-and-relatedmarket-developments, 2011. [WIL 67] WILENSKY H., Organizational Intelligence: Knowledge and Policy in Government and Industry, Basic Books, New York, 1967. [ZAR 08] ZARA O., Le management de l’intelligence collective, vers une nouvelle gouvernance, 2nd ed., M21 editions, 2008.

Internet sites http://data.gouv.fr http://opendatafrance.net/documentation/ http://openbiblio.net/principles/fr/ http://www.scoop.it/t/veille-open-data-france http://www.regardscitoyens.org/open-data-en-france/ http://donneesouvertes.info/ http://opendatafrance.net/category/data-publica/ http://www.lemondeinformatique.fr/start-up-128.html http://www.roberthalf.fr/id/PR-03954/jobs-en-or-2014 http://lesclesdedemain.lemonde.fr/business/a-qui-profite-l-open-data-_a-56– 1411.html http://fr.slideshare.net/lacantinerennes/open-data-garage-comment-valuerlimpact-conomique-de-lopen-data-local

118

Big Data, Open Data and Data Development

http://www.leparisien.fr/startup/ http://www.myfrenchstartup.com/ http://www.latribune.fr/technos-medias/informatique/20130208trib000747692/ l-open-data-est-a-la-mode-et-cherche-son-modele.html http://www.lemonde.fr/politique/article/2012/05/31/en-france-l-open-data-enmarche_1709874_823448.html http://business.lesechos.fr/entrepreneurs/idees-de-business/creneau-porteur-5exemples-de-start-up-open-data-57178.php?dxC0WjY1PjomkZJg.99 http://opendata.regionpaca.fr/donnees/detail/parkings-camping-cars-desbouchesdu-rhone.html http://www.handimap.org/ http://donneesouvertes.info/2012/03/12/ou-est-le-marche-de-lopen-data/ http://www.deciderensemble.com/nos-etudes/observatoire http://www.etalab.gouv.fr/ http://www.bibliobsession.net/2011/06/07/bibliotheques-publiques-et-donneesouvertes/ http://www.deciderensemble.com/nos-etudes/observatoire http://bbf.enssib.fr/ www.gapminder.org http://donneesouvertes.info/ http://www.redsen-consulting.com/ http://www.redsen-consulting.com/2011/11/quel-sens-donner-aux-motsdonnees-informations-connaissances-et-competences/ http://www.data-publica.com/ http://www.data-publica.com/content/2012/02/linria-ouvre-les-donnees-brutesde-son-sondage-sur-les-francais-et-le-numerique/ http://www.c-radar.com/ http://cadderep.hypotheses.org/94/ http://www.cairn.info/resume.php?ID_ARTICLE=MULT_049_0138

Bibliography

119

http://www.dunod.com/entreprise-economie/entreprise-et-management/strategieet-politique-de-lentreprise/ouvrages-professionnels/la-revoluti-1 http://www.datazen.com/ https://powerbi.microsoft.com/ http://www.lecfomasque.com/ http://www.bcg.fr/ https://support.office.com/ https://support.office.com/fr-fr/article/Compl%C3%A9ment-PowerPivota9c2c6e2-cc49-4976-a7d7-40896795d045?CorrelationId=4299016a384e-4367-842e-a5faab3d3384&ui=fr-FR&rs=fr-FR&ad=FR http://www.transport-intelligent.net/sti-en-france/article/handimap-uneapplication-pour http://www.commentcamarche.net/faq/29732-web-semantique-quellesapplications-aujourd-hui http://novaspivack.typepad.com/nova_spivacks_weblog/2004/04/new_version_ of_.html http://www.mckinsey.com/insights/mgi http://www.gartner.com/technology/home.jsp http://www.jouve.com/fr/la-data-visualisation-dataviz-dans-tous-ses-etats http://pro.01net.com/editorial/506931/le-web-intelligent-prend-forme/ http://www.mediabistro.com/alltwitter/infographics-on-twitter_b26840 http://www.cs.man.ac.uk/~horrocks/Slides/ed02.pdf http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpen Data http://www.orange.com/fr_FR/presse/communiques/cp111201fr.jsp http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid= 710000003034 http://www.decideo.fr/Le-Big-Data-un-cadeau-empoisonne-pour-les-DataMiners_a5457.html http://business-herald.com/enquete/yannick-poivey-big-data-consequencesen-intelligence-economique

120

Big Data, Open Data and Data Development

http://www.data-business.fr/big-data-definition-enjeux-etudes-cas/ http://owni.fr http://www.guardian.co.uk/open-platform http://lab.linkeddata.deri.ie/2010/star-scheme-by-example http://www.jobillico.com/blog/metier-avenir-scientifique-des-donnees/ http://www.indeed.com/jobtrends/Data-scientist.html http://dataconomy.com/seven-vs-big-data/ http://www.ssc.upenn.edu/~fdiebold/papers/paper112/Diebold_Big_Data.pdf http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpe nData http://www.statsoft.fr/concepts-statistiques/glossaire/d/data.html http://www.it-expertise.com/dsi-et-big-data-developpez-la-proximite-avec-lemarketing-et-embarquez-le-dans-la-strategie-data/

Index

A, B, C

D, E, F

accessibility, 27, 83 architecture of the semantic web, 89 assist decisionmaking, 66 association, 48, 69 AUNEGE, 59 automatic classification, 48, 71 autonomy, 27 big challenge, 14 Big Data 4v, 54 7v, 54 Bootstrap, 52 business intelligence, 8, 70 Canal-U, 59 cloud, 2, 11, 21, 46, 65, 81, 98 computing, 11, 46, 65 collaboration, 29, 40–42, 93 collective intelligence, 35, 51, 57, 61, 64 competitiveness, 53, 62–64, 73, 74

data analysis, 8–12, 14, 21, 33, 43, 56, 65, 70, 75, 83, 95, 96 analyst, 17, 19 gouv.fr, 24, 84, 87 governance, 54–60, 72 journalism, 30 management, 10, 15, 57, 72, 74, 95 marketplace, 45 mining, 8, 16, 43, 61, 65–71, 75 public, 24, 27, 29, 34–37, 39, 83, 87 quality, 14, 15, 21, 27, 57, 67, 96 reuse, 27, 29, 35, 38 science, 17–19, 46, 60 science, 17–19, 46, 60 scientist, 2, 15–21, 51, 67, 87, 94, 95 scientists, 2, 16–21, 67, 94, 95 statistica, 70 visualization, 74, 95–97, 100

Big Data, Open Data and Data Development, First Edition. Jean-Louis Monino and Soraya Sedkaoui. © ISTE Ltd 2016. Published by ISTE Ltd and John Wiley & Sons, Inc.

122

Big Data, Open Data and Data Development

warehouse, 10, 23, 66 web, 74, 90 dataset, 8, 10, 39, 40, 51, 67, 86 DataViz, 94–100 decisions trees, 48 defensive, 58 description, 68, 89 descriptive techniques, 48 development of information and communication technologies (ICTS), 82 discriminate analysis, 48 discrimination, 48 e-prospects, 70 economy, 23, 29, 30, 41, 58, 60, 74 ecosystem, 7, 33, 34, 39, 69, 80, 87 enterprise, 81 estimation, 69 etalab, 36 evaluation of a model, 67 exchange office, 74 exploitation of data, 32 freedom Act, 73 FUN, 59 G, H, I geolocalization data, 62 Google Car, 79 government data, 41 hadoop, 19, 20 handimap, 39 harvesting phasec, 86 heterogeneity, 69, 81 hurst, 52 hyperlink, 89 identifying sets, 90

improvement, 16, 19, 24, 40 input, 75 internet of things, 45 interoperability, 27, 53, 89 K, L, M knowledge pyramid, 75 linear regression, 49 linked data, 31, 88, 91 lobbying, 58 LOD, 90 log analysis, 56 logistic regression, 48 machine learning, 4, 16, 81 managerial approaches, 82 MapReduce, 19 Martre, 64 mash-ups, 40 mass data, 77 mediation, 64 metadata, 89, 92 model general linear, 49 N, O, P Neuronal method, 50 new economic value, 10 offensive, 58 open innovation, 81 knowledge foundation network, 84 page rank, 92 piloting activity, 56 PORTICES, 59 power of data, 56 powerset, 92 predictive techniques, 48 preparation of data, 67, 96

Index

product lifecycle analysis, 14 productivity, 2, 11, 14, 41, 45, 53, 75 public actors, 29 Q, R, S quantified self, 87 raw data, 27, 34, 37–39, 53, 87 RDF, 89, 90, 92 reporting, 10, 72 scientific research, 24, 29 segmentation, 48, 69 semantic approach, 71 web, 69, 88–93 smart city, 80 Smart Data, 12–16, 54 social innovation, 39

123

stage deployment, 96 modeling, 96 startups, 33, 36, 87 statistical approach, 71 stock, 1, 31, 43 T, U, V, W Text mining, 65–71 Tim Berners-Lee, 27, 42, 87–90, 93 top management, 2, 19, 94 TRIS laboratory, 49 trust rank, 92 unstructured data, 10, 14, 21, 35, 56, 69, 73, 77 variability, 4 W3C, 88, 90 Web 3.0, 87

Other titles from

in Innovation, Entrepreneurship and Management

2015 CASADELLA Vanessa, LIU Zeting, DIMITRI Uzunidis Innovation Capabilities and Economic Development in Open Economies (Smart Innovation Set – Volume 1) CORSI Patrick, NEAU Erwan Innovation Capability Maturity Model CORSI Patrick, MORIN Dominique Sequencing Apple’s DNA FAIVRE-TAVIGNOT Bénédicte Social Business and Base of the Pyramid GODÉ Cécile Team Coordination in Extreme Environments MAILLARD Pierre Competitive Quality and Innovation MASSOTTE Pierre, CORSI Patrick Sustainability Calling MASSOTTE Pierre, CORSI Patrick Operationalizing Sustainability

2014 DUBÉ Jean, LEGROS Diègo Spatial Econometrics Using Microdata LESCA Humbert, LESCA Nicolas Strategic Decisions and Weak Signals

2013 HABART-CORLOSQUET Marine, JANSSEN Jacques, MANCA Raimondo VaR Methodology for Non-Gaussian Finance

2012 DAL PONT Jean-Pierre Process Engineering and Industrial Management MAILLARD Pierre Competitive Quality Strategies POMEROL Jean-Charles Decision-Making and Action SZYLAR Christian UCITS Handbook

2011 LESCA Nicolas Environmental Scanning and Sustainable Development LESCA Nicolas, LESCA Humbert Weak Signals for Strategic Intelligence: Anticipation Tool for Managers MERCIER-LAURENT Eunika Innovation Ecosystems

2010 SZYLAR Christian Risk Management under UCITS III/IV

2009 COHEN Corine Business Intelligence ZANINETTI Jean-Marc Sustainable Development in the USA

2008 CORSI Patrick, DULIEU Mike The Marketing of Technology Intensive Products and Services DZEVER Sam, JAUSSAUD Jacques, ANDREOSSO Bernadette Evolving Corporate Structures and Cultures in Asia / Impact of Globalization

2007 AMMI Chantal Global Consumer Behavior

2006 BOUGHZALA Imed, ERMINE Jean-Louis Trends in Enterprise Knowledge Management CORSI Patrick et al. Innovation Engineering: the Power of Intangible Networks