Open Data Blueprint - Canada's Open Data Exchange

Open Data Blueprint Helping municipalities think differently about open data

codx.ca

Helping municipalities think differently about open data

About Canada’s Open Data Exchange Canada’s Open Data Exchange (ODX) is a partner network committed to making it easier for Canadians to access and commercialize open data. We’re here to help entrepreneurs, policymakers, data suppliers, data enablers and companies work together to bring data to market. ODX connects those who need data with those who have it, develops tools to share it, provides training and networking opportunities, and influences policy and methods. ODX aims to strengthen Canada’s technology brand and accelerate commercialization. Our founding partners – the University of Waterloo, D2L (formerly Desire2Learn), the Canadian Digital Media Network (CDMN), OpenText and Communitech – are committed to contribute over three years to building ODX. Funds have been matched by the Government of Canada.

Acknowledgements

Canada’s Open Data Exchange acknowledges the insights provided by the following organizations: City of Moncton Venn Innovation Province of Prince Edward Island Halifax Regional Municipality Volta Labs Ville de Montréal Montreal Center for Enterprise and Innovation InnoCité MTL City of Ottawa Invest Ottawa City of Toronto The DMZ at Ryerson University City of Kitchener City of Waterloo

Regional Municipality of Waterloo City of Winnipeg Information & Communication Technologies Association of Manitoba City of Edmonton Startup Edmonton Strathcona County City of Calgary Innovate Calgary City of Vancouver British Columbia Technology Industry Association Wavefront Accelerate Okanagan Technology Association Public Sector Digest

Report authored by John Chiappetta, Projects & Initiatives ODX

Andrew Robertson, Manager, Research Projects Communitech

Contributions by: Kevin Tuer, Managing Director ODX

Joseph Bou-Younes, Data Executive In Residence ODX

Michael Chan, Business Development Manager ODX

Anthony Reinhart, Director, Editorial Strategy Communitech

Tyler Jacques, Community Manager ODX

Kelsey Vere, Digital Marketing Coordinator ODX

codx.ca


Table of Contents MESSAGE FROM THE MANAGING DIRECTOR................................................................................................................. i EXECUTIVE SUMMARY.................................................................................................................................................. ii DEMAND-DRIVEN MATURITY MODEL............................................................................................................................ 1 NATIONAL DATA STANDARDS - A FOUNDATION FOR OPEN DATA ................................................................................. 2 RECOMMENDATIONS:..............................................................................................................................................2-13 DATA MANAGEMENT:......................................................................................................................................... 2 Recommendation 1: Launch a data inventory project................................................................................ 2 Open Data Catalogue.................................................................................................................. 3 Data Governance........................................................................................................................ 3 Recommendation 2: Adopt a data evaluation process............................................................................... 4 Data Sensitivities......................................................................................................................... 4 Standards.................................................................................................................................... 4 RESOURCE ALLOCATION:.................................................................................................................................... 5 Recommendation 3: Allocate appropriate resources to open data initiatives.............................................. 5 Resource Management................................................................................................................ 5 Recommendation 4: Use business intelligence applications to build momentum....................................... 6 Financial Strategy........................................................................................................................ 6 STRATEGY:.......................................................................................................................................................... 7 Recommendation 5: Create an open data strategy................................................................................... 7 Open Data Strategy..................................................................................................................... 7 Recommendation 6: Establish an advisory panel or working group........................................................... 8 Release schedule......................................................................................................................... 8 INTERNAL INGENUITY:......................................................................................................................................... 9 Recommendation 7: Spend time communicating internally and leveraging champions.............................. 9 Internal Engagement................................................................................................................. 10 Recommendation 8: Start with small, quick wins.................................................................................... 10 Repeatable Process Management.............................................................................................. 10 COMMUNITY ENGAGEMENT:............................................................................................................................ 11 Recommendation 9: Communicate with external stakeholders and employ feedback mechanisms on your data portal............................................................................................................................................. 11 Engagement Process................................................................................................................. 11 Consumer Support.................................................................................................................... 12 Recommendation 10: Work with CDMN hubs in your region.................................................................. 12 Leverage Existing Networks....................................................................................................... 12 CONCLUSION.............................................................................................................................................................. 13 APPENDIX:................................................................................................................................................................... 14 A) Atlantic Trip Report..................................................................................................................................... 14 B) Central Trip Report...................................................................................................................................... 17 C) West Trip Report......................................................................................................................................... 20

codx.ca


Message from the Managing Director

O

pen data is an enabling tool that helps government connect with citizens, organizations and companies. It can inform new solutions to complicated problems of governance and service delivery. The social impacts of open data have been explored in depth, yet its ability to create economic growth is often overlooked. How so? Think for a moment of the forestry sector. Harvesting trees does not, in itself, generate an economic gain. Rather, the demand for raw resources is driven by demand for finished products. A clear supply chain is needed in order to create jobs and wealth. The same could be said for open data. Releasing data costs money, yet it is of limited use until it is processed and incorporated into a saleable end product. Forward-thinking governments will ensure that any data they release is driven by economic demand. To use data to its best advantage, it is imperative that an open data strategy includes an understanding of which datasets to open, how to standardize data into a machine-readable format and the ability to forecast the economic return on investment. At whatever stage or speed your municipality is opening data, we encourage you to think differently. See open data for all that it could be – a precious resource to create jobs and wealth, now and in the future. Visit codx.ca to learn more about this initiative and see how you can become part of the open data supply chain.

Kevin Tuer, Managing Director Canada’s Open Data Exchange

ODX is here to unlock the commercial opportunity of Canada’s data supply chain.

codx.ca

i


Executive Summary

I A fundamental hurdle highlighted by companies across the country was the inability to scale their business at the provincial, national and international levels.

n early 2017, Canada’s Open Data Exchange travelled across Canada to find examples of how municipalities and companies work together to achieve commercial results from open data. The Cross-Canada Tour was initiated in response to the preliminary results of ODX’s OD150 survey – a research project to identify 150 companies in Canada that use open data. This initiative is being done in conjunction with GovLab, which launched the US OD500 in 2015. By identifying 150 companies in Canada that use open data as part of their business, ODX gained a deep understanding of the challenges companies face when trying to commercialize open datasets. The OD150 preliminary results infographic can be found here. In Canada, the open data environment should be viewed as a supply chain. The movement of open data from producers to consumers involves many different organizations, people, activities, projects and initiatives, all of which work together to push out a final product. Naturally, if there is a break in this supply chain, it doesn’t work efficiently. A fundamental hurdle highlighted by companies across the country was the inability to scale their business at the provincial, national and international levels. The commonly cited challenges are: • a lack of datasets across Canada, • a lack of common data standards permitting a repeatable business process, and • a n inability to create a dialogue between data providers and consumers for data discovery purposes. This blueprint aims to address these challenges by encouraging municipalities to launch open data initiatives, and by increasing the number of accessible datasets available within existing jurisdictions. The structured recommendations that follow are based on feedback and best practices seen in major cities across Canada.


codx.ca

ii


The research methodology for this project was simple: talk to the people who have experience launching and managing open data projects, record the best practices and disseminate what was learned. ODX began by interviewing municipal open data teams at cities highlighted by Public Sector Digest’s Open Cities Index. In total, the ODX team travelled in three segments: • Atlantic Canada: Moncton, Charlottetown and Halifax • Central Canada: Montreal, Ottawa, Toronto and Waterloo Region • Western Canada: Winnipeg, Edmonton, Strathcona County, Calgary and Vancouver Face-to-face interviews were conducted with municipalities to understand how they planned and implemented open data initiatives, followed by roundtable discussions where companies had the opportunity to discuss how both sides of the table could work together to advance open data. Municipal staff and open data companies both benefited from establishing personal connections, gaining insight into public and private perspectives, as well as participating in open discourse. Establishing direct relationships with public sector counterparts is invaluable to companies, especially when they run into challenges with public sector datasets.

Ten Recommendations for Governments: 1. Launch a data inventory project 2. Adopt a data evaluation process 3. Allocate appropriate resources to open data initiatives 4. Use business intelligence applications to build momentum 5. Create an open data strategy 6. Establish an advisory panel or working group 7. Spend time communicating internally and leveraging champions 8. Start with small, quick wins 9. C ommunicate with external stakeholders and employ feedback mechanisms on your data portal 10. Work with CDMN hubs in your region Bonus: A National Data Standard for Canada


codx.ca

iii


Demand-Driven Maturity Model At ODX, we hope this blueprint will inspire open data enthusiasts in government across Canada to launch their own open data initiatives, or to encourage those already engaged with open data to become more ambitious in launching new and improved data portals. Moreover, a successful open data program does not stop there. Open data programs are living, ever-changing initiatives which need to be able to adapt to new and innovative technology. This is essential in order to keep pace with consumers, who constantly expect governments to do more with less. With that said, how do municipalities know where their areas of improvement lie, and how do they measure progress against milestones? Following this blueprint, ODX will be launching a Demand-Driven Open Data Maturity Model – a self-assessment tool designed to help municipalities measure how well their open data initiatives align with business needs. The Demand-Driven Open Data Maturity Model will be heavily based on the Open Data Maturity Model originally created by the Open Data Institute. The Demand-Driven adaptation will help organizations assess how effectively they meet the publication and consumption demands of those who use open data. The demand model aids the assessment of operational and strategic activities associated with open data, establishes thought leadership, provides guidance on potential areas for improvement, and helps organizations compare themselves to one another to highlight their respective strengths and weaknesses. In doing so, the Demand-Driven Maturity Model will assist in the adoption of best practices, and an improved process framework. The Demand-Driven Maturity Model rests on five pillars – or themes – each representing a broad area of public sector activity: (1) data management, (2) resource allocation, (3) strategy, (4) internal ingenuity, and (5) community engagement. Organizations will use the Demand-Driven Maturity Model to set appropriate goals based on the current maturity of their open data initiative, resourcing and organizational goals. To attain the full, long-term economic benefits of open data, organizations must take the necessary steps which go beyond basic data publication. This demand-driven self-assessment of open data publishing and consumption is a strong starting point. It comes as no surprise that all organizations face similar challenges when becoming data publishers and suppliers. These challenges are generally technical, and primarily centered around publishing data in easyto-use, accessible formats with clear licensing. When it comes to the maturity of open data initiatives within Canada and internationally, organizations are at different stages. Some are only just initiating their program and begging to publish data to the general public, while others have already established themselves and undergone significant changes in regards to the ‘open by default’ model and movement. Regardless of the stage of your organization, all municipalities we spoke with are still seeking guidance about how to take their municipality to the next level. The goal of the ODX blueprint is to highlight issues commonly encountered by data suppliers across the country, and the goal of the Demand-Driven Maturity Model is to provide a means for organizations to assess their current situation, improve effectiveness as publishers and align with consumers. The Demand-Driven Maturity Model establishes a framework for identifying and understanding the different areas of organizational change towards open data, while also tailoring the strategic approach to be inclusive of open data consumers.


codx.ca

1


National Data Standards: A Foundation for Open Data This recommendation was so important we decided not to include it in the top 10. Instead, it transcends all, and is vitally important to the future success of companies attempting to extract value out of open data. Furthermore, if implemented, it would immediately establish Canada among the global leaders in open data adoption. Businesses expect two things from government: consistency and predictability. In the absence of national data standards, both consistency and predictability are difficult to ascertain with respect to open data. During the Cross-Canada Tour, ODX heard countless examples of companies investing significant human and financial capital into modifying their processes when datasets are unilaterally changed or abandoned. This can occur at any time, generally with little notice or understanding of the impact it will have on users. An example of this problem could be something as simple as a change in date format or the addition of a new column of data.

But what does a ‘national data standard’ mean? Having a ‘national data standard’ means co-ordinating a common set of standards for data files: file format, release date, metadata, release platform, contact point and feedback mechanism. It could also mean collaborating with partners to create a preferred list of datasets for release, hosting a series of white-labelled open data frameworks and policies, or providing short-term advisory support for jurisdictions launching open data portals. Based on what ODX heard from municipalities and companies, our team has a conclusion and a call to action – Canada needs a third-party organization, with a national mandate, to manage data standards in Canada. This is no small task. It will require significant skills, knowledge, and reputation to successfully: • Co-ordinate among all levels of government, • Perform constant environmental scans of international data standards, • O rganize working groups across the country to identify and understand emerging trends in data (not just open data), and • Understand the value of open data to companies. Although we may not have an immediate answer to this challenge, we are diligently working to bring stakeholders from the public and private spheres to the table to address how to move forward.

Data Management Recommendation 1: Launch a data inventory project A fundamental step towards launching an open data portal is the completion of a data inventory project. This initiative is meant to index all the existing datasets that are created through an organization’s business operations. Inventories can be a good method of determining which datasets exist and who is responsible for them. The ability to draw a straight line between dataset and data owner removes confusion over who the main contact point is, and will very likely improve the quality of data as accountability becomes transparent.


codx.ca

2


A data inventory project is not to be confused with an open data catalogue. An open data catalogue is considered to provide free access to a list of all open datasets owned or hosted by an organization. In contrast, a data inventory project is an internal activity undertaken to identify a complete index list of all data collected by an organization. Inventories are usually undertaken once, and used as a way of determining a starting point. Once satisfied that all available datasets have been identified, the process of creating an externally facing open data catalogue can begin. In the spirit of true transparency, the open data catalogue should publicly display all data the organization collects – even those datasets that will never be opened. The output of an inventory project is a full index of datasets produced and managed by the organization – including those which will not or cannot be released publicly. The index then becomes a reference point for internal and external data users who are looking for new data sources. This process of data discovery is incredibly important to companies as they would rather use this to identify datasets that might provide value, instead of endlessly searching through metadata or hundreds of data fields which may or may not contain the data they are after. At the end of the day, a product or app created using 100 datasets will be far richer than an app created using two or three. The importance of a data inventory was reinforced across the country, as ODX heard the following interaction between government and private sector stakeholders during the roundtables: • Government -- “What data do you want us to release?” • Companies -- “What data do you have?” It’s simple; data inventories are a valuable source of information which satisfies each party’s perspective in this interaction. Suppose an open data platform is a restaurant, and the users represent the patrons; without a menu of what the chef is prepared to cook, how are the patrons expected to know what to order? Truly satisfied patrons are those who can make small changes to their order by substituting higher value ingredients if they desire, all while receiving high quality service. The quality of the data inventory can be augmented by identifying additional criteria: • Frequency of data release, • Initial scan of data users, • Approvals required before releasing the dataset, • Existence of private/confidential information, • Current cost of acquisition by external entities, • Current cost to produce the dataset, • Current release status (ie. is it already open?) • Guarantee of future existence (is the publication permanent/repeatable?) A complete inventory list allows for greater data discovery, becomes more efficient at delivering data to users, and enables cross-departmental use of internal data. Getting to a place where an organization collects and monitors data upon release takes time, but optimizing the release process by allowing consumers (both internal and external) to select which datasets are prioritized is essential. A complete data inventory is ideal; however, the importance of data governance should be highlighted. Data governance is concerned with high quality data, and ensuring data assets have clear ownership. With an organization working to improve its data program it must also improve its data governance process, and identify and monitor where data governance may lapse. To successfully do this, organizations can


codx.ca

3


work in partnership with consumers to help identify, improve, and maintain data quality.

Recommendation 2: Adopt a data evaluation process The end goal for any open data program is achieving a frictionless and repeatable data release process. To achieve this means making data more accessible through an evaluation framework that considers the quality and characteristics of data. Through this framework, open data teams will be able to quickly identify datasets that are ready for release, and those that require a significant time investment before being classified as ‘release ready’. Applying a rigorous data evaluation process will reassure data owners that only the best quality data will be made available. Furthermore, since the evaluation process will be predictable and repeatable, there can be no claims of targeting departments or unfair treatment of data owners. It is therefore recommended that the data evaluation criteria be transparent so that data owners will have some insight into which datasets will likely be opened. It’s important to implement a data evaluation process as it is crucial to avoid working with datasets that either provide little value, or require lengthy intervention in order to meet release standards. Being able to release or update datasets at frequent intervals is far more important to new open data initiatives than releasing every dataset available.

Is your evaluation process consistent? Organizations that have launched an open data initiative have also developed a process to evaluate datasets to one degree or another. The evaluation criteria are typically composed of a series of questions, weighted by importance, resulting in an overall score. Based on ODX’s cross-Canada consultations, the most common include: 1. How clean/accurate is the data? 2. Who owns it? 3. What is the current demand for this data? 4. Is it in a data format that is machine readable? 5. How easy is it to transform into open data? 6. What is the value of the data internally and externally? 7. Is the data currently released outside the organization? 8. How much would it cost to maintain the dataset in a release-ready state? This process is simple and easy to implement. To optimize evaluation, organizations should broaden their scope and include consumer feedback into which elements should be weighted more heavily than others. Attention to data sensitivities is a familiar theme for governments. Where this narrative begins to depart from how the public sector traditionally treats sensitive data, is how one labels it. In the initial stages of the open data initiative it is very easy to ignore sensitive datasets for release and paint them all with the same brush; a dataset labelled ‘sensitive’ does not mean the conversation has to stop there. Organizations should dig deeper to identify what needs to be done to either anonymize the data or manage elements that raise red flags. Data owners can optimize their risk management mechanisms by performing routine


codx.ca

4


assessments of key datasets during the entirety of the life-cycle (opposed to at the end) to highlight, monitor and resolve sensitivity issues.

Resource Allocation Recommendation 3: Resource allocation and internal champions Providing adequate resources is one of the most important aspects to consider when launching, transforming or redesigning an open data initiative. To ensure an initiative is effectively managed and ultimately succeeds, resources must be strategically deployed. A recurring statement heard from municipalities was the importance of having a dedicated resource to manage and implement open data strategies. The big question becomes – how does one lay the foundation for such a request to management and/or city council? This question is especially difficult for smaller municipalities. Some of the most innovative open data programs across the country are built by those who are not able to fully staff or resource their open data teams. A strong recommendation ODX heard for smaller municipalities was to hire a consultant to write the strategy and policy, as well as the core framework supporting the release of open data. Despite this approach being costly, the leveraged expertise of a consultant can ultimately reduce costs, increase impact and mitigate organizational risk. In one case, the project consultant was subsequently hired to run the open data program full-time. Ultimately, regardless of the goals, the municipality must first clearly define what success means to them. Without this crucial step the program will not advance at the pace that it could, and should. During the tour, ODX heard that responsibility for open data typically resides in or begins with the IT department, under the responsibility of the CIO; this seems to be a natural fit for programs at the initial stages. That said, if open data is a ‘side of desk’ project and remains that way, insufficient attention is likely to be given to ensuring quality datasets are released at consistent intervals. In these circumstances, consumers quickly become uninterested, and disconnect from the process. It will also signal to others that open data is not a priority.

Leverage internal champions Of the municipalities interviewed, it was clear that those who could identify and leverage internal champions were the most successful. For the most part, these champions come from the senior levels of bureaucracy or city council. Open data initiatives are usually launched by the mayor or a city councilor, but as initiatives move into execution, these senior politicians will become less involved, as working-level civil servants are given new tasks. As a result, it is crucial for a municipality to cultivate internal champions (outside of the open data implementation team) to speak on their behalf. The more senior the champion, the better. There are many approaches one can adopt to find internal champions. Two of the most successful paths were: • C onducting presentations with each department highlighting how they fit into the open data strategy and what results they can expect. During these sessions, interested individuals who seem receptive to better data collection and distribution are identified


codx.ca

5


• Identify data owners and have informal discussions with them about why their data is valuable. Propose small projects using their datasets and demonstrate their value internally. At the end of the day, conversation around open data programs begins and ends with resource management and financial strategy. Most programs begin unfunded with no attempt to quantify the value of proactively publishing information, but eventually move to a more sustainable model. With time, the organization should begin to monitor the financial costs and benefits associated with the publication and consumption of open data and make sure they are in line with their definition of success. Over time, efficiencies around cost savings mechanisms and reduced data management overhead can be realized, leading to a more optimized program.

Recommendation 4: Business intelligence applications as a test case Gaining staff buy-in at the working level can be one of the most significant barriers to open data initiatives. Articulating the benefits of open data is no longer seen as a powerful enough reason to undertake open data initiatives, as citizens and staff alike desire return on investment (ROI) and statistics. In fact, the concept of transparency as a public good can actually increase organizational risk. As a result, one municipality ODX interviewed uses business intelligence (BI) case studies as the first step to engage working-level staff. By demonstrating how open data can be used to increase efficiency and achieve cost savings, the municipality was able to build a better argument for the impact of open data. As a result, the open data that was produced by the municipality was actually of higher quality for the consumer because of a use-and-improve quality check that was implemented through BI. Implementing these case studies and demonstrating the ROI is the gold standard.

What is business intelligence and how do I get it? Begin by looking for business intelligence examples in other jurisdictions and present them to internal teams. After demonstrating how they successfully used data to improve performance, an opportunity should exist to undertake an internal project, likely one of the more compelling case studies reviewed. It is important that these projects add some type of value to the end user (cost savings, time savings, improved analytical tools, etc). Simply automating data collection and dissemination with no added value will not result in the desired buy-in. Under these circumstances, open data is related to the realm of ‘nice to have’ but not essential. Regardless of how well-funded an open data program is, using open data to provide more efficient and cost-effective solutions to service delivery challenges should be the eventual goal. To build on this, automating the cost-savings calculations associated with open data solutions would allow data owners to see the ROI of releasing their datasets. Admittedly, this is simply not attainable for most municipalities at this point in time. Instead, municipalities should use BI methods to identify where optimization is possible, and begin implementing cost reduction methods. A simple way to demonstrate cost-reduction through automating data release would be to calculate the human resource costs associated with multiple requests for information (RFIs) for the same dataset. Interviews with departments can help provide insight into which internal datasets would be used the most for organizational improvement. During the interviews conducted by ODX, we heard that through open data, the departments that do not normally collaborate were able to leverage each other’s data to


codx.ca

6


improve workflow, reduce costs and provide more effective public services. These instances of internal collaboration increase the effectiveness of government, while at the same time having a measurable impact on citizens’ lives.

Strategy Recommendation 5: Create an open data strategy Without a defined plan or strategy, open data initiatives have no leadership or direction. Defining a strategy to guide open data initiatives is fundamental to success. As with any initiative, establishing a defined strategy is the first step, but it must be collaborative and rooted in desired outcomes. In this sense, it is also extremely important to include the needs and perspectives of those who consume the data. Without their input, a void exists when trying to define how open data will create economic opportunities for businesses. Creating such a strategy may seem like a daunting amount of work – but there’s no need to reinvent the wheel. Consider reaching out to one of the municipalities ODX interviewed (please reach out; we can help!) and asking for help, or look online for examples that can be replicated. This even includes white-labelled examples that can be repurposed based on your needs. During the Cross-Canada Tour, municipalities interviewed found a strategy document to be extremely important when communicating with internal stakeholders what final outcomes look like. As such, it was very important to include input from internal champions as it led to ownership through to implementation. It was then indicated that this document should not simply be created, agreed upon and set on a shelf – it should be a living, breathing document that evolves alongside the open data program. Simply put, this document should be able to be modified and updated as needed. It is also critical that the strategy includes specific mention of targets, goals and deliverables to ensure progress is effectively tracked. This type of analytics plays into the financial reporting mentioned earlier as well. As internal progress metrics are made readily available internally using business intelligence analytics, financial and strategic forecasting can be made more accurate. Ultimately, this will create a feedback loop which makes the organization run smoother and more predictably.

Where do I start? In the beginning, most organizations have no strategy or policy when it comes to open data – this is not unusual. These initial programs usually run based on individual effort and interest. Strategies should include many elements of the recommendations discussed in this blueprint, but depending on needs, some may be more prominent than others. For example, the open data strategy should be concerned with data valuation, governance and standards. When launching a strategy, clear responsibility delegation and a clear budget are necessary for the delivery of the strategy itself. This should be formulated and approved at the senior management level to ensure proper engagement and ownership of key processes. Some elements of the strategy, however, may need to be omitted at the early stages. This will likely include any mention of mandating release of all datasets, or tying the release and quality of datasets to performance assessments. Regardless of what is included, obtaining buy-in at the early stages is critical.


codx.ca

7


Due to the importance of open data strategies, ODX will be including a step-by-step section on open data strategy in the upcoming ODX Demand-Driven Maturity Model.

Recommendation 6: Establish an advisory panel or working group Open data programs can only be as successful as the strategy and framework they are founded upon. When designing open data strategies and programs, an emerging best practice is the inclusion of open data stakeholders on advisory panels or working groups to guide strategy implementation. These individuals bring a wide range of experience and knowledge to the table, ranging from which data formats to adhere to, to which datasets should be prioritized. They will also be familiar with which companies are currently leveraging open datasets. Having data release guided by advisors and civic engagement boosts community morale. As such, civic groups also gain a sense of ownership over the direction of open data in their jurisdiction when asked to be involved. Throughout ODX’s Cross-Canada Tour, the strained relationship between municipal governments and local companies became evident. This is no surprise given that companies require a level of speed from government that the bureaucracy is not structurally able to achieve. Another cause of the strained relationship stems from poor communication channels for entrepreneurs to convey their needs and perspectives into government units managing open data initiatives. By including these frustrated stakeholders in a formal structure that has an impact on decision making, governments can help mend this relationship. Moreover, if these advisory panels or working groups feel that their input is being ignored, these relationships can quickly and publicly crumble. By pulling back the ‘curtain’ and allowing external stakeholders have input into how the open data program is established and run, government decision makers will also get a sense of the wider open data community and the direction it is heading. It is important that this group of external stakeholders include entrepreneurs, as they will have a better understanding of the value certain datasets hold for the private sector, and can inform the public sector accordingly. The social benefits of open data have been well documented, but there is less understanding of which datasets hold the most value for companies. Even when the value is defined, it can change dramatically from jurisdiction to jurisdiction depending on strengths in the local economy.

Okay, I’ve talked to the community; now what? Understandably, when municipal governments begin to approach the topic of open data and systematic data release, there is no approach to managing the release of datasets. Eventually, the end goal is to have an organization-wide policy where the entire organization has implemented a dataset release automation process on core datasets. Not surprisingly, this will not happen overnight. Instead, many steps within the organization must first take place. Before departments can automate datasets, they must identify which ones are of the highest value. These core datasets can then be prioritized. From here, elements of the release catalogue may be publicly and openly shared for external input. Once a planned schedule is released to the public, consumers can then proactively weigh in on datasets that are being released, planned for release, or even those datasets that are not scheduled to be released. One surprising remark consistently heard at the roundtables was the fact that companies were willing to answer voluntary questions when downloading datasets as a mechanism to provide feedback. This is something that we did not expect, however companies said that this is an excellent way to generate feedback and gain insights into data


codx.ca

8


download and usage statistics. This process will also serve as the first step to creating a feedback loop within your civic community. Having an open data program where data release is guided by distinguished advisors as well as civic engagement should, without a doubt, be the end goal of any open data initiative. This fundamentally increases transparency through stakeholder involvement in the organization’s open data process, and will lead to a more streamlined data release process in the end.

Internal Ingenuity Recommendation 7: Spend time communicating internally Internal communication is the best way to ensure inter-departmental buy-in when it comes to your organization’s open data initiative. Communicating to each team how open data can be used to improve their work is not just a recommended first step, but essential for success. One challenge to open data adoption that has been frequently cited by municipalities across Canada is the constant friction that opening data creates internally. There are a variety of reasons why open data can receive pushback: • It may result in a loss or reduction in revenue, • Job descriptions can be significantly changed, • Fear that mistakes in data can result in demotion or job loss, • Assumptions that privacy would be at risk, • Worry that open data will be used maliciously, • Skepticism around the value of open data, and • An expectation that it will create more work. Any one of these reasons can stop open data in its tracks. Without significant buy-in from senior executives and council members, these underlying problems can intensify. Therefore, once senior decision makers are on board and the initiative has internal traction, it is important to approach open data from the perspective of consumers – what do they need from open datasets? Instead of going out to each stakeholder group as the open data team, it is highly valuable to leverage the insights of public servants who engage frequently with potential data consumers. One of the best examples of internal communication was an open data manager who created customized presentations for each department to demonstrate how other jurisdictions were using similar open data, and what types of successes they were experiencing. When the application of open data is measurable and tangible, data owners may become more willing to release their data.

Okay, but how do I get senior-level buy in? The most effective way of obtaining senior-level buy in is by demonstrating the benefits open data can create internally. For example, one of the interviewees calculated the staff time required to answer multiple


codx.ca

9


access-to-information requests for the same dataset, and used the resulting cost to demonstrate the cost savings by proactively releasing the dataset. To assist in identifying what type of information might be helpful in these internal discussions, consider the following questions : 1. How will open data be used to advance economic development? 2. Are there other jurisdictions where these datasets are used by companies? 3. How can releasing data help the department create efficiencies through reducing workload? For example, identify which datasets are frequently requested and require substantial time and effort to compile. 4. How might open data users augment the quality of data they’re working with?

Recommendation 8: Start with small, quick wins. Releasing open data can seem like a daunting (and perhaps compromising) task, which is why many of the municipalities ODX interviewed stressed the importance of small, quick wins. This serves two purposes: 1. It demonstrates to internal teams that the process of opening a dataset is not as challenging as they may think, 2. It validates the support champions have given. Regardless of whether you are just beginning your open data initiative or growing your existing program, quick wins are always welcome. Identify and use datasets that are already public-facing, and seek guidance from similar municipalities. A tool like the Open Cities Index has been referenced in many municipalities and serves this exact function. Alternatively, datasets that are public-facing have already been approved for release, and as a result, it becomes much easier to obtain final approvals. It was even recommended that data from annual reports are the simplest and most useful as they are already in the public domain. Annual reports will likely contain years of historic data, providing consumers with context as well as trends.

‘Open by Default’ Another intriguing recommendation put forth by municipalities was to be cautious of immediately adopting an open-by-default policy. This was a bit of a surprising discovery. The reasoning behind this recommendation reflected human nature; individuals who are already resistant to open data will become even more defensive if open-by-default manifests immediately. Rather, begin by working with internal champions around specific datasets and create a strategy. Creating an internal dialogue around datasets that could be problematic in the future is also beneficial. The municipality must discuss open-by-default in relation to their strategy – does open-by-default enable the current strategy? This type of open discourse can ease staff concerns when it comes to participating in open data initiatives. Should there be disagreement around which datasets should or should not be opened, productive discourse can be used to reduce objections to making data available. It should also be noted that while open-by-default may seem attractive to gain quick wins, those wins will by no means be repeatable. Data users will have high expectations when such a policy is adopted, and there may be too many datasets to manage effectively. It is better to be methodical about early wins to ensure continued momentum.


codx.ca

10


Identified the small wins? Great, now let’s pull it all together! A word of caution is needed – avoid releasing datasets your jurisdiction does not own. This topic has already been covered under data governance; however, in many jurisdictions data ownership can be quite complicated, and it is crucial that the data supplier have ownership of the raw data. If your jurisdiction does not have ownership, it becomes difficult to make any changes or updates if you rely on a third party to make the data available to you. Similarly, it is difficult to vouch for the accuracy and availability of data when it originates outside the reach of your organization. It all comes down to having a repeatable process management system in place. In the initial stages of adopting an open data program, there will not be a standard approach for capturing important learnings from projects, or a way to ensure knowledge transfer between staff at the end of projects. Over time, individual teams may begin to implement processes to capture useful information relating to their open data activities. Once established, many of these processes will begin to focus more on knowledge transfer at the end of a project, or when team members change, but still will not be carried out as an integral part of the entire lifecycle. Once an organization begins to allocate significant resources to mature their open data program, defined policies relating to knowledge capture, transfer and sharing will emerge. Here is where knowledge capture begins to apply throughout a project or dataset lifecycle opposed to just at the end. This will eventually lead to all staff having access to relevant documentation and policies as the organization actively encourages the sharing of data and knowledge internally. Admittedly, this is fairly ambitious. Nevertheless, establishing a repeatable process management system creates a strong foundation which enables long-term success. Such a system can lead to better internal use and awareness of how data can be applied, allows for the transfer of knowledge regarding core datasets, and actually mitigates risks when launching new open data initiatives.

Community Engagement Recommendation 9: Feedback mechanism and external communication One of the greatest resources a city can tap into is its local community. ODX found that citizens and entrepreneurs across the country are willing to share feedback and contribute their time to improve their city’s open data initiatives. This included contributing to strategy sessions, sitting on advisory boards and working with data owners to improve individual datasets.

Use your community Identifying members of the community is one of the first steps in developing a community engagement program. An early hurdle to overcome is the fact that one of the foundational principles of open data is the elimination of impediments to data access (eg. login, registration, providing contact information). As a result, it becomes difficult for municipalities to gather information about its users. For example, answers to questions regarding ‘who is using the data?’ and ‘what is it being used for?’ provide open data teams


codx.ca

11


with the ability to translate open data into tangible outcomes. Although no municipality has yet been able to solve this challenge, companies recommended at least some form of feedback mechanism where they can provide insight into how open data could be improved. Companies may not be aware that if they require a dataset, a small amount of background work to identify key economic, social or environmental outcomes of data release would help equip open data managers with information to advocate on their behalf. If they are looking for a dataset from a jurisdiction that does not have dedicated open data staff, doing some of the work for the municipality would go a long way, especially if they are in the early stages of their initiative. Similarly, it is important for consumers and municipal staff to connect so that both sides are aware of the kinds of projects happening in the civic community, how they are doing it, and what successes have been achieved all through the use of public sector data.

Get involved As previously mentioned, it is important for cities to identify local stakeholders and begin outreach. A good place to start would be open data book clubs, as these groups of open data enthusiasts will have a sense of who the local open data leaders are and where you might find pockets of interest. Once these stakeholders have been identified, the next step is to determine which method of outreach is the most beneficial. Typically, newsletters or emails have been the most efficient method of outreach. It is also valuable to conduct regular in-person outreach, by attending or hosting events where stakeholders can meet personally. We also heard from cities who found it valuable to include key community members on their advisory boards. Another group that should be included in any new open data initiative is other municipalities who have already engaged in such projects. As ODX found during the Cross-Canada Tour, the experience and knowledge these individuals possess is invaluable.

Recommendation 10: Work with CDMN Hubs in your region The Canadian Digital Media Network (CDMN) is a founding member of ODX. CDMN connects Canadian technology companies to a national ecosystem of 26 commercialization hubs where they have access to technical, legal, financial, marketing and business resources needed for success. In preparation for our Cross-Canada Tour, CDMN extended invitations to local data-enabled companies to attend roundtable discussions between municipalities and businesses. In convening the roundtables, we heard from stakeholders the value that local CDMN hubs brought to municipalities looking to connect with data-enabled companies. For example, in one of the cities, the local hub assisted with running a hackathon sponsored by the local municipality. Throughout the entirety of ODX’s Cross-Canada Tour, each of the municipalities we spoke with focused on improving the way they connect with the local business community, an area in which CDMN hubs excel. Rather than trying to build an outreach strategy from scratch, a quick conversation with local hub staff may prove fruitful as they may already have lists of companies by focus area (including open data). CDMN hubs are strong in the area of communication and collaboration. As members of a network of 26 like-minded organizations, they will have access to other regions where open data may be a strength. Their ability to make introductions across Canada should be leveraged by your open data team as it ma-


codx.ca

12


tures and creates more opportunities for companies.

Conclusion This blueprint has identified and addressed many of the challenges that organizations in the public sector face when becoming open data suppliers. Without a doubt, the commercialization of open data in Canada has a long, and relatively unstructured road ahead. Each recommendation articulated in this blueprint is based on feedback and best practices seen in major cities across Canada. Accordingly, each recommendation will help municipalities better align existing open data initiatives with private sector needs. However, there is no magic bullet; each company will be different, and each solution will have unique needs. It is therefore incredibly important to reach out to open data users and engage them. Municipalities and other public sector organizations should use this blueprint to guide the launch or advancement of their own open data initiatives. Given that open data programs are living, ever-changing initiatives that need to adapt to new and innovative technology, this blueprint may not hold the same value in years to come. As such, ODX has identified the Demand-Driven Open Data Maturity Model as a follow-up tool to this blueprint. The Demand-Driven Open Data Maturity Model serves to help organizations assess how effectively they meet the publication and consumption demands of their open data users. This aids the assessment of operational and strategic activities associated with open data, establishes thought leadership, provides guidance on potential areas for improvement, and helps organizations compare themselves against one another to highlight their respective strengths and weaknesses. Public sector organizations will use the ODX Demand-Driven Maturity Model to set appropriate resourcing and organizational goals. This self-assessment tool not only compares municipalities in regard to the amount of data published and consumed, but it identifies next steps no matter the stage of their respective initiatives at a particular point in time. Regardless of the level of maturity, each and every open data initiative must establish a framework for identifying and understanding the different areas of organizational change towards open data. This is something the ODX Cross-Canada Tour uncovered, and something the ODX Demand-Driven Maturity Model will answer.


codx.ca

13


Appendix A: Atlantic Canada Interviews and roundtables convened • Moncton, NB • Charlottetown, PEI • Halifax, NS Common themes • need for national data standards • overnment uncertainty of which data to prioritize • importance of internal open data champions in government • communicating the value of open data within government • provision of open data is a public service

Overall insights Overall, ODX heard a consistent desire to support open data businesses from stakeholders across the province. However, the need for better communication between businesses and governments was clear. One of the best examples of this emerged around which datasets to release; on the one hand, companies are asking for inventories of available data, while on the other hand, governments are asking for direction from entrepreneurs on which datasets they should prioritize. Addressing this misalignment represents a significant opportunity for both data suppliers and data users. It was also clear from our afternoon roundtables that innovation-support organizations like the Venn Centre and Volta provide value to the ecosystem by connecting government and entrepreneurs. In both Moncton and Halifax, municipalities had worked with Venn and Volta to run competitions/hackathons, communicate with entrepreneurs and raise awareness around the opportunities of open data. Moving forward, ODX sees an opportunity for organizations like the Canadian Digital Media Network to support and encourage interactions between companies and governments across the country.

What we heard from companies Key Issues 1. Need to know what datasets are available and where to find them 2. Need a Canadian standard for data formats, structure and release platforms 3. Single point of contact for questions and feedback 4. More equitable procurement process that supports startups and SMEs, including innovative procurement models 5. Remove pay-for-access and single-license access to government datasets Companies expressed difficulty scaling their businesses outside of local jurisdictions due to incompatible data formats and infrequent updates of datasets. Without reliable access to data that uses similar formats and structures, companies are stuck selling their products and services within confined markets. As there is currently no nationwide data standard, nor is there an entity mandated to co-ordinate standards, this is a persistent challenge that we expect to encounter as our tour progresses through central and western


codx.ca

14


Canada. When looking for datasets, we heard that companies prefer working directly with open data managers, rather than searching through data portals themselves. Companies felt it was more efficient to deal directly with the individuals who are responsible for monitoring data quality, updating data sets and maintaining the portals. It is for this reason that companies were eager to learn who their primary contact point in government was. A single contact point for questions about data sets was noted by companies as a highly desirable best practice. Companies were emphatic during the roundtables that data sets released by government did not need to be perfect – in fact, they wanted to work with government to help improve the quality of their data. In many cases, they were hoping to improve government data by including an additional layer of analysis, and sell the resulting insights back into government. This presents another critical issue identified by both companies and government: the procurement process is complex and lengthy, and in many cases not optimized for smaller vendors. Unfortunately, this is a structural issue within government that requires a large investment of time and effort to change.

What we heard from government Key Issues 1. Need a Canadian standard for data formats, structure and release platforms 2. Desire for direction from companies regarding data sets to prioritize 3. Open data is a ‘side of desk’ project, lacking budget and staffing 4. Better communication in government: Who will use the data? How will they use it? What will the social, economic and environmental benefits be? What is the cost? 5. Open data is often seen as a risk to government at the working level Across Atlantic Canada governments expressed a commitment to opening data sets and creating an environment that supports companies using it. Despite this enthusiasm to help the private sector, capacity and privacy challenges within government can create seemingly insurmountable hurdles to increasing the frequency and quality of data release. Governments were eager to adhere to a national data standard. They were straightforward in their request: ‘someone just let me know what format the data should be in, and I’ll try make it happen’. It is in this vein that one set of participants indicated the need for a national-level, 3rd party organization to set standards around data formats and structures. It was felt that a non-government organization with dedicated and knowledgeable staff would be able to more quickly assess which data standards would be appropriate. A key roadblock to opening more data sets was an inability to communicate to internal groups what open data is and how it can create value. We heard from multiple participants that due to an unfamiliarity with how the data can be used, government employees often oppose opening data sets; there is concern about how it can be used maliciously, or that mistakes found in the data can hurt their career. This presents an opportunity for companies to better communicate to government how they intend on creating value from open data sets scheduled to be released. By making this case, government open data champions will have a stronger argument when advocating for data releases. Despite these challenges, a dominant best practice emerged – leveraging internal champions is the best way to advance open data initiatives within government. Since open data strategies are usually initiated by


codx.ca

15


the premier, mayor or local councilors, these initiatives can quickly lose momentum at the working level. It is therefore important to identify champions outside of the open data implementation team, and have them speak to how opening data sets has had a positive impact on their departments and jobs.


codx.ca

16


Appendix B: Central Canada Interviews & Roundtables Convened • Ottawa, ON • Montreal, QC • Toronto, ON • Waterloo / Kitchener, ON Common Themes • Data Quality • Open by default • Need for national data standards • Business Intelligence and Analytics • Absence of a solution that tracks and measures impact

Overall insights In the Central portion of the Canadian Open Data Tour, ODX learned that some of the previous themes such as: the need for better communication between businesses and governments, extends to both inter and intra governmental relations as well. Not only are companies asking governments for data – but governments are also asking departments for data; both facing similar problems. A key solution that was proposed was to have two data catalogues; one catalogue for internal use showing all data that an organization has internally, and one catalogue which is visible to the public – the Open Data Catalogue. In this instance, an organization can post data to the internal catalogue and when they are sufficiently comfortable with the information, they can then flip the switch and publish it to the external facing Open Data Catalogue. Internal business intelligence and analytics departments are also gaining prominence in some of Canada’s most advanced cities. To date, municipalities have yet to find an adequate solution that allows them to track which companies actually use the information published by the city. To a large extent, the only time a municipality will hear about a company using the data which has been published is when the company attributes the data source, or when the company reports an error with the data – this is clearly problematic. To address this problem and to make sure departments are consistently putting out the best quality data, cities have established BI and analytics programs to analyze data that has been published and provide feedback in return to the city.

What we heard from companies Key Issues 1. Creation of a ‘Master list’ of all government information – even unreleased information 2. Understanding how to properly apply security measures 3. Lack of internal governmental communication 4. Reasonable procurement processes that allow startups and SMEs to view all RFPs without barriers 5. National data standards, formats, structures, and release platforms


codx.ca

17


So far, we have heard that companies are having difficulties around scaling businesses outside of their local jurisdictions due to incompatible data formats and infrequent data updates. Without reliable access to data sets that exists in standardized formats and structures, companies are confined within geographical markets. Although the lack of standardization is an overarching concern, it is not a concern when it comes to finding and using historical data. Companies have expressed a need and interest for historic and legacy data, however, concerns around this type of data differs as when it is released by municipalities it does not have to be perfect. Companies are willing to put their own time into historical data as when it is released it can be used to make more robust products and services. When it comes to searching for applicable RFPs, companies made it clear that any charges or transactional fees which add friction to the process severely limits the growth of start-ups and SMEs. This discussion was also framed around the fact that the current RFP process limits innovation by tailoring itself to larger, more established organizations. Companies in this portion of the tour also noted that data quality coming from municipalities is often very poor. This is a concerned that has been echoed in other cities across Canada as well. Generally, companies found that this issue stemmed from the fact that those within a municipality tasked with identifying and releasing data sets, often have done this off the side of their desk. This, along with the fact that a lot of the data published is operationalized and not standardized has resulted in poor quality data. It is for this reason that companies were eager to learn who their primary contact point in government was. As such, a single contact point for questions about data sets was noted by companies as a highly desirable best practice.

What we heard from government Key Issues 1. Inter-departmental communication around identifying and releasing data 2. Need a national and international standard for data formats, structure and release platforms 3. Desire for direction from companies regarding data sets to prioritize 4. Open data is a ‘side of desk’ project, lacking budget and staffing 5. Better communication in government: Who will use the data? How will they use it? What will the social, economic and environmental benefits be? What is the cost? Governments across Canada have expressed a commitment to making data more liquid and to foster an environment that supports the commercialization of open data. That said, to help the private sector integrate the data into products and services – capacity and privacy challenges within government must be addressed. Currently, these challenges pose as insurmountable hurdles to increasing the frequency and quality of data that is released. One common theme that has been carried forward through every interview thus far is that: governments are actively trying to find, establish, and adopt national data standards. While it is true that few individual municipalities actually have the capacity and resources available to them to establish standards and structure for releasing data sets themselves; they are still actively looking to adopt previously established standards. Without a doubt, a fundamental roadblock for governments of any level when it comes to opening various data sets, is an inability to communicate internally. Whether it be connecting the value or ROI of open


codx.ca

18


data, or establishing a process for doing so – its tedious, extra work for many government employees. Keeping in mind that government systems traditionally, and historically were built for single purpose processes – these legacy systems are not made for publishing data to any type of server. This fundamental problem unlocks an opportunity for companies. By communicating to government how they intend on using the datasets released, companies can successfully shape the future of how governments release. Open data champions within government will then be able to create a map for both future infrastructure investment as well as clearly establish processes around collecting and releasing municipal data sets. Notwithstanding the challenges that every municipality seems to be facing, best practices can be seen emerging across the board. As mentioned, municipalities have begun to incorporate two open data catalogues into their programs; an internal, and external catalogue. The internal catalogue hosts every single data set that exists while the external hosts only those approved. This practice is proving to be extremely efficient in establishing clear processes, and schedules within the municipality around publishing datasets.


codx.ca

19


Appendix C: Western Canada Interviews & Roundtables Convened • Winnipeg, MB • Edmonton, AB • Strathcona County, AB • Calgary, AB • Vancouver, BC Common Themes • Prioritize data sets seen as ‘low hanging fruit’ for release • Importance of open data champions • Communication and outreach to local open data community • Focus on data quality, not quantity • Full time staff dedicated to open data necessary

Overall Insights A consistent theme across our tour has been the willingness and eagerness for companies to work with their municipal counterparts. What’s important is that companies do not expect governments to have all the answers nor do they expect perfection. Rather, they seek ways to work collaboratively to advance the open data agenda. Companies are similarly interested in participating in advisory panels and roundtable discussions which have an impact on government decision making related to open data. If governments are interested in leveraging open data to its fullest impact, leaning on the data users will be a key method of capitalizing on emerging opportunities. In convening the roundtables, we also heard from municipalities they want to hear from companies how open data is being used. Herein lies a key challenge – municipalities are having a difficult time connecting with companies that are successfully commercializing municipal open data. These success stories can then be used by municipal staff to demonstrate the economic impact of open data in their region, which can then make more resources available to open data projects. It cannot be understated how important establishing clear lines of communication between companies and municipalities is to advancing open data. The only question which hasn’t been addressed is which outreach/communication method is the simplest and most effective.

What we heard from companies Key Issues 1. What data sets are available and where to find them 2. Importance of version and quality control of data sets 3. Need for common data sets across Canada The companies we heard from were very eager to work with governments releasing open data sets. Furthermore, they were eager to give back to government by helping to clean data sets and feed them


codx.ca

20


back in. The challenge, however, is government’s ability and receptivity to ingesting data that has been manipulated by an outside source. The question becomes: is it easier for government to clean up data sets internally (slowly), or is it easier to review and approve altered data sets that have been cleaned by interested parties? This presents an opportunity for improved and expanded partnerships between businesses and municipalities no matter what the decision is. There is also a large desire for consistency; among data sets, data quality, release dates, and contact points. For the most part, companies are unconcerned with the specifics, they simply want some consistency that they can plan for. Therefore, they suggest to municipalities to establish processes and stick with them, provide ample notification of impending changes, and only make changes when necessary. Companies already go through the process of manipulating open data sets for their own purposes, and any aberration from the norm represents a significant risk to their products or services.

What we heard from government Key Issues 1. Importance of champions at leadership levels (bureaucracy and council) 2. An OD policy is a valuable starting point 3. Begin with data that is easy to release 4. Communicate value of OD to internal stakeholders 5. Require better mechanisms to communicate with OD community A commonality across all municipalities has been the challenge of pitching open data to internal stakeholders, many of whom are struggling to with the cultural shift associated with greater transparency and openness. In response to this, a change of tone emerged in our discussions with some government OD stewards, who indicated they no longer pitch open data as an asset for the community. Instead, they focus on how OD can bring value to internal stakeholders by reducing paperwork, freeing staff time, and even cutting costs. Eventually the benefits of OD will flow to the wider community, but that is not a compelling value proposition for government stakeholders who are forced to change the way they work. An interesting struggle also exists for municipalities between finding out who is using their data sets and the importance of removing all impediments to data access (ie. no login, no registration, no email/name required). Above all, privacy is front of mind. However, contrary to popular narrative, companies were very eager for a platform to share thoughts and ideas and were in fact encouraging governments to gather names and emails to establish ongoing conversations with OD users. We heard from governments a simple approach to starting with open data – start with data sets that you already provide for free, and simply automate the process. By starting with these data sets, it becomes easy to quickly convert staff into internal champions. Other approaches to identifying data sets for release include those that are subject to frequent access to information requests, digitizing annual reports or quarterly updates, and talking with staff to identify ‘clean’ data that is produced with little effort.


codx.ca

21