CHALLENGES IN DATA-DRIVEN POLICYMAKING: USING SMART CITY DATA TO SUPPORT LOCAL RETAIL POLICIES

: In Flanders (Belgium), there is a growing awareness among city administrations and governments to turn their cities into smart cities by transforming their policy decisions into evidence-based and data-driven policymaking. Nonetheless, many Flemish cities still face several challenges related to the integration of data-driven policy evaluation research at different stages of their policy making processes. For that reason, the ‘Smart Retail Dashboard’-project was set up, which aims to develop a dashboard for helping Flemish cities in transforming their retail policies into data-driven policies. This paper reports on the initial findings of this project. The most important finding is the varying degrees of maturity across cities regarding their smart retail policies. This is illustrated by the current state of data-driven policy making in the retail sector in Flanders. Subsequently we provide an overview of the current challenges reported by mature and less mature cities, which range from the cost and the usability of data to the analysis and interpretation of such data. We discuss how a Smart Retail Dashboard could provide answers to these challenges. This discussion helps city administrations to transform their policymaking processes into data-driven processes and dashboard developers to optimise their development processes.


INTRODUCTION
The Smart City concept has been popularised (and hyped) for over at least a decade but turning its promises into practice remains a challenge for cities today. Most agree that technology has some role to play in supporting or implementing policy (Hollands, 2008), but how that role should be filled in remains unclear and is often the result of trial and error. Currently, city governments are exploring what the concept can actually contribute to their daily practices and which role technology can play in providing better or 'smarter' services to citizens.
One example of the rising popularity of technology in policy processes is the urge for data-driven policymaking, which refers to policy decisions made on the basis of objective empirical and evidence-based evaluation research about the context, need and efficacy of different policy programs rather than subjective intuition (Janssen & Helbig, 2018;Ruppert, Bernard & Kohlhammer, 2013). Indeed, while data-driven policy making has always been present to more or lesser extent in policy making, the increasing availability of vast amounts and new forms of data introduced by new information and communication technologies, as well as the increasing ability to combine data from different sources and domains, can provide new types of tools and insights to policy makers. This data can be captured from Internet of Things solutions (e.g. sensors in public parking garages), structured information in reporting systems internal to city administrations or detailed data on the public domain (e.g. from satellite imaging).
Also in Flanders, there is a growing awareness amongst policy makers and administrations in cities that data can play a crucial supporting role in the policy cycle (context, design, implementation, evaluation) (Van Compernolle et al., 2018;Walravens et al., 2019). This is translated into the way cities organise themselves and reflected in the profiles they attract: between 2017 and now, Flemish cities have hired smart city coordinators, data officers, data analysts and so on. They have begun to form working groups or networks within the city (and across departments) of people working with different types of data in their daily practice.
While the awareness of data-driven policy making is growing, a number of challenges remain. These include challenges related to data gathering, GDPR-compliance, digitisation in some cases, but most boil down to making data easy to interpret and actionable. In order to fully unlock the potential of this data, it needs to be more easily available and accessible than today. Not seldom, this requires complex decisions and solutions regarding data integration, data linkage, data standardization, data reporting and communication (Höchtl, Parycek & Schöllhammer, 2016). Moreover, these solutions are often not readily available as they need to be tailored to very specific resources, situations and policy questions.
The Smart Flanders program in Flanders , coordinated by 1 IMEC, defined and implemented a common open data policy for the largest cities in Flanders. Based on quantitative surveys and several rounds of interviews (Walravens et al., 2019) with representatives from the cities, four key challenges arise, which need discussion and answers if cities want to establish more sustainable data policies and start designing and evaluating policies using data. Summarised, the following topics are identified: • Data hygiene in the organization: In some cases, digitisation still is a significant challenge, but how can we generate awareness to the level of key individual public servants that work with data? How do we change working with data into an operational process that leads to good open data? • IoT and data: In the hype surrounding the Smart City concept a lot is made of the data generated by sensors and other IoT devices, but how do we access data from these sensors in a proper way, dealing with the real-time aspect, the sheer volume of the data, archiving of data and so on? • Centralised vs. decentralised: This is a debate related to which level of government is responsible for which aspect of the data gathering, processing, publishing and so on. How do we turn this into processes that work? This requires agreement on the roles of different levels of government. • Government and the market: where does the role of the government regarding data end? When do private actors come into play? Handing over too much control over the public domain to private companies raises concerns regarding democracy and the commodification of the public space (Greenfield, 2013;Townsend, 2013).
The cities involved in the Smart Flanders programme are all experimenting with new tools and data analytics for data-driven policymaking, but struggle to make these methods part of their daily operations. To dig into some of these challenges this paper will explore the potential for more data-driven policymaking in a specific domain: the local retail sector. It will discuss the current state of data-driven retail policy among Flemish cities as well as the challenges cities encounter in transforming their retail policy into data-driven policymaking.
The paper is structured as follows. The next section describes the Smart Retail Dashboard project. It also discusses the research methodologies used within the initial phase of the project for surveying the current state and needs among Flemish cities for introducing data-driven retail policy processes. Subsequently, the third section provides an overview of the current state of data-driven retail policies in Flanders, while the fourth section discusses uncovered challenges within these policies. The paper ends with points for discussion for the next stages of the project and for research on data-driven (retail) policies in general.

The challenge of local retail
In Flanders, the local retail sector is increasingly under pressure due to a shrinking market and changing consumer habits caused by e-commerce (Cardinaels et. al, 2019). Local retailers face decreasing numbers of customers, increasing operating costs compared to e-commerce companies and, as a consequence, lowering margins. Many small shop owners go bankrupt and get replaced by large chain stores. Additionally, online shops have a tremendous amount of data at their disposal, creating a huge competitive advantage compared to policymakers and local retailers. For that reason, local retailers and policymakers need to adapt in order to remain relevant in the market.
Data can help cities to customise policies in favor of the local retail. For example, if a city knows about the profile of its visitors through data, it can tailor its city marketing campaign towards these profiles. Also, if a city understands the optimal mix of shops based on the demand of consumers, it can adjust its policies to attract certain types of retailers. Retailers on the other hand can utilise data in order to, among others, determine optimal opening hours or optimize store supply.
Currently, data is already used in different public policy decision processes regarding local retail. For example, local administrations in Flanders are incentivised to create a strategic commercial plan for their city in order to define future policies. Cities already use data made available by the Flemish government to monitor retail offering and vacant retail real estates. Nonetheless, IoT data and data from private sources such as telecom companies provide new opportunities, enabling local retailers to adapt to a changing environment and allows local governments to monitor and evaluate the implementation of an urban retail plan. To date, limited IoT data is used in common policy cycles. Additionally, cities face challenges in interpreting data and evolving it into a retail policy.

Data-driven policy through the retail dashboard
The increased demand for data-driven policy making in the retail sector led to the 'Smart Retail Dashboard' project, initiated by the Flemish Governmental Agency 'VLAIO' (Vlaams Agentschap Innovatie en Ondernemen), which is responsible for innovation and entrepreneurship. Within the scope of this project, IMEC is managing a feasibility study to assess the need for gathering different existing data sources both from within the government (including socio-demographic data, data on opening hours etc.) as from external/ private/smart city data sources and visualizing this data in a Smart Retail 2 Dashboard. This dashboard would support policy makers and eventually retailers in making decisions based on actual smart city data.
IMEC performs the feasibility study in different phases: • Phase 1: Defining the needs and challenges regarding use of data of cities and retailers; • Phase 2: Defining the requirements of cities and retailers for a Smart Retail Dashboard; • Phase 3: Defining the conditions and availability of private data sources; • Phase 4: Open call for smart city data sources and IT 3 providers to build and pilot the Smart Retail Dashboard; and • Phase 5: Run a pilot version of the Smart Retail Dashboard to test it in two cities.
At the end of March 2020, phase 1 of the project was finalised, which resulted in an overview of the current needs and challenges that city administrations face when using smart city data to support local retail policies. These findings will be described in this paper.

Methodology
In the first phase of the project, explorative in-depth expert interviews were performed with 20 cities in order to define the needs and challenges they face and the requirements they have regarding a 'smart retail dashboard'. Within this project, focus was restricted to the two groups of largest cities in Flanders as defined by the VRIND categorisation ("centrumsteden" and 4 "structuurondersteunende steden"). Additionally, cities were also selected taking into account their geographical spread. More specifically, all Flemish provinces were represented by 2 We will refer to these three data sources as 'smart city' data throughout this article, i.e. data coming from sensors, wifi trackers, cameras in the city or telecom data 3 A form of public procurement through companies can apply alone or together to build the data platform. 4 Vlaamse Regionale Indicatoren The interviewees were mainly civil servants responsible for local economy and/or city center management , sometimes 6 supported by civil servants responsible for urban planning, mobility and/or ICT. The interviews were held with a semi-structured topic guide. Questions asked to the interviewees concerned the current use of data in their policymaking processes, the challenges they faced when utilizing data and their requirements for a Smart Retail Dashboard. In total, 20 interviews with different cities were held, each lasting at least two hours. Transcripts were made for analysis.
Based on the interviews, an overview was made of a) existing use of data-driven retail policy applications and b) challenges regarding such policies. Both overviews are discussed in the next sections.

RESULTS
Based on the current state of smart retail policy making, a variation in the maturity levels was identified across the cities regarding the use of data for retail policy, as further described in section 3.1. This results in different challenges as described in section 3.2. The maturity level is defined by 1) the extent internal data is currently used in the policy cycle; 2) the extent 7 private data is utilised in the policy cycle and 3) whether civil servants are able to interpret the data. This was measured by looking at the amount of (internal and private) data sources already used in the policy cycle and by the frequency of usage of the data platform allotted by the provinces (for more information see below). This brings us to the following profiles: direct communication with local retailers.
7 This is a combination of data sources that are provided by the different governmental levels in Belgium (Federal, regional, provinces), data from within the city administration or data from own existing city applications like loyalty cards.

Current state of smart retail
Results from our interviews made clear that, in Flanders, cities already started experimenting with (IoT) data for retail policy development. We will illustrate this by following the phases of a regular policy cycle, which are agenda-setting, decision-making, implementation and impact-assessment phase (Rossi, Lipsey & Henry, 2018;Stufflebeam, 2003). Nonetheless, there are still numerous challenges and barriers that cities face, which prevent actual data-driven policy making in the retail sector.

Agenda-setting phase
In the agenda-setting phase, policy problems are identified and put on the political agenda. In this phase, evidence can be collected about societal needs, problems, assets, opportunities and resources to support the need for political action and to grasp the attention of important stakeholders.
In Flanders, data is already used in the agenda setting phase for retail policy development because many cities use data in order to analyse the current state of their retail sector, to define the amount of shops in their area, the amount of vacancies of retail real estate and the current catchment area of consumers . Most 8 of the currently used data is delivered through a data platform provided for free by the Flemish Provinces . Several cities make 9 use of this data platform on a monthly basis, but less mature cities do not know how to use the data and how to include it in their policies.

Decision-making phase
In the decision-making phase, a decision is made about the policy programs that will be implemented for tackling the policy problems at stake. In order to take such a decision, evidence can be collected about the expected efficiency, feasibility and public support of competing policy programs (Höchtl at al., 2016).
Also here, Flemish cities already use data during the decision making phase of their retail policies when they create a strategic commercial plan. This plan includes the vision on retail for the city and it often relies on the same source of data provided by the Flemish Provinces. This plan might comprise determining the city's level of ambition towards retail and how this will be reflected in relevant policy measures, the creation of city marketing campaigns or in policies to develop the urban area of a specific space in the city. Data is utilised to predict the impact of the new policies, and to identify which policy program needs to be focused on.
The decision-making phase was mainly relevant for mature cities, where the use of data is already widely adopted. In the case of less mature cities, this phase was not performed with the use of data. In that case, decisions were primarily made based on "gut feeling". In the case where data was used, it was usually outsourced to external consulting agencies in the form of vision documents for the city. In this case, the vision statements they produced often were not implemented, kept internally and possibly shelved.

Implementation phase
In the implementation phase, policy makers try to monitor the implementation of policy programs in order to assure whether the implementation is conducted properly and whether it does not cause unintended side-effects (Khan & Rahman, 2017). Empirical evidence can be used to keep track of temporal targets being achieved, to make judgments about the program performance and to suggest adaptations or revisions of the program implementation.
Data is already used in the implementation phase of retail policy making in Flemish cities in the process of granting permits to different large retailers wanting to enter an area. When a new shop wants to open in an area, this can have a considerable impact on the sales of other shops, on mobility, the attraction of consumers from a city center to the periphery etc. Hence, data analytics and predictions are required to predict impact. Private companies specialised in geo-marketing provide prediction software to define the future catchment area, the impact on buying streams and on mobility.
Additionally, several mature cities also use data for city marketing. In order to define the catchment area and the profiles of (potential) visitors, cities engage with telecommunications companies selling data on the profiles of the visitors. The cities can define the origin of the visitor, socio-demographic profiles, profiles on social standing,... This helps mature cities to define social media campaigns and other marketing efforts with an increased return on investment.

Impact assessment phase
During the impact assessment phase, evidence is collected about whether the selected policy program tackles the policy problem at stake, to what extent it meets the targeted needs, whether it outperforms competing policy programs and whether it doesn't introduce unintended outcomes. The results of this evaluation research should lead to the decision to continue, stop, be revised or improve the program.
Also here, data is increasingly used in a number of mature cities to evaluate the success of implemented policies. A good example is the evaluation of the impact of events. To define the succes, the amount of visitors and the amount of transactions are the major KPIs. Currently, cities evaluate the success of an event based on subjective measures, such as the satisfaction of a (small) number of retailers, or by getting a perception on the amount of bags people carry on the street. As these measures are prone to errors and to misjudgement and bias, some cities are experimenting with the use of IoT technologies, such as counting the passengers with WiFi tracking or with cameras.

Challenges and solutions
Throughout the interviews, five major challenges have been observed. We discuss the challenges that occur, as well as potential solutions which will be provided within the Smart Retail Dashboard program. Table 1 provides an overview of the challenges and the potential solutions.

Lack of perceived data value
Design Dashboard to answer policy questions based on the policy cycle Include different types of dashboards

Cost of data
Affordable data: group purchase of data

Data Quality
Standardisation Data quality assessment Reliability measures

Combination of different data sources
Data platform, analysis based on different data sources

Skills for and interpreting data
Support system to educate Visualisation of dashboard Pre-defined data analysis and visualisation based on maturity of the city

Lack of perceived data value
The lack of data usage occurs most in cities with a low maturity level. A first reason is the lack of political will from the city government to utilise data in the policy cycle. These cities also lack resources. In many smaller cities only a limited amount of civil servants work in the local economy department and/or combine this with other responsibilities. Lastly, there is the perception of a lack of value derived from utilizing data in the policy cycle.
In order to increase the usage of data, the value of the data and the data dashboard needs to be clear for the different cities, with different maturity levels. Thus, the Smart Retail Dashboard will need to clearly answer the policy questions linked to the different stages of the policy cycle in order to put the usage of data higher on the political agenda.
Because different types of evaluation questions may arise at different stages of a policy-making process, such processes may also require very different dashboards at these various stages. As a result, it is important to clearly define the main goals of the program evaluation task well in advance and to meticulously translate these goals into appropriate data collection strategies and dashboard design instructions (Sarikaya et al., 2018;Yigitbasioglu & Velcu, 2012;Bartlett & Tkacz, 2017).
Challenges in dashboard development can be addressed by assuring that the dashboard addresses a clear goal from the start. The policy cycle can be used as an underlay for this. The different policy questions need to be clearly defined in advance and fit within the policy cycle. A clear goal will help the development team to make guided decisions about data collection, processing and visualization much earlier in the process. As such, the cost of dashboard development is reduced.

Cost of data
The cost of (private IoT) data is a major bottleneck for cities. The access to transaction data, passenger data and buying streams data is of high interest to several cities, but the high cost of acquiring the data is a large barrier for these cities. Some cities allocate budgets to the acquisition of data, but the quality of the data is not yet sufficiently high to reach an adequate return on investment for the less mature cities.
The Smart Retail Dashboard will need to be accessible to the different cities. Making the access to data affordable for the cities with high needs can be overcome through the utilisation of economies of scale of data generation, which might result in cost reductions when common data-collection strategies are utilised in different cities, as opposed to utilising different buying strategies separately.

Quality of data
Many sources still provide data of low quality. The comparability of data historically and geographically, for e.g. assessing the effect of a new circulation plan on the retail sector, is of a high importance, as benchmarking is required. Yet, many data suppliers change the technologies or algorithms used, which results in unreliable data, and the lack of standardization does not allow to compare different datasets delivered by different companies. Additionally, the cities perceive the access to data as inadequate as they perceive several errors in the data.
The need for standardisation and thorough agreements with the cities and the data sources is required in order to increase the interoperability of the different data sources. By standardizing the agreements on a higher level, the Smart Retail Dashboard will be able to offset the individual efforts of cities to come to these agreements with the data sources.
Additionally, a thorough data evaluation assessment will need to be put in place in order to validate the quality of the data. Transparency of companies is required in how they deliver data , which technologies they use and which potential bias might arise. Additionally, confidence intervals for data reliability are required to indicate data trustworthiness.

Combination of data sources
In order to enable policy decisions and to determine whether a policy was effective or not, different data sources need to be combined. For example, to determine the impact of new mobility plans on the local retail, data of the amount of visitors 10 by car and bike needs to be compared, and needs to be coupled with the amount of transactions. External factors, such as the weather, the seasons, the city marketing efforts and so on are also parameters that might influence the effectiveness of the policy. Thus, different data sources need to be combined in order to make a thorough analysis and decision.
The Smart Retail Dashboard will combine different data sources in one data platform, enabling the combination of insights. This will enable cities to come to conclusions through pre-defined analytical tools.

Skills for and interpreting data
Several cities do not have the skills to analyse data. Officials have a limited amount of time to utilise data. Additionally, several civil servants lack technical and/ or analytical skills. Cities face difficulties to link the analysis to actual policy making. Cities do not know which potential policies can be effective, and several cities are prone to make errors in their interpretation which might lead to ineffective and even counterproductive policies. An example given during the interviews is that cities might see that there is a high amount of vacancies of shops. The cities could conclude that more high level brands need to be attracted, which could attract more consumers. A more accurate conclusion might have been that the market space is too small and thus not interesting to attract new retailers.
When data is gathered, the optimal visualization and reporting strategy for each piece of information will be required, in order to facilitate the right decisions. Through analytical tools designed to answer the most pressing policy questions, the relevance of the Smart Retail Dashboard will be sufficient, as it will offload the need for high analytical skills of the city officials. When designing the Smart Retail Dashboard, high importance will be put in ensuring that it answers the questions of both the mature and the less mature cities, which will prove to be a balancing act between leaving degrees of freedom to analyse the data and with providing sufficient guidelines and structures to simplify the analytical process.
In order to support the less mature cities, support programmes of regional governmental agencies will be required, to ensure civil servants get adequate training in analysing and interpreting the data.

CONCLUSIONS
Cities are increasingly using data to support decision making. In order to do this, data need to be incorporated into the agenda setting, decision making, implementation and evaluation phases of current policy cycles. The Smart Retail Dashboard project identified the challenges cities face to use data in the policy cycle of the local economy. The five challenges that were identified are (1) the lack of data usage, (2) the cost of data, (3) quality of data, (4) the combination of data sources and (5) the analysis and interpretation of data.
There are different maturity levels in data driven policy in local retail policies in cities, as there are mature, medium mature and less mature cities regarding the way they already implement smart retail policy programs. The challenges and needs of the different cities differ greatly linked to the maturity level of the city.
In overcoming these challenges, the designers of dashboards should incorporate five different strategies in order to enable the usage and the value for the civil servants in cities.
1) Focused problem statement: In order to make the data relevant and valuable, a process needs to be set up with different policy makers and civil servants ensuring that the relevant policy questions are incorporated in the dashboard. Thus, the most crucial parameters will be added to the dashboard 2) Data cost : through the use of group buying, different smart city data sources can be purchased by different cities. This enables the affordability of smart city data. 3) Data quality : by standardising data parameters through agreements between different cities and smart city data sources, the quality and usability of data will be increased. 4) Combination of data sources: By combining data sources, additional context and interpretation can be given to outcomes of the dashboard. This ensures to provide more useful insights and conclusions 5) Visualisation and analytical tools : through the visualisation of outcomes, and through the incorporations of analytical tools, pre-defined analytics can be provided to civil servants, allowing the facilitate the development of skills regarding the analysis and interpretation of smart city data.
Further research within the Smart Retail Dashboard project will validate solutions to overcome the challenges discussed in this paper. More specific, the following research questions will be addressed: "Are the 5 proposed solutions the most adequate for solving challenges in data driven policy making?" and "Can a Smart Retail Dashboard support data driven policy making in the 4 phases of a policy cycle?".
Our research was restricted to retail policy and to cities in the region of Flanders, which forms a serious limitation in order to make statements about smart, data-driven policy making in general. Future research beyond the Flemish context and beyond the topic of smart retail may confirm whether our findings can or cannot be generalized to a wider context. We can refer here to projects in Flanders like VLOCA ( Flemish Open City Architecture ) or the above mentioned project Smart