Forecasting in the Czech public administration - preliminary findings

post by janklenha · 2023-03-16T14:47:28.670Z · EA · GW · 1 comments

Contents

    TL;DR
  Intro
  Our work so far
    1. Scoping
    2. Forecasting
    3. Implementation
  Our next steps
None
1 comment

TL;DR

Intro

In this post, we discuss some of our preliminary findings from the FORPOL (Forecasting for Policy) project run by Czech Priorities in cooperation with Metaculus. Our main goal is to provide other organizations with real-life experiences and recommendations aimed at making forecasting more policy-relevant and accepted by the public administration.

In Czech Priorities (EA-aligned NGO), we consider ourselves to be in a good position to advance this knowledge for three main reasons: 1/ we have experience in organizing large-scale local forecasting tournaments, 2/ the Czech Republic is a small enough country for us to have developed good connections with government analysts and politicians at most of the public institutions, and 3/ our team has a strong consulting expertise.

Based on expert consultations and preparatory work in the summer of 2022, we developed a strategy of individually reaching out to a large number of selected policy partners (in forecasting-relevant areas), working with them intensively, and monitoring the interactions.

Contrary to what might be expected, we did not work with organizations in the realms of foreign policy or the local intelligence community, as we wanted to explore possible use cases beyond these areas and we have stronger links to potential “champions” of forecasting (i.e. senior officials) in other governmental ministries, departments, and agencies. 

The expected value of our approach lies in delivering impact and success stories and, more importantly, in understanding precisely what works in various situations and interactions. This will be the main content of our study, which will come out in September 2023. Until then, some of our previous suggestions on how to use forecasting tournaments in policy can also be found in our methodology manual (České priority, 2021).

Our work so far

These are the Czech institutions (12 public inst., 2 NGOs, 1 foreign ministry) that we have partnered with so far, and the topics we forecasted for them (2-4 predictions per topic):

  1. Ministry of Regional Development: Accommodation for Ukrainian refugees
  2. Ministry of Regional Development (different section): Urbanization
  3. Ministry of Education: Deferrals in Czech elementary schools
  4. Π Institute (Institute of the Pirate political party): Digitization of the society
  5. Ministry of Health - The fade-out of Covid-19 (tests, vaccines, & hospitalizations)
  6. (undisclosed) - Labor market
  7. STEM (Public polling agency & research institute) - Perceptions of security & trust
  8. Ministry of Education - The demand for teachers and principals
  9. Czechitas (NGO providing IT courses for women) - Women in the IT industry
  10. Slovak Ministry of Finance - The future of the European economy
  11. Ministry of Labour and Social Affairs - Insolvency & Foreclosures
  12. Ministry of Trade and Industry - The future of Industry
  13. Ministry of Education - Lifelong learning
  14. Technological Agency (main public R&I funder) - Financing science and R&I
  15. Ministry of Justice - Forensic experts

On a broader scale, we discussed the potential and the need for the use of foresight methods in public policy with many other stakeholders. This included the newly elected president Petr Pavel (the conversations took place during his campaign), who remains interested in using participative foresight methods to predict national risks & opportunities.

Our tournament is still running and we are in the process of helping selected policy partners in leveraging the predictions, but we wanted to share some of the findings early on. Our model of cooperation with policy partners has three main phases, along the lines of which we structure the remainder of this post.

  1. Scoping - Multiple meetings to specify the real needs, motivations, and bottlenecks, explain the Theory of Change, and discuss the future use of the predictions.
  2. Forecasting - Submitting questions (2-4 in each thematic “challenge”) to the platform, where an average of 100 forecasters discuss & predict for 3 weeks.
  3. Implementation - Delivering a written report with aggregate predictions and the main identified causalities or drivers, and organizing “implementation follow-ups”.

1. Scoping

2. Forecasting

3. Implementation

Our next steps

Starting in Q2/23, we plan to split our forecasting process into two separate mechanisms. We are now considering a few design choices for both the recruiting mechanism (long ongoing vs. short intensive tournaments) and the expert forecasting mechanism (general vs. topic-specific, publicly visible vs. on-demand access, etc).

We are in the process of working with policy partners, conducting individual talks with the 30+ expert forecasters who will participate in our expert forecasting team, and mapping the demand for “rapid predictions“ as a product even outside of the policymaking sector. Recently, we also had the opportunity to publish forecasts and talk in one of the largest Czech weeklies, Respekt (podcast in Czech), and to work with a network of private schools (SCIO) on running tournaments for elementary and high school students, which are both streams that we want to continue working on.

Any advice, recommendations, or experience sharing would be appreciated, let me know! We’re also happy to help or consult with folks starting similar projects.
 

1 comments

Comments sorted by top scores.

comment by Lizka · 2023-03-22T10:45:11.570Z · EA(p) · GW(p)

Thanks for sharing this! I found it interesting to read about your process. In case someone wants to read a summary — Zoe has one [EA · GW]! 

Assorted highlights/insights I pulled out while reading:

  • Useful for engagement ("Drop-off across the six months of our tournament is around 75% (from ∿160 to ∿40 weekly active participants)"): prizes for the most informative rationales in each thematic “challenge” (every 3 weeks), having some questions designed to resolve quite soon after opening to provide early feedback (although it's important to avoid making these distracting)
  • This was an interesting section: "The involvement of domain experts was useful especially to increase trust and prevent future public criticism"
  • "when the Russian invasion of Ukraine started, it became clear that many of the refugees would flee to the Czech Republic. Our forecasters quickly made a forecast, and we used it to create 2 scenarios (300k and 500k incoming refugees). We then used these scenarios in our joint analysis with PAQ Research on how to effectively integrate the Ukrainian immigrants. The study was then used by multiple Czech ministries in creating programs of support for housing, education, and employment. This happened at a time when widely circulated estimates spoke of tens of thousands of such arrivals by the end of 2022. In reality, it was over 430 thousand people."