When Metrics Become Targets

An exploration of various ways Goodhart's Law shows up in Canadian innovation policy frameworks

The common theme of most of Canada’s innovation challenges is captured in Goodhart’s Law:

“When a measure becomes a target, it ceases to be a good measure”

Misaligned metrics that have since become the targets underlies most of the issues of policy fragmentation and risk intolerance that are the subjects of so much of my writing, creating negative feedback loops that reinforce the underlying problems.

In this article, I get into the weeds of how Goodhart’s Law manifest in several closely-connected areas of Canadian innovation policy, including our approach to risk management, how we think about the relationship between government and competition among emerging technology firms, procurement, and the ever-fraught issue of foreign acquisitions.

Risk Management

Within Canadian public funding for innovation, there is a deeply entrenched idea that sound governance of taxpayer dollars means that every expenditure of public funds has to create a success story.

On the surface, this is an admirable and rational goal: if the government is going to spend taxpayer dollars on anything, it should be able to point to clear benefit to the taxpayer arising from that specific use of public funds.

For an infrastructure project that will be delivered over years by a single lead bidder, this approach makes sense. Technical risk is low, we understand everything that needs to happen, and no new inventions are required. All the risk is in the execution and timeline, and there is only one attempt being made. We either get it right, or the money is truly wasted.

Emerging technology, has a completely different risk profile. Many attempts can be made at low cost, most attempts will fail for reasons completely unrelated to the technology, and it is effectively impossible to predict winners early. A majority of such projects ultimately fail, and the minority that succeed create so much value that they more than offset the cost of the failures. The risk management approach used for infrastructure projects is not fit for purpose in this case but we continue to apply it anyway, treating both risk profiles as nails to be driven in with the same risk management hammer, which leads us to the logical but wrong conclusion that we should simply not invest taxpayer dollars in high-risk innovation.

The need to tell a success story for each individual investment actually creates aggregate under-performance as under-investment guarantees missed outliers, resulting in exactly the outcome the policy exists to avoid.

Technological innovation requires that risk be managed at the portfolio level, considering the aggregate value created by many projects together, rather than evaluating the success or failure of each individual project in isolation. Through this lens, the decision of whether or not to invest becomes less about avoiding blame for the failure of individual projects, and refocuses on the outcomes that matter: developing domestic technological capabilities and broad socioeconomic impact.

Tech Transfer

One of the many challenges that stands between technologies developed in federal labs and commercialization of those technologies is concern over accusations of favouritism by the government when licensing technologies to spinout companies. The thought process is that if a spinout of a government lab competes with an existing company in the market, any support for that spinout represents an unfair advantage, and may become the basis for lawsuits. Internally, there is also potential for conflict of interest when public servants are involved in supporting or founding that company.

These are rational fear that reflects the realities of the trade policy constraints under which the public sector operates, but together, they mean that the government often shies away from directly funding or supporting its own spinouts, or does not deal with the commercialization step that should follow some research at all. As a result, grants and funding are spent on research, but little goes to the early stages of turning that research into economic impact, in part to avoid the possibility of accusations of favouritism or the possibility of failure. Nobody gets fired over projects that don’t happened.

This is problematic, since domestic spinouts are more effective vehicles for disruptive technology than incumbent firms, and not engaging means we are leaving intellectual property (IP) collecting dust on federal shelves.

The United States solved this problem in 1980, but their approach is not the only one that has worked. China’s approach to the problem is also interesting. Instead of shying away from challenges related to competition, they make it a central feature of their approach to technological innovation. When the Chinese government wants to develop national capability in a technology area, they deploy funds toward many competing companies at the same time and let them fight it out in the domestic market. As Aman Narain puts it in his recent article:

“[T]he centre sets objectives, localities experiment with methods. The result is dozens of parallel experiments running at once. Survivors scale. The rest disappear, and no one writes their obituary.”

Most of those companies fail. In Canada, this would be viewed as a failure and a waste of taxpayer dollars. In China, it is considered the cost of developing domestic technological capability. Far from being wasted, the talent and IP developed by companies that fail are simply absorbed by their competitors, which become more economically fit in the process, until the handful of survivors are internationally competitive. ITIF refers to this approach as “economic Darwinism”, and state that:

“China’s subsidies are still on steroids, in part because of the intense competition between cities and provinces, which shell out massive funds to support local champions. In 2022, 99 percent of listed firms in China received direct government subsidies.

China’s extreme approach does not translate directly to Canada, for many obvious reasons. We operate on an entirely different scale, and cannot afford to fund hundreds of competitors in the same space, and our approach to government/industry relations is fundamentally different and much less hands-on than that of China. However, we can still learn from it. The way to avoid being accused of favouritism is not to avoid funding anyone, it is to adopt a portfolio approach that funds multiple parallel competing experiments in the domestic market, accepts failure as a feature, and allows projects to fail when it becomes clear they are not keeping up.

The other signatories to our free trade agreements do this regularly.

This is also something that can only be effectively done by the public sector, despite lamentations that the Canadian private sector is not investing enough in innovation. It makes little sense for a private investor to back multiple companies competing in the same space, but the public sector, being interested in economic development broadly while being disinterested in the specific outcome of a battle between competing companies, is actually able to engage in economic development in a way that the private sector never will.

Money spent on projects that ultimately fail will not be wasted if we support a vibrant ecosystem in which talent and IP can be absorbed when companies fail. If one company fails in such an ecosystem, most of the talent and IP will simply add to the strength of the winner.

Foreign Acquisition

While Canada presently leaks IP on exits to the United States and other jurisdictions, this is a direct consequence of a lack of early stage support, not a fundamental truth of Canadian innovation.

One of the many causes of this issue is, like most of our innovation challenges, the result of a self-fulfilling prophecy. There is a persistent concern that taxpayer funding for domestic companies just goes to supporting companies that later exit, subsidizing foreign benefit with Canadian taxpayer dollars. As a result, many Canadian funding programs for innovation wait to see signals that a company is viable and intends to stay in Canada before investing.

A good recent example is the change in eligibility requirements for the CanExport SMEs program. Previously, companies could access this after having $100,000 in revenues in the previous year. This year, that threshold was increased to $300,000, further freezing out the very pre-revenue companies that could be the basis for the next generation of Canadian innovation.

This fear itself contributes to the rate of exits: by denying companies that might exit access to domestic support, we force them to seek support abroad for their first cheque, greatly increasing the probability of the very outcome we are ostensibly trying to avoid.

It’s also worth noting that not all foreign acquisitions are bad outcomes. Given the neighbours, some attribution through exits is probably inevitable, but that does not mean it is necessarily a problem. Supporting companies that later exit is only a waste if the net benefit created in the process is negative. As with the commentary on competition above, a vibrant ecosystem that allows companies to start, grow, and scale domestically before exit is one that will retain the talent that made that company an attractive acquisition target and recycle the capital that resulted into the next generation of companies.

Procurement

Government procurement of emerging technologies is a powerful tool to direct innovation when used effectively. However, Canada’s approach has long been criticized for being overly prescriptive and rigid, typically following a waterfall model that involves fully specifying a solution before beginning the project.

Needless to say, this is not an effective way to engage with emerging technologies, which generally require iteration and experimentation to reach a final useful form, and for which full capabilities may not be clear from the start. (In fairness, the government recognizes the problem and has signalled that it intends to start addressing it).

Other jurisdictions recognize this. The American SBIR and DARPA programs, for example, both fund multiple competing, high-risk innovations that address the same problem, expecting 80–90% to fail, and are recognized as cornerstones of American technological dominance.

Canada has Innovative Solutions Canada, which is a structural clone of the SBIR program, but we lack the scale of demand of the American government, which is central to its success,. While Canadian government departments are technically mandates to spend a portion of their budget on innovation through ISC, most departments missed those targets without consequence and instead continue to buy from safe, established foreign multinationals. Instead of penalizing under-spending departments we simply changed the targets, a pattern that has repeated itself several times in other under-performing programs. It remains to be seen whether proposed changes to ISC can overcome these issues, but they are certainly a step in the right direction.

Outside of ISC, the fundamental difference between the American model and the approach taken in Canada is focusing on defining the problem that needs to be solved, rather than detailing how the solution must do it. The rationale is the same as that which I discussed in the risk management section above: the government uses long, detailed Request for Proposals (RFPs) as a shield against risk. If the project fails but the vendor followed the exact specifications, nobody gets fired. The misaligned metric for success is process compliance, whereas procurement systems that effectively engage with emerging technology and innovation instead measures problem resolution and domestic economic value creation.

The solution, clearly born out in other jurisdictions, is the same as with the other two issues above: take a portfolio approach to procurement risk management, specifying problems rather than solutions, and funding multiple competing attempts to address it. Allow some to fail en route when it becomes clear that they are not going to deliver. By incrementally engaging with emerging technology as it progresses toward a solution, the cost of early failures is minimized while the chance of actually achieving a solution is greatly increased. The Canadian government as an anchor customer would go a long way toward keeping Canadian innovation at home.

A good example of such an opportunity is in the development of quantum sandbox to help retain our world-class quantum expertise and IP.

The Path Forward

In risk management, public servants are evaluated on fiscal responsibility, and a costly failure can end a career. The target naturally becomes avoidance of failure at the project level, which does not allow for public engagement with high-risk, high-reward innovation.

In federal tech transfer, the intention is to create a fair and impartial playing field. The target becomes avoidance of the risk of a lawsuit or accusations of favouritism or conflict of interest, which can only be achieved by not engaging at all. This leads to starving domestic startups, and IP developed in federal labs never making it off the shelf.

In procurement, the goal of the framework is supposed to be fair and transparent sourcing. This devolves into measuring process compliance rather than problem resolution, which ironically leads to favouring large companies, often foreign multinationals, who know how to fill out the paperwork over less bureaucratically-inclined domestic startups.

In foreign direct investment, we decry foreign exits without considering the context, nor what happens after the acquisition, painting with the same brush an exit that simply loses research IP before it can be developed and one that seeds the next generation of domestic companies. This leads us to avoid supporting our own companies for fear they will later leave, creating an incentive, if not an existential requirement, for them to do exactly that.

The people who administer these programs are doing what the metrics on which their performance is evaluated incentivize them to do, guided by perfectly rational goals. The issue lies with the metrics themselves, metrics that are embedded in rigid policy frameworks that punish exactly the type of risk-taking Canada needs to move forward.

This structural gridlock creates a market failure. Private venture capital flows to jurisdictions where public policies supports the best balance of risk against reward. Because Canada’s policy frameworks hinder technological innovation, private capital looks elsewhere and domestic companies defect early to places with policies better suited to supporting them.

In short, Canada’s innovation challenges are self-inflicted. Changing this must start at the structural level in the public sector, by shifting from project-level to portfolio-level risk management, from avoiding complaints to funding parallel and competing experiments, from specifying rigid solutions to articulating the problems, and from fearing foreign exits to creating conditions in which those exits can become the basis for the next generation of innovation. Only when the government acknowledges its role as an investor of first resort and a disinterested anchor customer for homegrown innovation will private investment follow and allow Canadian innovation to stay and build at home.