This research project is focused on how well suited are regulatory models in adapting to dynamic change, a more interventionist state or exogenous shocks like covid or the financial crisis. Can the regulatory environment accommodate strategic policy initiative like decarbonisation, promotion of technological innovation or activist industrial policy without its function being impaired? This leads to questions concerning the nature of the regulator’s institutional ecology and will demonstrate the potential for truly effective reform. It would also provide an understanding of their organisational capacity to achieve its key objectives.
The research can be divided into four themes:
- Drivers for intervention: Government intervenes in the regulated industries if there are compelling reasons to do so. This includes extreme economic crisis, pending industrial crisis for a particular sector, externalities, the prospect of long term price rises for regulated monopolies or performance issues with the regulator. Each event creates different reactions either by the creation of a new agency, suspension of an existing regulatory framework or the regulator gets tasked with new primary duties or objectives.
- Decision making: Research on how intra regulatory relationships impact on decisions identifies the complex interactions between independent regulators. This includes compromise, joint initiatives and disagreement. How could this insight improve the quality of regulatory output?
- International Comparison: Comparisons between other regulated systems. Can the four models identified in the UK reflect similar issues in other countries? Are the drivers for regulatory change similar?
- Refinement: We present four case studies. Further refinement would be to include more sectors for example, the regulation of medicines in a pandemic (crisis); residual regulation in rail (policy failure); communications (fully integrated merger of smaller regulators); food standards?
Taken together the reassessment of regulatory independence is set to become an issue as the commitment to independent regulation still remains strong but is set against forces that lead to interventionism. Paradoxically this commitment has led to the creation of more regulatory agencies.
Independence from Government is the defining characteristic of regulation. It is distinct from Government policies administered through ministries and their agencies as it represents a transfer of decision making powers bounded by legislation. In the last ten years developments have meant that, for some commentators, ensuring competition as the primary objective of economic regulation has been relegated to a lower status than it once had. It is now one of the many competing objectives that are more interventionist in nature than creating market rules.
This paper argues that the biggest issue for independent regulation is not direct political interference as originally imagined in the debates over regulatory independence. Instead, we argue that it is the organisational and policy ecology regulators inhabit that materially influences the ability of regulators to make decisions for some sectors. From this perspective and in retrospect, it is difficult to imagine that at any one period of time regulators were ever fully independent. This is not because of the regulators were experiencing overt political interference, in the form of direct ministerial intervention in a particular decision, as seen in the Westland Affair (which was very much an exceptional case). It is a better overall characterisation of the UK regulatory model to observe that regulators are operating in an often turbulent and contradictory environment. This includes interventionist policies triggered by “crisis,” “emergencies” or the gradual realisation that a policy is “failing.” This leads to the conclusion that the scope for regulatory reform has changed significantly as Government, and bodies independent of Government, have an impact on each others’ operations.
We interpret what could look like a holding pattern between pro and anti-independent forces as a reflection of the numerous interactions between the state and regulators that has led to long term cohabitation and tolerance. From this perspective we argue that UK regulation is a product of non-linear development and is not cyclical dance of independence versus political interference. A better characterisation is that the British model appears to be multi layered and heterogenous. Competing policies with repeated interactions between them may or may not have an impact on relative independence. Not all sectors have experienced the same degree of complexity and indeed the original thinking and arguments for independent regulation still hold as they have done so for over 120 years. This paper uses four case studies to illustrate four stylised models that though they have their limitations identify relationships between institutions that would not have been noticed otherwise.
Origins of the debate for and against independence and why it still matters
In his book The Independent Regulatory Commissions Robert E Cushman¹ traced the origins of the arguments for and against independent regulation to the creation of the US Federal Trade Commission and Interstate Commerce Commission made between 1887- 1906. At that time, the word “independence” was not used very often. It was the arguably better concept of “impartiality” that covered what is effectively the same ground as independence. The arguments used in the debates at the turn of the last century now frame the core principles behind independent modern regulatory design which are now taken for granted.
The case for independent regulation from political interference via a body such as a Commission revolves around four main arguments. At the core of the case for independent regulation is balancing competing interests within a pre agreed legislative frame work. At turn of the nineteenth century, it was argued that the Federal Trade Commission would be able to defend the public against business interests, in effect becoming a “poor man’s court.” Without independence, politicians would intervene in the regulatory process to disrupt decisions either by influence from business or pressure from consumers. This would prevent rational technical decisions and could favour big business which was an important issue at the time and still is. In overtly favouring consumer interests, we could now also add that the investment climate for regulated firms would change if political interventions created regulatory uncertainty potentially increasing the cost of capital or prevent any investment at all in a particular regulated sector.
An independent regulator would also act as a court for disputes between industrial rivals and would render a valuable service with its recommendations to the legal system which would ultimately make final decisions. It is well worth noting that the UK experience of independent commissions that were set up to deal with a particular market problem were cited as evidence that this model could work in practice².
An independent Commission could also be flexible and provide expert oversight of a complex regulated sector. This would be superior to using an inflexible statute and the courts. This expertise could also be useful in advising the legislature in the form of future legislation.
Besides legality and independence principles the debate over the institutional design of competition authorities at that time did not consider a number of issues that we now think are important³. Transparency, that is fair accessible institutions and open with provision for stakeholder involvement, are vital for regulation to work. Enforcement powers such as remedies that the independent agency is able to apply and the design of the appeals structures were also omitted.
The case against independent regulator comes from two very different forms of criticism. In the American debate over the regulation of railroads, there were concerns about the actions of an “uncontrolled” competition authority that could impose its ideological views of markets on industry. The underlying issue was the degree of discretion the Commission could command. This would also have the effect of diluting the original intensions of the statute over the medium term.
From the arguments above, the appointments process itself becomes an issue. For its critics, an independent Commission has a number of factors that make impartiality almost impossible. For example, the expert recruitment pool is often limited for what are specialist economists; former members of the business community could be partisan and members of the “great and the good” would not understand the technical issues.
Other arguments at the time included the concern that the Interstate Commerce Commission would become the “football of politics” as a result of the American two party system. It was also argued that railroad regulation was too vast an area for one agency of the Government to control.
From a more modern radical political economy perspective, independence is impossible for substantially different reasons than the American objectors in the late nineteenth century. Hubert Buch-Hansen and Angela Wigger take a “critical political economy” perspective to apprise competition policy4.
They argue that economic frameworks that drive competition are not inherently neutral but are in fact political, arguing that “there is no such thing as a politically innocent capitalist market”. They argue that EU and by implication UK competition policy is neo-liberal in nature to the exclusion of other alternatives and therefore can never be politically neutral or free from interference.
Three Models of relative institutional autonomy
As the UK regulatory system has developed, we have seen the evolution of the debate and the realities of modern Government changing our understanding of independent regulation. There two very different rationales for market intervention. Independent regulation has rules are set out in statute to achieve specific objectives, it seeks coherence, requires a stable political and intellectual environment (including technical norms) to work. Regulatory discourse is based on case law, canvased by reasoned consultation and implemented by licence change, code modification or direction. By contrast where independence has not been respected, we can say high politics is incoherent, fast moving, unstable and intellectually random. High politics is based on broad principles, is inherently vague, legitimised by public opinion and is based on ministerial discretion, legislation or persuasion. Yet both high and domain politics have lived and continue to live together.
The simplest model is where there is only the Government and regulator and not much else as characterised by Cushman. The Office for Statistics Regulation demonstrates perfectly the importance of independent regulation in its purest form. UK competition policy is a case where the CMA has explicit powers over other regulators regarding certain cases and more generally takes a leading role in this area, though it is itself an independent regulator sandwiched between industrial and consumer policy. The third case study illustrates how strategic decisions are largely set by Government and other state sponsored independent organisations other than the regulator. Finally, water regulation has the most complex arrangements that has numerous Government agencies has led to the development of a hybrid organisation RAPID to encourage investment involving regulated water companies and third parties (summarised in Figure 1).
Figure 1 Four Stylised Models of institutional relationships that have an impact on independence
Case Study 1 Regulating official statistics
On the face of it, there is a difference between the oversight and regulation of official statistics on the one hand and the economic regulation of sectors, like energy and water. Official statistics are not traded in a market, with consumers paying for services from providers. They are not transmitted down expensive physical infrastructures. Official statistics may not seem as essential to contemporary life as energy, water or telecommunications.
But in fact, there are similarities. Official statistics are seen as a fundamental building block of democracy – for example by the UN Fundamental Principles of Official Statistics – and in that way as much as an essential feature of life as traditional utilities. Official statistics have public good characteristics. And while government statistical offices may not have a complete monopoly – private organisations can collect data and aggregate them into statistics – there are good reasons to regard official statistics producers as facing some of the same risks of pursuing producer interest over the consumer interest as have long concerned regulators of utilities.
In the statistics context, this producer interest can be redefined as the vested political interest of the Government of the day, and the consumer interest as the public good that is served by reliable, freely available statistics. One of the main objects of statistics legislation and regulation is to protect official statistics from political interference.
As the table shows, the independence of statistical production and regulation has evolved through a series of phases. It started in World War 2, with the creation of the Central Statistical Office, which aimed to ensure greater coherence in the information provided to the Government during the war. Since then, there have been various institutional developments that aimed to strengthen the role of professional statistical production (and a significant mis-step in the 1980s, when a review led by Derek Rayner introduced a doctrine that official statistics should meet the needs of Government, not the wider public). But the most significant development was the creation of the UK Statistics Authority in 2007 through the Statistics and Registration Services Act 2007, with the dual purposes of:
- Taking oversight of the ONS away from Ministers, so that the ONS reports directly to the UK Statistics Authority as a non-Ministerial Department, and thence to the UK’s Parliaments (the UK Parliament and the devolved Parliaments in Scotland, Wales and Northern Ireland)
- Creating a statutory regime of oversight, based on a statutory Code of Practice, that is empowered to review official statistics, and report publicly any concerns about the extent to which they “serve the public good” – the Act’s equivalent of the overarching duty on regulators to further the interests of consumers.
The Code of Practice for Statistics forms the basis of this regulatory regime. The current version of the Code highlights three key principles:
- trustworthiness – that statistics must be produced free from vested interests, and requires Government bodies to demonstrate freedom from political interference through a range of measures – for example by pre-announcing the publication of statistics, so that the statistics are not vulnerable to day-to-day Government communications management
- quality – the statistics must be of sufficient quality and not be materially misleading;
- value – the statistics should meet the needs of their users.
The Statistics Authority, through its Office for Statistics Regulation, upholds these principles through a range of tools, from formal assessments to reactive public statements about particular uses of statistics by Government Ministers and others in the public domain, for example in speeches or in Parliament.
Case Study 2 Independent Regulation in Energy
The importance of independent regulation and markets free from political intervention is a precondition for the substantial investment in the sector particularly during the acquisition of formally nationalised energy companies in the late nineteen eighties and nineties. Any threat of interference would be priced in to the cost of capital for investment. However, consumer representatives may argue for the exact opposite in the form of lower prices, higher service standards, protections for vulnerable customers and more information about the market. Hence the idea of a neutral umpire in energy.
The development of UK energy regulation broadly followed telecoms in the design of price control using the the RPI-X model and the institutional arrangements associated with independent regulator. From an independence perspective the two activities of market design and price control require must be free from intervention yet they also represent the highest risk from any potential political interference. This period is on reflection, the high water mark of independent energy regulation (1999- 2003).
It is worth noting that there is a greater degree of continuity in energy processes after liberalisation than might be expected that are coincidently suited to independent regulation. Initially, what were later to become industry codes existed as internal company policy and procedures in the nationalised Central Electricity Generating Board (CEGB) and the Regional Electricity Companies (RECS). Activities such as the calculation of energy balancing charges form the basis of the balancing and settlement code (BSC) for very specific ancillary services and other charges of balancing the system. Independent regulation is in this case is closely linked to the commercial neutrality of the regulator within the development of market rules. However, this model is under threat as there are now low participation rates of many smaller suppliers in the industry codes. The new mid-tier and challenger suppliers are not replacing the effort of the now diminished former state owned suppliers that are in drawing close to terminal decline.
Energy market regulation begins with competition enforcement and not just from the energy regulator. The Monopolies and Mergers Commission recommending breaking up vertically integrated British Gas in 1993 and Ofgas investigating monopoly transportation and tariffs opened up the market.
In contrast to the quiet world of price controls and industry codes, the retail market has been highly politicised. Its performance has always been under some form of political scrutiny. This was initially related to industry failures in the switching process and misselling scandals between 1999- 2003 when the regulator was under pressure from critics in the media and politicians. However, the major changes to the perceptions of the independence of regulation were driven by the global commodities super spike that drove energy prices upwards between 2000- 2006. Ever since there has been a constant stream of investigations with large numbers of proposed remedies by parties other than the energy regulator:
- DG Competition 2005-7: ultimately led to the third package and independent European “super” regulator ACER
- Ofgem Energy Supply Probe 2008: Simpler Tariff Choice rules (four tariff rules), Secure and promote (regulated Bid/Offer spreads in wholesale markets), clearer information on switching and bills.
- CMA 2014- 2016: market investigation remedies- price control tariffs for default tariffs, market database, gas settlement project
- Government: next day switching, energy efficiency schemes, renewable obligations
During this time the functions that really relied on an independent Regulator also came under political pressure. In particular price controls for networks in the context of rising final bills and political pressure for investment in renewables. This is combined with pressure from consumer groups who have accused the regulator of being too generous to networks. However, by promising a hard price control it does not mean to say that the decisions will not be challenged. All the gas distribution and transmission networks are currently appealing their control. It seems likely that distribution companies will follow in which case the entire sector will have appealed.
Ofgem is becoming an agent of change in order to hit the 2050 zero emissions target. It is busy creating incentive and regulatory structures top encourage this with sandbox initiatives and price control incentives. This is in effect industrial policy administered by Ofgem but using traditional regulatory levers- perhaps the natural state of affairs in the regulation of energy?
In conclusion, the regulation of energy markets has been driven to a large extent by competition institutions investigating the sector. No more is this evident than the Monopolies and Mergers Commission inquiries, the CMA 2016 investigation and the impact of DG Competition mentioned above. The CMA remedies are still in force today setting price caps, regulating markets and industry codes. We can conclude that the independence of Ofgem is less important than the regulatory environment it inhabited especially as it starts to pick up more energy policy tasks.
Case study 3 Competition policy
Whilst in energy competition agencies have made a big impact in energy in retrospect, we can now see with more clarity that competition policy in UK has always been sandwiched between consumer and industrial policy. Independent competition regulation reached its highwater mark in its neo liberal form between 1998 to 2012. Before 1998 competition policy was competing with other policies particularly industrial policy especially during the 1970’s. At that time, competition policy was combined with consumer policy in the Fair Trading Act in 1973. As the legislation was so widely drafted in 1973 using a concept of “public interest” it could accommodate such diverse politics of the Heath, Callaghan and Thatcher Governments. The Competition Act 1998 gave competition powers to the economic regulators with an explicit competition mandate through the doctrine of concurrency. After 2012 The Enterprise and Regulatory Reform Act acknowledged a number of other political priorities in competition enforcement including case selection for market investigations and consumer policy.
The institutional ecology of Competition Policy 1948- 1973
During 1948- 1972 organisational arrangements were set up with an independence in mind. The Monopolies Commission (MC) based on administrative model developed in the 19th century, that of an independent commission investigating a specific instance of market failure. Competition policy requires independence- technical legal and economic analysis for its legitimate functioning. Yet at the same time industrial policy is at its zenith between 1948-1970 starting with the nationalisation program. In 1948 the Monopolies and Mergers Commission coexisted with the Economic Planning Board and the Industrial Redevelopment corporation (IRC) 1966- 1971 that was actively restructuring British industry using mergers and the Prices and Incomes Board busily advising on wage policy. Table 1 identifies the key organisations that had potentially conflicting roles with competition policy.
TABLE 1 Cohabitation and Co-existence: Industrial policy organisations 1947-1979
Sources: Fells, A. The British Prices and Incomes Board (1972), Hague, D. Wilkinson, G. The IRC and Experiment in Industrial Corporation (1983)
By 1971, abolition of previous industrial policy organisations in table 1 was completed yet between 1971-79 “Industrial Crisis” forces Government bail outs. After 1979 large scale privatisation begins and any residual industrial policy refocused on encouraging innovation. It is at this point the space for independent competition policy emerges as the industrial policy organisations are abandoned. The development of a supportive policy momentum towards independence is helped with the “Tebbit Doctrine” that links consumer welfare to competitive markets. In the 1973 model the Tebbit doctrine could have easily been overridden by another Secretary of State. However, with the passing of the 1998 Competition Act there is a full legislative commitment to competition policy.
In 2013 the Enterprise and Regulatory Reform Act the UK’s neo liberal competition model that required an independent regulator was changed. In this reform there are more avenues exist for ministerial discretion to introduce new cases into the system but this does not challenge the supremacy of economic reasoning over political intervention but rather administrative priority. This reform contained in the reintroduction of the term “public interest” into UK legislation. For independent regulators the functioning of concurrent powers meant the potential loss of independent powers related to interpreting competition law. The CMA takes the lead on competition matters over other regulators and can take cases from other regulators and more significantly provides intellectual leadership to other regulators on these matters. A side effect of this legislation was that market investigation were initially the weapon of choice and an ambitious plan for investigations, including energy, was developed but this was ultimately overtaken by new and challenging issues associated with the digital economy.
The changing face of Intervention?
A new class of remedies became available to competition authorities as Behavioural Economics has meant different perspectives applied to retail markets forcing intervention as consumers themselves may have difficulties in making rationale choices. It has also meant a new synthesis of combining competition and consumer policies when compared with the older standard economic model that assumes rational choice.
There are specific issues that competition policy is always overridden particularly related to industrial or financial crisis and issues related to national security. This fundamentally no different to the drivers of interventions in the 1970’s. By 2016 state intervention returns for full scale aid for renewable technologies, arguments for security of supply and network resilience in energy. This is linked to a wide spread acceptance market will not deliver renewables such as flexible power or adequate energy storage. The nuclear debate and the move to the Contracts for Difference model to help finance major energy infrastructure was another sign of the move towards government support along with other major infrastructure projects such as HS2. However, these events now seem minor compared with Covid-19 induced large scale Government intervention in the economy. This is combining with the growing sense of national security and the willingness to pay for resilience. Table 2 summarises some of the interventions have actually extended in scope from table 1. For example, the Prices and Incomes Board does not have the same level of influence as the Furlough scheme, nor was the extensive IRC sponsored mergers seen trivial compared to the banking mergers.
TABLE 2 Selected Interventions and new industrial policy
There number of Governmental organisations is still large that form a similar advice role to Government identified in table 1. Current organisations include the Committee on climate Change, Regulatory Policy Committee. Others are busy encouraging investment in infrastructure such as the National Infrastructure Commission.
Paradoxically the 1948 control of monopoly and the 1973 public interest formulations do not seem as outdated as they once were in 1998.
Case study 4 Water
Regulating water industries has its own long standing regulatory tradition set in medieval law significantly predating the development of applied economic theory. This developed the definition and control of property rights for water abstraction. A second characteristic is that different forms of both public and private ownership coexisted before privatisation. There is also a long tradition of price regulation. Government has used a number tools to control prices including placing statutory limits on dividends and how much cash water companies put into their reserves5. Regulation of the water sector covers three regulated activities drinking water quality (Drinking Water Inspectorate), environmental considerations river basin management including flooding etc (Environment Agency) and modern economic regulation (Ofwat) being the latest activity.
The sectoral regulators have largely been seen by Whitehall as more a technocratic than political activity warranting intervention. Equally until recently there has been no big environmental driver for change as there has been in energy. Perhaps more significantly, a combined waste and water domestic final bill is approximately half an energy bill so never attracted the same amount of politicisation. In any event bills have historically been rising below inflation. The impact of climate change and the need for increased resilience combined with population growth has changed this.
The development of the Regulators’ Alliance for Progressing Infrastructure Development (RAPID) is perhaps the first example of how the regulatory ecosystem has both closed down regulatory independence for any one individual regulator but created a single organisation for cooperation. RAPID’s approach is to encourage companies to develop plans that would not otherwise been thought of as they required cooperation between water companies. These Strategic Resource Options (SRO) includes schemes to transfer water between rivers, build new reservoirs or desalinisation plant. In each case there will be a number of regulatory hurdles to be climbed from financing to impact on water quality. This multiagency body seeks to effectively coordinate the most strategic investment in the sector that could not be attempted by a single company. This model is unique to water but is illustrative of how the processes described above created what is a unique organisation.
Table 3 compares our case studies. The arguments for independent regulation are common across the sectors and remain strong and are still more or less well established. The water industry has had the most enduring tradition of regulatory intervention and has the most complex coordination issues between regulators. In contrast statistics is based on the original model. However, it is possible to argue that the aspiration of the UK’s independent model of regulation developed in the nineteen eighties has truly ended. Intervention in the economy after the financial crisis in 2007 and changes to competition policy in 2013 seem to be more permanent than originally thought and confirm this. From the CMA market investigation of energy, to the debate over reporting Coronavirus testing rates, there appears to be a drift towards deeper and more frequent state intervention.
On the other hand, the complete degrading of regulatory independence has not come to pass either. Regulators still conduct the price controls of natural monopolies free from direct political influence. The arguments for the desirability of independent regulation remain unchallenged and well defended. The model still remains a powerful regulatory norm that has been refined by the OECD’s work on the independence of regulators that provides further support for independent regulation.
Summary Table 3
The interaction between regulatory independence and politics relate to the stability of the conditions that justify intervention with the ultimate result of creating a more complex regulatory landscape. Recently we have seen that if it looks to politicians there is systematic market failure (whatever this may mean) or a wider political event will eventually result in intervention for example:
- Extreme economic “crisis” forcing even more extreme action across many sectors to preserve the economies functioning e.g., response to global pandemic
- Pending industrial crisis for a particular sector e.g., the car industry in the 1970’s, banking crisis 2007/8
- Externalities not being addressed by the market e.g., climate change
- Anticompetitive market structure (vertical integration/ long term contracts)
- Failure of regulated industries to perform as expected (e.g., poor customer service)
- Regulator is captured by the industry it regulates
- Regardless of the causes above, when it looks like high utility prices will not go down
- After 2013, another more “senior” regulator (CMA) thinks there is a performance issue with a regulator and conducts a market investigation or reversed a decision with a successful appeal such as a price control
- Core national interest is at stake due to a crisis terrorism, trade dispute
- Regulatory structures taken over by events (pandemic forces reassessment of drug approval regime or accident e.g. Potters Bar 2002).
The form of intervention is more or less linked to the fears associated with the impending crisis or a feeling that a policy is failing. This builds the consensus that intervention is necessary and welcome activity in these cases.
This paper has sought to argue that no regulator is an island. To best preserve its independence – which is crucial for supporting investment, and minimising the more unstable gyrations of high politics – the regulator needs to recognise and engage with its own ecosystem. For that reason, we describe the most effective interaction of regulatory and political spheres as one of ‘cohabitation’. This means that each works its own sphere. Both the regulator its regulatory peers and the Government have to respond to crisis that forces cohabitation between the desire for competition and direct intervention. They clearly have different basis of legitimacy between Government and regulation. This includes technical understanding vs popular mandate – yet both are able to control industry. They can both make changes to the industry via different means price control, licence, legislation, and persuasion that could potentially conflict. We have seen (table 3) that they share the same rationale for state intervention they can choose to emphasise or not. The utility characteristics themselves form the main rationale for control such essential service or natural monopoly. Justification for intervention for policy objectives are also present though more likely to be used by Government such as redistribution, other social policy objectives, externalities including environment and sustainability. Both Government and regulators cannot easily or quickly change this state of affairs, so in the short to medium term they are stuck with each other. Each has to accommodate each other in a number of complex decisions and in meeting their individual objectives but can also signal to each other both formally and informally
The challenge from the Penrose report is to design a competition policy that no longer has to follow the EU model or its norms. We can now find our own idiosyncratic boundaries between competition and other policies that to some extent they are competing with. An explicit understanding of this issue will go some way to defining the true boundaries between regulators and the state. Recognising the fact of multi layered cohabitational relationships between provides a focus on what to reform in the future policy debates regarding regulatory independence that may arise in the competition reform debate.
For those groups wanting to influence the reform of regulation in and general and UK competition policy, considerations of cohabitation and adaptation are critical for any meaningful change. We urge reformers to bring this observation to be brought into their thinking. The very scope for change has limits placed on it by the complex network of other Government organisations.
 The Independent regulatory Commissions Robert E Cushman Oxford University Press 1941
 Cushman Independent Regulatory Commissions Ch. IX
 Independence is also mentioned in Regulation of British Telecommunications’ Profitability HMSO Feb. 1983.
 The Politics of European Competition Regulation: A Critical Political Economy Perspective Routledge 2011
 David Kinnersley Troubled Water Shipman 1988.