Evaluation of the Patented Medicine Prices Review Board

Final Report

June 30, 2012

Submitted by:
Mira Svoboda, Managing Partner
Beechwood Consulting and Research
Ellie Beals, Principal
Beals, Lalonde & Associates

Submitted to:
Marian Eagen and Pauline Lahey
Patented Medicine Prices Review Board
333 Laurier West, Suite 1400
Ottawa, ON K1P 1C1

Table Of Contents

• List of abbrs
• Executive Summary
• 1. Introduction
• 2. This Evaluation
• 3. Relevance
• 4. Achievement of Outcomes - Regulation Program
• 5. Achievement of Outcomes - Pharmaceutical Trends Program
• 6. Efficiency and Economy 1
• 6. Efficiency and Economy 2
• 7. Summary of Conclusions
• Appendix A: Logic Models for PMPRB’s Regulatory and Reporting Functions
• Appendix B: Budget Allocation and Actual Spending by Program Activities (2008-09 to 2010-12)
• Appendix C: Evaluation Matrix
• Appendix D: Master Interview Guide
• Appendix E: Survey Questionnaire
• Appendix F : Bibliography

List of abbrs

• EBP: Employee Benefits Plan
• CADTH: Canadian Agency for Drugs and Technology in Health
• CDR: Common Drug Review
• CEO: Chief Executive Officer
• CIHI: Canadian Institute for Health Information
• CPI: Consumer Price Index
• F/P/T: Federal/Provincial/Territorial
• FTE: Full-time Equivalent
• HC: Health Canada
• HDAP: Human Drug Advisory Panel
• NGO: Non-government Organization
• NPDUIS: National Prescription Drug Utilization Information System
• OECD: Organization for Economic Cooperation and Development
• OTC: Over the counter (drug)
• PLA: Product Listing Agreement
• PMPI: Patented Medicines Price Index
• PMPRB: Patented Medicine Prices Review Board
• PMS: Performance Measurement Strategy
• R&D: Research and Development
• Rx&D: Canada's Research-Based Pharmaceutical Companies (Industry Association)
• SR&ED: Scientific Research and Experimental Development
• SPA: Special Purpose Allotment
• TBS: Treasury Board Secretariat
• VCU: Voluntary Compliance Undertaking

Executive Summary

This Evaluation

Objectives

This evaluation addresses the relevance and performance of both the Patented Medicine Prices Regulation Program and the Pharmaceutical Trends Program, during the period from 2008-09 to 2011-12. Along with the issues and questions shown below, it addresses the extent to which increased resources granted by Treasury Board Secretariat in 2008-09 allowed PMPRB to effectively deliver on its mandate.

Source: Patented Medicine Prices Review Board Performance Measurement Strategy (January 2011).

Methodology

The methodologies employed in conducting evaluation research included:

• Interviews with 46 stakeholders, including PMPRB managers and Board Members, and representatives from patentee organizations and associations, patient advocacy groups, health insurance firms, provincial and territorial governments, Health Canada, and researchers from agencies and think-tanks;
• A stakeholder survey. Invitations were extended to 700 individuals. 74 responses were received;
• A document and literature review; and
• An analysis of PMPRB performance data.

These multiple lines of inquiry were used in order to yield a high level of confidence in the findings that emerged. However, survey results have had to be treated with considerable caution because overall results were dominated by patentees. This was not problematic for the Regulation Program, for which patentees are the principal target audience. It was problematic for the Pharmaceutical Trends Program, which is designed primarily for policy-makers, whose interests often are very different and sometimes opposed to those for patentees.

Relevance

A program's relevance must be assessed relative to the environment in which that program operates. Thus, changes that have occurred in the environment since programs were initiated must be articulated and understood. There have been significant changes in the pharmaceutical environment in the past three to five years. Those changes include:

• The end of the era of “blockbuster” drugs, and the emergence of lower-cost generics introduced when patents on blockbusters expire;
• The emergence of (often very costly) drugs/treatments for niche markets;
• Burgeoning drug costs and assertive drug plan efforts to control those costs;
• Increased globalization; and
• Changes in the pharmaceutical industry, which include reduced price transparency as a result of post-purchase rebates.

For the Regulation Program, the changes in the pharmaceutical environment have had an impact. But as shown below, the most significant of those impacts tend to balance one another out so that the program remains relevant:

• The rise of generics and the aggressive provincial cost-containment efforts have reduced the relevance of the program for people covered by public plans. However, only 32% of patented medicine expenses are covered by public plans.
• The emergence of very costly drugs for the niche markets has increased the need for and relevance of the program.

The Pharmaceutical Trends Program is also still relevant but its relevance could be improved. The factors most implicated in diminishing the relevance of this program is a lack of timeliness in developing and disseminating information products, and significant patentee concerns over the appropriateness of aspects of the program which are mandated by law (i.e. reporting on patentee research and development expenditures).

Both the Regulation and the Pharmaceutical Trends Programs are appropriate for delivery by a federal agency and are well-aligned with government priorities and with PMPRB's Strategic Outcome.

Outcome Achievement: The Regulation Program

The Regulation Program has performed well. The extent to which the PMPRB has achieved each of the program's intended outcomes is detailed below:

• Building knowledge and awareness: PMPRB efforts have been successful in regard to this outcome. The PMPRB has taken numerous measures to build knowledge and awareness of the Act, Regulations, policies and Guidelines and consequences of non-compliance. A large proportion of stakeholders are satisfied with PMPRB efforts to enhance their knowledge/awareness of the legislative framework, Guidelines and policies.
• Compliance with Regulations: PMPRB has achieved a very high level of success at achieving this outcome. There has been a very high level of patentee compliance during the evaluation period.
• Clarity, Transparency and Predictability: The PMPRB has experienced a good level of success at attaining the clarity aspect of this intermediate outcome. Stakeholders found that the clarity provided by the PMPRB (timely and coherent explanations/interpretations) was good. However, the Guidelines and procedures need to be simplified to be more transparent. This lack of transparency has resulted in industry being less able to predict outcomes which affect them. Thus, the existing level of complexity has impeded the PMPRB's ability to fully achieve the outcome of providing industry with a transparent and predictable regulatory environment.
• Reasonableness of Guidelines: The PMPRB has experienced moderate success at achieving this outcome. Though stakeholders input on the revised Guidelines was good, there was still considerable stakeholder concern over the appropriateness and/or complexity of the revised Guidelines. Some of this concern is occasioned by aspects of mandate and methodology established by law. However, there does appear to be some potential for and value in the PMPRB seeking to further simplify the Guidelines to the extent possible.
• Prices Not Excessive: The PMPRB has been highly successful at achieving its ultimate outcome - that of ensuring that prices charged by patentees for patented medicines sold in Canada are not excessive, as specified in the Act.

Outcome Achievement: The Pharmaceutical Trends Program

The Pharmaceutical Trends Program has been successful in working towards some of its intended outcomes and less successful in regard to others.

• Accessibility, Comprehensiveness, Timeliness and Accuracy: Legislative constraints result in the PMPRB having little control over the accessibility, comprehensive, timeliness and accuracy of Research and Development spending information. The PMPRB has done reasonably well at providing accessible, comprehensive and accurate information products for the products it can control: NPDUIS and Price Trends Information products. However, none of these products are sufficiently timely.
• Utility of the PMPRB Research Products: None of the Pharmaceutical Trends Program's information products are well-utilized. For research and development spending information, once again legislative constraints are implicated. In the case of NPDUIS and Price Trends Information, the poor utilization appears to be at least partially predicated on the lack of timeliness, and it seems likely that these products would be better utilized for policy and decision-making if they were timelier. The fact that there has been a trend toward more rather than less utilization in recent years reinforces this likelihood.
• Trend Awareness: Though stakeholders claim and demonstrate a high awareness of trends in the pharmaceutical sector, the Pharmaceutical Trends Program offerings do not appear to have contributed to building that awareness in any significant fashion.

Efficiency and Economy

The evaluation findings support the current level of funding for salaries and operating expenses at the PMPRB. Based on the evaluation findings, the incremental funding approved by Treasury Board Secretariat in 2008 has been appropriately utilized to achieve the results for which it was authorized.

Lacking any similar programs to which the PMPRB programs could be compared, the only valid conclusions that can be drawn about the efficiency and economy of the programs being evaluated, is that none of the data occasions any concern or alarm. There are no indications that PMPRB is not operating in a cost-effective way. While no formal cost-benefit analyses were conducted as part of this evaluation, anecdotal evidence suggests a reasonable cost/benefit balance.

Although operational efficiency appears to be adequate, there is potential for improving the PMPRB's internal efficiency by considering simplifying the Guidelines and revising them more often, and decreasing patentee reporting requirements from twice to once yearly. Beyond simplifying the Guidelines, the evaluation findings also suggest that it would be appropriate to reconsider some methodological aspects of the Guidelines to ensure that they are well-matched to the current environment in which the PMPRB operates.

Overall Summary Conclusions

The overall summary conclusions presented below are derived from the topic-specific conclusions presented above. These summary conclusions address the key evaluation issues and questions identified earlier in this Executive Summary.

Relevance

Continued Relevance of the PMPRB Programs

The Regulation Program continues to be relevant. Though the pharmaceutical environment has changed considerably, the Regulation Program continues to meet the needs it was established to address.

The Pharmaceutical Trends Program is also still relevant, but less so than the Regulation Program. Some of its reduced relevance is attributable to the evolution of the pharmaceutical sector, which is no longer consistent with important program parameters like the definition of Research and Development. Another significant factor in the program's reduced relevance is a lack of timeliness in product development and dissemination.

Alignment with Government Priorities and Federal Roles and Responsibilities

Both PMPRB Programs are appropriate for delivery by a federal agency and are well-aligned with both government-wide priorities and with PMPRB's Strategic Outcome.

Performance

Achievement of Expected Outcomes for the Regulation Program

The Regulation Program has performed very successfully. It has achieved a high level of compliance and has ensured that prices are not excessive as determined by legislated criteria. In addition, the program has succeeded at enhancing stakeholder knowledge and awareness, and at creating a regulatory environment perceived by stakeholders to be increasingly clear and transparent. The program's success regarding the predictability of the environment is more guarded, primarily because the Guidelines, which have been improved considerably, require further simplification.

Achievement of Expected Outcomes for the Pharmaceutical Trends Program

The PT Program has been quite successful in some aspects of its operations. When considering how well the program has met the needs of the policy-makers for whom it was designed, it has performed well in terms of the accessibility, comprehensiveness and accuracy of the products over which it has control - NPDUIS and Price Trends outputs. However, the program has not produced these products in a manner which is sufficiently timely for policy makers who are the key audience for these products.

The program has not been very successful at producing information utilized in policy-making and decision-making. The failure regarding timeliness is implicated here, and it is likely that utilization will improve significantly if and when the timeliness issue is addressed.

Efficiency and Economy

The evaluation supports the current level of funding for salaries and operating expenses at the PMPRB. The incremental funding received in 2008-09 has been well-utilized, having achieved the expected results for which it was approved. Though formal cost-benefit analyses were not undertaken, anecdotal evidence suggests that PMPRB operations are cost-effective.

Some gains in efficiency and economy are likely to be achieved by modernizing or streamlining aspects of both the PMPRB's operational and methodological framework. The most important of these is simplifying the Guidelines.

1. Introduction

1.1 Background

1.1.1 Mandate

The PMPRB is an independent, quasi-judicial body established by Parliament in 1987 as a result of revisions to the Patent Act (the Act) which were designed to provide patentees with a greater period of market exclusivity during the term of their patents. Subsequent amendments to the Act in 1993 effectively established that a patentee had market exclusivity for the entire term of its patent, except in cases of a national emergency and increased the PMPRB's remedial powers.

The PMPRB's mandate is two-fold:

• Regulatory – to ensure that prices charged by patentees for patented medicines sold in Canada are not excessive, thereby protecting consumers and contributing to Canadian health care; and
• Reporting – to report on pharmaceutical trends of all medicines, and on research and development (R&D) spending by pharmaceutical patentees, thereby contributing to informed decisions and policy making.

1.1.2 Jurisdiction

a. The Regulatory Function

Under the Act and the Patented Medicines Regulations, patentees are required to inform the PMPRB of their intention to sell a new patented drug product, and to file relevant price and sales information. The PMPRB is responsible for ensuring that the average prices that patentees charge for prescription and non-prescription patented drugs sold in Canada to wholesalers, hospitals, pharmacies or others, for human and veterinary use, are not excessive. To make this determination, staff in the Regulatory Affairs and Outreach Branch review information submitted by patentees to ensure that prices they charge comply with the regulatory framework.

Although patentees are not required to obtain the PMPRB's approval of the price of a patented drug before it is sold, they are required to comply with the Act to ensure that prices of patented medicines sold in Canada are not excessive. If the PMPRB finds, after a public hearing, that a price is or was excessive, it may order the patentee to reduce the price and take measures to offset any excess revenues it may have generated.

b. The Reporting Function

Section 88 of the Act requires pharmaceutical patentees to report their research and development spending for the year to the PMPRB. Section 90 of the Act authorizes the Minister of Health to request that other inquiries be undertaken by the PMPRB.

The PMPRB reports annually to Parliament through the Minister of Health on pharmaceutical trends and research and development spending by pharmaceutical patentees. The reports on research and development spending are intended for government policy makers.

Through the National Prescription Drug Utilization Information System (NPDUIS) initiative, the PMPRB collaborates with federal, provincial and territorial stakeholders and the Canadian Institute for Health Information (CIHI) to produce critical analyses of price, utilization and cost trends for both patented and non-patented prescription drugs. The PMPRB also publishes specific NPDUIS reports based on the research priorities identified by the NPDUIS Steering Committee consisting of representatives of provincial/territorial public drug plans and Health Canada.

Through its studies and analysis of pharmaceutical price trends, the PMPRB aims to contribute to informed health policy making at the federal and provincial levels.

1.1.3 Organizational Structure and Programs

The Board consists of not more than five members who serve on a part-time basis, appointed by the Governor-in-Council, including a Chairperson and a Vice-Chairperson. The Chairperson is also designated under the Act as the Chief Executive Officer of the PMPRB with the authority and responsibility to supervise and direct its work.

The Executive Director is responsible for overall advice to the Board and for the leadership and management of the Staff.

The Senior Management team reports to the Executive Director. The members of the Senior Management team are: the Director of Regulatory Affairs and Outreach Branch, the Director of Policy and Economic Analysis Branch, the Director of Corporate Services Branch, the Director, Board Secretariat and Communications and the Director of Legal Services and General Counsel.

The Regulatory Affairs & Outreach Branch is responsible for the operational aspects of the Patented Medicine Prices Regulation Program (the Regulation Program). The Branch reviews the prices of patented drug products to ensure that they are not excessive; encourages patentees to comply voluntarily with the Board's Guidelines; implements related compliance policies and investigates complaints into the prices of patented medicines. This Branch also informs and educates patentees on the Board's Guidelines and filing requirements.

The Policy and Economic Analysis Branch is responsible for policy and economic analysis related to the Patented Medicine Prices Regulation Program, which includes maintaining the Guidelines that delineate the scientific review, price review, and investigative processes, conducting policy analyses, providing policy advice, and preparing economic analysis. The Branch also undertakes the majority of the activity related to the PMPRB's second program activity, the Pharmaceutical Trends Program which consists of the reporting on the analyses described above. The Legal Services Branch advises the PMPRB on legal matters and leads the prosecution team in proceedings before the Board.

The Board Secretariat and Communications Branch manage the Board's meeting and hearing process, including the official record of proceedings in support of the Hearing Panels. The Branch also develops and manages the PMPRB's communications program, media relations and public enquiries, including the production of the PMPRB's quarterly publication, Annual Report and Pharmaceutical Trends Reports.

The Corporate Services Branch provides management advice and services including strategic and financial planning and reporting, human resources, information management and technology services.

Detailed descriptions of both PMPRB programs are provided in the Patented Medicine Prices Review Board Performance Measurement Strategy. The logic models included in that document are included as Appendix A to this report.

1.1.4 Program Resources

The total Budget allocated to the PMPRB for the four fiscal years from 2008-09 to 2011-12 excluding the Special Purpose Allotment (SPA), Employee Benefits Plans (EBP) and accommodation charges is $33.4M of which:

• $15.5M was allocated to the Regulation Program;
• $5.7M was allocated to the Pharmaceutical Trends Program; and
• $12.2M was allocated to Internal Services.1

The SPA funds available range from $2.2M in 2008-09 to $3.1M in 2010-11 onwards. The total Budget allocated to the SPA for the same period is $10.9M. The ability to hold public hearings when needed is a core component of the PMPRB's mandate and authority. Due to the difficulty in forecasting the number and complexity of hearings in any given year, the amounts related to external hearing costs (legal counsel, expert witnesses, etc.) are placed in the SPA so that they are reserved strictly for that purpose. Any unspent amount lapses at year end and is returned to the Consolidated Revenue Fund.

The total budget for the PMPRB and each of its three activity areas is summarized for 2008-09 to 2011-12 and shown in Table 1.1. A more detailed comparison of the total PMPRB's budget and actual spending is provided in Appendix B.

Source: PMPRB financial data, May 2012.

1.1.5 Governance

The PMPRB reports to Parliament through the Minister of Health. The Minister of Health is responsible for the pharmaceutical provisions of the Act as set out in sections 79 to 103. The PMPRB is part of the Health Portfolio, which also includes Health Canada, the Public Health Agency of Canada, the Canadian Institutes of Health Research, the Hazardous Materials Information Review Commission, and Assisted Human Reproduction Canada. The Health Portfolio supports the Minister of Health in maintaining and improving the health of Canadians. The PMPRB contributes to the federal government's strategic outcome of “Healthy Canadians.”

Although part of the Health Portfolio, the PMPRB carries out its mandate at arm's length from the Minister of Health. It also operates independently of the other organizations which are part of the overall Health Portfolio.

The Members of the Board are collectively responsible for the implementation of the applicable provisions of the Act. Together, they establish the guidelines, rules, by-laws and other policies of the Board as provided by the Act and consult as necessary with stakeholders including provincial and territorial ministers of Health, representatives of consumer groups, the pharmaceutical industry and others.

1.1.6 Stakeholders

The PMPRB's key stakeholders include:

• Consumers;
• The pharmaceutical industry (in particular companies selling patented medicines in Canada whether they be in the brand, generic or biotech sectors);
• Provincial, territorial and federal public drug plans;
• Private health insurance firms;
• Wholesalers, pharmacies, and hospitals;
• NGOs such as (but not limited to) advocacy groups for patients (such as Patients Canada, the Canadian Health Coalition and Best Medicines Coalition, Canadian Diabetes Association), the disabled, and seniors;
• Researchers and think tanks (e.g., Fraser Institute, Canadian Centre for Policy Alternatives, Pharmaceutical Policy Research Collaboration);
• Parliamentarians, to whom the PMPRB reports through the Minister of Health; and
• Foreign government and international organizations such as the Organization for Economic Cooperation and Development (OECD); who in the past, have sometimes incorporated PMPRB Pharmaceutical Trends Program findings and analyses into their decision-making.

1 Internal Services are groups of related activities and resources that are administered to support the needs of programs and other corporate obligations of an organization. These groups are: Management and Oversight Services; Communications Services; Legal Services; Human Resources Management Services; Financial Management Services; Information Management Services; Information Technology Services; Real Property Services; Materiel Services; Acquisition Services; and Travel and Other Administrative Services. Internal Services include only those activities and resources that apply across an organization and not to those provided specifically to a program.

2. This Evaluation

2.1 Need for this Evaluation

In 2008-09, the Treasury Board Secretariat (TBS), after having funded the increased workload at PMPRB through Program Integrity Funding for two years approved additional permanent funding of approximately $6 million per year, in order to allow the Board to deliver activities deemed essential to the PMPRB mandate. TBS approval included a condition requiring that an evaluation be conducted in fiscal year 2011-12 in order to assess the extent to which the increased resources allowed PMPRB to more effectively deliver on its mandate.

2.2 Scope of the Work

This evaluation is intended to assess the relevance and performance of both the Patented Medicine Prices Regulation Program and the Pharmaceutical Trends Program, during the period from 2008-09 through 2011-12. Detailed information on the evaluation plan for this study is contained in the Patented Medicine Prices Review Board Performance Measurement Strategy (January 2011). Though no part of the additional TBS funding described above was directed toward PMPRB's NPDUIS-related efforts that work has still been addressed in this evaluation in order to yield a more comprehensive assessment of how the PMPRB has performed in recent years.

In October of 2011, the Principals of Beechwood Consulting and Beals, Lalonde & Associates were contracted by PMPRB's Corporate Services Group (the Project Authority) to conduct this evaluation. The evaluation was conducted in accordance with Treasury Board Secretariat's Policy on Evaluation and the Directive on the Evaluation Function, which was implemented April 1, 2009. Under Treasury Board requirements, federal government evaluations must address relevance and performance. The table below shows the five key issues that were addressed by this evaluation.

Source: Patented Medicine Prices Review Board Performance Measurement Strategy (January 2011)

The evaluation team worked with the Project Authority to update the evaluation matrix included in the Patented Medicine Prices Review Board Performance Measurement Strategy to yield the matrix shown in Appendix C.

The anticipated primary audience for this evaluation report includes:

• The Chairperson, Board members and staff of the PMPRB;
• Treasury Board Secretariat - which was consulted during the orientation phase of the study to ensure that its expectations were factored into the planning and conduct of the study; and
• Health Canada.

2.3 Methodology

This evaluation featured multiple lines of evidence. This approach facilitates assessing the evaluation issues from a variety of perspectives and using a range of data collection and analysis approaches, and is consistent with Treasury Board standards for evaluation. Using multiple sources and multiple methods provides a more rounded assessment with the potential for greater confidence in the results. In Chapters 3 -6 of this report, findings from all lines of inquiry are woven together to provide a balanced analysis. The combination of qualitative and quantitative evidence establishes a strong foundation to support conclusions and recommendations.

2.3.1 Interviews

Interviews with stakeholders contributed to a nuanced understanding of the perceptions and opinions of individuals who have had a significant role in or experience with the PMPRB, or who have a key stake in it. A separate interview guide was developed for each of the following categories of interviewees. In total, 46 interviews were conducted as follows:

• Board members (current and former), selected PMPRB senior managers and staff (9);
• Representatives from patentee organizations (11);
• Representatives from consumer groups/advocacy groups (5);
• Representatives from private health insurance firms (3);
• Provincial and territorial government representatives and representatives of Health Canada (14); and
• Researchers from agencies/groups anticipated to make use of PMPRB research products (4).

The interviewee categories are somewhat artificial since many individuals interviewed could fit into more than one of the above categories. In cases where interviewees self selected into more than one category they were asked with which category they identify best.

The Master Interview Guide, which shows the questions each category of interviewees were asked to address is included as Appendix D to this report.

2.3.2 Survey of Stakeholders

An Internet-based survey was developed to collect data from stakeholders. The survey questionnaire, attached as Appendix E to this report, included a combination of open and close-ended questions to address issues identified for this project and was pre-tested and refined as required prior to the time that potential survey respondents were invited to participate.

Email invitations to participate in the survey were directed to approximately 700 individuals, seventy-four (approximately 10%) of who completed the survey, broken into categories as follows:

• 39 patentees or organizations representing patentees;
• 4 consumer and/or patient advocacy organizations;
• 7 provincial, territorial or federal drug plan providers;
• 8 researchers from think tanks, universities, etc.; and
• 16 others who self-identified as consultants, lawyers, and pharmacists.

2.3.3 Document and Literature Review

The document review component of the evaluation helped the evaluators develop a thorough understanding of the PMPRB and address many of the evaluation issues. The documents reviewed were gathered from the PMPRB, interviewees as well as through a range of targeted Internet searches.

The bibliography included as Appendix F to this report lists all of the documents reviewed for this study. The evaluators developed a working/technical document in which summaries or extracts from documents that were responsive to specific evaluation questions/performance indicators were maintained.

2.3.4 Analysis of PMPRB Performance Data

The PMPRB has a Performance Measurement Strategy (PMS) which was finalized in January 2011 and which has been, or is in the process of being implemented by most branches within the PMPRB. Given how recently the PMS was finalized, it yielded only limited performance measurement data. However, the PMS has been substantially bolstered by data provided in the PMPRB's Annual Reports and/or in a range of other PMPRB documents listed in the bibliography and by data maintained and provided by the various PMPRB activity areas.

2.3.5 Methodological Constraints/Limitations

All parties to this evaluation recognized from the outset that the numbers of both interviewees and survey participants would be too small to yield real statistical validity – either for the overall group or categories within it. This does not obviate the value of the information gathered. One of the merits of multiple lines of evidence is that it allows a higher degree of confidence if/when different lines of evidence yield consistent and congruent information. For this study, it was anticipated that the combination of input from interviews and surveys combined would provide a reasonable degree of confidence in the input received.

This anticipation proved accurate for the Regulation Program. However, the survey responses were dominated by patentees, which thus skewed results for the Pharmaceutical Trends Program (PT Program) because it is targeted at government policy-makers, not patentees. Given that these two groups (government policy-makers and patentees) may have conflicting priorities, overall survey input about the PT Program must be interpreted with caution.

For categories of interviewees/survey respondents other than patentees and F/P/T representatives, the very small number of responses may indicate a lack of connection between those groups and the PMPRB, perhaps because the PMPRB Price Regulation Program does not directly impact these groups. This is explored where possible and appropriate, in the chapters that follow.

2.4 This Report

The remainder of this report is structured as described below. In each of chapters 3, 4, 5, and 6, relevant findings are presented first and then conclusions drawn from those findings end the chapter.

• Chapter 3 – Relevance: Assesses the extent to which the two PMPRB programs are still relevant and needed, and the degree of their alignment with government and agency priorities and with federal roles and responsibilities.
• Chapter 4 – Achievement of Outcomes – Regulation Program: Assesses the extent to which the Regulation Program has achieved the intended outcomes for the program.
• Chapter 5 – Achievement of Outcomes – Pharmaceutical Trends Program: Assesses the extent to which the PT Program has achieved the intended outcomes for the program.
• Chapter 6 – Efficiency and Economy: Assesses the efficiency and economy of the performance of both programs.
• Chapter 7 – Summary Conclusions: Summary responses to the key evaluation issues and questions are presented. This chapter also identifies a number of issues that may warrant consideration by the PMPRB as it responds to this evaluation report.

3. Relevance

This chapter addresses evaluation issues and questions related to the on-going need for the PMPRB and its key programs – the Patented Medicine Prices Regulation Program and the Pharmaceutical Trends Program.

3.1 Changes in the Environment

The patented drug product environment is complex, international, and has evolved rapidly in recent years. Evidence from stakeholder interviews and the literature review converged to identify a number of major changes that have occurred in recent years, including:

• End of the Blockbusters/Emergence of Generics: In recent years, patents for some of the top-selling “blockbusters”2 have started to expire, leading to increased competition from less-costly generic versions. This was cited as a major factor driving change by most of the interviewees from the PMPRB, and by several of the other interviewees. Many of these interviewees established this “patent cliff” as the precipitating factor driving the emergence of less costly generics as alternatives to the brands going off patent.
• The Emergence of Drugs for Niche Markets: The recent past has also seen an emergence of very expensive drug products and more targeted testing and treatment for less common diseases. Niche market products include (but may not be limited to) biologics/small molecule products, enzymes, injectables, and vaccines. This was noted by the majority of interviewees from the PMPRB and federal/provincial/territorial representatives and by several others.
• Burgeoning Drug Costs and Drug Plan Responses: The high-cost of drugs in niche markets exacerbates demographic impacts described in Generic Drug Pricing and Access in Canada: What are the Implications? (2010)3 “An aging population, along with an increasing number of individuals living with multiple chronic conditions, means that many more Canadians will require access to prescription medications”. This demographic change, along with utilization trends and shifts in the therapeutic mix, are probably among the key factors resulting in burgeoning drug costs, as reported in The National Pharmaceuticals Strategy Progress Report (June 2006)4 “After hospital care, Canada spends more on drugs than any other major category of the health care system. Since 2000, the total public and private expenditure on prescription drugs has grown by approximately 12 per cent annually. This rapid escalation in drug costs threatens the sustainability of public drug programs”. These research findings are also echoed by CIHI's report titled Drug Expenditure in Canada, 1985 to 2011 with drug expenditures reported to have increased from 9.5% in 1985 to 16.2% in 2010.5 This reference is consistent with input from the majority of interviewees who are patentees, who cited aggressive formulary efforts to contain costs (particularly in regard to generics) as yet another major change in the environment.
• Increased Globalization: Several patentees also identified the contraction of the manufacturing sector, as mergers have resulted in fewer and larger manufacturers/patentees – most of which are international, as a major change. Some of them linked this to the impact of increased globalization, as described in an OECD document Pharmaceutical Pricing Policies in a Global Market6 “The market for pharmaceutical products is increasingly a global one, with trade policy practices making market segmentation and corresponding price differentiation by country difficult – particularly within Europe, where multinationals have encouraged their subsidiaries to set prices within narrow price corridors”.
• Changes in the Pharmacy Industry/Reduced Price Transparency: A number of interviewees also cited an evolution in how pharmacies operate as a significant change in the environment, and one which has reduced price transparency. They were referring most often to rebate arrangements between manufacturers and pharmacies, which result in list prices being unaffected because the rebate occurs post-purchase. This was mentioned by the majority of federal/provincial/territorial interviewees and by several patentees. Interviewees also noted the increased use of product listing agreements (PLAs) that provincial formularies negotiate with manufacturers.

3.2 Relevance of the Regulation Program

The implications of the changes identified above are well-described in two cogent extracts from the literature review:

• “Canada is faced with the challenge of optimizing the benefits of prescription drugs for Canadians while managing the risks and complexities associated with this rapidly evolving sector.”7
• “The importance of containing costs over the long term should not be understated given the projected impact of demographics on drug use.”8

This suggests that the Regulation Program remains relevant. The realities of federal/provincial responsibilities for health care reinforce that relevance. Under the federal system of health care, each province has its own formulary and each provincial formulary is free to negotiate prices directly with drug manufacturers. However, the provinces' negotiating power varies by size of the province, with larger provinces having a much stronger negotiating position than smaller provinces. Based on interview findings the PMPRB serves an important regulatory function for small provinces, even in the context of formularies seeking to negotiate drug prices directly with manufacturers. In fact, some interviewees reported that the PMPRB's pricing is often used by provinces as a starting point in their product listing agreements (PLA) negotiations with manufacturers.

The majority of interviewees supported the relevance of the Regulation program, though some of that support was qualified:

• Twelve of the 44 interviewees expressed relatively unqualified support for the need for and appropriateness of continued regulation at the federal level. This support was most pronounced among interviewees from the federal/provincial/territorial and patient advocacy categories, and least evident among patentees.
• A few interviewees felt that the program is more relevant than ever because of the emergence of high-cost niche-market products like biologics.
• Another few felt that the Regulatory function is less relevant than in the past because of the success public formularies have had in cost-containment, but that it remains very relevant for those not covered by public plans. According to the Impact of Federal Regulation of Patented Drug Prices9 – this is a large group. It reports that:
  • 32% of total patented drug expenditures are public
  • 56% are covered by third party payers and employee benefit plans
  • 12% are assumed by cash-paying customers.

A few interviewees spread throughout the stakeholders categories, thought the relevance of the Regulation Program was reduced or negated by both concerns about the appropriateness of the basket of comparator countries (which they felt feature too narrow a price range), and by the impact of globalization. They felt that international manufacturers use these factors to their advantage when introducing drugs. This was noted in an OCED study10 which found that “….manufacturers have responded to the increasingly global market for their products in a strategic way. In response to external price referencing, they launch their products first in countries where they can set prices freely or can negotiate relatively high prices, delay or refrain from launching in relatively lower-price countries and maintain artificially high list prices, even when they are willing to consent to confidential rebates.”

The input from survey respondents was reasonably consistent with the views of interviewees. Overall, survey respondents feel there is a need for a program to ensure that prices charged by patentees for patented medicines are not excessive. However, there is a clear split in opinion between patentees (i.e. the regulatees) and other groups of survey respondents with other groups of survey respondents indicating a high level of need for such a program. Respondents who self-identified as belonging to the researcher category indicated that there is a “great need” for such a program, likewise F/P/T drug plan representatives noted a great need. Survey results for this question are summarized in Figure 3.1.

The literature review provided support for the on-going rational of the Regulation Program with the PMPRB's Annual Reports for 1995 to 2003 indicating that sales of patented drugs grew at an annual rate exceeding 10% while average annual rates of change for prices was less than 1%.”11

3.3 Relevance of the Pharmaceutical Trends Program

The literature review provided a few indications of the relevance of the Pharmaceutical Trends (PT) Program. The most recent example is the citing of statistics/information provided by the PMPRB in Ontario's 2012 Drummond Report. The fact that a high-profile research/investigative project like this one relied on PMPRB data provides some validation of the relevance of the program that produced it.

Perspectives gleaned from interviews from two specific categories of interviewees - patentees and F/P/T drug plan representatives were (not surprisingly) divergent.

Thirteen of the fourteen F/P/T interviewees thought the PMPRB's information products were relevant. Additional insights offered by F/P/T interviewees included the following:

• Over half of the F/P/T interviewees said the information and analysis provided by the PMPRB was well-aligned with their policy concerns.
• Some of the F/P/T interviewees were able to identify specific products they had found particularly useful. Four of them cited work done on generic prices, two cited work done on cost drivers, two noted that they found the NPDUIS data very useful for comparative analysis, and one found the PMPRB work done on wholesale markups to have been very useful.
• However, more than half of the F/P/T interviewees who thought the PMPRB's information products are relevant said that their relevance is reduced by lack of timeliness. A number of these noted that they often need to make decisions before the relevant PMPRB reports are available. As a result parallel research is often conducted by policy makers who then use PMPRB reports to support decisions already made rather than using the PMPRB research to directly inform decision-making. Concerns about timeliness (or lack thereof) of information products eroded their value and therefore reduced their relevance.

In contrast to F/P/T representatives, none of the eleven patentee interviewees expressed strong support during their interviews concerning the PMPRB's Pharmaceutical Trends Program, which as previously explained, is targeted at government policy-makers, not patentees. Reasons cited for the perceived lack of relevance included perceived inappropriateness over the reporting on patented generics and perceived inappropriateness of the legislatively mandated definition of R&D used by the PMPRB. As already noted the requirement to report on R&D expenditures and the definition of R&D are set out in the Patent Act and Patented Medicines Regulations and are not within the PMPRB's direct control.

Almost half of the interviewees in other categories indicated that the PMPRB information products are relevant. Overall, just under half of all interviewees felt the PT Program is relevant.

The picture that emerged from electronic survey results, which provide a more detailed breakdown of two categories of information products, is a little more enlightening. These results are described in the sections below.

3.3.1 Perceived Relevance of NPDUIS

Of the 74 survey respondents, 45 (or 61 %) indicated familiarity with the analysis and research produced under the National Prescription Drug Utilization Information System (NPDUIS). Survey respondents were asked to rate the extent to which they feel the information provided under the NPDUIS initiative is relevant. Overall, respondents indicated they find the information relevant; however there was significant variation in the ratings provided across categories of respondents. Respondents representing F/P/T public drug plans and researcher/industry/think tank representatives rated the relevance of the information highly while consumer/patient advocacy representatives rated the information as being of little relevance to them. These results can be attributed to the fact that the NPDUIS program was created pursuant to an agreement by federal, provincial and territorial Ministers of Health, in order to meet their need for critical analyses of price, utilization and cost trends to inform policy-making. Representatives of F/P/T public drug plans comprise the NPDUIS Committee which provides advice and guidance to the PMPRB on research priorities and topics. Although the vast majority of NPDUIS reports are publicly available, they are not produced for general consumption. This is also reflected in the awareness of NPDUIS analysis and reporting among stakeholders responding to the survey. Survey results for this question are summarized in Figure 3.2 below.

3.3.2 Perceived Relevance of R&D Spending Data

The PMPB is mandated to collect information on patentees' R&D spending in order to enable the tracking of the industry's R&D performance against the initial commitment of a 10% R&D-to-Sales ratio made with the adoption of the 1987 amendments to the Act.12

Of the 74 survey respondents, 62 (or 84%) indicated familiarity with PMPRB reporting of information on R&D spending by patentee. Patentees who indicated familiarity with R&D spending data rated the information as less than somewhat relevant while respondents representing consumer/patient advocacy groups and researcher/industry group/think tank rated the information as more than somewhat relevant. Survey results for this question are summarized in Figure 3.3 below.

Survey respondents who feel that PMPRB information on R&D spending by patentees is not relevant were asked to explain why they held this view. The most frequent explanations focused on the view that the current definition of R&D used by the PMPRB is outdated and/or inaccurate. The definition used by the PMPRB, as required by the Regulations, is the one used by Canada Revenue Agency for tax purposes in 1987. Some survey respondents and interviewees contended that the definition is no longer relevant because the pharmaceutical industry functions differently now than it did when the existing definition was set.13 For example, patentees are undertaking less direct basic research but providing research funding through other channels such as university research chairs – this is not reflected in the current definition of R&D. That said, the Government recently announced, in Budget 2012, that it intends to move to a more restrictive definition of R&D under the Canada Customs and Revenue Agency Scientific Research and Experimental Development (SR&ED) program in the near future.

3.4 Alignment with Priorities and Strategic Outcomes

3.4.1 Alignment with Federal Government Priorities

In the recent past, there have been numerous iterations of federal government priorities related to the technology and infrastructure of innovation; the provision of health care services generally and prescribed medicines in particular; and to the provision of information:

• From the National Pharmaceuticals Strategy14:
  • "First Ministers agreed that no Canadian should suffer undue financial hardship in accessing needed drug therapies and that affordable access to drugs is fundamental to equitable health outcome for all our citizens.
  • … the strategy should "enhance analysis of cost drivers and cost-effectiveness, including best prices in drug plan policies."
• From the First Minister's meeting on the future of Health Care 2004 A 10-Year Plan to Strengthen Health Care15 :
  • “Affordable access to drugs is fundamental to equitable health outcomes for all our citizens".
  • (federal priorities include) "Continued accountability and provision of information to make progress transparent to citizens"; and “…enhanced analysis of cost drivers and cost effectiveness."
• The 2007 Speech from the Throne included a focus on promoting innovation and growth in key sectors of the economy and stated that it will support, “Canadian researchers and innovators in developing new ideas and bringing them to the marketplace through Canada's Science and Technology Strategy.”
• From Mobilizing Science and Technology to Canada's Advantage16: “Canada has worked to increase the impact of federal S&T investments. In some cases, we have made new investments in regulatory activities, scientific research or infrastructure that is in the public interest".17
• From the 2010 Speech from the Throne: “It will ensure that families have the information they need to make informed choices and it will hold those who produce, import and sell goods in Canada accountable for the safety of Canadians”.
• From the 2011 Speech from the Throne:
  • “Canadians want and expect their health care system to be there when they and their families need it most. Canadians want better results from the health care system, at the same time as an aging population is putting unprecedented pressure on the system's ability to deliver.”
  • “Our Government is committed to respecting provincial jurisdiction and working with the provinces and territories to ensure that the health care system is sustainable and that there is accountability for results.”
  • “Canadians rightly expect fairness and accountability in the full range of government institutions that serve them.”

Input from key interviewees was consistent with these documented indications of the appropriate alignment of both PMPRB programs with federal government priorities. This was particularly evident among the F/P/T representatives - the interviewee category one would expect to be most knowledgeable about federal versus provincial responsibilities. Most of these interviewees expressed comfort with the fit between the PMPRB programs and the legislative obligations of the federal government.

The PMPRB was designed to act as a counterbalance to the extension of patent protection power granted to industry. The creation of the Board occurred in the context of the then five pillars of pharmaceutical patent policy – intellectual property, relationship to industrial policy, multilateral relations, consumer protection and health of Canadians.

3.4.2 Alignment with PMPRB Strategic Outcomes

The Strategic Outcome for the PMPRB is two-pronged. It is that:

• Canadians are protected from excessive prices for patented drug products sold in Canada.
• Stakeholders are informed on pharmaceutical trends.

The Regulation Program, whose objective is to review the prices of patented drug products to ensure that they are not excessive, aligns with the first component of the PMPRB's Strategic Outcome. The Pharmaceutical Trends Program, whose objective is to report on pharmaceutical trends and on R&D spending by patentees; and to provide critical analyses of price, utilization and cost trends of prescription drugs, aligns with the second component of the Strategic Outcome.

None of the research gathered, through the literature/document review, interviews or survey, countermanded this very straightforward alignment between the PMPRB's programs and its Strategic Outcome.

3.4.3 Relationship to Agencies with Related Mandates

The PMPRB is a federal agency established by legislation with legislated reporting requirements. As such, its mandate is pan-Canadian. In contrast, CADTH is a voluntary F/P/T collaboration and CIHI is an independent (non-legislated) organization funded by the federal government.

Interviewees were asked if they thought the PMPRB programs either complement or duplicate the work of other organizations/levels of government. Of those who responded to this question, a significant majority indicated that they perceived some overlap but no duplication between the research/reporting work of the PMPRB and other groups like CIHI, CADTH, and provincial/territorial/federal formularies. Of these interviewees, several mentioned the need to continue and perhaps enhance cooperation and collaboration between the PMPRB and these other groups, because of the potential, so far largely avoided, for duplication in regard to the reporting and analysis function. (All seemed to agree that the regulatory function is unique). Quotes from a number of PMPRB interviewees explain the potential for duplication or perceptions of duplication:

• (Talking about groups like CIHI, CADTH and the CDR): “We look at the same data but with a different perspective”
• “We are dependent on national agencies like CDR and CADTH, and understand the intricacies of their separate but related mandates. I suspect that for other sectors like patentees, this is probably more confusing and that answers will be different.”

While the surmise above regarding patentees' perspectives was correct, the extent of the difference was less radical than might have been anticipated. Half of the patentees agreed with the majority of interviewees who thought there was some overlap but no duplication. Another breakdown of interviewee input in this regard is quite positive. Most F/P/T interviewees thought there was some overlap but no redundancy.

When discussing either perceived duplication or the potential for duplication in the reporting function, interviewees identified CIHI and provincial formulary research groups as the organizations with whom PMPRB work was most likely to be duplicative.

Although most interviewees' suggestions/comments focused on the Regulatory Program, a few interviewees suggested that the Pharmaceutical Trends Program could be transferred to another federal department or agency. The most frequently suggested alternative was transferring the PMPRB's research activities to the Canadian Institute for Health Information (CIHI). Although this was put forward as a suggestion by some interviewees, others specifically noted that the PMPRB has a legislated mandate to collect and report on R&D spending and pharmaceutical pricing trends and is well placed to conduct additional research related to the pharmaceutical industry.

3.5 Conclusions

3.5.1 The Regulation Program

Based on findings in this evaluation, changes in the patented drug environment have had an impact on the degree to which the Regulation Program continues to be relevant with respect to preventing excessive pricing for patented drugs sold in Canada:

• The emergence of generic drugs and implementation of aggressive cost-containment approaches in public formularies appears to have reduced the relevance of the Regulation Program for those covered by those drug plans. However, only 32% of patented medicine expenses are covered by public drug plans, implying that the approach of public formularies has NOT reduced the relevance of the Regulation Program for the majority of patented medicine expenses.
• The emergence of extremely expensive products for niche markets appears to have increased the need for and relevance of the Regulation Program.

Overall, there continues to be a need for a program to prevent excessive prices for pharmaceutical drugs in Canada. In that regard, the PMPRB's Regulation Program is responsive to the needs of Canadians since there is evidence that prices could be excessive in the absence of the PMPRB.

Based on the evaluators' assessment of evaluation findings described in this chapter, the Regulation Program is appropriate for delivery by a federal agency. The PMPRB was designed to act as a counterbalance to the extension of patent protection power granted to industry. It appears that the PMPRB continues to serve this role, one that could not be served by a non-federal entity. This evaluation found no evidence of duplication of roles with respect to the prevention of excessive pricing of patented pharmaceutical drugs in Canada.

The Regulation Program is well-aligned with both government-wide priorities and with PMPRB's Strategic Outcome as evidenced by findings from a review of federal and PMPRB documentation. Numerous references to health care, affordability and the need for innovation in Speeches from the Throne since 2007 reflect the alignment of PMPRB with the federal government's current priorities. The Regulation Program aligns precisely with one of the PMPRB's Strategic Outcomes, “Canadians and their health care system are protected from excessive pricing for patented drug products sold in Canada.”

3.5.2 The Pharmaceutical Trends Program

Based on the evaluators' assessment of evaluation findings described in this chapter, the PT Program continues to be relevant. However, this is a more qualified pronouncement than the assessment of the Regulation Program. This assessment reflects the fact that the PT Program is largely intended to meet the needs of F/P/T stakeholders. The factors most implicated in diminishing the relevance of the PT information products is a lack of timeliness in development/dissemination, and with respect to patentees, concern about the appropriateness of aspects of the program which are mandated by law (i.e. reporting on R&D expenditures). The PMPRB could improve the relevance of the PT Program by improving, where possible, the timeliness of the information, specifically for NPDUIS.

Under Section 88 of the Act pharmaceutical firms must report their research and development spending to the PMPRB. In addition Section 90 of the Act authorizes the Minister of Health to request that other inquiries be undertaken by the PMPRB. Given that both these sections of the Act continue to exist, the PMPRB's PT Program appears to continue to align with federal government needs and priorities with respect to research related to pharmaceutical pricing trends. Furthermore, the NPDUIS Steering Committee includes representatives from the provincial, territorial and federal government who provide input into the selection of NPDUIS research topics. This serves as a link between NPDUIS research and federal priorities related to pharmaceutical pricing trends research.

Findings from the evaluation indicate that the PT Program is well-aligned with one of the PMPRB's Strategic Outcomes, “Stakeholders are informed on pharmaceutical trends.” While there is the potential for duplication in regard to research work done by closely related agencies like CIHI and CADTH, that potential does not appear to have been realized. To date it appears that there is some overlap but little or no duplication in the efforts of the PMPRB and other federal research organizations.

2 Blockbuster drugs are commonly defined as extremely popular drugs that generate annual sales of at least $1 billion for a company. Examples of blockbuster drugs include Vioxx, Lipitor and Zoloft. Blockbuster drugs are commonly used to treat common medical problems like high cholesterol, diabetes, high blood pressure, asthma and cancer.

3 Bell C, GrillerD, Lawson J, Lovren D. (2010) Generic Drug Pricing and Access in Canada: What are the Implications? Toronto: Health Council of Canada. www.healthcouncilcanada.ca., page 4.

4 Federal/Provincial/Territorial Ministerial Task Force on the National Pharmaceuticals Strategy. (2006). National Pharmaceuticals Strategy Progress Report Ottawa: Publications Health Canada. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/alt_formats/hpb-dgps/pdf/pubs/2006-nps-snpp/2006-nps-snpp-eng.pdf, page 8.

5 Canadian Institute for Health Information (CIHI). (2012) Drug Expenditure in Canada, 1985 to 2011. Ottawa: CIHI. Retrieved from https://secure.cihi.ca/free_products/DEIC_1985_2011_EN.pdf

6 OECD. (2008). Pharmaceutical Pricing Policies in a Global Market. E. Docteur, V. Paris, and P. Moise. Page 11.

7 Federal/Provincial/Territorial Ministerial Task Force on the National Pharmaceuticals Strategy. (2006). National Pharmaceuticals Strategy Progress Report Ottawa: Publications Health Canada. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/alt_formats/hpb-dgps/pdf/pubs/2006-nps-snpp/2006-nps-snpp-eng.pdf, page 6.

8 Bell C, Griller D, Lawson J, Lovren D. (2010) Generic Drug Pricing and Access in Canada: What are the Implications? Toronto: Health Council of Canada. www.healthcouncilcanada.ca., page 4.

9 Patented Medicine Prices Review Board. (1997). Impact of Federal Regulation of Patented Drug Prices. Patented Medicine Prices Review Board. Ottawa.

10 OECD. (2008). Pharmaceutical Pricing Policies in a Global Market. E. Docteur, V. Paris, and P. Moise. Page 11.

11 Patented Medicine Prices Review Board. (2011) Annual Report 2010. Ottawa: Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca, page 16.

12 As published in the Regulatory Impact Assessment Statement (RIAS) of the Patented Medicines Regulations, 1988, published in the Canada Gazette, Part II, Vol. 122, No. 20 – SOR/DORS/88-474

13 The PMPRB 2010 Annual Report noted that the PMPRB was participating as a member of a working group examining mechanisms for capturing and reporting on research and development investments by Rx&D and its member companies.

14 Federal/Provincial/Territorial Ministerial Task Force on the National Pharmaceuticals Strategy. (2006). National Pharmaceuticals Strategy Progress Report Ottawa: Publications Health Canada. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/alt_formats/hpb-dgps/pdf/pubs/2006-nps-snpp/2006-nps-snpp-eng.pdf.

15 Health Canada. Health Care System First Minister's Meeting on the Future of Health Care 2004 A 10-Year Plan to Strengthen Health Care. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/delivery-prestation/fptcollab/2004-fmm-rpm/index-eng.php

16 Public Works and Government Services. (June 2009). Mobilizing Science and Technology to Canada's Advantage Progress Report 2009. Ottawa: Publishing and Depository Services Public Works and Government Services http://ic.gc.ca/epublications.

17 Public Works and Government Services Mobilizing Science and Technology to Canada's Advantage Progress Report 2009. Ottawa: Publishing and Depository Services Public Works and Government Services http://ic.gc.ca/epublications.

4. Achievement of Outcomes - Regulation Program

This chapter addresses the achievement of program outcomes for the Patented Medicine Prices Regulation Program.

4.1 Knowledge and Awareness of Guidelines and Related Policies

The immediate outcome of the Regulation Program is that “Patentees have enhanced knowledge and awareness of the Act, Regulations, policies and Guidelines and consequences of non-compliance.” The degree of success the PMPRB has achieved in working toward this outcome is explored below.

The Board's Guidelines were revised in 2010. As part of the process of revising the Guidelines the PMPRB consulted extensively with stakeholders. The PMPRB has also undertaken on-going efforts at ensuring that patentees are aware of the Guidelines and related policies in order to understand their legal requirements for reporting and complying with the Guidelines and related policies.

Since 2007 PMPRB has participated in or held 13 outreach events including seminars, outreach sessions, and more recently webinars. These outreach events have been generally well attended by patentees and consultants who work with patentees, with attendance ranging from 114 patentees and consultants at outreach sessions in Toronto and Montreal in October 2009 to 37 participants in a webinar held on DIP methodology in April 2011. Table 4.1 summarizes the total number of stakeholder consultations held by the PMPRB over the past four years.

Source: PMPRB Performance Measurement Data, 2012.

As part of implementing its new performance measurement strategy developed in January 2010, the PMPRB has recently implemented feedback forms for outreach events held with patentees and consultants. Thus far feedback forms have been collected from four outreach sessions – Montreal and Toronto in March 2011 and Montreal and Toronto in February 2012. For all four of these sessions, the feedback from participants was almost entirely positive with the vast majority of participants indicating that the session answered their questions on the Guidelines and related topics.

In addition to making in-person presentations to patentees and stakeholders on various aspects of the Guidelines and related policies, the PMPRB has published 10 articles in its newsletter clarifying the Guidelines and the processes followed by Board staff when reviewing the price of a patented drug. Some examples of articles aimed at clarifying the Board's Guidelines and related policies include: Clarification of Offset of Excess Revenue (October 2010); Block 5 Foreign Price Verification (January 2011); Back Out Methodology (July 211); Backing-Out Formulas for Foreign Price Verification (January 2012).

Survey findings indicate that a large proportion (79 per cent) of patentees and organizations representing patentees are satisfied with the PMPRB's efforts to provide them with information on the legislative framework, Board Guidelines and policies. Those respondents who indicated that they felt PMPRB efforts have been insufficient were asked to explain their perspectives. Most of them focused on a need for the PMPRB to increase communication. One of these respondents explained that the Guidelines are complex and require good understanding and there are numerous grey areas that are open to interpretation that require clarification.

Likewise almost all patentees and organizations representing patentees who were interviewed for this evaluation indicated that PMPRB efforts have contributed to patentees' understanding of their obligations under the Patent Act. In fact a number of patentees noted that the relationship between patentees and the PMPRB has improved significantly in the past year or two with some describing what they see as a “regime change” at the PMPRB, which features considerably more open communication between patentees and PMPRB staff. Although most patentees interviewed are satisfied with PMPRB efforts with respect to communication with patentees, a few noted some generalized dissatisfaction, another few noted that there remain some grey areas with respect to application of the Guidelines and one interviewee commented that the website has been reorganized too frequently, making it difficult to find things on the website.

Although all categories of survey respondents find the Guidelines and other PMPRB documents intended to provide guidance on the Act, Regulations, and policies meet their needs at least to some degree, the extent to which survey respondents feel these documents meet their needs is variable. Patentees and organizations representing patentees and respondents in the “other” category tend to rate the extent to which documents provided by the PMPRB meet their needs as lower than other categories of respondents. Survey results for this question are summarized in the Figure 4.1 below.

Those respondents who indicated that they do not feel the Guidelines and other PMPRB documents meet their needs were asked to explain why. Responses provided focused on the lack of clarity of the Guidelines, and the need for interpretation of the Guidelines.

Survey results indicate that a large proportion (92%) of patentees and organizations representing patentees feel they have sufficient knowledge and awareness of the Act, the Regulations and Guidelines to enable them to comply. Patentees and organizations representing patentees who were interviewed for this evaluation were likewise satisfied that information and guidance from the PMPRB meets their needs. However, interviewees noted some gaps or shortcomings. The most frequently cited gap relates to the selection of comparator drugs, particularly for new drugs without obvious comparators. One interviewee noted a lack of clarity with respect to the Human Drug Advisory Panel (HDAP) decision-making and a desire on the part of patentees to have access to more detailed reports of summaries of deliberations/decisions (not just for their own medicines).

4.2 Compliance with the Act, Regulations and the Guidelines

One of the intermediate outcomes of the Regulation Program is that “Patentees comply with the Act, Regulations and Guidelines” The degree of success the PMPRB has achieved in working toward this outcome is explored below.

4.2.1 Trends in Compliance with Guidelines

Patentees are required by law to file information pertaining to the sale of their drug products in Canada. The Patent Act and the Patented Medicines Regulations set out the filing requirements, and Board staff reviews the pricing information on an on-going basis to ensure that prices are not excessive. Patentees are required to file information at introduction and then twice a year until the patent expires.

The number of patented drugs that fell within the Guidelines, the number not in compliance with the Guidelines and the number outside the Guidelines but which did not trigger an investigation are summarized in Table 4.2.

Source: PMPRB Performance Measurement Data, 2012.

The extent to which Form 2 and Form 3 filings are complete and accurate and submitted within the established timeframe is, in part, reflective of the extent to which patentees are aware of and understand their obligations under the Guidelines and the Act.

The submission of price and sales data (Form 2 filing) is required on a semi-annual basis (January 30th and June 30th). Where companies fail to file, a letter is sent by the PMPRB to the president of the company providing the company with an additional seven days to comply. A Board Order is issued to companies that do not subsequently file their Form 2. According to senior staff at the PMPRB, there is a relatively high rate of turnover in employees who are responsible for Form 2 filing at pharmaceutical companies and so failures to file are often the result of inexperienced staff.

The PMPRB has an internal system whereby Form 2 fillings are submitted electronically. The Form 2 filings submitted are run through an electronic verification system that checks whether the format of the components of Form 2 is as required. If the filing contains no errors the data is automatically entered into the PMPRB database. Where an error is contained in the Form 2 filing, an automated message is sent to the patentee requesting corrections.

Under subsection 89(3) of the Act, the PMPRB is required to report the identity of patentees that fail to file information (through Form 3) on their revenue from sales of drugs (including revenue from sales of non-patented drugs and from licensing agreements) and research and development expenditure in Canada related to medicines. Under the Act a patentee that does not file complete information would also be considered to be in “failure to file” status. The PMPRB has not reported an incident of failure to file since the one reported in 2008. The number of Failures to File Form 2 and Form 3 are summarized in Table 4.3 below.

Source: PMPRB Performance Measurement Data, 2012.

Overall the data strongly support the view that the level of compliance among patentees is very high.

4.2.2 Investigations and Voluntary Compliance Undertakings (VCUs)

Board staff reviews the prices of all patented drug products sold in Canada. When it finds that the price of a patented drug appears to exceed the Guidelines, and the circumstances meet the criteria for commencing an investigation, Board staff conducts an investigation to assess if the price may in fact be considered excessive. Thus, the number of investigations can be viewed not only as an indicator of compliance, but also of knowledge/awareness - the higher the number the more likely it is that there is a knowledge gap regarding the Guidelines, the lower the number the more likely it is that the Guidelines are well-understood. The number of investigations opened and on-going has been steadily decreasing while the number of investigations completed has increased since 2007. Increased funds provided to the PMPRB to reduce backlogs and provide increased outreach thus appear to have had the intended results. The number of investigations undertaken by the PMPRB since 2007 is summarized in the Table 4.4 below.

Source: PMPRB Performance Measurement Data, 2012.

A Voluntary Compliance Undertaking (VCU) is a written undertaking by a patentee to comply with the Board's Guidelines including adjusting its price to a non-excessive level and off-setting excess revenues. Patentees are given the opportunity to submit a VCU when Board staff concludes, following an investigation, that the price of a patented drug sold in Canada may have exceeded the Guidelines.

In 2010 and up to May 31, 2011, the Chairperson approved 16 VCUs and an amendment to an existing VCU. Of the 34 patentees who responded to the survey, 19 indicated that they have been involved in VCUs. Overall, patentees feel they were treated somewhat fairly in the VCU process. Patentees responding to the survey who were involved in VCUs were asked what, if any, improvements could be made to improve the VCU process. Their comments included:

• Timeframe is too short;
• Payments should be made to patient groups rather than the Federal Government;
• Price increases not taken should offset excess revenues;
• Set timeline for decision after hearing; and
• Increase flexibility.

Interviewees generally agree with most survey respondents that the VCU process is fair. Some patentees interviewed qualified their response by indicating that the VCU process is fair only if it is truly a negotiation. Others noted that although they see the process as fair, PMPRB staff generally has all the power in the development of a VCU or the VCU process negotiations.

4.2.3 Hearings

In the event that the price of a patented medicine appears to be excessive, the Board can hold a public hearing. If during the hearing, the Board finds that the price is excessive, it may issue an order to reduce the price and to offset revenues received as a result of the excessive price. Board decisions are subject to judicial review at the Federal Court of Canada.

Since its creation, the Board initiated 25 hearings, 17 of which were between 2006 and 2010. As of May 31, 2011 there were decisions pending on three matters.

Board management and members agree that the principles of fairness are applied at hearings. The process is quasi-judicial and as such, all parties are represented by legal counsel.

Of the 34 patentees who responded to the survey, six indicated that they have been involved in hearings. Of these six patentees, two felt the principles of fairness20 were not applied, three felt the principles of fairness were applied somewhat and one felt the principles of fairness were applied. Although asked to provide an explanation, only one patentee who was interviewed was able to provide one. That person felt that the Chair displayed a bias that worked against the patentee.

The six patentees who were involved in a hearing were asked how the hearing process could be improved. Suggestions include:

• The Chair of the Hearing Panel must remain objective and not offer comments and take positions that indicate a bias against manufacturers;
• More transparency;
• Extend timelines to allow more time to submit information; and
• Set timeline for decision.

4.3 Clarity, Transparency and Predictability

One of the anticipated intermediate outcomes of the Regulation Program is that “Industry has a clear, transparent, predictable patented drug pricing regulatory environment”. The degree of success the PMPRB has achieved in working toward this outcome is explored below.

Survey participants were asked to assess the clarity (understandability) and procedural transparency of the regulatory framework. Overall respondents indicated that the regulatory framework is clear. Those who indicated that they felt the regulatory framework lacked clarity were further asked to explain their opinion. The most frequent explanation provided relates to the need for pharmaceutical firms to hire external consultants and lawyers to help them interpret the Guidelines.

Though survey respondents tended to rate the transparency of the regulatory framework positively, patentees and respondents in the “other” category tended to rate the transparency as much lower than other survey respondents. Explanations as to why survey respondents felt the regulatory framework lacks transparency include the PMPRB not making all decisions public; lack of consultation with industry; and the belief that some decisions seem purposefully vague and don't provide a rationale for the decision.

Input from interviewees was somewhat consistent with survey results. “I think the Board has transparent rules, and when they make changes they are open/consultative.” A number of interviewees noted that they appreciated recent changes which reduced the complexity of the Guidelines. The nature of this input is well captured in this quote from one interviewee: “The Act and Regulations are not particularly clear – the Act is purposefully vague. But the new Guidelines …(are) carefully crafted. The work the Board continues to do with industry will improve them even further.”

Other interviewees agreed that the new Guidelines improved the clarity and transparency of the regulatory framework but noted that further simplification would be appreciated. The challenge inherent in this task was well-summarized in the following quote from an interviewee, “(The regulatory framework is)….very clear to those in the pharmaceutical environment, who are really the only ones who need to really get it.”

Survey respondents overall indicated that the regulatory framework is somewhat predictable. Researchers and F/P/T drug plan representatives find the regulatory framework to be predictable but other categories of respondents are less positive. Survey results are summarized in Figure 4.2.

Those who indicated they felt the regulatory framework lacks predictability were asked to provide an explanation. Various explanations were provided, including:

• Lack of predictability regarding comparators that will be used in the HDAP process, leading to unpredictable outcomes;
• Guidelines are applied rigidly but the interpretation of the Guidelines by PMPRB staff is unpredictable;
• Continuous changes to the Guidelines that impact the pricing of products. In the short-term these are predictable, however, long-term the impacts are seen as unpredictable and impacting the long-term strategy of the firm; and
• Mandate concerns expressed by a few are well summarized with this quote: “Recent court decisions indicate that there is uncertainty around the boundaries of PMPRB's mandate.”

Again, interviewee input was consistent with the survey results and the issues related to predictability echoed those of survey respondents. The factors cited most often by interviewees as reducing the predictability of the regulatory framework were related to methodology and mandate, both for the most part established by law.

The tone of interviewee input on the issue of predictability is noteworthy. While the criticism in some comments was unalloyed, a number of interviewees appeared to sympathize with the PMPRB in confronting the challenge of trying to create a predictable regulatory environment: “I want predictability but you have to understand that this is a sophisticated field and there will be some shades of grey, particularly when dealing with newer products. We take this view: we need to understand the rules and have technical expertise to manage through the grey in terms of how we price our products. Other patentees want only black/white.”

Source: Survey of PMPRB Stakeholders, 2012 Source: Survey of PMPRB Stakeholders, 2012

4.4 Balance between Predictability and Flexibility

One of the intermediate outcomes of the Regulatory Program is that Stakeholders view the Guidelines as properly reflecting the context of the PMPRB's legislative and regulatory framework. An important aspect of this outcome is stakeholders' perceptions of the Guidelines' ability to balance the predictability of the price review process against the flexibility required to address case-by-case situations.

Patentees who were interviewed were asked to comment on the balance between predictability and flexibility in the Board's Guidelines. As noted in the previous section, patentees interviewed generally feel the Guidelines are not very predictable or flexible nor do they balance predictability with flexibility. The key area of unpredictability noted by patentees relates to the selection of comparator drugs. This issue relates almost exclusively to new drugs with few, if any, comparators. As one patentee explained, there is uncertainty over which drugs the PMPRB will use as a comparator for setting the price of a new drug and this creates risk and uncertainty for firms. As a result of this uncertainty, firms may choose not to introduce a product in Canada first. Despite most patentees perceiving the Guidelines to be lacking in predictability, a few noted that there is some predictability and that there is an expectation that the situation will improve with the new Guidelines.

With respect to flexibility, some patentees noted that they expect the PMPRB to be more flexible in applying the new Guidelines, and in fact a few patentees noted that they have seen increased flexibility already. The lack of flexibility was described by patentees in the context of the Special Access Program (SAP) and the inability of staff to make exceptions. However other patentees did note some flexibility on the part of the PMPRB in applying the Guidelines noting that sometimes patentees are able to present their case to staff.

Overall, patentees feel that the Guidelines lack both sufficient flexibility and predictability. Key issues noted include the complexity of the Guidelines leaving them open to interpretation and resulting in a lack of predictability. There are also issues related to the selection of comparators for new drugs, and to the current approach, which is felt by some patentees to be unpredictable and inflexible. A few patentees noted that there is a need to implement a simpler, more “common sense approach” to how the Guidelines are implemented.

4.5 Alignment of Guidelines and Legislative and Regulatory Framework

As previously noted, one of the anticipated intermediate outcomes of the Regulatory Program is that Stakeholders view the Guidelines as properly reflecting the context of the PMPRB's legislative and regulatory framework. General evaluation findings related to this outcome are presented below.

The large majority of patentees interviewed do not feel that the current Guidelines are consistent with the mandate for the PMPRB established in the Patent Act. Explanations provided focus on a perceived mandate or “scope creep” with some patentees contending that the current mandate of the PMPRB has extended beyond what was envisioned in the Patent Act. The argument made by these patentees is that the PMPRB has been increasingly controlling prices rather than reviewing prices – which is perceived to have been the intended mandate of the PMPRB as per the Patent Act. As one patentee explained, “(PMPRB) are looking at it in a very granular way, but the spirit was a national average.” Patentees representing generic firms/organizations held that it was not the intent that the Act give the PMPRB jurisdiction over the prices of patented generic drugs and so the PMPRB has no mandate with respect to patented generic drugs, despite a recent Board decision to the contrary21.

However, despite their opinion that the Guidelines extend beyond the mandate of the PMPRB, a few patentees noted that the situation has somewhat improved recently with the new Guidelines and what they perceive as a “regime change” at the PMPRB with a new Chair and Executive Director. Despite the recent changes in the Guidelines and some sense among patentees that the new Guidelines represent an improvement, most do not feel the Guidelines are reasonable given the Board's mandate and enabling legislation. Reasons provided for this perspective include the lack of certainty in the application of the Guidelines, specifically around the selection of comparators; a lack of a “common sense approach” to the application of the Guidelines; and an emphasis on price control rather than price review.

All survey respondents were asked to what extent they feel the Board's Guidelines are reasonable given the PMPRB's mandate and enabling legislation. Overall, respondents from categories other than patentees tended to rate the reasonableness of the Guidelines higher than patentees. Survey respondents representing researchers tended to rate the reasonableness of the Guidelines fairly highly. The survey findings are summarized in Figure 4.3 below.

When asked to explain why they felt the Guidelines were not reasonable in the context of the PMPRB's mandate and enabling legislation, survey respondents echoed many of the same concerns as patentees who were interviewed. Specifically survey respondents noted the rigidity and complexity of the Guidelines, a perceived focus on price control rather than making sure prices were not excessive and the perception that the Board is acting as both judge and jury based on recommendations from PMPRB staff.

Survey respondents from all categories except patentees, tended to rate the extent to which the Guidelines are consistent with the PMPRB's mandate very positively. Those who indicated that they do not feel the Guidelines are consistent with ensuring that prices for patented medicines are not excessive were asked to explain their view. Responses focused on the issue of price control versus preventing excessive pricing with respondents arguing that the selection of comparators and the complexity of the Guidelines often results in forcing lower prices. The survey results for this question are summarized in Figure 4.4.

4.6 Extent to Which Prices are Not Excessive

The ultimate outcome of the Regulation Program is that “Prices charged by patentees for patented medicines sold in Canada are not excessive according to the factors of the Act.” The degree of success the PMPRB has achieved in working toward this outcome is explored below.

According to the PMPRB Annual Report for 2010, the sale of patented drug products amounted to $12.9 billion in 2010. The PMPRB uses the Patented Medicines Price Index (PMPI) to monitor trends in prices of patented drug products sold in Canada. The PMPI measures the average year-over-year change in the ex-factory price using a formula that takes a sales weighted average of price changes observed at the level of individual products. This is similar to the approach used by Statistics Canada to calculate the Consumer Price Index (CPI). Over time the trend has been for changes in the PMPI to be lower than the CPI indicating that patented drug prices are, on average, increasing at a lower rate than the CPI. This is also an indication that patentees are not always increasing prices of individual patented drugs by the CPI, as is permitted under the Guidelines. The trend in the PMPI relative to the CPI is illustrated in the Figure that follows.

Source: Patented Medicine Prices Review Board (2011). Annual Report 2010

Under the Act and the Regulations, patent holders are required to report publicly available ex-factory prices of their patented drug products for seven foreign comparator countries: France, Germany, Italy, Sweden, Switzerland, the United Kingdom and the United States. This information is used by the PMPRB to conduct its international price comparison tests. In general, average Canadian prices have been slightly lower than average prices in most other comparator countries in 2008 to 2010. Canadian prices are significantly lower (on average) than prices in the US and higher than most European comparator countries, with the exception of Germany. The average foreign to Canadian price ratios for 2008 to 2010 are illustrated in the Figure 4.5 below.

Source: PMPRB Performance Measurement Data, 2012.

As noted above in section 4.1 the number of drugs found to be excessively priced as defined in the Act and corresponding Regulations has remained very low since 2007. To determine if the price of a patented drug sold in Canada is excessive, the PMPRB applies factors set out in the Patent Act and in its price guidelines. In summary:

• Most new patented drug prices are limited so that the cost of therapy is in the range of the cost of therapy for existing drugs sold in Canada which have the same approved indication or use as the new patented drug product under review;
• Prices of moderate and substantial improvement drugs and breakthrough drugs are also restricted by a variety of tests;
• Existing patented drug prices cannot increase by more than the Consumer Price Index (CPI) as defined by the PMPRB methodology; and
• The Canadian prices of patented medicines can never be the highest of the seven comparator countries as defined by the Regulations.

4.7 Unintended Outcomes

On the regulatory side, the only unintended consequence frequently mentioned by interviewees was that manufacturers may decide not to introduce a drug into Canada because they don't want to be subjected to the PMPRB's regulatory process and results. Among the interviewees who raised this as an issue:

• There was a divergence on whether this was a positive or negative unintended outcome with some indicating that the availability of fewer drugs was a negative impact because it limited consumer choice and may result in the most effective drugs not being available to Canadians. One interviewee noted that having fewer drugs is potentially a positive impact in terms of contributing to keeping prices low as per the New Zealand approach to preventing excessive prices.
• Most presented this as apocryphal – something they'd often heard, but of which they'd seen little evidence.

There were no other unintended consequences of the Regulation Program that were frequently identified. The few others mentioned were negative and appeared to be speculative, in that no supporting evidence was cited:

• A few interviewees felt that regulation had resulted in a reduction of research and development spending in Canada. One of these, who contended that the Board involvement in patented generic products is inappropriate, was more specific, speculating that the high legal costs incurred by manufacturers in dealing with the “generics issue” might be implicated in reduced research and development spending.
• The PMPRB approach might create a disincentive to decrease prices on older products because doing so risks having subsequent products set at a lower price because existing products are used as comparators by the PMPRB in price-setting.

4.8 Conclusions

Knowledge and Awareness

Evaluation findings provided a strong indication that the PMPRB has put significant effort into enhancing stakeholder knowledge and awareness of the Act, the Regulations, policies and Guidelines, and of the consequences of non-compliance. These efforts have been focused on providing information to patentees through documents and outreach events as well as an enhanced web-site. Overall, these efforts have been successful with evaluation findings indicating that a large proportion of patentees and other stakeholders are satisfied with the PMPRB's efforts to provide them with information on the legislative framework, Board Guidelines and policies. The evaluators are comfortable concluding that based on evaluation findings, the PMPRB's recent efforts have resulted in patentees having enhanced knowledge and awareness of the Act, Regulations, policies and Guidelines.

Compliance with Regulations and Guidelines

Evidence from the evaluation indicates that patentees are highly compliant with their legal obligation to file sales information with the PMPRB. There is also evidence of a positive trend in the degree to which prices charged for patented drugs comply with Guidelines, with the number of investigations initiated yearly decreasing since 2007. Evaluation research related to hearings is considered too limited to contribute in any meaningful way because the number of hearings is relatively small, unpredictable and may vary significantly from year-to-year. Based on the evidence, it can be concluded that overall there is a high degree of patentee compliance with the Act, Regulations and Guidelines.

Clarity, Transparency and Predictability

Evaluation findings indicate that the PMPRB has done well at creating a regulatory environment viewed by industry stakeholders as clear and transparent. However, while the overall stakeholder assessment was positive, stakeholder input indicated that yet more efforts at simplification (not clarification) would be appreciated. In other words, the evidence suggests that stakeholder concerns are more focused on the inherent complexity of the Guidelines, than on the PMPRB's efforts to explain and clarify them.

There is evidence to suggest that these inherent complexities appear to impede the predictability of the regulatory process, which stakeholders did not assess as favourably as they did the clarity and transparency of the process.

Overall, the PMPRB has done a good job at creating a regulatory environment that is clear and transparent. However more efforts are needed in simplifying the Guidelines and thus making them more predictable (less subject to interpretation/misinterpretation).

Alignment of Guidelines and Legislative and Regulatory Framework

Evaluation findings indicate that for the most part, patentees do not feel that the current Guidelines are consistent with the mandate for the PMPRB established in the Patent Act, or that they provide the degree of flexibility and predictability sought by patentees. The issues cited most often as limiting the reasonableness, flexibility and predictability of the Guidelines were related to mandate and methodology, both of which are established by law. While other categories of stakeholders were less negative than patentees, overall stakeholder perception of the reasonableness of the Guidelines was only in the lower ranges of “somewhat reasonable”, supported by reasons similar to those voiced by patentees. Some patentees noted an increasing focus on price control rather than ensuring prices are not excessive (i.e. what's been referred to by some as “mandate creep”) on the part of the PMPRB. This evaluation did not find strong evidence to support or refute this contention.

Based on the evaluation findings there appears to be some potential for the PMPRB to further simplify the Guidelines, but most of the concerns expressed by stakeholders are related to issues that are embedded in the legislative mandate of the PMPRB. However, it must also be noted that it is in the interests of patentees to seek Guidelines and legislative and regulatory framework that are as unrestrictive to patentees as possible and so the PMPRB must balance the needs of patentees with those of the Canadian public.

Extent to Which Prices are Not Excessive

The framework's criteria for determining whether prices are excessive are:

• On average, patented drug prices have increased at a lower rate than has the CPI;
• Canadian prices for patented drugs have been slightly lower than average prices in most comparator countries; and
• The number of drugs for which prices were found to be excessive has remained very low during the evaluation period.

18 This measure started to be tracked as part of changes to the Guidelines in 2010. Prior to 2010 these drugs were considered to be within the Guidelines. The criteria for commencing an investigation are:

1. The National Average Transaction Price or any Market-Specific Average Transaction Price of a new drug product exceeds the Maximum Average Potential Price during the introductory period by more than 5%.
2. The National Average Transaction Price of an existing drug product exceeds the National Non-Excessive Average Price by more than 5%.
3. Excess revenues for a new or existing drug product are $50,000 or more.
4. PMPRB receives a complaint.

19 This measure may include investigations or price hearings that were not yet completed but the drug product was no longer sold or no longer patented in the year.

20 The principles of fairness in administrative law ensure that parties appearing before a tribunal receive sufficient information and notice of the proposed action, have the opportunity to present their case fully and fairly and have decisions affecting their rights and interests made using a fair, impartial and open process.

21 In the Matter of the Patent Act R.S.C. 1985, c . P-4, and the Matter of Ratiopharm Inc., Decision: PRB-08-D3-ratiopharm – Merits, dated June 30, 2011, PMPRB web site: www.pmprb-cepmb.gc.ca, Hearings and Decisions, List of Board Decisions, Ratiopharm Inc.

5. Achievement of Outcomes - Pharmaceutical Trends Program

This chapter addresses the achievement of program outcomes for the Pharmaceutical Trends (PT) Program. For the PT Program:

• The Immediate outcome is that: Stakeholders have access to comprehensive, relevant and accurate information regarding pharmaceutical trends in a timely manner.
• The Intermediate outcome is that: Stakeholders are more aware of trends in the pharmaceutical sector.
• The Ultimate outcome is: Informed decision-making by stakeholders.

The main outputs or products produced through the PT Program are:

• Analysis of price trends for pharmaceuticals in Canada;
• Reporting on research and development spending in Canada related to medicines; and
• Reporting/analyses of price, utilization and cost trends in Canada under the National Prescription Drug Utilization Information System (NPDUIS) initiative.

The nature of the evaluation evidence collected allows some, but not all of these outcomes to be addressed on a product-by-product basis. Thus, there is a mix of topical headings in this chapter - some by outcome, and others by product. Most of the more general findings are presented first, followed by the product-by-product findings.

5.1 General Findings re: Accessibility of PMPRB Research

This section deals with the accessibility element of the immediate outcome, for the whole of the PT Program. The other elements of the immediate outcome (comprehensiveness, timeliness and accuracy) are dealt with on a product-by-product basis later in this chapter.

Upon release, publications are posted on the PMPRB website, an announcement is made on the PMPRB home page under What's New @ PMPRB, and an e-bulletin is sent to electronic subscribers (currently approximately 900 registered). The PMPRB has been tracking web statistics since August 2011. The number of page views and number of unique page views for key pages on the PMPRB website are summarized in Table 5.1. All new releases are also reported in the PMPRB's quarterly NEWSletter. In addition, the PMPRB Annual Report and key research reports are mailed to subscribers (approximately 1,000).

Source: PMPRB Performance Measurement Data, 2012.

As part of its 2012-13 stakeholder engagement plan, PMPRB staff plan to initiate an enrolment campaign for the e-Bulletin directed at patient advocacy groups, health professional associations, academics and third party payers. It is the evaluators view that this is appropriate, given that a number of interviewees in the patient advocacy and third party payer categories noted that they had not been aware of the kind of information/analysis provided by the PMPRB, and that the introduction they gained in preparing for the evaluation interview had rendered them eager to be better-informed about PMPRB efforts. In addition, the PMPRB has recently launched a Twitter account aimed at ensuring that the PMPRB remains open and accessible to Canadians and maximizing stakeholder engagement. The PMPRB intends to announce all releases via Twitter. All presentations by PMPRB staff at conferences and outreach events have a pharmaceutical trends component.

Stakeholders interviewed representing F/P/Ts, research users, advocacy groups and third party payers are divided as to whether the PMPRB pharmaceutical trends information is going to the right people. Some who feel the research is finding its way to the right people qualified their response by noting that in some cases the information is outdated by the time it is released. The NPDUIS Steering Committee members noted that they do receive the reports in advance of publication. Those who do not feel the information is going to the right people represent advocacy groups and 3rd party payers, who were not parties to the original NPDUIS agreement.

5.2 General Findings re: Contribution to Trend Awareness

The intermediate outcome of the PT Program is that “Stakeholders are more aware of trends in the pharmaceutical sector”. The degree to which the PMPRB has achieved this outcome is explored below.

Interviewees were questioned about the extent to which they feel they are aware of trends in the pharmaceutical sector, and the extent to which their awareness is contingent upon information provided by the PMPRB. All of the fifteen interviewees who responded to this query felt they were highly aware of trends in the sector, but only three of them felt the Board played a role in that awareness. They all described the role played by the Board in creating awareness as minimal or marginal.

5.3 General Findings re: Utility of Research

The section that follows describes general findings related to the ultimate outcome of “informed decision-making by stakeholders”. It focuses on the perceptions of interviewees. Additional product-specific findings based on survey data are presented in the product-specific sections that follow.

Interviewees were asked about the degree to which PMPRB information facilitated their decision-making. Among the interviewees, a very small number (all of them from the F/P/T sector for whom this information is targeted) said that PMPRB information definitely did facilitate their decision-making. Within that sector, there was a fairly even split between those who said PMPRB information facilitated their decision-making at least to some extent, and those who said that it rarely or never facilitated their decision-making. In both groups, there were interviewees who noted that their response would be considerably more positive if the information was timelier. However it must be noted that the PMPRB's Annual Report cannot be released until it has been tabled in Parliament with the Minister having only 15 “sitting” days to table the report. In addition the Regulations require filing of pricing information on January 30th and R&D data on March 1st.

A number of interviewees were able to cite examples of information products which had facilitated their decision-making. Two of them cited NPDUIS reports on generic pricing, two noted that NPDUIS information had been utilized in decision-making, and one person said that PMPRB efforts related to wholesale mark-ups had facilitated decision-making.

Interviewees were queried about any changes experienced in the usage patterns of PMPRB information products in recent years. Most reported that their usage pattern has remained unchanged.

5.4 NPDUIS

This section presents findings related to both the immediate and ultimate outcome for the PT Program that are specific to the range of NPDUIS Information products.

5.4.1 Comprehensiveness, Timeliness and Accuracy

a. Interviewee's Perspectives

Stakeholders interviewed representing F/P/T drug plans, research users, advocacy groups and third party payers were asked to comment in general terms on their satisfaction with the comprehensiveness, timeliness and accuracy of the PMPRB's pharmaceutical trend information. The majority of the interviewees indicated that they find the PMPRB's information to be comprehensive and accurate. A few of those who found the research comprehensive and accurate (generally members of the F/P/T category who are members of the NPDUIS Steering Committee) noted that they appreciate the opportunity to review and provide feedback on the research products produced under the NPDUIS initiative before they are finalized. Other interviewees who expressed some dissatisfaction with the comprehensiveness and accuracy of the PMPRB's pharmaceutical trends information noted that they would like the PMPRB to consult more with potential users.

Since the 2008-09 13 studies, all of which were identified as research priorities by the NPDUIS Steering Committee, have been completed through the Pharmaceutical Trends Program. All but two of these studies were intended for publication and to date, 10 of the 11 intended for publication have been published. According to data provided by the Pharmaceutical Trends Program, projects undertaken through the NPDUIS initiative generally span over two fiscal years22 with the average time being 8 months with 70 per cent of studies completed within the established timelines.

The following studies have been completed since the 2008-09 fiscal year:

• New Drug Pipeline Monitor, Published July 2008
• New Brunswick Drug Plan Costing, Not intended for publication
• Baby-Boomer Effect on Prescription Expenditures and Claims, Published December 2010
• Use of the World Health Organization Defined Daily Dose in Canadian Drug Utilization and Cost Analyses, Published December 2010
• Generic Drugs in Canada: Market Structure — Trends and Impacts, Published December 2010
• Generic Drugs in Canada: Price Trends and International Price Comparisons, 2007, Published December 2010
• New Drug Pipeline Monitor, Published July 2011
• Public Drug Plan Dispensing Fees: A Cost-Driver Analysis, 2001/02 to 2007/08, Published September 2011
• Generic Drugs in Canada: International Price Comparisons and Potential Cost Savings, Published September 2011
• The Impact of Generic Entry on the Utilization of the Ingredient, Published September 2011 (Revised May 2012)
• Wholesale Up-charge Policies of Canada's Public Drug Plans, Published December 2011 (Revised January 2012)
• Patent Cliff or Patent Slope? Cost Savings from Past & Future Generic Entry, Not intended for publication
• Diabetes Test Strips: Actual Utilization vs. Recommendations, Publication forthcoming.

Interviewees generally do not find the PMPRB research to be timely enough. This is especially true of F/P/T representatives. Interviewees noted that the information is often outdated by the time it is released, making it less useful for policy making and planning purposes.

The majority of interviewees representing F/P/Ts, research users, advocacy groups and third party payers are satisfied with the reporting vehicles used by the PMPRB. Some of these interviewees noted that they find the briefing notes useful. However, others who are not entirely satisfied with the PMPRB's reporting vehicles noted a need for shorter, less complex, plain language documents.

Stakeholders interviewed representing F/P/Ts, research users, advocacy groups and third party payers are divided as to whether the PMPRB Pharmaceutical Trends Program outputs/reports are going to the right people. Some who feel the research is finding its way to the right people qualified their response by noting that in some cases the information is outdated by the time it is released. For example, the NPDUIS Steering Committee members noted that they do receive the reports in advance of publication. Those who do not feel the information is going to the right people represent advocacy groups and 3rd party payers, who were not parties to the original NPDUIS agreement. A PMPRB staff member reports that members of the NPDUIS Steering Committee receive PowerPoint presentations summarizing key results as soon as they are available, which is often many months prior to receiving the draft NPDUIS report.

b. Perspectives of Survey Respondents

The electronic survey provided more detail on NPDUIS research products, which is presented below. However, in considering this information, readers should recognize that the largest group of survey respondents is patentees, while these products are principally intended to address the research priorities and needs of F/P/T policy-makers, not patentees.

Survey respondents were asked whether they were familiar with the information produced under the NPDUIS initiative. Overall 61 per cent of respondents indicated familiarity with the NPDUIS initiative and its research products. Those who indicated familiarity were further asked to rate the extent to which the information produced under the NPDUIS initiative is comprehensive, accurate and timely. Overall, respondents tended to rate the NPDUIS work as somewhat comprehensive and timely and slightly more positively with respect to accuracy. Notably, the provincial/territorial respondents who are the intended audience for these products, tended to rate the accuracy of the NPDUIS information highly. Results are summarized by respondent category in Figure 5.1 that follows.

Source: Survey of PMPRB Stakeholders, 2012

Those who rated the comprehensiveness, accuracy or timeliness of information produced under the NPDUIS initiative as low were asked to provide an explanation. Those who feel the NPDUIS initiative information was lacking in comprehensiveness noted that the NPDUIS information does not always reflect changes in the pharmaceutical market and the need for more/better data from all the provinces. Survey respondents who feel the information produced under the NPDUIS initiative is not accurate noted that data are often outdated and so the reports are inaccurate. Those respondents who feel that NPDUIS reports are not timely explained that the reports are not published quickly enough and so are often outdated by the time they are published.

Survey respondents were asked to rate their overall satisfaction with the information produced under the NPDUIS initiative on a scale of 1-to-7 where 1 is “to little extent” and 7 is “to a great extent”. The Provincial/territorial representatives for whom these products are directly targeted tended to rate the quality of the information produced under the NPDUIS initiative as moderate to high. Results from other categories of respondents indicated a low to moderate level of satisfaction.

All survey respondents who indicated they were aware of the NPDUIS initiative were asked how the PMPRB could improve reporting under NPDUIS. Suggestions for improving included:

• Improve timeliness of reports;
• More collaboration and engagement with stakeholders and policy makers to identify topics;
• Include Quebec23 in the analysis; and
• More in-depth, nuanced analysis.

5.4.2 Utility of NPDUIS Products

Overall, survey respondents reported fairly low use of this type of information product. Given that this information is targeted more to policy-makers than patentees, it is not surprising that researchers, F/P/T representatives and patient advocacy groups use it more than patentees. Of those who reported using NPDUIS information/analyses, the highest percentage (35%) used it for information and analysis to inform research, and another 27% used it to inform jurisdictional policy and strategic planning.

Survey results on use of the NPDUIS information and analysis in decision-making mirror the results described above for use of the data. Overall use in decision-making is low with the F/P/T drug plan provider representatives and researcher/industry groups/think tank representatives using it the most.

Survey respondents indicating they were familiar with the information produced under the NPDUIS initiative, were asked whether their use of this PMPRB product has increased, decreased or stayed the same in the past few years. A majority (67%) of respondents indicated that their use of information provided under NPDUIS has remained the same but a significant proportion of consumer/patient advocacy group representatives, F/P/T drug plan representatives and researcher/industry groups/think tank representatives indicated that their use has increased in recent years.

5.5 Price Trends Information

The sections that follow are based on input from survey respondents. These responses were dominated by patentees, who are not the target audience for the price trend information, which is geared to the needs of policy-makers.

5.5.1 Comprehensiveness, Timeliness and Accuracy

Survey respondents were asked whether they were familiar with the PMPRB's analysis of price trends for pharmaceuticals sold in Canada. Overall 86 per cent of respondents indicated familiarity with the PMPRB analysis of price trends. Those who indicated familiarity were further asked to rate the extent to which the PMPRB trends reporting is comprehensive, accurate and timely. Overall respondents tended to rate the PMPRB analysis of price trends as somewhat comprehensive and timely and slightly more positively with respect to accuracy. Results are summarized, by respondent category in Figure 5.2 that follows.

Source: Survey of PMPRB Stakeholders, 2012 Source: Survey of PMPRB Stakeholders, 2012

Those who rated the comprehensiveness, accuracy or timeliness of the PMPRB analysis of price trends as low were asked to provide an explanation. Those who feel the trends analysis was lacking in comprehensiveness focused on the need for more detailed or granular data and the need for a broader analysis of price trends at the level of retail prices. Survey respondents who feel the trends data are not accurate commented that the data reported by the PMPRB are inaccurate because they do not reflect actual prices negotiated through agreements, only published prices are reflected. Those respondents who feel that the PMPRB trends analysis are not timely explained that the data often comes too late for policy and/or decision-making. As one survey respondent explained, “Too often the data analysis is too outdated to be useful as the regulatory environment has already changed.”

Survey respondents were asked to rate their overall satisfaction with the PMPRB's pharmaceutical trends analysis on a scale of 1 to 7, where 1 is “to little extent” and 7 is “to a great extent”. Satisfaction ratings from the categories of respondents for whom this information is targeted ranged from between 5 to 5.65 (moderate to fairly high). As anticipated, patentee satisfaction was lower (4.57), but even with this factored in, the overall average satisfaction rating for comprehensiveness, timeliness and accuracy was 5.1 out of 7.

All survey respondents who indicated they were aware of the PMPRB's pharmaceutical trends analysis were asked how the PMPRB could improve the trends analysis. Suggested improvements included:

• Provide more timely reporting;
• Better engagement with stakeholders, including patients, to identify needs;
• Include prices paid in other countries;
• Make the data downloadable;
• Provide price trends by therapeutic area;
• Include commentary and data on publicly funded drug plan pricing and reimbursement; and
• Make clearer that these are factory gate prices and not prices paid by consumers.

5.5.2 Utility of Price Trends Information

Overall survey respondents reported low use of this information included in the PMPRB Annual Report. Among the survey categories, researchers tended to indicate the highest level of use of Price Trends analysis. Of those who reported using price trend information and analyses, 30% reported using it for pricing strategy and budget forecasting, and 23% reported using it for strategic planning and comparative analyses. Other uses noted include market studies; presentation and teaching purposes; and preparing reports/policy.

Based on survey responses, the four researchers who responded were most likely to believe that PMPRB Price Trends information facilitates their decision-making. Overall, survey respondents feel that Price Trends analysis does not make a large contribution to their decision-making.

Finally survey respondents indicating familiarity with PMPRB Price Trends information, were asked whether their use of this information has increased, decreased or stayed the same in the past few years. Most of the researcher/industry groups/think tank representatives indicated that their use has increased in recent years. Overall, the majority (69%) of respondents indicated that their use of Price Trends information has remained the same, while 19% indicated that their usage has increased in recent years.

5.6 R&D Expenditure Data

The information that follows is specific to the use of R&D expenditure data. Patentees are required to submit this information via Form 3 by March 1 of every year and the information is published in the PMPRB's Annual Report, which is normally published in May or June. Because of the constraints this life-cycle imposes on timeliness, comprehensiveness and accuracy, the focus in this evaluation is solely on the utility of this data.

The 1987 amendments to the Patent Act were “an attempt to balance considerations respecting the protection of intellectual property in Canada, the encouragement of research and development in Canada, international negotiating pressure and concern for the cost of the health care system. The Pharmaceutical Manufacturers Association of Canada (now Rx&D), which includes most of the patent-holding companies, nearly all of which are not Canadian-owned, had undertaken to raise the research and development to sales ratio for patentees in Canada from less than 5% in 1984 to 8% in 1991 and 10% in 1996.”(Eastman, 1992)

“It was noted [in section 2.1] that the Board reports annually to Parliament on the research and development expenditures of patentees. The Board has no responsibility for research and development performance in Canada beyond the reporting function.” (Eastman, 1992)24

The R&D spending data are largely intended for federal government stakeholders at Health Canada and Industry Canada. Unfortunately, no one from Industry Canada was interviewed for this evaluation. Based on survey results respondents from provincial/territorial or federal drug plan providers tend to use the R&D data the least. Consumer/patient advocacy and researchers tend to be the primary users of R&D spending data. Of those who reported using R&D spending data, the highest percentage (66%) said they used it for comparative industry analysis and market studies. Overall survey respondents reported very low use of data on research and development spending.

Survey results on use of R&D spending data in decision making closely mirror the results described above for use of the data. Consumer/patient advocacy organizations and researcher/industry group/think tank representatives used the data most. Overall, usage by survey respondents was low.

Email communication from the PMPRB indicates that although not used extensively (based on survey results) users of PMPRB R&D spending data believe it is the best source of data despite lacking in detail or granularity. These users tend to be university-based researchers. As one researcher noted in an email, “In Canada the most consistent data source for estimating expenditures is maintained by the Patented Medicine Prices Review Board (PMPRB).”

Survey respondents indicating familiarity with R&D spending data published by the PMPRB, were asked whether their use of this information has increased, decreased or stayed the same in the past few years. A large majority (71%) of respondents indicated that their use of R&D spending data has remained the same. Notably, a significant proportion (33%) of researcher/industry group/think tank representatives indicated that their use has increased in recent years.

5.7 Unintended Outcomes

Interviewees were asked if they had encountered or could identify any unintended outcomes (positive or negative) as a result of either of the PMPRB's programs.

For the Pharmaceutical Trends Program, a number of interviewees who praised the comprehensiveness of the PMPRB-produced information products noted that the reports produced tended to be so complex and lengthy that some potential readers found them daunting and failed to study them. These interviewees stressed that more plain language summary products would offset the impact of this negative unintended consequence.

A positive unintended outcome cited by one interviewee was that the members of the NPDUIS Steering Committee, the organizations they represent, and the sector as a whole benefit from the opportunity the Committee offers for a group of diverse experts to come together and share their perspectives.

5.8 Conclusions

Accessibility, Comprehensiveness, Timeliness and Accuracy

Evaluation findings indicate that the PMPRB has been effective and proactive in its efforts and plans to make its research accessible to stakeholders.

Comprehensiveness, accuracy and timeliness are not relevant for research and development spending information, because timelines are established by legislation and the PMPRB's role is simply to transmit information supplied by pharmaceutical firms. Although some stakeholders indicated that they feel the research and development information could be timelier, the PMPRB can do little to make this information more quickly accessible. The definition of research is contentious with many believing that the current definition is outdated. It is beyond the power of the PMPRB to alter this definition because it is embedded in legislation.

The evaluation was able to assess the comprehensiveness and accuracy on a product-by-product basis for NPDUIS and Price Trends Information products.

• NPDUIS Products were found to be sufficiently comprehensive and accurate. This assessment is predicated on the fact that the federal/provincial/territorial stakeholders for whom this information is principally targeted, rated the comprehensiveness and accuracy NPDUIS as fairly high. On an overall basis (which includes stakeholders for whom the products are not intended) NPDUIS Information was perceived as reasonably comprehensive and accurate.
• The evaluation found that Price Trends Information products are also sufficiently comprehensive and accurate. The categories of stakeholders for whom these products are targeted rated comprehensiveness and accuracy as fairly high and even patentees assessed these products as better than “somewhat” comprehensive and accurate.

Based on evaluation findings, neither NPDUIS nor Price Trends Information products are sufficiently timely. Numerous stakeholders explained that by the time they receive this PMPRB information, it is “too late” for their purposes. Some of this may be a result of how rapidly the pharmaceutical sector is evolving and how quickly policy makers need information in order to react. It can thus be concluded that the PMPRB must assess options for improving the timeliness of these products to the extent possible.

Trend Awareness

Based on evaluation findings, though stakeholders claim and demonstrate a high awareness of trends in the pharmaceutical sector the PMPRB PT Program does not appear to have contributed to building that awareness in any significant fashion. That is, the PT Program is generally not seen by stakeholders as playing a key role in contributing to their awareness of trends in the pharmaceutical sector.

Utility of the PMPRB Research Products

The evaluation found that the PMPRB Research products are not well-utilized. This emerged on an overall basis, and on a product-by-product basis for NPDUIS and Price Trends Information products. However, stakeholder input suggests that the utility of these products would be significantly increased if timeliness improved.

The research and development spending information collected and disseminated by the PMPRB is very poorly utilized across all categories of stakeholders and this is unlikely to change so long as the existing definition of research and development stays as it is currently. As noted previously, the definition of research and development used by the PMPRB is legislated and the PMPRB alone cannot modify the definition.

On the positive side, it can be concluded that despite the overall poor showing regarding utilization, there has been a trend toward increased stakeholder utilization of all PT Program products during the evaluation period. It seems likely that this is a result of the successful efforts made in regard to accessibility.

22 The priority setting exercise for the NPDUIS initiative occurs at the annual NPDUIS Steering Committee meeting, which is usually held in October or November – the 3rd quarter of the fiscal year.

23 As set out in both A 10-Year Plan to Strengthen Health Care and Asymmetrical Federalism that Respects Quebec's Jurisdiction, Quebec is maintaining its own pharmacare program and, consequently, does not participate in nor provide data to the NDUIS initiative.

24 Eastman, Harry C., Pharmaceutical Price Review in Canada, PharmacoEconomics 5(4): 278-285, 1994

6. Efficiency and Economy 1

This chapter addresses the extent to which the PMPRB's programs are delivered with respect to efficiency and economy. Effectiveness is defined by the Treasury Board Secretariat as “the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. The level of input and output could be increases or decreases in quantity, quality, or both.” The Treasury Board Secretariat defines economy as being achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes. 25

6.1 Resources

The PMPRB has three key program activities – the Pharmaceutical Trends Program; the Patented Medicine Prices Regulation Program; and the Internal Services that support them such as human resources, financial, information technology and information management services and communications. Executive Office and Board member expenses not specifically related to hearings are also included in the Internal Services category.

Prior to 2008-09, the PMPRB had an A-base of $5.8 million, which had been supplemented for a number of years with temporary program integrity funding.

In 2008-09, through a Treasury Board (TB) submission, the PMPRB was granted permanent funding in the amount of $6.1 million, in addition to its core A-base of $5.8 million. The additional funds were intended to enable the PMPRB to meet its workload pressures and continue ongoing initiatives related to the delivery of its mandate.

The operating budget for the PMPRB remained stable over the period under review (2008-09 to 2011-12) with the exception of some targeted funds received from 2008-09 to 2010-11 for a major IT project and a retrofit of the PMPRB office space.

FTEs have also remained stable since 2009-10, at 76, although not all positions have been filled. It took about two years to staff the new indeterminate positions for which the PMPRB received funding in December 2008, resulting in lapses in the salary budget in the earliest years of this review. More recently, a few selected positions have remained intentionally vacant in order to manage within the restraints of the operating budget freeze introduced in Budget 2010 and the deficit reduction action plan.

The total budgets and expenditures for each program activity for fiscal years 2008-09 to 2011-12 are summarized in Table 6.1.

Source: PMPRB financial data, May 2012.

The PMPRB operates in an environment where workload and related outputs can vary significantly. In particular, the number of investigations, VCUs and hearings is difficult to predict. This is illustrated in Table 6.2 which summarizes the level of outputs relative to the inputs for the Patented Medicine Prices Regulation Program. Although the number of outputs relative to inputs provides one measure of the on-going efficiency of a program, the measure is crude in that it does not reflect the broad range in complexity of individual investigations or hearings, nor the time required to complete a VCU and other variations in the nature of outputs. It also does not reflect the output of the considerable resources dedicated to consultation on, and development of the new Guidelines, or the effort with respect to implementation, including outreach and training activities over the period under review.

Source: PMPRB Performance Measurement Data, 2012.

* The number of hearings ongoing in the fiscal year includes those for which a Hearing Panel decision is pending

** Following a hearing of the Board conducted in 2008-09 pursuant to the Patent Act, the Board concluded that the patentee had sold two patented medicines in Canada at excessive prices. The patentee was ordered by the Board to pay the amount of $2,512.9 thousand to the Crown. The patentee complied with the Board Order but applied for judicial review of the Order. The Federal Court quashed the Board Order and directed in its judgment that the sum of $2,512.9 thousand be returned promptly to the patentee with appropriate interest and specified costs.

Available data on outputs for the Pharmaceutical Trends Program are summarized in Table 6.3. We note that the measure of outputs relative to inputs for this program is also a relatively crude measure of the effectiveness of use of resources since the complexity of outputs (e.g. reports and/or research studies) can vary significantly.

Source: PMPRB Performance Measurement Data, 2012.

6.2 Impacts of Incremental Funding in 2008-09

6.2.1 Funding Request

The 2008 Treasury Board submission which resulted in an increase of $6.1 million to the PMPRB's reference levels identified four priority areas where additional funding was required:

• Policy and Economic Analysis, including the review, and modernization of the Guidelines - ($787 thousand annually ongoing);
• Price Review Process and Investigations, including maintaining the timeliness of price reviews and investigations and additional resources for anticipated workload increases ($1.29 million annually ongoing);
• Hearings ($3.5 million annually ongoing, including $2.8 million in the Special Purpose Allotment dedicated to hearing expenses); and
• Internal Services ($554 thousand annually ongoing).

Six specific priorities were identified:

• Review and modernize the Guidelines;
• Strengthen regulatory policy and economic analysis capacity;
• Maintain the timeliness of ongoing price reviews and investigations;
• Address the increase in workload anticipated as significantly more drug products come under PMPRB jurisdiction when patents are issued to generic companies;
• Hold hearings into excessive prices as needed; and
• Replace its mission critical database.

Additional funding received in 2008-09 and ongoing directly impacted the Patented Medicine Prices Regulation Program and Internal Services. No additional funding was provided for the Pharmaceutical Trends Program.

6.2.2 Results of Incremental Funding

The impacts of the incremental funding received by the PMPRB in 2008-09 are summarized in the sections below.

Policy and Economic Analysis, including the review, and modernization of the Guidelines

The revised Guidelines were implemented in 2010 and the impacts of changes are being closely monitored through the development and implementation of a Guidelines Monitoring and Evaluation Plan. This Plan provides the Board with guidance designed to help ensure that the Guidelines remain relevant and appropriate, thereby supporting the Board's objective of upholding the principles of fairness, transparency, openness, and predictability. Looking forward, the PMPRB has identified a number of Guidelines-related issues that will require analysis over the coming years. There are plans to undertake additional minor revisions to the Guidelines on an ongoing and as-needed basis to reflect changes in the pharmaceutical environment.

The current resourcing level is considered appropriate by PMPRB management to meet the challenges of the ongoing work described above.

The PMPRB has strengthened capacity in policy and economic analysis. A key achievement since the receipt of additional funding includes new information being made available (including but not limited to back-out formulas, and the soon-to-be released work on foreign price verifications).

According to PMPRB staff, implementation of many of the above-described achievements took longer than expected. A specific area where things took longer than anticipated was the revision of the Guidelines. The rigour and comprehensiveness of the consultation process meant that the process overall was much more iterative and collaborative than initially planned adding to the time required to develop and finalize the new Guidelines.

Price Review Process and Investigations

According to PMPRB staff, increased capacity in the Regulatory Affairs and Outreach Branch has allowed the PMPRB to eliminate backlogs and improve efficiency and timeliness. The number of investigations completed has greatly increased since 2007-08 as is evidenced in Table 6.2. Although the number of patented medicines under the PMPRB's jurisdiction has remained constant, the number of investigations has been decreasing due in part to changes in the Guidelines and the new DIP methodology. The increased workload expected from patents issued to generic companies has not yet materialized as the issue of whether patented generic drugs fall under the jurisdiction of the PMPRB is currently before the Federal Court. However, increased capacity in this area has contributed to improved efficiency and timeliness for both reviews and investigations and has allowed the PMPRB to focus on new drugs that need to be fast tracked. One resource has been reallocated to human resources to supplement staffing needs in this area.

Hearings

As a quasi-judicial tribunal, a core component of the Board's mandate is to hold public hearings, when needed, to determine whether the price of a patented drug product is or was excessive. The principles of administrative law dictate that a patentee has the right to have a fair and timely hearing. The following table provides an overview as to the number of hearings from 2005-06 to 2011-12:

Source: PMPRB Performance Measurement Data, 2012.

* Hearings (new & ongoing): hearings that were either initiated in the given fiscal year and/or were ongoing during the given fiscal year. Please note that in the event that a matter is returned to the Board for reconsideration or redetermination, or if a company has requested a supplementary Order, the case has been added in the fiscal year in which the matter or issue was ‘re-heard' by a given panel.

** Hearings before the FC, FCA & SCC: are proceedings initiated by a patentee.

As a result of the additional funding received in 2008-09 (ongoing), the PMPRB has been able to meet the needs related to hearings, both through increases to the SPA dedicated to external hearing costs, as well as through the hiring of legal, support and communications staff required.

According to PMPRB staff and performance data collected for this evaluation, the PMPRB currently has the organizational capacity to conduct hearings in a fair, transparent and timely way within current resource levels. Hearings in recent years have not been as numerous as expected at the time of development of the Treasury Board submission. However, currently there are five matters outstanding; one hearing panel decision is pending; two matters are ongoing before the Board; and three matters are before the Federal Court. The current initiative to examine alternative dispute resolution models may increase activities by Board members, the levels of support required from the Secretariat and may possibly increase the support or nature of the support needed from the Legal Branch.

Because hearing expenditures are very difficult to predict, unused funds from the SPA are returned to the Consolidated Revenue Fund at fiscal year-end. The PMPRB has recently proposed that the SPA fund be reduced as one of the organization's contributions to the deficit reduction action plan. This proposal was accepted and reflected in Budget 2012.

One of the challenges in this area is to maintain appropriate human resources capacity to hold hearings as required, while ensuring that staff is fully and appropriately utilized during times of fluctuating workload. This has been achieved by undertaking additional projects as time allows, such as the development and launch of a new website which includes an interactive complaints process in addition to information required by patentees and other stakeholders on hearings, Guidelines, Board orders, etc. Several other initiatives have resulted in the enhancement of PMPRB communications and stakeholder engagement.

Internal Services

Funding provided to replace the mission-critical database has now sunset and the project is in the final stages with the old and new databases being run concurrently for the last two semi-annual filing periods. The old database will be decommissioned in late summer/fall 2012. According to PMPRB staff, the new database now meets Government of Canada requirements related to security and bilingualism, has improved data integrity, allows for better reporting and will better position the PMPRB for increased interchange and direct electronic filing of drug pricing data in the future. PMPRB staff report that additional resources in information technology have allowed the organization to refresh its aging IT infrastructure and to support a conversion to the PeopleSoft HR system - a shared services initiative undertaken in conjunction with Health Canada, the Public Health Agency and Agriculture Canada.

PMPRB staff report that funding provided for enhanced information management has resulted in the hiring of an Information Manager, a review of IM policies, and the development of an electronic records management system that is currently being piloted in the Corporate Services Branch. Efforts are underway to become fully compliant with the new Directive on Record Keeping.

Funding provided for the retrofit of PMPRB office space was received in January 2009 which was too late for Public Works to action the project in the timeframe expected, and resulted in a lapse in that year. However, the project was completed in 2009-10 through the reallocation of some resources while the staffing of new positions was undertaken.

With respect to the appropriateness of current funding, PMPRB staff reports that the organization found that requirements in the areas of Finance and Human Resources were underestimated at the time of the development of the Treasury Board submission, in light of the need to support a larger employee body with HR services, and to meet increased requirements in accounts payable and contracting support while meeting new reporting and monitoring requirements with respect to core controls, the auditing function, investment plans, quarterly financial reports etc. The organization was required to reallocate some resources from program areas to fund two critical positions in the Corporate Services area. Even at this level, benchmarking studies performed by the Canadian International Trade Tribunal in 2008 and 2011 on the resourcing of corporate services in similar tribunals show that the PMPRB's corporate services are the leanest of the seven tribunals included in the study. That said, PMPRB management feels that in these times of fiscal restraint, services commensurate with the needs of the organization can be provided within the current level of funding.

25 Treasury Board Secretariat, Policy on Evaluation, 2009, http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?section=text&id=15024. (accessed March 26, 2012)

6. Efficiency and Economy 2

6.3 Cost versus Benefits

The cost-benefit assessment below is not the result of a formal cost/benefit analysis. It is an effort to provide a reasonable, though anecdotal assessment of the overall costs versus benefits of PMPRB operations.

The total PMPRB budget is approximately $8.8 million per year, excluding approximately $3 million set aside in the SPA for hearings. Almost all non-industry stakeholders consulted feel that this represents good value relative to the benefits associated with the PMPRB. Explanations provided include the amount of money collected by the PMPRB via VCUs and Board Orders which exceeds the cost of operating the PMPRB. Other interviewees noted that without the PMPRB patented drug prices in Canada would be similar to those in the United States, i.e. much higher, and so the money spent on the PMPRB represents a small amount relative to the potential savings. The cost of the PMPRB is seen as very small relative to the value of the pharmaceutical industry in Canada which is in the billions of dollars.

Patentees were split in their opinion on benefits of the PMPRB relative to the cost ($8.8 million per year). Industry representatives who feel the PMPRB represents good value noted that the VCUs collect substantial funds and that the cost of the PMPRB is small relative to the potential savings. One patentee went further noting that increasing the funding to the PMPRB would be a good investment if it decreased delays.

Patentees who feel the PMPRB is not a good investment argued that prices are unlikely to be excessive without the PMPRB because of negotiated contracts with provinces/formularies and other market forces. However the data from the PMPRB indicates that the Board has collected $81M in excess revenue from patentees since 2008-09.

The actual amount of money collected by the PMPRB through VCUs and Board Orders has tended to vary significantly from year-to-year since 2008-09. VCUs collected have ranged from a low of $8,394,13926 collected in 2011-12 to a high of $24,202,467 collected in 2009-10, while Board Orders have ranged between $2.5M and $12.5M over the same period. In addition to the money collected through VCUs and Board Orders there are also price reductions of the price of the drug that is the subject of the VCU or Order and possible reductions of the prices of other drugs produced by the patentee, in an effort to offset excess revenues earned as a result of excessive pricing. These factors are difficult to measure at the time of the approval of the VCU or the issuance of a Board Order.

Over the past four years the value of VCUs has totaled $62,538,74027 and Board orders $18,089,747. The total expenditures for the Regulation Program over the same period of time were $12,843,043 excluding hearing expenses and $3,286,135 in hearing expenses. Some would argue that the “value” of the PMPRB in preventing excessive prices for patented drugs in Canada is the total monies collected less the cost of the Regulatory Program or $64,449,309 over four years, and in fact, a number of interviewees made this argument. However, we note that this is a very crude measure of the benefits of the PMPRB in that it does not reflect:

• The complex environment in which the PMPRB and the pharmaceutical industry operate;
• The full benefits of the PMPRB (e.g. formularies use of PMPRB price ceilings as a starting point for negotiating with pharmaceutical firms, decreased costs for beneficiaries because of the impact of VCUs); and
• The full costs (e.g. industry compliance costs).

The PMPRB's mandate is to ensure consumers not pay excessive prices, preferably at the outset, i.e., when the drug comes under the Board's jurisdiction. When that is not possible, price reductions are sought in order to have immediate benefit to consumers. If this measure was used, the true “value” or impact (difference in the price the medicine would have sold at and the price it is now sold at due to PMPRB intervention) would have to be measured over the length of the patent. It is simply not possible to calculate this benefit in an accurate and meaningful way.

All of the evidence cited above focuses on an informal assessment of the costs versus the benefits of the Regulation Program. The absence of the sort of surrogates used thus precluded utilizing a similar approach for the value or benefits of PMPRB reports.

6.4 Modernizing and Streamlining

6.4.1 PMPRB Processes

From the perspective of patentees, proposed more efficient alternatives focus on simplifying the approach used by the PMPRB, including setting a ceiling price for a particular drug and then monitoring only to ensure that prices charged do not exceed this ceiling. According to interviewees, such an approach would be more efficient from both the perspective of patentees and the PMPRB. It would be less resource intensive from the perspective of the PMPRB because of decreased need for monitoring and on-going analysis. Patentees would not be required to report detailed pricing data twice per year. In fact a few interviewees from all categories suggested that significant efficiencies could be gained, particularly for patentees, if the Board required reporting on prices only once per year rather than every six months as is currently the case.

Another suggestion for simplifying the approach used by the PMPRB is focused on using a risk based approach to price reviews. The suggestion came from both patentees and the PMPRB staff/management/Board members. Currently the PMPRB reviews the prices of all drug products in the same way but using a risk-based approached would focus the PMPRB resources on drugs that are higher cost and/or of particular interest to provincial formularies from a cost perspective.

Currently, the PMPRB does not reassess the price of a drug when it is approved for different purposes than that for which it was initially reviewed. As one interviewee explained, in some cases manufacturers first introduce a drug for treatment of a disease/condition with a limited patient base thus warranting a higher price. However, over time the drug is approved for the treatment of other diseases/conditions that impact/affect a larger patient base which generally warrants a lower price. A reassessment of prices in such situations would, it is believed, result in lower prices for some drugs. Some interviewees representing F/P/Ts and third party payers feel that the PMPRB should be reassessing drugs in such cases.

The basket of comparator countries identified in the Act was noted by interviewees from groups other than PMPRB staff, management and Board members as something that needs to be modernized (the comparator countries were selected when the relevant legislation was enacted). Specifically some interviewees take issue with the U.S. being included as a comparator country since the list price/published price in the U.S. is seldom the price paid due to the plethora of negotiated agreements – some argue that countries that are more similar to Canada in terms of market structure should replace the U.S. as a comparator country. Patentees argue that the countries identified in the Regulations and used as comparators by the PMPRB are purposely those with the lowest prices (with the exception of the U.S.).

There was a suggestion on the part of management/staff/Board members interviewed that the PMPRB could implement a more efficient approach with respect to renewing/revising Guidelines. Currently the Board is required by legislation to undertake a consultation process with stakeholders when changes are proposed. There is a sense that given the increasingly dynamic environment of the pharmaceutical industry, more frequent changes to the Guidelines may be required. In response to this changing environment it was suggested that the Board could consider replacing the broad consultations with a Stakeholder Advisory Committee which would include patentees, F/P/T and other key stakeholders. This Stakeholder Advisory Committee would provide feedback and advice to the PMPRB on changes to the Guidelines.

Other suggestions from interviewees for modernizing and streamlining the PMPRB processes and administration included:

• Perform less intensive data collection – requiring reporting from patentees only once per year and sampling rather than evaluating all drug prices. Increasing the trigger amount for reporting would also result in less data collection since data collection could be simplified for drugs sold below a certain range.
• Review and revise the definitions of R&D – the current definitions are not seen by patentees as being reflective of the realities of the industry and lead them to believe that this results in under-reporting R&D spending by the industry.
• Engage with stakeholders beyond just the provinces/territories and patentees. Better engagement with the research community is expected to improve the quality and utility of research products. Engagement with third party providers, advocacy groups and others would facilitate a better sharing of information and more awareness of the PMPRB and its role.
• Improve outreach with patentees to educate PMPRB staff about how the industry works. There is a sense among some patentees that PMPRB staff lack an understanding of the realities of the pharmaceutical industry.
• Simplify the Guidelines – as noted throughout this report, a broad range of interviewees feel the Guidelines are too complex, despite recent changes which are seen to have decreased the complexity somewhat.
• Simplify the approach to generic drugs by moving to a complaint-based approach such as the one used for over the counter (OTC) drugs and veterinary drugs.
• Enhance coordination with CDR, CIHI and Health Canada – increased coordination across federal departments and agencies involved in research related to the use and sale of pharmaceuticals and in the regulation/review of pharmaceuticals is seen as being potentially beneficial to all agencies.

6.4.2 Perceived Alternatives

The evaluation also featured a limited focus on potential alternatives to the PMPRB programs.

From the perspective of smaller provinces, a national formulary or a national negotiating and purchasing strategy for drugs, which may include generics, was proposed as a viable alternative to the PMPRB by some F/P/T representatives, advocacy groups, and researchers who contended that:

• A national formulary or purchasing strategy would also serve to counteract the incomplete pricing information resulting from non-public agreements between pharmaceutical firms and formularies whereby prices paid are not made public; and
• A single formulary would mean that all provinces would benefit equally.

Research into alternatives to the PMPRB included delving briefly into how the Canadian regulatory framework compares to that of other nations. The countries most often suggested as providing different and perhaps better models by interviewees were the United Kingdom, New Zealand, and Australia.

The most compelling lesson learned from this research was that considerable caution must be brought to bear in attempting to assess the massive body of information available on global comparisons of regulatory approaches and the outcomes they produce. This field is tremendously complex as is the associated data which different analysts can use to support opposing points of view. This point is well-made28 by Danzon (2000) “Taking Canada as the base, we found prices in the United States to be 218 percent higher; taking the United States as the base, we found prices in Canada to be 171 percent higher. Such contradictory results are possible, especially in a small sample, because relative prices can vary widely from drug to drug…”

The Canadian context is unique, as noted in this extract from the Canadian Pharmacists Association (2007)29: “In addition, the provinces together amass more revenue than the federal government, an almost unique structure among federations in the world. This uniqueness means that there will likely always be a need to have “made in Canada” pharmaceutical policies rather than adopting those developed in the United States, the United Kingdom, France, Germany or other developed countries. ”

6.5 Conclusions

Benefits of Incremental Funding

Although additional funding pressures are being experienced by the organization due to economic measures introduced in Budget 2010, including but not limited to the freezing of operating budgets, overall, the evaluation findings support the current level of funding for salaries and operating expenses at the PMPRB. Based on evaluation findings, the incremental funding has been appropriately utilized having achieved the expected results specified when it was approved by TBS in 2008-09.

The PMPRB's submission to TBS identified six priorities for where the additional funds requested were required. Based on evaluation findings, it appears that the funds were used appropriately and the expected results related to the six priority areas identified were achieved.

PMPRB Economy and Efficiency

Lacking any similar programs that would facilitate a comparative analysis of the PMPRB resources in the program areas, the only conclusion to be drawn from the resource information assembled for this evaluation is that none of the data occasions any specific concern or alarm. There is no evidence to indicate that the PMPRB is not operating in a cost-effective manner.

The lack of comparative data or a counter factual to the PMPRB means that an assessment of the costs versus the benefits of the PMPRB cannot be estimated. Likewise assessing the cost-effectiveness of the PMPRB requires detailed data on the cost per output and outcome of the PMPRB. Although the cost per VCU, hearing, report, and so forth could technically be estimated, the cost relative to the strategic outcomes of the PMPRB (non-excessive prices for patented drugs and stakeholders being informed on pharmaceutical trends) cannot be calculated since there is no way to estimate the counter factual (e.g. what would the prices of patented drugs be in the absence of the PMPRB).

As a very rough estimate of the benefits of the PMPRB, the value of VCUs collected and Board Orders issued over 2008-09 to 2011-12 totaled $80,628,487 while the total budget for the Regulation Program over the same period of time was $12,843,043 (excluding $3,286,135 for the SPA). Some would argue that the “value” of the PMPRB in preventing excessive prices for patented drugs sold in Canada is the total value of VCUs and Board Orders less the cost of the Regulatory Program or $64.5M over four years, however, it is our view that this is a relatively unsophisticated measure of the benefits of the PMPRB vis à vis preventing excessive pricing of patented drugs sold in Canada.

Improving Operational Efficiency

Although evaluation research did not yield any evidence of inefficiency within the PMPRB, it did reveal some potential for improving the internal efficiency of the PMPRB approach, by considering:

• Simplifying the Guidelines: Throughout the evaluation, stakeholder input consistently positioned the new Guidelines as improved, but still too complex to work with efficiently.
• Revising the Guidelines more often, which would be viable if the consultation process were modified and streamlined. The suggestion to use a permanent Stakeholder Advisory Committee for such consultations is one that could be considered.
• Decreasing reporting requirements on the part of patentee - currently patentees are required to file price data twice per year. Changing this to once per year would save time and resources within both the industry (patentees) as well as the PMPRB. This evaluation found no evidence that this would be detrimental to the PMPRB's ability to appropriately monitor prices. However, it would require a change to the Regulations.

Potential Methodological Efficiencies

Beyond simplifying the Guidelines, the evaluation findings suggest a need to revisit some methodological aspects of the Guidelines to better reflect the current environment in which the PMPRB operates.

One of the issues that numerous stakeholders suggested should be reconsidered is the choice of comparator countries. However, the fact that both patentees and other groups of interviewees feel the comparator countries should be changed, seems likely to indicate that the comparators used by the PMPRB are appropriate since the current choices are felt to result in prices which are too low on the part of patentees and too high on the part of many other interviewees/stakeholders.

26 The PMPRB actually collected $10,907,018 in excess revenues through VCUs in 2011-12. However, following a hearing of the Board conducted in 2008-09 pursuant to the Patent Act, the Board concluded that the patentee had sold two patented medicines in Canada at excessive prices. The patentee was ordered by the Board to pay the amount of $2,512.9 thousand to the Crown. The patentee complied with the Board Order but applied for judicial review of the Order. The Federal Court quashed the Board Order and directed in its judgment that the sum of $2,512.9 thousand be returned promptly to the patentee with appropriate interest and specified costs.

27 A VCU can be submitted following the issuance of a Notice of Hearing, but at this point, must be approved by the Hearing Panel. Revenues collected from a VCU submitted following the Notice of Hearing are reported in the funds collected through VCUs even though the VCU is approved by the Hearing Panel.

28 Danzon, Patricia. (2000) Health: Making Sense of Drug Prices. Regulation Magazine, Volume 23, 56-62. Retrieved from http://www.cato.org/pubs/regulation/regv23n1/danzon.pdf, page 57-58.

29 Canadian Pharmacists Association. (2007). Safe and Effective: The Eight Essential Elements of an Optimal Medication-Use System. Ottawa, Ontario, page 120.

7. Summary of Conclusions

The answers to the key evaluation issues and questions (identified in Chapter 2 Table 2.1) are derived from the conclusions presented at the end of Chapters 3, 4, 5 and 6.

7.1 Relevance

7.1.1 Continued Relevance of the PMPRB Programs

The Regulation Program continues to be relevant. Though the pharmaceutical environment has changed considerably, the Regulation Program continues to meet the needs it was established to address.

The Pharmaceutical Trends Program is also still relevant, but less so than the Regulation Program. Some of its reduced relevance is attributable to the evolution of the pharmaceutical sector, which is no longer consistent with important program parameters like the definition of Research and Development. Another significant factor in the program's reduced relevance is a lack of timeliness in product development and dissemination.

7.1.2 Alignment with Government Priorities and Federal Roles and Responsibilities

Both PMPRB Programs are appropriate for delivery by a federal agency and are well-aligned with both government-wide priorities and with PMPRB's Strategic Outcome.

7.2 Performance – Achievement of Expected Outcomes

7.2.1 The Regulation Program

The Regulation Program has performed very successfully. It has achieved a high level of compliance and has ensured that prices are not excessive as determined by legislated criteria. In addition, the program has succeeded at enhancing stakeholder knowledge and awareness, and at creating a regulatory environment perceived by stakeholders to be clear and transparent. The program's success regarding the predictability of the environment is more guarded, primarily because the Guidelines, which have been improved considerably, still need to be simplified.

7.2.2 The Pharmaceutical Trends Program

The PT Program has been quite successful in some aspects of its operations. When considering how well the program has met the needs of the policy-makers for whom it was designed, it has performed well in terms of the accessibility, comprehensiveness and accuracy of the products over which it has control - NPDUIS and Price Trends outputs. However, the program has not produced these products in a manner which is sufficiently timely for policy makers who are the key audience or these products.

The program has not been very successful at producing information utilized in policy-making and decision-making. The failure regarding timeliness is implicated here, and it is likely that utilization will improve significantly if and when the timeliness issue is addressed.

7.3 Performance – Efficiency and Economy

The evaluation supports the current level of funding for salaries and operating expenses at the PMPRB. The incremental funding received in 2008-09 has been well-utilized, having achieved the expected results for which it was approved. Though formal cost-benefit analyses were not undertaken, anecdotal evidence suggests that PMPRB operations are cost-effective.

Some gains in efficiency and economy are likely to be achieved by modernizing or streamlining aspects of both the PMPRB's operational and methodological framework. The most important of these are simplifying the Guidelines, which in turn requires that the Guidelines be revised more frequently, which in turn may require developing an expedited consultation process.

7.4 For Further Consideration

In light of the conclusions above, the PMPRB may want to consider these options:

1. Expedite all PMPRB processes to facilitate a better fit between those processes and the very rapid pace of change that is now the norm for the pharmaceutical sector.
2. Further simplify the Guidelines, including methodologies used and explained in the Guidelines.
3. Expand plain language offerings throughout all PMPRB communications to enhance the value of all information products.
4. Expand the target audience for outreach efforts, with a strong focus on establishing ongoing working relationships with third party payers and patient advocacy groups.

Appendix A: Logic Models for PMPRB’s Regulatory and Reporting Functions

Logic Model – PMPRB Regulatory Function

Appendix B: Budget Allocation and Actual Spending by Program Activities (2008-09 to 2010-12)

Appendix C: Evaluation Matrix

30 throughout document * = data analysis to be performed by PMPRB

Appendix D: Master Interview Guide

31 Includes senior managers from SSHRC, HC, TBS and any other govt orgs

32 Includes organizations representing patentees

33 The principles of fairness in administrative law ensure that parties appearing before a tribunal receive sufficient information and notice of the proposed action, have the opportunity to present their case fully and fairly and have decisions affecting their rights and interests made using a fair, impartial and open process.

Appendix E: Survey Questionnaire

Patented Medicine Prices Review Board (PMPRB) Survey of Stakeholders

Welcome to the Patented Medicine Prices Review Board (PMPRB) survey of stakeholders. The Patented Medicine Prices Review Board (PMPRB) is an independent quasi-judicial body established by Parliament in 1987 under the Patent Act (Act).

The PMPRB protects the interests of Canadian consumers by ensuring that the prices of patented medicines sold in Canada are not excessive. It does this by reviewing the prices that patentees charge for each individual patented drug product in Canadian markets. If a price is found to be excessive, the Board can hold public hearings and order price reductions and/or the offset of excess revenues. The PMPRB regulates the “factory gate” prices and does not have jurisdiction over prices charged by wholesalers or pharmacies, or over pharmacists' professional fees.

The PMPRB is also responsible for reporting on trends in pharmaceutical sales and pricing for all medicines and for reporting research and development spending by patentees.

The PMPRB has commissioned Beechwood Consulting and Research and Beals, Lalonde & Associates to conduct an evaluation of the delivery of its programs. As part of this evaluation we are conducting a survey of stakeholders to gather the perceptions and opinions of individuals who have had a significant role or experience with the PMPRB. The stakeholder survey is one component of an evaluation aimed at assessing the relevance, performance and efficiency and economy of the PMPRB's programs in accordance with the Treasury Board Policy on Evaluation.

Please be assured that your responses to the questions in this survey will be kept in the strictest of confidence and that any information you provide will be administered in accordance with the Privacy Act and any other applicable privacy laws. If you have any questions or concerns regarding this survey or this evaluation, please contact Ms. Mira Svoboda, Managing Partner at Beechwood Consulting and Research at 613-424-5476, or at MiraSvoboda@rogers.com.

1. First please indicate your primary role with respect to the PMPRB. (Please select only one. This survey does not accept multiple or nil responses.)
   1. Patentee
   2. Industry Organization representing patentees
   3. Consumer/patient advocacy organization
   4. Provincial, territorial or federal drug plan provider
   5. Third party insurance providers
   6. Researcher, industry group, think tank
   7. Foreign government, international organization
   8. Federal/provincial/territorial researcher or analyst
   9. Other ________________________________

Patented Medicine Prices Regulation Program

1. (ASK ALL) Are you aware of the PMPRB Patented Medicines Regulation Program (the program intended to ensure that prices of patented medicines sold in Canada are not excessive)? (yes, no, don't know/no answer) – If no or don't know/no answer, then skip to next section.
2. (ASK ALL) To what extent do you believe there is a continued need for a program to ensure that prices charged by patentees for prescription and non-prescription patented medicines sold in Canada are not excessive? (7 point rating scale – little, somewhat, great) (Indicator 1.1.1)
3. (ASK A, B) As a patentee or organization representing patentees, to what extent do you feel you have sufficient knowledge and awareness of the Patent Act, Patented Medicines Regulations, and Board Guidelines to enable you to comply? (7 point rating scale – little, somewhat, great) (Indicator 2.1.2)
4. (ASK A, B) As a patentee or organization representing patentees, to what extent do you feel the PMPRB has made sufficient efforts to provide you with information on the legislative framework, Board Guidelines and policies with respect to patented medicines, and the measures required to comply with them? (7 point rating scale – little, somewhat, great) (Indicator 2.1.2)
   1. If effort insufficient (1, 2, 3– please explain. (open ended) (Indicator 2.1.2)
5. (ASK A, B) To what extent do you find the Board's Guidelines reasonable given the PMPRB's mandate and enabling legislation? (7 point rating scale – little, somewhat, great) (Indicator 2.4.1)
   1. If response is 1, 2, 3 – please explain. (Indicator 2.4.1)
6. (ASK A, B) To what extent do you feel the Guidelines are consistent with the PMPRB's mandate to ensure that the prices charged for patented medicines sold in Canada are not excessive? (7 point rating scale – little, somewhat, great) (Indicator 2.4.2)
   1. If response is 1, 2, 3 – please explain. (Indicator 2.4.2)
7. (ASK A, B) To what extent do the Guidelines and other PMPRB documents that provide guidance on the Act, Regulations, and policies meet your needs? (7 point rating scale – little, somewhat, great) (Indicator 2.1.8)
   1. If the documents referred to above don't meet need (1, 2, 3) – please explain. (open ended) (Indicator 2.1.8)
8. (ASK A, B) To what extent do you feel the regulatory framework governing the pricing of patented medicines is clear (i.e. understandable to patentees)? (7 point rating scale – little, somewhat, great) (Indicator 2.3.1)
   1. If response 1, 2, 3 – please explain why you feel the regulatory framework governing the pricing of patented medicines is not clear. (open ended) (Indicator 2.3.1)
9. (ASK A, B) To what extent do you feel the processes and decisions associated with the regulatory framework governing the pricing of patented medicines are transparent? (7 point rating scale – little, somewhat, great) (Indicator 2.3.1)
   1. If response 1, 2, 3 – please explain why you feel the regulatory framework governing the pricing of patented medicines is not transparent. (open ended) (Indicator 2.3.1)
10. (ASK A, B) To what extent do you feel the regulatory framework governing the pricing of patented medicines is predictable? (7 point rating scale – little, somewhat, great) (Indicator 2.3.1)
    1. If response 1, 2, 3 – please explain why you feel the regulatory framework governing the pricing of patented medicines is not predictable. (open ended) (Indicator 2.3.1)
11. (ASK A) Have you been involved in the PMPRB's VCU process? (yes, no, don't know/no answer) ) (Indicator 2.5.4)
    1. If yes – To what extent do you feel your firm received fair treatment from the PMPRB during this process? (7 point rating scale – little, somewhat, great) (Indicator 2.5.4)
    2. What improvements, if any, would you propose to the VCU process? (Indicator 2.5.4)
12. (ASK A) Have you been involved in a hearing of the PMPRB? (yes, no, don't know/no answer) ) (Indicator 2.5.5)
    1. If yes – To what extent do you feel the principles of fairness were applied during the hearing? The principles of fairness in administrative law ensure that parties appearing before a tribunal receive sufficient information and notice of the proposed action and have the opportunity to present their case fully and fairly and have decisions affecting their rights and interests made using a fair, impartial and open process.(7 point rating scale – little, somewhat, great) (Indicator 2.5.5)
    2. (ASK ONLY If response is 5, 6 or 7 in 12a) Which elements of the principles of fairness do you feel were not applied during the hearing? (Indicator 2.5.5)
    3. What improvements, if any, would you propose to the PMPRB hearing process? (Indicator 2.5.5)

PMPRB Reporting & Analysis – Pharmaceutical Price Trends

1. Are you aware of the analysis of price trends for pharmaceuticals in Canada performed by the PMPRB? (yes, no, no response/don't know) If response is no, don't know/no answer, skip to next section
2. To what extent do you use information and/or analysis of price trends for pharmaceuticals disseminated by the PMPRB in your work? (7 point rating scale – little, somewhat, great) (Indicator 2.6.3/1.2.4/1.2.5)
   1. If response is 3, 4, 5, 6, 7 - Please describe how you use this information.
3. To what extent do you feel the information and/or analysis of pharmaceutical price trends provided by the PMPRB are relevant to your policy development and/or research needs? (7 point rating scale – little, somewhat, great) (Indicator 1.2.1)
   1. If response 1, 2, 3– Please explain why you feel this information is not relevant. (open ended) (Indicator 1.2.1)
4. To what extent do you believe that PMPRB information and/or analysis of price trends for pharmaceuticals facilitates your decision-making? (7 point rating scale – little, somewhat, great) (Indicator 2.8.1)
5. In the past three years, has your use of information and/or analysis of price trends for pharmaceuticals from the PMPRB (increased, decreased, stayed the same, don't know/no answer)? (Indicator 2.8.3)
6. To what extent do you feel the information and/or analysis of pharmaceutical price trends provided by the PMPRB are comprehensive? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – Please explain why you feel this information is not comprehensive. (open ended) (Indicator 2.2.9)
7. To what extent do you believe the information and/or analysis of pharmaceutical price trends provided by the PMPRB is accurate? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – please explain why you feel the information provided by the PMPRB is not accurate. (open ended) (Indicator 2.2.9)
8. To what extent do you feel the information and/or analysis of pharmaceutical price trends provided by the PMPRB are timely? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – please explain why you feel the information provided by the PMPRB is not timely. (open ended) (Indicator 2.2.9)
9. To what extent are you satisfied with the overall quality of the information and/or analysis of pharmaceutical price trends provided by the PMPRB? (7 point rating scale – little, somewhat, great) (Indicator 2.2.8)
   1. Please explain – How could we improve? (Indicator 2.2.8)

PMPRB Reporting & Analysis – Research and Development Spending by Patentees

1. Are you aware of the reporting on research and development (R&D) spending by patentees performed by the PMPRB? (yes, no, don't know/no answer) If response is no or don't know/no answer, skip to next section.
2. To what extent do you feel the information on R&D spending by patentees collected by the PMPRB is relevant? (7 point rating scale – little, somewhat, great) (Indicator 1.2.1)
   1. If response 1, 2, 3– Please explain why you feel this information is not relevant. (open ended) (Indicator 1.2.1)
3. To what extent do you use data on the R&D spending by patentees disseminated by the PMPRB in your work? (7 point rating scale – little, somewhat, great) (Indicator 2.6.3/1.2.4/1.2.5)
   1. If response is 3, 4, 5, 6, 7 - Please describe how you use this information.
4. To what extent do you believe that the data collected by the PMPRB on R&D spending by patentees facilitates your decision-making? (7 point rating scale – little, somewhat, great) (Indicator 2.8.1)
5. In the past three years, has your use of the information collected by the PMPRB on R&D spending by patentees (increased, decreased, stayed the same, don't know/no answer)? (Indicator 2.8.3)

PMPRB Reporting & Analysis – National Prescription Drugs Utilization Information System (NPDUIS)

INTRO FOR THIS SECTION: The National Prescription Drug Utilization Information System (NPDUIS) is a research initiative jointly conducted by the PMPRB and the Canadian Institute for Health Information (CIHI). NPDUIS seeks to provide policy makers and drug plan managers with information and insights on trends in prices, utilization and costs.

1. Are you aware of the reporting and analysis performed under the National Prescription Drugs Utilization Information System (NPDUIS) initiative? (yes, no, don't know/no answer) If no or don't know/no answer, skip to next section.
2. To what extent do you feel the information provided under the NPDUIS initiative is relevant? (7 point rating scale – little, somewhat, great) (Indicator 1.2.1)
   1. If response 1, 2, 3– Please explain why you feel this information is not relevant. (open ended) (Indicator 1.2.1)
3. To what extent do you use information and/or analysis provided under the NPDUIS initiative in your work? (7 point rating scale – little, somewhat, great) (Indicator 2.6.3/1.2.4/1.2.5)
   1. If response is 3, 4, 5, 6, 7 - Please describe how you use this information.
4. To what extent do you believe that the information provided under the NPDUIS program facilitates your decision-making, policy development and/or address your research needs? (7 point rating scale – little, somewhat, great) (Indicator 2.8.1)
5. In the past three years, has your use of information provided under the NPDUIS initiative (increased, decreased, stayed the same, don't know/no answer)? (Indicator 2.8.3)
6. To what extent do you feel the information provided under the NPDUIS initiative is comprehensive? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – Please explain why you feel information provided under the NPDUIS initiative is not comprehensive. (open ended) (Indicator 2.2.9)
7. To what extent do you believe the information provided under the NPDUIS initiative is accurate? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – please explain why you feel the information provided under the NPDUIS initiative is not accurate. (open ended) (Indicator 2.2.9)
8. To what extent do you feel the information provided under the NPDUIS initiative is timely? (7 point rating scale – little, somewhat, great) (Indicator 2.2.9)
   1. If response 1, 2, 3 – please explain why you feel the information provided under the NPDUIS initiative is not timely. (open ended) (Indicator 2.2.9)
9. To what extent are you satisfied with the overall quality of the information provided under the NPDUIS initiative? (7 point rating scale – little, somewhat, great) (Indicator 2.2.8)
   1. Please explain – How could we improve?(Indicator 2.2.8)

General

1. Do you have any other comments to add?

Appendix F : Bibliography

• Bell C, Griller D, Lawson J, Lovren D. (2010) Generic Drug Pricing and Access in Canada: What are the Implications? Toronto: Health Council of Canada. www.healthcouncilcanada.ca.
• Canadian Institute for Health Information. (2001). Prescription Drug Utilization Information System for Canada: Business Case. Ottawa: Canadian Institute for Health Information.
• Canadian Institute for Health Information. (2012) Drug Expenditure in Canada, 1985 to 2011. Ottawa: Canadian Institute for Health Information. Retrieved from https://secure.cihi.ca/free_products/DEIC_1985_2011_EN.pdf
• Canadian Pharmacists Association. (2007). Safe and Effective: The Eight Essential Elements of an Optimal Medication-Use System. Ottawa, Ontario.
• Cancer Research UK. (Jan 6 2009) The Pharmaceutical Price Regulation Scheme - Cutting the Cost of Cancer Drugs. Retrieved from http://scienceblog.cancerresearchuk.org/2009/01/06/the-pharmaceutical-price-regulation-scheme-cutting-the-cost-of-cancer-drugs/
• Cancer Society. Te Kahui Matepukupuku 0 Aoetearoa (2009) .Submission to the High Cost Highly Specialized Medicines Review Panel. New Zealand: Cancer Society Te Kahui Matepukupuku 0 Aotearoa
• Danzon, Patricia. (2000) Health: Making Sense of Drug Prices. Regulation Magazine, Volume 23, 56-62 . Retrieved from http://www.cato.org/pubs/regulation/regv23n1/danzon.pdf
• Department of Health (UK). (December 11 2008) Publications policy and guidance: The Pharmaceutical Price Regulation Scheme. Retrieved from:http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_091825
• Department of Justice Canada (2006-09-21, current to 2012--2-07) The Patent Act. Ottawa: Department of Justice Canada Retrieved from Retrieved from: http://laws-lois.justice.gc.ca/PDF/P-4.pdf
• Department of Justice Canada (2008-07-01, current to 2011-09-21) Patented Medicines Regulations SOR/94-688. Ottawa: Department of Justice Canada Retrieved from http://laws-lois.justice.gc.ca./eng/regulations/SOR-94-688/page-1.html
• Federal/Provincial/Territorial Ministerial Task Force on the National Pharmaceuticals Strategy. (2006). National Pharmaceuticals Strategy Progress Report Ottawa: Publications Health Canada. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/alt_formats/hpb-dgps/pdf/pubs/2006-nps-snpp/2006-nps-snpp-eng.pdf
• Government of Canada. (3 June 2011) Speech from the Throne. Retrieved from http://www.speech.gc.ca/eng/media.asp?id=1390.
• Government of Canada. (3 March June 2010) Speech from the Throne. Retrieved from http://www.speech.gc.ca/eng/media.asp?id=1388.
• Health Canada - Health Care System First Minister's Meeting on the Future of Health Care 2004 A 10-Year Plan to Strengthen Health Care Retrieved from http://www.hc-sc.gc.ca/hcs-sss/delivery-prestation/fptcollab/2004-fmm-rpm/index-eng.php
• Health Canada (22 March, 2010) Minister's Response to the Speech from the Throne. Retrieved from http://www.hc-sc.gc.ca/ahc-asc/minist/speeches-discours/_2010/2010_03_22-eng.php
• Health Canada. Health Care System National Pharmaceuticals Strategy. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/pharma/nps-snpp/index-eng.php.
• Health Council of Canada. (May 2011) Progress Report 2011: Health Care Renewal in Canada. Toronto: Health Council of Canada. Retrieved from http://www.healthcouncilcanada.ca/docs/rpts/2011/progress/2011Progress_ENG.pdf
• Lindberg, C (Sept 21 2011). The Patented Medicine Prices Review Board and Its Role In Our Health care System. Notes for an Address by MC Lindberg. Ottawa: Patented Medicines Prices Review Board www.pmprb-cepmb.gc.ca
• Management Accountability Framework - archived. (2008) Retrieved from http://www.tbs-sct.gc.ca/maf-crg/assessments-evaluations/2008/pxr/pxr-eng.asp
• National Patented Medicine Prices Review Board. NPDUIS Steering Committee Terms of Reference. Ottawa: Patented Medicine Prices Review Board.
• OECD. (2008). Pharmaceutical Pricing Policies in a Global Market. E. Docteur, V. Paris, and P. Moise.
• Office of Fair Trading. (2007). The Pharmaceutical Price Regulation Scheme. UK. Office of Fair Trading (http://www.oft.gov.uk/shared_oft/reports/comp_policy/oft885.pdf)
• Office of the Auditor General Chapter 12 Follow-Up of Recommendations in Previous Report - Patented Medicine Prices Review Board (1998 Chapter 17) Ottawa: Office of the Auditor General.
• Office of the Auditor General. 1998 September Report of the Auditor General of Canada - Chapter 17 - Patented Medicine Prices Review Board. Ottawa: Office of the Auditor General. Retrieved from http://www.oag-bvg.gc.ca/internet/English/parl_oag_199809_17_e_9323.html
• Patented Medicine Prices Review Board Executive Committee Terms of Reference. Ottawa: Patented Medicine Prices Review Board
• Patented Medicine Prices Review Board. (2008) Report of CGPA/PMPRB Working Group on Options for Changes to the PMPRB's Excessive Price Guidelines to Reflect the Unique Nature of Patented Generic Medicines Sold in Canada. Retrieved from:.
• Patented Medicine Prices Review Board. 2011-2012 Report on Plans and Priorities. Ottawa. Patented Medicine Prices Review Board
• Patented Medicine Prices Review Board. Roles and Responsibilities of Board Members Board Profile. Retrieved from.
• Patented Medicine Prices Review Board. (23-12-2011) Patented Medicine Prices Review Board NPDUIS Communication Strategy. Ottawa: Patented Medicines Prices Review Board
• Patented Medicine Prices Review Board. Impact of Federal Regulation of Patented Drug Prices. (1997). Patented Medicine Prices Review Board. Ottawa.
• Patented Medicine Prices Review Board.(2010-2011)Patented Medicine Prices Review Board 1010-2011 Departmental Performance Report. Ottawa: Patented Medicine Prices Review Board
• Patented Medicines Prices Review Board (January 16 2012) National Prescription Drug Utilization Information System. Retrieved from http://www.pmprb-cepmb.gc.ca/english/View.asp?x=1058&mid=834
• Patented Medicine Prices Review Board Patentee's Guide to Reporting March 2008. Ottawa: Patented Medicine Prices Review Board Retrieved from http://www.pmprb-cepmb.gc.ca/CMFiles/guide-e15LIS-4122003-3911.pdf
• Patented Medicine Prices Review Board Role Function Methods of the Board. Ottawa: Patented Medicines Prices Review Board
• Patented Medicine Prices Review Board. (2011) Annual Report 2010. Ottawa: Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (2011) Departmental Performance Report 2010-11. Patented Medicine Prices Review Board http://publications.gc.ca/collections/collection_2011/cepmb-pmprb/H79-2-2011-eng.pdf
• Patented Medicine Prices Review Board. (December 2010). Baby-Boomer Effect on Prescription Expenditures and Claims: Impacts of Demographic Change on Provincial Public Drug Plans in Alberta, Saskatchewan, Manitoba, New Brunswick and Nova Scotia. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicines Prices Review Board. (December 2010). Generic Drugs in Canada: Market Structure - Trends and Impacts. Ottawa: NPDUIS - Patented Medicines Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (December 2010). Use of the World Health Organization Defined Daily Dost in Canadian Drug Utilization and Cost Analysis. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (December 2010) Generic Drugs in Canada: Price Trends and International Price Comparisons, 2007. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicines Price Review Board. (December 2011/Revised January 2012). Wholesale Up-charge Policies of Canada's Public Drug Plans. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (July 2011) New Drug Pipeline Monitor: July 2011. Retrieved from http://www.pmprb-cepmb.gc.ca/english/View.asp?x=1528&mp=124
• Patented Medicine Prices Review Board. (June 2011) Compendium of Policies, Guidelines and Procedures. Ottawa: Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (September 2011). Generic Drugs in Canada: International Price Comparisons and Potential Cost Savings. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (September 2011). The Impact of Generic Entry on the Utilization of the Ingredient. Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. (September 2011). Public Drug Plan Dispensing Fees: A Cost-Driver Analysis 2001/02 to 2007/08.Ottawa: NPDUIS - Patented Medicine Prices Review Board. www.pmprb-cepmb.gc.ca
• Patented Medicine Prices Review Board. Annual Report 2009 - Voluntary Compliance Undertakings and Hearings Ottawa: Patented Medicine Prices Review Board. Retrieved from
• Patented Medicine Prices Review Board. Annual Report 2010. Ottawa: Patented Medicine Prices Review Board.
• Patented Medicine Prices Review Board. Monitoring and Evaluation Plan for the Major Changes in the Guidelines. Retrieved from
• Patented Medicine Prices Review Board. The Prime Minister's Red Tape Reduction Commission. Ottawa: Patented Medicine Prices Review Board
• Paterson, M., Suleimanm, A., Hux, J., Bell, C. (2008). How Complete Are Drug History Profiles That Are Based on Public Drug Benefit Claims?. Canadian Journal of Clinical Pharmacology Volume 15 (1) Winter 2008 108-116.
• Pharmaceutical Management Agency (of New Zealand) - PHARMAC Website - all pages. New Zealand. Retrieved from: http://www.pharmac.govt.nz/patients/AboutPHARMAC/procedures
• Public Works and Government Services Mobilizing Science and Technology to Canada's Advantage Progress Report 2009. Ottawa: Publishing and Depository Services Public Works and Government Services http://ic.gc.ca/epublications.
• Secretary of State for Health. (2010) Equity and Excellence: Liberating the NHS. UK. The Stationery Office Limited.
• The Fraser Institute. (Spring 2001). Seeing Through the Snow. Health and Medicine. Graham, J., pages 46-50.
• The Treasury Board Canada (no year shown) Treasury Board Submission - Funding for activities that are essential to the delivery of the Patented Medicine Prices Review. Ottawa. Treasury Board Canada.
• Trottier M, Boyle M. (April 15, 2008). PMPRB Governance Review. Ottawa: Institute on Governance.
• (UK) Department of Health. Cancer Drugs Fund will provide 200 million (pounds) a Year. Retrieved from http://www.dh.gov.uk/en/MediaCentre/Pressreleases/DH_120941
• Wright, Donald J. (2006). The Drug Bargaining Game: Pharmaceutical Regulation in Australia. Journal of Health Economics (2004). Retrieved from http://www.fgcasal.org/politicafarmaceutica/docs/Donald_Wright.PDF