Leitlinienwatch

Criterion 1: Transparency

0 points: No information about conflicts of interest *
1 point: Conflicts of interest are listed in general terms for each author in the guideline documents (yes/no)
2 points: Conflicts of interest are documented in detail (with information about the companies)
3 points: Conflicts of interest are evaluated by the coordinators or a separate committee

*This criterion refers to financial conflicts of interest: proprietary interest and shares, consulting contracts (advisory boards), lecture fees, or other commissions from manufacturers related to the products to be evaluated, in addition to private partner relationships with employees of the manufacturing company. Conducting research for the manufacturer qualifies as a conflict of interest, even if the researcher did not receive a personal fee. Three points are awarded if all authors are free from conflicts of interest, and no further evaluation is required.

Criterion 2: Composition of the Guideline Working Group

0 points: > 50% have financial conflicts of interest *
1 point: 25% to 50% have conflicts of interest
2 points: < 25% have conflicts of interest
3 points: No guideline authors have conflicts of interest

*0 points will be awarded if no detailed information is provided regarding the individual authors’ conflicts of interest.

Criterion 3: Independence of the Lead Authors/Coordinators/Chairs

0 points: > 50% have conflicts of interest*
1 point: 25% to 50% have conflicts of interest
2 points: < 25% have conflicts of interest
3 points: None of the of the lead authors/coordinators/chairs have conflicts of interest

*If detailed information about the individual authors’ conflicts of interest is not provided, 0 points are awarded.

Criterion 4: Abstention from Voting

0 points: No information or no evidence of abstentions from voting when conflicts of interest exist
1 point: There is a rule regarding abstention but voting results are not documented
2 points: Partial abstentions
3 points: Fully documented abstentions by members with conflicts of interest

Note

3 points are awarded if none of the authors has conflicts of interest, because in this case abstention is not required.

Criterion 5: External Review of the Guideline

0 points: No external review
2 points: Consulting on the draft guidelines by the scientific community or patients via a website
3 points: Review of the draft guideline by the scientific community or patients and documentation of how the suggestions were handled

Note

1 point is not awarded here.

Bonus Points

Further measures for reducing conflicts of interest can be awarded a maximum of 3 bonus points. Examples of such measures include:

Documented efforts to recruit independent authors
Transparent system for evaluating the relevance of conflicts of interest
Non-participation of authors with major conflicts of interest after assessment of the information regarding conflicts of interest
Open discussion of conflicts of interest in the guideline working group at the beginning of the cooperation process, including discussion of consequences such as abstention
Pluralistic composition of the guideline working group including methodologists and patient representatives
Study evidence prepared by methodologists
Independent internal review process that goes beyond approval from scientific medical societies
Strong consensus on recommendations

Overall Score

The maximum possible score is 15 points plus 3 bonus points.

11 to 18 points: Good! Independent guidelines with properly regulated conflicts of interest

6 to 10 points: Warning! Good approach, but only partial regulation of conflicts of interest

0 to 5 points: Needs reform! Insufficient regulation of conflicts of interest

Notes on the Evaluation Criteria

The initiators of GuidelineWatch are aware that the independence of a given set of guidelines is just one of many criteria indicating quality – but it is an important criterion that has not yet been thoroughly evaluated anywhere else. A good tool for assessing methodological quality is the Deutsche Leitlinien-Bewertungs-Instrument (German Guideline Evaluation Tool, DELBI).
The evaluation criteria go beyond what the AWMF required when the guidelines were drawn up. Instead, they reflect the state of the international discussion (see links below) and can contribute to promote reform in Germany as well.
Quantitative assessment of conflicts of interest can be equivocal as on the one hand, many conflicts of interest are not disclosed, and on the other hand, not every relationship with the pharmaceutical industry necessarily results in conflicts of interest – say, for example, a company does not manufacture the medications relevant to the guidelines in question. However, the vast majority of conflicts of interest disclosed in guidelines are connected to the subject of the guidelines.

*Links

Evaluation Criteria of GuidelineWatch