Critical Appraisal of Evidence in Medical Education: A Systematic Review of Methodological Approaches, Evidence Types, and Appraisal Frameworks

Ersin Akpınar · Dogu Akdeniz Aile Hekimleri Dernegi (DAAHED) · 2025 · medical-education · 10.64514/vmsjournal.2025.13 · record akp-nar2025-critical-appraisal-of-evidence-in
tag layer · index · concepts · authors · עברית
Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type was a close call between 'method' and 'critique': the paper's stated deliverable is the appraisal matrix (tagged method), but its central conclusion is negative — "traditional biomedical hierarchies of evidence are insufficient - and at times misleading - when applied uncritically to medical education research." and "Quotation withheld" claim relations is empty: the review names frameworks (BEME, MERSQI, CASP, PRISMA, COREQ) as reviewed objects and cites only by number, never positioning itself against a specific named study. The PDF extraction is column-scrambled in the abstract and discussion (fragments such as "into it The present systematic review underscores"), so quotes were taken only from stretches that are contiguous and coherent; the two tables are extracted as broken cell runs and were not quoted. Search yield and inclusion counts are reported ("The database search yielded 1,246 records.", "Eighty-four studies met inclusion criteria.") but no PRISMA flow figure or study list appears in the text. 2026-08-25 schema-0.3 pass: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.
This record’s own method is systematic-review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.
How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Other fields define these differently the same word, defined another way in another discipline
context 10 fields · systematic-review 4 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses.
Educational interventions sit inside complex social systems, which blocks causal attribution and so makes biomedical evidence hierarchies inapplicable. Absent an explicit theoretical frame, a study stays descriptive, which drains its explanatory power and stops findings accumulating across the field. transferability context systematic-review

Legend

claim typeclaim_typethe shape of the claim: existence, mechanism, method, critique, prediction, null
content tagcontent_tagsa finding, definition, quantity, statistic or variable named in the paper
controlcontrolsan experiment run to eliminate a rival explanation
limitationlimitationswhat the paper says its evidence cannot support
mechanismmechanismshow the reported effect is said to come about
methodmethodhow the paper produced its evidence
phenomenonphenomenonthe effect or regularity under study
transfertransfersa method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.64514/vmsjournal.2025.13. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_type 1 · content_tags 13 · controls 1 · limitations 2 · mechanisms 2 · method 1 · phenomenon 1 · transfers 1

This record holds no relations between concepts, relations between claims or senses of a term. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facettagfieldconf.whose words
claim_typemethod
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_typethe paper's words
phenomenoncritical appraisal of evidence in medical education
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
phenomenonthe paper's words
methodsystematic-review
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
methodthe paper's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[0]the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[1]the tagger's words
limitationsRestricted to English-language publications and excluded grey literature.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
limitations[0]the tagger's words
limitationsThe narrative synthesis approach may introduce interpretive bias.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
limitations[1]the tagger's words
transfersThe review's whole object is the import of evidence-based medicine's appraisal principles and evidence hierarchies into medical education, an import it judges conceptually and methodologically problematic and then reworks into an education-specific matrix.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
transfers[0]the tagger's words
controlseditorials without methodological content and purely clinical studies excluded before appraisal
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
eliminates: an appraisal of evidence quality carried out on documents that make no methodological claim · outcome (stated by the tagger, no quotation): held
controls[0]0.85the tagger's words
content_tagsOf 1,246 records retrieved by the database search, 84 studies met the inclusion criteria.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[0]the tagger's words
content_tagsEvidence in medical education research falls into four types: quantitative, qualitative, mixed-methods, and design-based research.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[1]the tagger's words
content_tagsBiomedical evidence hierarchies imported into medical education failed to capture educational relevance and contextual complexity.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[2]the tagger's words
content_tagsBEME, MERSQI, CASP and realist review frameworks were the appraisal tools most often cited, but they were applied inconsistently across the included studies.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[3]the tagger's words
content_tagsRandomised controlled trials were frequently impractical or educationally reductive in education settings, while qualitative and theory-driven studies were undervalued despite being methodologically rigorous.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[4]the tagger's words
content_tagsTheory is underused throughout the reviewed medical education research and appraisal literature.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[5]the tagger's words
content_tagsThe review develops an integrative appraisal matrix from the synthesised literature.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[6]the tagger's words
content_tagsThe proposed matrix scores studies on six domains - relevance, methodological rigor, theoretical grounding, outcomes meaningfulness, context, and impact - instead of reducing quality to one number.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[7]the tagger's words
content_tagstransferability
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.definitions[0]the tagger's words
content_tagscontext
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
about this quote: THE QUOTE STOPS WHERE THE EXTRACTED TEXT GOES OUT OF ORDER: the words that follow it in the extraction ('review educational reinforces that generalizability …') belong to another line of the page, so the sentence cannot be completed from this copy.
also in: context
content_tags.definitions[1]the tagger's words
content_tags1,246 — records returned by the database search before de-duplication and screening
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.quantities[0]the tagger's words
content_tags84 — studies that met the inclusion criteria and were synthesised
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.quantities[1]the tagger's words
content_tags2000 and 2024 — publication window for eligible studies in the four databases searched
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.quantities[2]the tagger's words

Interpretive choices

Tags that could not be placed in the text:

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.

© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824