Overview
When a clinician submits a query, OpenBook Clinical does not generate responses from an AI model's training memory. Instead, it executes real-time queries against verified academic databases, retrieves candidate papers, grades them against the Australian NHMRC evidence hierarchy, and synthesises findings using Anthropic Claude (claude-3-5-sonnet-20241022).
Every citation presented in the synthesis must be traceable to a retrieved database record. Claims that cannot be attributed to a retrieved paper are not surfaced to the clinician. This architecture is designed to minimise — though not eliminate — the risk of fabricated citations.
Evidence Sources (in priority order)
Queries are dispatched in parallel to all four sources. Results are deduplicated by DOI before being passed to the synthesis step. Sources are queried in this priority order when resolving conflicts between records.
| Source | Database | Coverage | Query Method |
|---|---|---|---|
| Semantic Scholar | 200M+ papers | All disciplines | Semantic search + AHPRA discipline filters |
| PubMed Central | Full-text articles | Biomedical focus | MeSH terms + allied health filters |
| CORE | Open access papers | Global research | Keyword + discipline filter |
| Unpaywall | Full-text detection | N/A | DOI resolution for open-access PDFs |
Deduplication: Records sharing the same DOI are collapsed into a single entry. Where metadata differs between sources for the same DOI, the record from the highest-priority source is used.
Australian weighting: Records from Australian authors or institutions, or papers that explicitly address Australian practice guidelines, receive a relevance boost. Australian sources (1.0), international (0.8), non-matching national guidance (0.5).
NHMRC Evidence Hierarchy
OpenBook Clinical grades retrieved papers using the Australian National Health and Medical Research Council (NHMRC) evidence hierarchy — the standard evidence classification framework used in Australian clinical practice guidelines. Grading is performed automatically by analysing study design descriptors in the abstract and metadata.
Systematic review of RCTs
A systematic review of all relevant RCTs. The highest level of evidence for questions of intervention effectiveness.
Randomised Controlled Trial (RCT)
A properly conducted RCT. Considered the gold standard for individual studies of treatment effectiveness.
Pseudo-RCT
A quasi-randomised controlled trial with some allocation mechanism that is not truly random.
Comparative study with concurrent controls
Prospective or retrospective cohort studies, case-control studies, or interrupted time series with a control group.
Comparative study without concurrent controls
Historical control studies, two or more single-arm studies, or interrupted time series without a parallel control group.
Case series / Expert opinion
Case series with or without pre-test/post-test outcomes, cross-sectional studies, and expert clinical opinion including clinical practice guidelines based on expert consensus.
Source: NHMRC (2009). NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. National Health and Medical Research Council, Canberra.
AI Synthesis Process
Evidence synthesis is performed by Anthropic Claude (claude-3-5-sonnet-20241022). The model is not permitted to generate citations from training memory — all citations must be grounded in the retrieved paper set. The synthesis pipeline runs in six sequential stages:
- 01
Query reformulation
The user's natural-language query is reformulated for academic database search. The AI extracts clinical concepts, identifies relevant MeSH terms and allied health taxonomy, and constructs structured queries optimised for each source database. The original query intent is preserved.
- 02
Multi-source retrieval
Parallel queries are dispatched to Semantic Scholar, PubMed Central, CORE, and Unpaywall simultaneously. Results from all sources are pooled. Records are deduplicated by DOI. Metadata is normalised to a common schema.
- 03
Relevance filtering
Papers are filtered by year range, study type, and discipline based on active filters applied by the clinician. Papers below a relevance threshold (cosine similarity to the query embedding) are discarded. Australian clinical context is upweighted where applicable.
- 04
Evidence grading
Each retrieved paper's study design is assessed from title, abstract, and publication type metadata. An NHMRC evidence level (I through IV) is assigned automatically. Grading is heuristic — see the Limitations section for known accuracy bounds.
- 05
Synthesis
Claude synthesises findings from the retrieved and graded paper set. The model is instructed to highlight agreements and conflicts in the evidence, foreground higher-quality evidence, note gaps or limitations in the literature, and write in plain clinical English appropriate for an Australian allied health professional. The synthesis is grounded — every claim is traceable to the retrieved papers.
- 06
Citation integrity
All claims in the synthesis are traced to source papers in the retrieved set. Citations that cannot be matched to a retrieved record are excluded. DOIs are validated to resolve to a real record where technically feasible. The synthesis is not presented to the clinician until this step is complete.
Limitations (Important)
The following limitations must be understood by all users of OpenBook Clinical before relying on its output in any clinical context.
AI-generated synthesis may contain errors
AI-generated synthesis may contain errors. All output must be verified by the treating clinician before being applied in any clinical decision. OpenBook Clinical is a decision support tool — it does not replace clinical judgement.
Database coverage is not exhaustive
Grey literature, conference proceedings, non-English papers, paywalled journals without open-access versions, and papers from databases not covered by our four sources may be missed. The absence of evidence in a search result does not confirm the absence of evidence in the literature.
NHMRC grading is automated and may misclassify
Automated NHMRC grading may mis-classify study designs in ambiguous abstracts, non-standard paper formats, or papers with atypical methodology descriptions. Complex multi-arm trials, adaptive designs, and N-of-1 studies are particularly prone to misclassification.
AI models can hallucinate citations
Despite our citation-grounding architecture, AI models may occasionally generate plausible-sounding but incorrect citation metadata (e.g. incorrect author order, year, or journal). Always verify DOIs before citing in clinical documentation or reporting.
Evidence currency — indexing lag
Papers are retrieved in real-time from academic databases, but database indexing may lag 1–6 months for recent publications. Very recently published papers (less than 4 weeks old) may not be indexed and will not appear in results.
This tool does not replace clinical judgement
OpenBook Clinical does not replace clinical judgement, professional supervision, direct patient assessment, or institution-specific clinical protocols. Evidence synthesis is one input into clinical decision-making, not a standalone clinical recommendation.
Known Hallucination Rate
Honest disclosure — validation status
We have not yet published a formal validation study. Informal internal testing shows a citation accuracy rate of approximately 85–92% for returned DOIs (i.e., the DOI resolves to a real paper with the stated title and lead author). The remaining 8–15% of DOIs either resolve to a different paper, do not resolve, or resolve to a paper with partially incorrect metadata (typically author order or year).
We are pursuing a formal validation study with a university partner to quantify hallucination rates across disciplines, query types, and evidence levels. Results will be published on this page when available.
What this means for clinical use
Clinicians should independently verify all citations before using them in clinical documentation, academic writing, or formal clinical reasoning. Do not cite an OpenBook Clinical-generated reference without first confirming the source paper exists and matches the attributed claim. DOI verification can be performed at doi.org or via PubMed.
Retraction Screening
Automatic real-time retraction detection
Every retrieved paper that carries a DOI is automatically checked against the Crossref retraction database in real time before results are displayed. Retracted papers are prominently flagged with a red ⚠ RETRACTED banner, and clinicians are warned not to cite or act on them.
The check queries the Crossref REST API (api.crossref.org/works/{DOI}) for each DOI and inspects the update-to field for entries of type retraction. Where a retraction date is recorded by Crossref, it is displayed alongside the warning.
Scope
All retrieved results with a DOI. Results without a DOI (e.g. grey literature, book chapters) are not screened.
Data source
Crossref is the primary retraction data source. Crossref indexes retraction notices from publishers and from Retraction Watch via their partnership. Coverage is comprehensive for major journals but may lag for small publishers or predatory journals.
Latency and currency
The check is performed at search time against the live Crossref API. Results reflect the state of the Crossref database at the moment of the query. Retraction notices that have not yet been submitted to Crossref by the publisher will not be detected.
Best-effort, not exhaustive
Retraction screening is best-effort. If Crossref is unavailable or returns an error, the check fails silently — the search result is still displayed but without retraction status. Clinicians should independently verify the status of any paper before clinical use, particularly for older papers in disciplines with historically lower Crossref coverage.
Performance
Retraction checks are parallelised across the top-10 DOI-bearing results using concurrent API calls. Results are cached for the session to avoid duplicate requests. The checks run server-side and do not delay the AI synthesis step.
Comparison with Other CDS Tools
OpenBook Clinical is designed specifically for Australian allied health professionals and integrates Australian evidence standards — specifically the NHMRC evidence hierarchy. This distinguishes it from generic AI tools and from established international clinical decision support databases.
| Feature | OpenBook Clinical | UpToDate / DynaMed | General LLM (ChatGPT etc.) |
|---|---|---|---|
| Evidence source | Real-time database retrieval | Curated editorial database | Training data memory (not real-time) |
| Citation grounding | Retrieved papers only | Editorial citations | No grounding — hallucination-prone |
| Evidence hierarchy | NHMRC (Australian standard) | Oxford CEBM / editorial | None |
| Australian focus | Yes — AHPRA disciplines, NHMRC, RACGP | International (limited AU content) | No |
| Currency | Real-time (1–6 month indexing lag) | Editorial update cycle (months) | Training cutoff (typically 1–2 years) |
| Synthesis quality certainty | Lower (AI-generated) | Higher (editorial peer review) | Very low (no grounding) |
| Formal clinical validation | In progress | Yes (extensive) | No |
Requesting Methodology Review
University ethics committees, hospital procurement teams, and research organisations may request a methodology review, data processing agreement, or detailed technical documentation.
Contact clinical governance
Methodology enquiries
clinical@openbookclinical.com.auGeneral governance enquiries
openbookclinical@gmail.comWe respond to formal governance and methodology enquiries within 2 business days.