MedSkillAudit: A Quality Audit Framework for Pre-Deployment of Medical AI Agent Skills — A New Standard for Digital Health Infrastructure

Overview

On June 29, 2026, AIPOCH, in collaboration with the Department of Pathology of Zhongshan Hospital, Fudan University, officially released MedSkillAudit, a pre-deployment domain-specific audit framework for medical AI agents. The framework aims to identify scientifically unreliable capability modules—those with fabricated references or reasoning errors—before deploying AI agent skills in medical research environments. This initiative marks a new phase in quality control for digital health infrastructure, analogous to the introduction of independent technical audits and due diligence processes in infrastructure projects.

Medical AI Agents: Modular Components of Digital Health Infrastructure

Current medical research agents are increasingly composed of modular skills, covering literature screening, statistical analysis, protocol design, and even manuscript drafting. These skills resemble subsystems and microservices in smart infrastructure; without rigorous quality gates, they may introduce systematic errors into clinical research pipelines. Like bridges or power grids in physical infrastructure, AI agent skills must undergo assessment of structural integrity, functional reliability, and safety margins before being put into production.

MedSkillAudit’s Two-Layer Veto Gate and Two-Stage Evaluation

MedSkillAudit introduces a “two-layer veto gate” review process. The first layer assesses operational stability, structural consistency, result certainty, and system safety. The second layer evaluates four dimensions of scientific integrity: scientific completeness (no fabricated references, DOIs, sample sizes, or p-values), practical boundaries (no direct diagnostic conclusions; must include medical disclaimers), methodological baselines (no logical fallacies, such as confusing correlation with causation), and code availability (generated code must have no syntax errors or missing core dependencies). Any skill that fails to meet critical requirements will be blocked from deployment.

In terms of evaluation methodology, the framework adopts static evaluation (design quality, 40%) and dynamic evaluation (runtime performance, 60%), combining skill design and source code review with execution tests in simulated research scenarios. The final score classifies skills into four readiness levels: “Production Ready,” “Limited Release,” “Beta Only,” and “Rejected.”

Validation Results: 57.3% of Skills Fail to Reach the Limited Release Threshold

In a validation study covering 75 skills (across five medical research categories: evidence insight, protocol design, data analysis, academic writing, and others), 57.3% scored below the “Limited Release” threshold. This result underscores the urgency of such gating mechanisms. The study also shows that MedSkillAudit’s evaluations are highly consistent with expert review and stable across different assessments.

Quality Control Logic from an Infrastructure PerspectiveFrom the perspective of global infrastructure project financing, MedSkillAudit serves as a "technical due diligence" tool, similar to the review of design documents and construction plans by an independent engineer in PPP projects. In the digital health infrastructure domain, AI agent skills are capital-intensive "software assets," and their reliability directly impacts clinical research investment returns and patient safety. AIPOCH CEO Huimei Wang noted: "AI agents are becoming part of scientific workflows, yet the skills they rely on lack quality control checkpoints. MedSkillAudit helps researchers identify scientific, methodological, and ethical risks before deployment."

Regional Cooperation and Trends in Digital Infrastructure Development

This release was jointly completed by Singapore-based AIPOCH and Zhongshan Hospital Affiliated to Fudan University in China, reflecting collaborative innovation in the digital health infrastructure field within the Global South. As a regional digital hub, Singapore is actively promoting the integration of AI governance frameworks into infrastructure standards. With the rapid deployment of healthcare AI agents in Global South markets such as Southeast Asia, the Middle East, and Africa, a pre-audit framework like MedSkillAudit will become an essential component of digital infrastructure investment.

Conclusion

MedSkillAudit is not only a quality tool in the medical AI field but also an important milestone in the standardization of digital health infrastructure. It draws on the tiered acceptance logic used in physical infrastructure engineering, providing a quantifiable safety baseline for the modular deployment of AI agents. In the future, as AI agents become foundational components of medical research, domain-specific audit frameworks will go hand in hand with traditional model evaluation methods, forming the core pillar of digital infrastructure risk management.