Engineering Dilemmas of Digital Infrastructure: Global Lessons from AI Project Failures in the RAND Report

The Engineering Dilemma of Digital Infrastructure: Why Do AI Projects Fail at Scale?

A Global Capital Adventure

As global capital floods into artificial intelligence, a sober study has sounded the alarm for the frenzy. A RAND Corporation report released in August 2024 points out that more than 80% of AI projects fail to meet expectations—a failure rate twice that of traditional IT projects without AI involvement. This means that AI development is not an ordinary technology upgrade, but a form of infrastructure engineering with far greater complexity.

The report is based on in-depth interviews with 65 senior data scientists and machine learning engineers. The interviewees come from enterprises and institutions of varying sizes, with an average of more than five years of experience. Their consensus is that AI project failures do not stem from algorithms being insufficiently advanced, but rather from the entire construction process violating the basic principles of infrastructure engineering.

Data, Algorithms, and Deployment: The Three Pillars of Digital Infrastructure

Any infrastructure—whether a transportation network, an energy pipeline, or a data center—requires close coordination among the planning, construction, and operation phases. AI systems are no different: data is the raw material, algorithms are the core equipment, and the deployment environment is the operating site. A disconnect in any one of these links can lead to systemic collapse.

RAND's research distilled five recurring "anti-patterns" that correspond precisely to five categories of structural defects in engineering projects. When we place them within an infrastructure analysis framework, it becomes clear that their destructive power far exceeds that of ordinary technical failures.

First: Problem Definition Deviation—Mistaken Site Selection and Misjudged Requirements

In physical engineering, mistakes in site selection or misjudged requirements often cause a project to be unable to function after completion. AI teams frequently receive vague business directives, such as "we need to predict sales" or "make the process intelligent." But without a rigorous definition of the target scenario, the model optimizes the wrong metrics, and the deliverables fail to integrate with real workflows. The report shows this is the most commonly cited cause of failure among respondents. In infrastructure terms, this is equivalent to building a cargo airport in a residential area—no matter how high the engineering quality, it cannot be used.

Second: Insufficient Data Assets—Supply Chain Disruption

Infrastructure requires a reliable supply of materials; AI requires high-quality training data. Many organizations assume that simply accumulating more data is enough, while neglecting data quality, annotation accuracy, timeliness, and legal compliance. Data is AI's "concrete"—substandard raw materials inevitably produce fragile models. The report notes that possessing large volumes of data does not mean possessing usable data assets; a broken data pipeline is more fatal than outdated algorithms.

Third: Technological Supremacy—Equipment Worship

Third: Techno-supremacy — Device worship

In the engineering field, there is often a phenomenon of "technology worship." For example, new building materials and smart systems are blindly adopted without asking whether they suit local geology and climate. AI teams face the same temptation: pursuing the most cutting-edge large models and deep learning frameworks while ignoring targeted solutions to real problems. RAND's interviewees observed that some projects choose the "flashiest" technology merely to show superiors that they are "making progress," and ultimately cannot be put into production. This tendency turns projects from problem-solving into technology performance.

Fourth: Missing support systems — Incompatible operational infrastructure

Traditional infrastructure projects cannot operate without power grids, roads, and maintenance teams. AI projects similarly depend on complete data management, model deployment, monitoring, and update infrastructure. Many organizations invest heavily in the development stage but lack the engineering capability to integrate models into business systems. The report points out that without appropriate MLOps tools and management processes, high-accuracy models in the laboratory rapidly degrade in real-world environments, or even become unusable.

Fifth: Misjudgment of problem complexity — Starting construction on impossible terrain

AI is not omnipotent; some tasks are inherently difficult to automate. The RAND report reminds us that when technology is used to solve problems beyond current AI capabilities, failure is inevitable. This is like forcibly building large projects on steep cliffs or geological fault zones, ignoring the limits of physics and technology. Interviewees stressed that AI's "intelligence" has boundaries, and managers must acknowledge this rather than deifying AI as a magical tool.

AI Risks and Capital Allocation from a Project Financing Perspective

If we evaluate AI projects as a type of infrastructure asset, their financing failure rate is as high as 80%, enough to deter any infrastructure investor. RAND's data shows that although 97% of business leaders feel urgency to deploy AI, only 14% of organizations are truly ready. This huge gap between "willingness and capability" reflects a structural mismatch in capital allocation: a large amount of funding is poured into poorly justified pilot projects, while the foundational investment needed to sustain these projects over the long term is severely insufficient.

Infrastructure financing has mature concepts of "feasibility study" and "whole-life cycle cost." AI projects should not only focus on the initial cost of training models, but also must consider the costs of data governance, model governance, continuous learning, and compliance auditing. The interviewees in the RAND report largely come from the engineering execution level; they see large budgets consumed on model tuning, while the most basic data pipelines and system integration receive no funding. This mismatch prevents projects from crossing the "valley of death" to move from model prototype to large-scale deployment.

Digital Infrastructure in Strategic CompetitionThe report specifically mentions that the U.S. Department of Defense invests $1.8 billion annually in military AI applications and regards AI as a key technology for future warfare. This indicates that AI is not only a commercial competitive advantage but also part of national strategic infrastructure. As major economies worldwide compete to build computing centers, data platforms, and intelligent decision-making systems, the success or failure of AI projects directly affects long-term industrial competitiveness and national defense security. The high failure rate consumes not just capital, but also time windows and the foundation of trust. If success rates cannot be improved from an engineering perspective, national strategies will also face the risk of "paper systems."

From Anti-Patterns to a Blueprint for Success: Building Resilient AI Infrastructure

The RAND report offers success principles based on interviews. For industry, leaders must first clarify the boundaries of the problem and avoid "black-box" tendering; second, cultivate data awareness within the organization and establish genuine data assets; third, choose mature, stable, and maintainable technology paths rather than chasing trends; fourth, invest in MLOps infrastructure so that models can be continuously deployed and monitored; and finally, maintain realistic expectations about the boundaries of AI capabilities.

For academia, the report recommends that research projects focus more on implementable solutions under real-world constraints and strengthen the integration of data science with domain knowledge. This is crucial for AI infrastructure construction, because theoretical breakthroughs must be translated into deliverable engineering outcomes.

These principles are not new. Like the success of all large-scale infrastructure, they emphasize order, discipline, and a long-term perspective. AI is not magic, but an emerging infrastructure that requires meticulous engineering management. As governments and the private sector compete in the digital domain, it must be understood that real infrastructure must not only be built, but also used and maintained.

Conclusion: From High Failure Rates to High Maturity

The current high failure rate of AI projects does not mean that AI is worthless, but rather indicates that the field is in a painful transition from "technological experimentation" to "reliable engineering." Historically, railways, power grids, and telecommunications networks have all gone through similar stages. Early railway construction saw numerous planning errors and financial losses, and only through standardization, engineering norms, and government regulation did railways become the arteries of modern civilization.

AI development today is no different. RAND's research provides a valuable checklist for reflection. For global infrastructure investors and policymakers, the important thing is not to avoid AI, but to treat every AI initiative with the rigorous attitude of building infrastructure. Only by viewing data, algorithms, computing power, and organizational processes as an integrated system, and following the complete cycle from definition to operation, can AI's potential be transformed into lasting productivity. The next era of global competition will be defined by the countries and institutions that can successfully build digital infrastructure.