AI goes mainstream—but governance gaps and legacy chaos threaten enterprise ambitions

Large U.S. enterprises are decisively moving beyond isolated analytics and AI experiments, embedding these technologies into the very fabric of their core business systems and decision-making processes. As Kathy Rudy of ISG Data, Analytics and Technology Office observes, organizations are no longer content with marginal pilots; instead, AI is now central to how decisions are made, processes automated, and performance measured. This shift is underscored by a dramatic surge in investment, with AI spending nearly tripling over the past two years to account for almost 6% of IT budgets, signaling AI’s new role as a linchpin in digital transformation efforts.

To support this enterprise-wide integration, leading organizations are adopting formalized operating models that consolidate leadership, governance, and architectural ownership of advanced analytics and AI initiatives. Rather than treating AI as a series of disconnected projects, companies are building continuous operational capabilities—an approach that, according to ISG principal analyst Gowtham Sampath, is key to linking AI investments directly to measurable business outcomes. This organizational maturity is further reflected in the move to unified platforms and standardized processes, particularly in sectors like CPG, where 82% of companies are prioritizing data and process standardization to enable scalable AI adoption.

The transition from experimentation to integration is marked by enterprises progressing through distinct stages of AI adoption, with most now moving from assisted AI pilots to broader rollouts featuring guarded autonomy and measurable, auditable outcomes. This evolution is not just technological but operational, as organizations increasingly invest in robust data infrastructure, employee training, and AI security to ensure readiness for enterprise-scale deployment. As Dave Vellante predicts, 2026 will be the year when enterprises begin to see and report significant AI ROI at scale, a milestone enabled by massive investments in AI infrastructure from industry giants like Meta and Microsoft.

As large U.S. enterprises grapple with sprawling IT landscapes—averaging nearly 2,000 systems—unified data foundations have become the linchpin for scaling predictive analytics and ambitious AI initiatives. This shift is driven by the urgent need to tame data and application fragmentation, with companies like Aramco and McDonald's reaping the rewards of early, disciplined investments in their data infrastructure. However, the journey is far from over: over 90% of the necessary data work remains globally, and vendor sprawl—averaging seven data management and up to nine AI management providers per enterprise—continues to introduce complexity, costs, and security risks that threaten long-term scalability and ROI.

To move beyond scattered pilots and realize enterprise-wide AI, organizations are formalizing operating models that consolidate leadership, governance, and architectural ownership. This approach embeds AI value into leadership objectives and financial metrics, as seen in industry leaders who treat data, analytics, and AI as continuous operational capabilities rather than isolated projects. Yet, the lack of clear accountability frameworks—such as human verification workflows and audit trails—has triggered a governance crisis, exposing enterprises to regulatory and reputational risks, underscoring the need for disciplined data practices and robust compliance mechanisms.

Midsize enterprises, meanwhile, are charting their own path with hybrid operating models that blend central coordination with domain autonomy, allowing different business units to advance analytics and AI projects at their own pace. Supported by adaptable service providers offering accelerators and migration frameworks, this approach balances agility with oversight—though it requires careful orchestration to avoid the pitfalls of fragmentation and to ensure consistent governance across the organization.

Across industries, the push for unified data strategies is also a response to legacy system fragmentation and manual processes—challenges cited by 82% and 64% of CPG companies, respectively. Overcoming these hurdles demands not just modern infrastructure, but a holistic focus on people, process, and data: comprehensive employee training (72%), high-quality data infrastructure (66%), and strong AI cybersecurity and compliance (64%) are now seen as critical enablers for AI readiness and value realization, as emphasized by Veeva QualityOne’s David Maher.

As large U.S. enterprises grapple with the daunting complexity of integrating analytics and AI at scale, service providers have become indispensable partners in this transformation. Organizations are increasingly turning to these external experts not just for data modernization, but to orchestrate the integration of analytics and AI across sprawling IT environments—often spanning nearly 2,000 systems. Specialist providers, particularly those with deep expertise in regulated sectors like healthcare, financial services, and manufacturing, are now critical to unlocking advanced analytics and AI capabilities that meet stringent industry requirements.

The infrastructure demands of enterprise AI have triggered a surge in investment and innovation, with data center construction spending up 338% since 2020 and data centers now accounting for roughly two-thirds of infrastructure deal volumes, according to Citadel and Wells Fargo. Service providers such as Crusoe Cloud are stepping in to address the scarcity and complexity of powering AI-first infrastructure, leveraging creative solutions like renewable energy and repurposed supersonic jet turbines to energize massive GPU clusters. However, this build-out is not without hurdles: supply chain constraints for critical components like gas turbines, as cited by Crusoe Cloud’s CEO, and the rapid obsolescence of AI models—exemplified by OpenAI’s $5B investment in GPT-5 being eclipsed within a quarter—underscore the need for robust, adaptable infrastructure strategies and deep provider expertise.

The proliferation of vendors—averaging seven for data management and up to nine for AI management in large enterprises—has introduced significant complexity, cost, and security risks, fueling a shift toward platform integration and unified data foundations. While many data leaders once believed that a diverse vendor ecosystem was necessary, mounting evidence suggests that consolidating around integrated platforms with core capabilities like cataloging, privacy, and access management is more effective for sustainable AI governance. Service providers such as Accenture are increasingly valued not just for technology delivery, but for helping enterprises rethink operating models, embed AI value into leadership objectives, and prioritize 'must-win battles' as they modernize data infrastructure.

As the scale and complexity of AI investments soar, enterprises are demanding more flexible engagement models from service providers, including consumption-based and outcome-linked pricing that ties provider compensation to measurable business results. Midsize providers are also carving out a niche by offering accelerators, pre-configured data models, and migration frameworks tailored to specific business contexts, enabling hybrid operating models and domain-specific AI advancement. With over 90% of global data work still ahead, according to recent research, these new approaches and partnerships are essential to building the robust data foundations required for enterprise-wide AI adoption.

Scaling AI across large enterprises is hampered by a formidable mix of compliance and security concerns, high costs, and the daunting challenge of integrating new technologies with legacy systems—a reality underscored by 60% of CPG leaders citing compliance and security as top hurdles, and 58% struggling with integration complexity. Overcoming these barriers requires more than just technical upgrades; it demands a coordinated organizational push, with comprehensive employee training (72%), robust data infrastructure (66%), and rigorous cybersecurity and compliance practices (64%) emerging as critical enablers for effective, trustworthy AI deployment. As companies like Veeva QualityOne emphasize, only by balancing people, process, and data can enterprises unlock AI’s full potential while meeting rising regulatory and operational demands.

The financial calculus of scaling AI is fraught with risk and uncertainty, as evidenced by OpenAI’s $2 billion net loss on GPT-5 and the rapid obsolescence of even the most advanced models—GPT-5 was eclipsed within a quarter by rivals like Gemini and Claude Opus 4.5. Despite massive capital outlays—Meta and Microsoft alone are driving CapEx past half a trillion dollars—returns remain modest, often in the single digits, with Dave Vellante predicting that meaningful, enterprise-wide AI ROI will only begin to materialize at scale by 2026. This volatile landscape, where no clear leader sustains an advantage for long, forces enterprises to weigh the promise of future productivity gains against the reality of escalating infrastructure costs and fleeting competitive edges.

Governance and accountability have emerged as the Achilles’ heel of enterprise AI, with 25% of organizations identifying governance as the missing link in scaling autonomy and a growing crisis as AI agents gain decision-making authority without clear accountability frameworks. The temptation to blame AI for errors—the so-called 'hallucination defense'—is increasingly untenable, as regulators demand transparent chains of responsibility and courts reject 'the bot did it' as a legal excuse. Best practices now mandate human-in-the-loop verification, permissions audits, and comprehensive logging to mitigate legal and reputational risks, while Responsible AI frameworks and governance-by-design approaches are becoming prerequisites for compliant, explainable, and sustainable AI at scale.

Enterprises are evolving their approach to measuring AI ROI, moving beyond raw data collection to focus on governing outcomes and embedding explainability, compliance, and sustainability into their AI programs. Frameworks such as governed decision fabrics and enterprise data meshes are gaining traction, enabling organizations to align AI initiatives with broader business and societal goals—connecting, for example, sustainability and health via impact ledgers and insight hubs. This shift reflects a growing recognition that trustworthy, effective AI deployment hinges not just on technical prowess, but on the ability to deliver transparent, auditable value across the enterprise.