AI Incrementality: Is Your 2026 Strategy Flawed?

The Challenge of Agent-Driven Purchases: A Campaign Teardown

Measuring the true impact of marketing efforts has always been challenging, but the advent of agent purchases, where AI autonomously makes buying decisions on behalf of a user, introduces a new layer of complexity. These aren’t just sophisticated chatbots; we’re talking about AI systems that can research, compare, negotiate, and complete transactions without direct human intervention. This shift makes traditional attribution models, which often rely on user journeys and touchpoints, increasingly obsolete. I’ve seen firsthand how companies misattribute success to their AI initiatives simply because they’re looking at the wrong metrics.

Last year, we partnered with “Quantum Quills,” a B2B SaaS platform offering AI-powered content generation for mid-market businesses. Their primary goal was to increase enterprise-level subscriptions, and they had recently integrated an advanced AI purchasing agent into their system. This agent was designed to identify potential clients, evaluate their needs against Quantum Quills’ offerings, and even initiate trial sign-ups or direct purchases based on predefined parameters. The internal team was convinced their agent was a goldmine, but their existing analytics were just showing a surge in direct traffic conversions, without differentiating between human and agent-led activity. It was a classic case of correlation mistaken for causation.

Campaign Strategy: Isolating Agent Impact

Our objective was clear: determine the true AI incrementality of their purchasing agent. We needed to understand if the agent was genuinely driving new conversions or simply cannibalizing existing human-driven sales. Our hypothesis was that while the agent was generating volume, its incremental value might be lower than perceived, especially for high-value enterprise subscriptions that often require human-to-human sales interactions.

We designed a rigorous A/B test over a 12-week period, focusing on their target enterprise segment in the Atlanta metropolitan area. Specifically, we targeted businesses with 500+ employees located within a 50-mile radius of downtown Atlanta, particularly those in the Midtown and Buckhead business districts. We used a segment of their existing database, cross-referenced with publicly available firmographic data for companies listed on the Georgia Department of Economic Development’s business registry, to ensure our control and test groups were as similar as possible.

Experiment Design and Targeting

• Budget: $150,000 (allocated specifically for this incrementality test, separate from ongoing marketing efforts)
• Duration: 12 weeks
• Target Audience: Enterprise businesses (500+ employees) in the Atlanta metro area (specifically zip codes 30309, 30305, 30308, 30303, 30326).
• Control Group (50%): Exposed to standard marketing campaigns (display ads, search ads, email nurturing) but with the AI purchasing agent intentionally disabled for their accounts.
• Test Group (50%): Exposed to standard marketing campaigns AND the AI purchasing agent was fully active, empowered to initiate contact and purchase processes.

The key here was the technical implementation. Quantum Quills’ engineering team developed a custom flag within their CRM that allowed us to activate or deactivate the purchasing agent’s capabilities for specific accounts. This ensured a clean separation between the two groups. We also made sure to use unique tracking parameters for all agent-initiated interactions to distinguish them from human-initiated ones.

Creative Approach and Messaging

For both groups, the standard marketing creatives focused on the benefits of AI-powered content generation: efficiency, scalability, and quality. Examples included “Scale Your Content Production by 50% with AI” and “Automate Your Blog, Boost Your SEO.” The agent, when active, used a more direct, data-driven approach in its initial outreach. It would identify a potential client’s content gaps based on publicly available data and propose specific Quantum Quills solutions, often including a personalized trial offer. Its messaging was less about brand building and more about problem-solving and transaction facilitation.

Realistic Metrics and Outcomes

Here’s how the numbers broke down after the 12-week period:

What Worked

The data clearly indicated a significant incremental lift from the AI purchasing agent. The test group saw a 50% increase in enterprise trial conversions compared to the control group (90 additional conversions). This translated to a substantial 1.6x improvement in ROAS. The agent’s ability to directly engage, qualify, and initiate transactions for high-value prospects was undeniably effective. I was particularly impressed by the reduction in CPL and CPC; the agent seemed to streamline the sales funnel, cutting down on the human effort traditionally required for initial qualification.

One of the key successes was the agent’s ability to identify “cold” but highly relevant leads that human sales teams might have overlooked or deprioritized. It acted as an always-on, data-driven scout, surfacing opportunities that would otherwise remain hidden. We also observed that the agent-initiated trials had a slightly higher conversion rate to paid subscriptions post-trial, suggesting better initial qualification.

What Didn’t Work (or Needed Improvement)

While the overall numbers were positive, we identified some areas for improvement. A critical finding was that for truly complex enterprise deals (those requiring significant customization or integration), the agent’s transactional approach sometimes fell flat. Several prospects in the test group, while initially engaged by the agent, expressed frustration at the lack of human interaction when their needs went beyond the agent’s predefined parameters. It’s a common trap: assuming AI can handle everything. It can’t, not yet anyway.

Another issue was related to brand perception. A small percentage of test group prospects reported feeling “impersonal” or “like a number” when their initial interactions were solely with the AI agent. This highlights a delicate balance between efficiency and maintaining a human touch, especially in high-touch B2B sales. We also noticed that while the agent was great at initiating trials, the conversion rate from trial to full subscription wasn’t dramatically higher than the control group’s human-initiated trials, suggesting the agent excelled at the top of the funnel but didn’t necessarily improve the mid-to-bottom funnel conversion process without human sales support.

Optimization Steps Taken

Based on these findings, we implemented several key optimizations:

1. Hybrid Sales Model: We didn’t scrap the agent. Instead, we reconfigured it to act as a highly efficient lead qualifier and initial engagement tool. Once a prospect reached a certain engagement threshold or indicated complex needs (e.g., asked about specific API integrations), the agent would seamlessly hand off the lead to a human sales representative with a detailed summary of all prior interactions. This preserved the human element where it mattered most.
2. Enhanced Agent Scripting: We refined the agent’s conversational flows to include more empathetic language and acknowledge its AI nature upfront, managing expectations. For example, “I’m Quantum Quills’ AI assistant, designed to help you quickly find the best content solutions. For highly customized requests, I can connect you with a human expert.”
3. Post-Trial Nurturing: We beefed up the human-driven post-trial nurturing sequence for agent-initiated trials, ensuring that a dedicated account manager followed up promptly to address any lingering questions or concerns that the AI might have missed.
4. Continuous A/B Testing on Agent Parameters: We established an ongoing testing framework for the agent’s parameters, constantly refining its targeting criteria, messaging, and hand-off triggers to maximize its incremental value while minimizing potential friction. This included testing different thresholds for what constitutes a “complex” query requiring human intervention.

This campaign taught me a valuable lesson: AI is a multiplier, not a replacement. Its true power isn’t in automating everything, but in intelligently augmenting human capabilities. Without rigorous incrementality testing, Quantum Quills would have continued to pour resources into an “effective” channel without truly understanding its net new contribution. We needed to look beyond superficial metrics and really dig into the causal relationships, especially with something as transformative as agent purchases. I had a client last year, a fintech startup, who believed their AI chatbot was single-handedly responsible for a 30% increase in sign-ups. We ran a similar incrementality test, and it turned out nearly two-thirds of those sign-ups would have happened anyway through their organic channels. The chatbot was helpful, yes, but its incremental lift was only about 10%.

The Future of Attribution with AI

As AI agents become more sophisticated and prevalent, the need for robust AI incrementality testing will only grow. Traditional attribution models, like first-touch or last-touch, are inherently flawed when an algorithm is making decisions. Even multi-touch models struggle to assign fair credit when the “user” itself is an AI. This is why we increasingly rely on methodologies like incrementality testing, using control groups and statistical significance, to truly understand the causal impact of these new technologies.

Attribution is shifting from simply tracking touchpoints to understanding the causal relationship between a marketing intervention (whether human or AI-driven) and a desired outcome. This requires a strong foundation in experimental design and a willingness to challenge assumptions. We need to embrace advanced techniques, like Shapley values, to fairly distribute credit across complex, multi-agent, multi-human journeys. According to a 2023 IAB report on the state of data, nearly 60% of marketers still rely on last-click attribution, which is simply inadequate for today’s fragmented, AI-influenced customer journeys. We need to do better.

My editorial aside here: many companies are rushing to implement AI agents because everyone else is doing it. They’re investing millions without a clear strategy for measuring actual ROI. This isn’t just a waste of money; it’s a dangerous path that can lead to skewed insights and poor strategic decisions. Always, always, always test for incrementality, especially with AI.

The landscape of agent purchases is still evolving, but one thing is certain: marketers who master the art of measuring AI incrementality will be the ones who truly understand their return on investment and build sustainable growth strategies in the coming years.

To truly understand the impact of AI in your marketing, you must move beyond vanity metrics and commit to rigorous incrementality testing. It’s the only way to isolate the causal effect and ensure your AI investments are genuinely driving new value, not just shuffling existing conversions around.