#99: AI Is Only as Ethical as the Companies Using It
The Phia scandal surfaces an uncomfortable reality around incentives
Author’s Note: On November 1, I’ll be running the 2026 New York City Marathon in memory of my uncle Chris and in support of the Parkinson’s Foundation. As this newsletter lands in your inbox, I’m likely out on a hot, humid training run. If you’d like to learn more about Parkinson’s disease or support my fundraising efforts, you can visit my fundraising page here.
Have you heard about the Phia scandal? The AI shopping start-up accused of manipulating affiliate attribution? You know, the one founded by Bill Gates’ daughter?
To get you up to speed, here’s an oversimplified explanation of the scandal.
Phia is an AI shopping browser extension that helps users compare prices for clothing and accessories across the web. The concept itself isn’t new. It’s built on the affiliate marketing model, where partners earn commissions for referring shoppers to brands. Having AI search the internet for the best deal sounds like a win.
But Phia allegedly took the concept too far. According to research highlighted by Bloomberg, the browser extension silently overwrote affiliate cookies on iOS, taking credit for purchases even when it hadn’t influenced the sale or replacing referrals from other affiliates like Wirecutter. If true, the practice would violate affiliate ecosystem rules and could potentially raise legal concerns as precedents have been set in past cases surrounding eBay and Honey. For a deeper recap, I recommend Michael McNerney’s Adweek article.
To be clear, Phia has denied wrongdoing and has not faced known legal action related to these allegations. The claims remain allegations based on independent research (albeit, very compelling research).
But this week’s article isn’t really about Phia. It’s about the uncomfortable reality exposed by the scandal: AI does not remove incentives. It amplifies them.
Think about how quickly AI is advancing. It seems as if there’s a new frontier model update every month. As exciting as these times are, the pace has caused plenty of concern around trust and safety.
Although the frontier labs have come under plenty of well-deserved scrutiny, they have invested materially in model safety research. For instance, Anthropic has built its brand around how integral trust is to model creation and AI adoption. Entire research teams evaluate areas like model misuse, deception, cybersecurity risks, and dangerous capabilities across companies like Anthropic, OpenAI, Google, and xAI. But thousands of start-ups are building upon these models. And those companies have different incentives. Thus, the vertical layer is where I believe one of AI’s biggest trust problems lies.
It’s a fair assumption that engineers at an AI start-up in 2026 are using agentic coding tools like Claude Code or Codex. So, here’s the uncomfortable truth about Phia: an engineer could theoretically instruct an AI coding assistant to write code that executes cookie stuffing (i.e., deceptively adding in Phia’s tag for affiliate attribution).
The Claude model didn’t decide to overwrite affiliate cookies, the company’s implementation choices did. Even if Anthropic aims to build ethical AI, it’s impossible for them to think of every way its models can be leveraged for bad behavior. I don’t see a scenario where frontier labs are expected to think of every way businesses can exploit each other and then try to install the necessary guardrails to prevent these actions.
If AI faithfully executes the goals we give it, what happens when those goals are unethical? Phia recently raised a large round of funding, and like many venture-backed startups, it faces pressure to grow quickly. If a company’s incentives prioritize growth over ethics, AI becomes a force multiplier for those incentives.
I think back to a phrase a former manager used to say:
“As a business grows, both good and bad things scale.”
AI amplifies incentives, as agents are programmed to execute specific tasks. One employee looking to cut corners can have an exponentially negative effect on the business because they have such a powerful tool in their hands. The AI model didn’t create unethical incentives; it simply followed instructions from its user and amplified the incentives at scale.
What’s concerning is how much easier AI makes it to even “test out” a grey-area business idea. Completely speculating here, but perhaps Phia’s cookie stuffing was an A/B test that showed promise and the company decided to keep it. AI allows for fast, nimble experimentation. Unfortunately, it reduces the cost of experimentation for both ethical and unethical tests.
There’s plenty of angst in society around whether AI is trustworthy. But I believe it’s more important to focus on whether the companies building on top of the AI models are worth our trust as they have their own set of incentives.
We’ve talked a lot about delegating shopping decisions to AI agents in past agentic commerce articles. However, the agentic commerce ambition will slow if consumers begin to suspect that AI shopping agents optimize for affiliate commissions instead of their interests. Most markets function because participants trust that incentives are aligned enough for transactions to happen. Everyone must trust each other; otherwise, nothing will get done.
Bad actors will slow AI adoption, and technological progress more broadly. Every high-profile incident like Phia makes consumers a little less willing to delegate decisions to AI.
I’m less concerned about whether the frontier labs are building ethical AI models. See, they have stronger incentives to invest in safety because they’re highly visible and reputationally exposed. The bigger problem is not the foundational layer. It’s in the vertical layer.