When Engagement Optimisation Becomes Illegal AI Manipulation

If you optimise a recommender purely for engagement, you can drift into territory the EU now bans outright. Article 5 of the EU AI Act prohibits manipulative and deceptive AI and the exploitation of people's vulnerabilities, and those bans have been in force since 2 February 2025. The Digital Services Act adds duties on recommender transparency and a ban on dark patterns. The phrase "cognitive warfare" gets the headlines, but the board-level version of the story is simpler. The same engagement-maximising logic that powers a feed can, past a point, become a prohibited practice. That's not a content-moderation problem. It's a governance and liability problem, and it sits with the people who sign off the system.

What does "engagement optimisation crossing into manipulation" actually mean?

A recommender or personalisation system is usually trained to maximise one thing: time, clicks, shares, watch-through. Nobody sets out to manipulate anyone. The objective function does the work.

A pre-registered randomised experiment by researchers at UC Berkeley found that an engagement-based ranking algorithm amplified emotionally charged, out-group hostile political content, and that users did not actually prefer the content the algorithm served them. Engagement ranking selected more polarising material than users' own follow choices would predict (Milli et al., PNAS Nexus, 2025). So the system optimised for a metric, and the metric pulled it toward content people said made them feel worse.

That gap is the governance risk in one sentence. The system is doing exactly what you trained it to do, and what it's doing is no longer something you'd defend in front of a regulator.

It compounds because false and emotionally loaded content travels further on its own. The Vosoughi, Roy and Aral study in Science analysed roughly 126,000 stories shared on Twitter and found falsehoods were 70% more likely to be retweeted than true stories (Science, 2018). An engagement objective doesn't have to be malicious to learn that outrage performs. It just has to be left alone.

Where is the legal line under the EU AI Act?

Article 5 of the AI Act sets the line, and it's an outright ban, not a risk tier you can manage your way around.

Two prohibitions matter most for anyone running recommendation, personalisation or persuasion systems:

Provision	What it bans	The trigger
Article 5(1)(a)	AI using subliminal, purposefully manipulative or deceptive techniques that materially distort behaviour	Causes or is reasonably likely to cause significant harm
Article 5(1)(b)	AI that exploits vulnerabilities due to age, disability, or a specific social or economic situation	Causes or is reasonably likely to cause significant harm

Source: EU AI Act, Article 5. These prohibitions entered into force on 2 February 2025, and the penalty regime applies from 2 August 2025.

The penalties are the heaviest in the whole Act. Up to 35 million euro, or 7% of total worldwide annual turnover, whichever is higher (Article 99). That's a board-level number by design.

The threshold most teams misread is "significant harm". The harm doesn't have to be physical. The European Commission's February 2025 guidance is explicit that it includes psychological, financial and economic harm (Commission guidelines on prohibited practices, 4 Feb 2025). A system that nudges a person with a known gambling vulnerability toward more spend, or one that distorts a user's choices in a way they couldn't reasonably resist, is squarely in scope.

Doesn't the Digital Services Act already cover this?

It covers a different slice, and the two regimes overlap. Treat them as one compliance surface, not two filing cabinets.

The Digital Services Act (Regulation (EU) 2022/2065) does three things that bite on recommender design:

• Recommender transparency (Article 27). Platforms must set out, in plain language, the main parameters their recommender uses and any options users have to change them (DSA Article 27).
• Dark patterns (Article 25). Platforms must not "design, organise or operate their online interfaces in a way that deceives or manipulates" users or otherwise impairs their ability to make free and informed decisions (DSA Article 25). This is the first EU law to name dark patterns directly.
• A non-profiling option (Article 38). Very large platforms must offer at least one recommender option that isn't based on profiling.

The AI Act bans the manipulative technique. The DSA forces you to explain the recommender and gives users a way out. A persuasion system can clear one and fail the other. The honest read for a board: if your interface quietly steers and your recommender is a black box even to your own team, you're exposed on both fronts at once.

Why is this a board-level risk and not an engineering ticket?

Because the people who control the objective function are usually not the people who'll answer for it, and the gap between them is where this goes wrong.

The harm here isn't a bug an engineer ships and an engineer fixes. It emerges from a business decision: what the system is told to maximise. That decision is commercial. It gets made in product strategy and growth meetings, often without anyone framing it as a legal exposure. By the time it surfaces, it's a 7%-of-turnover question and a press question.

The regulatory direction is only tightening. Addictive-design rules are moving through legislatures, and the DSA's recommender duties were built precisely because self-policing didn't hold. A system that's compliant on a narrow read today can age into a prohibited one as guidance and case law fill in around it.

My view, plainly: any business running recommendation, personalisation or persuasion AI in a regulated or consumer context should treat engagement-only optimisation as a red flag in its own right. Not because engagement is bad. Because an objective that ignores user welfare is one regulator's interpretation away from "manipulative". You want to have asked the question before they do.

What should a responsible board actually do?

Govern the objective, not just the output. Most controls watch what the model produces. The risk lives one layer up, in what it was told to want.

A workable starting set:

• Audit the objective function. Write down what every recommender and personalisation system is optimising for. If the answer is "engagement" with nothing balancing it, that's your first finding.
• Run a manipulation and vulnerability review. Test specifically for Article 5 exposure: does the system exploit age, disability, or economic situation? Can it distort choices a user couldn't reasonably resist? Document it.
• Map AI Act and DSA duties together. One register, both regimes, owned by someone who reports to the board.
• Build the off-ramp. Recommender transparency and a non-profiling path aren't only DSA compliance. They're evidence you took user autonomy seriously.
• Put a human accountable. Significant harm is a judgement call. Someone senior should own that call, on the record, before launch.

This is the work VerityAI does as a Responsible AI advisory. We assess where an AI system sits against Article 5 and the DSA, document the exposure, and prescribe the fix. We don't run your platform. We make sure the people who do can defend it.

Frequently asked questions

Does the EU AI Act ban recommender systems?

No. It bans specific techniques: manipulation, deception, and exploitation of vulnerabilities that cause or are likely to cause significant harm (Article 5). A recommender is lawful. A recommender engineered to distort behaviour in a way users can't reasonably resist is not.

Is "engagement optimisation" illegal on its own?

Not by itself. The risk is that an engagement-only objective can drift into prohibited manipulation without anyone deciding it should. Optimising for engagement isn't the offence. Producing a system that materially distorts behaviour and causes significant harm is. The first can quietly become the second.

What counts as "significant harm" under Article 5?

Per the European Commission's February 2025 guidance, harm isn't limited to physical injury. It includes psychological, financial and economic harm. A system that pushes a vulnerable user toward financial loss can meet the threshold.

How do the AI Act and the Digital Services Act fit together here?

The AI Act prohibits the manipulative technique. The DSA requires recommender transparency, bans dark-pattern interfaces, and forces a non-profiling option on large platforms. A system can satisfy one and breach the other, so they need governing as a single compliance surface.

The bottom line

The "cognitive warfare" framing makes this feel like someone else's geopolitics problem. It isn't. The version that lands on your desk is far more mundane and far more expensive: a recommender you built, doing what you trained it to do, drifting into a practice the EU now bans, carrying a fine worth up to 7% of global turnover.

The fix isn't to abandon personalisation. It's to govern the objective behind it. Decide what your AI is allowed to want, document it against Article 5 and the DSA, and put a named person on the line for the judgement calls. Engagement is a fine thing to measure. It's a dangerous thing to worship.

If you're deploying recommendation, personalisation or persuasion AI, the question to ask this quarter is the one a regulator will ask later: what is this system optimising for, and who decided that was acceptable?

Related reading: the engagement trap and the metrics failing society, AI safety under the EU AI Act, and cognitive warfare threats to democratic institutions.

References

• EU AI Act, Article 5: Prohibited AI Practices
• EU AI Act, Article 99: Penalties
• European Commission guidelines on prohibited AI practices, 4 February 2025
• Digital Services Act, Article 25: Online interface design (dark patterns)
• Digital Services Act, Article 27: Recommender system transparency
• Milli et al., Engagement, user satisfaction, and the amplification of divisive content on social media, PNAS Nexus, 2025
• Vosoughi, Roy and Aral, The spread of true and false news online, Science, 2018

For the board-level view, see VerityAI's manipulation risk in regulated gaming AI.

If you want support with this, VerityAI offers AI governance and compliance.