Social Engineering in the Age of AI: New Threats, New Defences

21 Jul 2025Sotiris SpyrouUpdated on 3 Jul 2026

Share this article

Social Engineering in the Age of AI: New Threats, New Defences

The Evolution of Human Manipulation

Social engineering in the AI age is the practice of manipulating the humans who train, deploy, monitor, and govern AI systems, rather than attacking the AI system directly, in order to introduce vulnerabilities or bypass oversight. Social engineering has long been recognised as one of the most effective attack vectors in cybersecurity. By exploiting human psychology rather than technical vulnerabilities, attackers can bypass even the most sophisticated security systems. But the rise of artificial intelligence has fundamentally changed the landscape of social engineering in ways that most organizations haven't fully comprehended.

Recent intelligence from a Bank of England cybersecurity expert highlighted social engineering as a key concern in AI security discussions, indicating that even sophisticated financial institutions are grappling with new forms of human manipulation in AI-enabled environments.

The reality is that AI systems don't eliminate the human element in security - they transform it. AI systems are designed, trained, deployed, and monitored by humans. They interact with humans, learn from human-generated data, and make decisions that affect humans. Each of these interactions creates opportunities for social engineering attacks that are more sophisticated and potentially more damaging than traditional approaches.

Understanding the evolution from traditional social engineering to AI-era social engineering is crucial for developing effective defences:

Traditional Social Engineering

Traditional social engineering attacks target humans to gain system access:

Phishing: Deceiving individuals into revealing credentials or sensitive information.
Pretexting: Creating false scenarios to manipulate individuals into providing information or access.
Baiting: Offering something enticing to prompt individuals to compromise security.
Tailgating: Following authorized personnel into secure areas without proper authentication.
Quid Pro Quo: Offering services in exchange for information or access.

AI-Era Social Engineering

AI-era social engineering attacks exploit the human elements in AI systems:

Training Data Manipulation: Influencing humans responsible for AI training data to include malicious or biased information.
Deployment Deception: Manipulating humans involved in AI system deployment to create vulnerabilities.
Monitoring Subversion: Influencing humans responsible for AI monitoring to overlook suspicious activities.
Policy Manipulation: Influencing humans responsible for AI governance to create permissive policies.
Trust Engineering: Manipulating human trust in AI systems to enable malicious activities.

The Human-AI Interface Vulnerabilities

AI systems create new interfaces between humans and technology that attackers can exploit:

AI Training Personnel

Individuals involved in AI training are particularly vulnerable targets:

Data Scientists: Researchers and analysts who prepare training data may be influenced to include malicious examples.
Labelling Teams: Personnel who label training data may be manipulated to create biased or incorrect labels.
Quality Assurance: Teams responsible for validating training data may be influenced to approve compromised datasets.
External Contractors: Third-party contractors involved in AI training may be targeted for manipulation.

AI Deployment Teams

Personnel responsible for AI deployment face unique social engineering risks:

DevOps Engineers: Engineers deploying AI systems may be manipulated to implement insecure configurations.
System Administrators: Administrators managing AI infrastructure may be influenced to create security vulnerabilities.
Integration Specialists: Personnel integrating AI systems may be manipulated to create insecure connections.
Testing Teams: Teams responsible for AI testing may be influenced to overlook security issues.

AI Monitoring Personnel

Individuals responsible for monitoring AI systems present attractive targets:

Security Analysts: Analysts monitoring AI behavior may be influenced to ignore suspicious activities.
Compliance Officers: Personnel responsible for AI compliance may be manipulated to approve non-compliant systems.
Incident Response Teams: Teams responding to AI incidents may be influenced to mishandle security events.
Audit Teams: Personnel conducting AI audits may be manipulated to provide favorable assessments.

Attackers have developed sophisticated techniques specifically designed to exploit AI systems:

Synthetic Data Seeding

Attackers manipulate humans to introduce synthetic data into AI training processes:

False Data Sources: Creating fake data sources that appear legitimate to data collection teams.
Credential Impersonation: Impersonating credible sources to provide malicious training data.
Gradual Influence: Slowly introducing biased data over time to avoid detection.
Authoritative Deception: Using apparently authoritative sources to validate malicious data.

Behavioural Conditioning

Attackers gradually condition humans to accept abnormal AI behaviour:

Normalisation: Gradually exposing monitoring personnel to abnormal AI behaviour until it seems normal.
Desensitisation: Reducing human sensitivity to AI security alerts through repeated exposure.
Trust Building: Building trust in AI systems to reduce human oversight and intervention.
Expectation Management: Managing human expectations about AI behaviour to hide malicious activities.

Policy Manipulation

Attackers influence humans responsible for AI governance:

Regulatory Capture: Influencing personnel involved in AI regulation to create permissive policies.
Standards Manipulation: Manipulating individuals involved in AI standards development.
Governance Subversion: Influencing AI governance committees to approve risky policies.
Compliance Circumvention: Manipulating compliance personnel to overlook violations.

The Psychological Dimensions

AI-era social engineering exploits specific psychological vulnerabilities related to AI systems:

AI Mystification

Many individuals have limited understanding of AI systems, creating vulnerability:

Technical Intimidation: Attackers exploit individuals' fear of appearing technically incompetent.
Complexity Overwhelm: Using AI complexity to discourage thorough investigation.
Authority Deference: Exploiting deference to apparent AI expertise.
Black Box Acceptance: Encouraging acceptance of unexplained AI behaviour.

Trust Bias

Humans tend to develop trust in AI systems that can be exploited:

Automation Bias: Tendency to over-rely on AI systems and under-question their outputs.
Anthropomorphism: Tendency to attribute human-like qualities to AI systems.
Confidence Inflation: AI systems may appear more confident than they actually are.
Halo Effect: Positive impressions of AI systems may generalize to specific decisions.

Cognitive Overload

AI systems can create cognitive overload that impairs human judgment:

Information Overload: Too much information from AI systems can impair decision-making.
Decision Fatigue: Constant AI-human interaction can lead to decision fatigue.
Attention Splitting: Managing multiple AI systems can split human attention.
Complexity Burden: Complex AI systems can overwhelm human cognitive capacity.

Organizations must develop comprehensive defensive strategies that address the human elements in AI security:

Human-Centered Security Design

User Experience Security: Designing AI systems with security-focused user experiences.
Cognitive Load Management: Managing cognitive load on humans interacting with AI systems.
Trust Calibration: Helping humans develop appropriate trust levels in AI systems.
Transparent Communication: Providing clear communication about AI capabilities and limitations.

Enhanced Training and Awareness

AI Security Training: Comprehensive training on AI-specific social engineering threats.
Psychological Awareness: Training on psychological vulnerabilities related to AI systems.
Scenario-Based Training: Training using realistic AI social engineering scenarios.
Continuous Education: Ongoing education about evolving AI social engineering threats.

Organisational Safeguards

Segregation of Duties: Separating responsibilities for AI training, deployment, and monitoring.
Multi-Person Authorisation: Requiring multiple people to authorise critical AI decisions.
Independent Verification: Independent verification of AI training data and configurations.
Audit Trails: Comprehensive audit trails for all AI-related human activities.

Technical Countermeasures

Behavioural Monitoring: Monitoring human behavior for signs of manipulation.
Anomaly Detection: Detecting unusual patterns in human-AI interactions.
Automated Verification: Automated verification of human decisions related to AI systems.
Continuous Monitoring: Continuous monitoring of AI systems for signs of manipulation.

Paradoxically, AI systems can also be used to defend against social engineering attacks:

Behavioural Analysis

AI systems can analyze human behaviour to detect potential manipulation:

Pattern Recognition: Identifying unusual patterns in human behavior.
Deviation Detection: Detecting deviations from normal human behavioral patterns.
Anomaly Alerting: Alerting on unusual human activities related to AI systems.
Risk Scoring: Scoring human activities for social engineering risk.

Automated Verification

AI systems can automate verification of human decisions:

Decision Validation: Validating human decisions related to AI systems.
Data Verification: Verifying the integrity of AI training data.
Configuration Checking: Checking AI system configurations for security issues.
Compliance Monitoring: Monitoring compliance with AI security policies.

Predictive Analysis

AI systems can predict potential social engineering attacks:

Threat Modelling: Modelling potential social engineering threats.
Vulnerability Assessment: Assessing human vulnerabilities to social engineering.
Risk Prediction: Predicting the likelihood of social engineering attacks.
Mitigation Planning: Planning mitigation strategies for predicted threats.

Understanding real-world examples helps illustrate the sophistication of AI social engineering:

Training Data Manipulation

Scenario: Attackers target data labelling teams to introduce biased labels into training data.
Method: Social engineering attacks convince labelling personnel that certain biases are acceptable or necessary.
Impact: AI systems develop biased decision-making that favours attacker objectives.
Defence: Independent validation of training data and labels by multiple teams.

Deployment Deception

Scenario: Attackers manipulate deployment teams to implement insecure AI configurations.
Method: Social engineering attacks convince deployment personnel that security measures are unnecessary or harmful.
Impact: AI systems are deployed with vulnerabilities that attackers can exploit.
Defense: Mandatory security reviews and automated configuration validation.

Monitoring Subversion

Scenario: Attackers influence monitoring personnel to ignore suspicious AI behavior.
Method: Social engineering attacks convince monitoring personnel that abnormal behavior is normal or expected.
Impact: AI systems exhibit malicious behavior without detection.
Defense: Automated monitoring systems and independent verification of human decisions.

Integration with Comprehensive AI Security

Social engineering defenses must be integrated with comprehensive AI security strategies:

Connection to Pattern Mismatching

Pattern mismatching approaches can detect when humans are behaving differently due to social engineering influence.

Behavioral Monitoring Integration

Synthetic profile generation can model normal human behavior patterns and detect anomalies.

Red Team Testing

Red team operations should include social engineering attacks against AI systems.

Organizational Framework

The Bank of England's TRUSTED AI framework provides context for addressing social engineering in AI systems.

The Regulatory Perspective

Regulators are increasingly recognizing the importance of addressing social engineering in AI systems:

EU AI Act Implications

The EU AI Act includes provisions related to human oversight that address social engineering:

Human Oversight: Requirements for meaningful human oversight of AI systems.
Transparency: Transparency requirements that help humans understand AI behavior.
Training Requirements: Training requirements for personnel involved in AI systems.
Governance Requirements: Governance requirements that address human factors in AI security.

DORA Requirements

DORA includes specific requirements related to human factors in operational resilience:

Personnel Training: Training requirements for personnel involved in critical systems.
Incident Response: Human factors in incident response for AI systems.
Governance: Governance requirements that address human vulnerabilities.
Testing: Testing requirements that include human factors assessment.

Building Organizational Resilience

Organizations must build comprehensive resilience against AI social engineering:

Cultural Change

Security Culture: Building a culture that prioritizes security in AI systems.
Awareness Culture: Creating awareness of social engineering threats in AI environments.
Learning Culture: Fostering continuous learning about AI security threats.
Collaboration Culture: Promoting collaboration between security and AI teams.

Organisational Structure

Clear Responsibilities: Defining clear responsibilities for AI security.
Accountability Mechanisms: Implementing accountability mechanisms for AI-related decisions.
Communication Channels: Establishing clear communication channels for AI security issues.
Decision-Making Processes: Implementing robust decision-making processes for AI systems.

Continuous Improvement

Threat Intelligence: Continuous monitoring of AI social engineering threats.
Lesson Learning: Learning from AI social engineering incidents.
Process Improvement: Continuously improving AI security processes.
Technology Evolution: Evolving defensive technologies to address new threats.

The landscape of AI social engineering will continue to evolve:

Emerging Threats

Deepfake Attacks: Using deepfakes video generation to impersonate authority figures.
AI-Generated Content: Using AI to generate convincing social engineering content.
Behavioral Mimicry: Using AI to mimic trusted individuals' behavior patterns.
Psychological Profiling: Using AI to create detailed psychological profiles for targeted attacks.

Evolving Defenses

AI-Powered Detection: Using AI to detect social engineering attacks.
Behavioral Authentication: Using behavioral patterns to authenticate humans.
Continuous Monitoring: Continuous monitoring of human-AI interactions.
Adaptive Defenses: Defenses that adapt to evolving social engineering threats.

Strategic Recommendations

Organizations should implement comprehensive strategies to address AI social engineering:

Immediate Actions

Risk Assessment: Assess current vulnerability to AI social engineering attacks.
Training Programs: Implement comprehensive training on AI social engineering threats.
Policy Development: Develop policies that address human factors in AI security.
Monitoring Systems: Implement monitoring systems for human-AI interactions.

Medium-Term Strategy

Cultural Change: Implement cultural change initiatives focused on AI security.
Technology Investment: Invest in technologies that detect and prevent AI social engineering.
Process Development: Develop robust processes for AI security management.
Capability Building: Build organizational capabilities to address AI social engineering.

Long-Term Vision

Industry Leadership: Lead industry efforts to address AI social engineering.
Standards Development: Contribute to standards development for AI security.
Research Collaboration: Collaborate with researchers on AI social engineering defense.
Global Cooperation: Participate in global efforts to address AI social engineering threats.

The VerityAI Advantage

The complexity of AI social engineering highlights the need for independent validation and specialized expertise. VerityAI's approach addresses human factors in AI security:

Human-AI Interaction Testing: Testing how humans interact with AI systems under various conditions.
Behavioral Validation: Validating that AI systems behave appropriately in human contexts.
Independent Assessment: Providing independent assessment of AI security including human factors.
Continuous Monitoring: Continuous monitoring of AI systems for signs of human manipulation.

For organizations seeking to address AI social engineering, VerityAI provides the specialized expertise and tools needed to understand and mitigate human factors in AI security.

The Human Element Endures

Despite the sophistication of AI systems, the human element remains crucial in AI security. Social engineering attacks that exploit this human element represent some of the most serious threats to AI systems. Organizations that recognize this reality and implement comprehensive defenses will be better positioned to protect their AI systems from sophisticated attacks.

The question for security leaders is not whether to focus on technical or human factors in AI security, but how to integrate both into comprehensive defense strategies that address the full spectrum of AI social engineering threats.

Ready to address social engineering vulnerabilities in your AI systems? Contact VerityAI for comprehensive AI security assessment and strategic guidance that transforms human factors from vulnerabilities into security strengths.

If you want support with this, VerityAI offers board-level AI governance.

Frequently asked questions

What is social engineering in the context of AI systems?

Social engineering in AI security means manipulating the people who train, deploy, monitor, or govern an AI system, rather than attacking the model directly. Because AI systems are built and overseen by humans at every stage, each of those human touchpoints is a potential target.

Who inside an organisation is most exposed to this kind of attack?

Data scientists and labelling teams who shape training data, engineers who deploy AI systems, and analysts who monitor AI behaviour are all attractive targets. Each role sits at a point where a manipulated human decision can create a vulnerability that technical controls alone would not catch.

How is this different from traditional phishing or pretexting?

Traditional social engineering aims to get direct system access, a password, or a document. AI-era social engineering often aims at something subtler: getting a person to accept a biased dataset, wave through an insecure configuration, or treat abnormal AI behaviour as normal.

What is the most effective defence against AI social engineering?

Segregating duties so no single person can introduce and approve a change unchecked, combined with independent verification of training data and configurations, closes off much of the opportunity. Ongoing awareness training that covers AI-specific scenarios, not just generic phishing, matters as well.