Intro
Artificial intelligence is no longer a side topic in security. It sits at the centre of how attacks are planned and how they are stopped. In a recent survey, 80% of CISOs named AI-powered cyber attacks as their top concern, up from fifth place the year before.
Attackers are already using AI to write more convincing phishing emails, to scale social engineering, and to test defences faster than human teams can react. At the same time, security leaders are under pressure to use AI on the defensive side, from faster analysis of alerts to quicker incident response.
This is the new reality. AI is now on both sides of the field. The real question for leaders is not whether to use it, but how to combine AI with people and services in a way that stays in control.
How attackers are already using AI
Attackers are not waiting. They are already using AI in ways that change the scale and speed of crime.
Generative models help write phishing emails that sound natural, with correct spelling and tone. Microsoft has reported that threat actors are using large language models to improve phishing content and craft more believable lures.
Deepfakes are moving from theory to practice. In March 2025, a finance director at a multinational in Singapore authorised a transfer of about US$499,000 after a Zoom call where every “executive” on screen was a deepfake copy of their real colleagues. tookitaki.com The criminals used AI to mirror faces and voices well enough to pass as genuine.
AI is also part of more technical work. Microsoft and OpenAI have reported state-linked groups using large language models to help with social engineering, vulnerability research and scripting.
The pattern is simple. AI does not invent new types of fraud. It makes existing attacks cheaper, faster, and harder to spot at human speed. And the money at stake is growing. One recent analysis found that deepfake-enabled fraud alone caused more than 200 million dollars in losses in the first quarter of 2025.
Why classic defensive thinking is struggling
AI has changed the tempo of attacks. Many security teams have not kept up.
In a global study by BCG, AI powered attacks jumped to the number one concern for CISOs in 2025. This was a 19 point rise in a single year. Another survey by SoSafe found that 87% of security professionals say their organisation has already faced at least one AI driven cyber attack in the past twelve months.
Yet most organisations still rely on controls and thinking built for a slower world. Accenture’s 2025 State of Cybersecurity Resilience report shows that only 13% of companies have advanced capabilities to defend against modern, AI driven threats. SandboxAQ reports a similar gap. 79% of organisations are using AI in production, but only 6% have a proper AI native security strategy.
At the same time, alert volumes and tool outputs keep rising. Recent research on security operations shows that 88% of teams saw alert volume increase, and many report burnout and long investigation times. Traditional rules, signatures, and manual workflows were not designed for this speed and scale. They are still useful. They are just no longer enough on their own.
Defensive AI as co-pilot, not autopilot
If attackers use AI to move faster, defenders cannot stay manual. Many teams are already turning to AI to keep up.
In most cases, this starts in simple ways. AI helps group similar alerts. It suggests which ones to check first. It pulls context from past incidents and threat reports, so analysts do not start from zero each time. Used well, this does not replace people. It gives them a clearer picture, faster.
Some organisations also use AI to support investigations. For example, by summarising long log files, mapping relationships between users, devices and events, or drafting first versions of incident reports that experts then correct. Large language models are good at reading and rephrasing large volumes of text. That saves time when minutes matter.
This is what “co-pilot” really means in security. AI does the heavy lifting on repetitive work. People still ask the hard questions. They decide what to trust, what to escalate, and when to act.
The organisations that benefit most from defensive AI are not the ones with the most tools. They are the ones who design services and workflows around it. They decide where AI can safely help, where human review is always needed, and how handovers between the two should work in practice.
The risks of over-trusting AI in security decisions
AI can be wrong. And when it is wrong in security, the impact is not a typo. It can be a breach.
Security vendors and governments are already warning about “hallucinations”. AI systems sometimes produce outputs that look confident but are false or misleading. In 2025, experts noted that AI used on bad or outdated data can invent non-existent vulnerabilities or misread threat intelligence, sending teams after fake issues while real risks stay hidden.
There is also a growing problem with how people use AI, not just how it works. IBM’s 2025 Cost of a Data Breach report shows that breaches involving “shadow AI” tools. AI used without approval or controls. cost organisations about 670,000 dollars more on average than other incidents. Over-reliance on ungoverned tools makes it harder to know where data went and what the AI actually did with it.
New attack patterns are emerging as well. Researchers recently described “slopsquatting” attacks, where criminals exploit hallucinations in AI coding assistants to insert malicious packages into software supply chains.
For leaders, the main risks of over-trusting AI in security are clear:
- False confidence. Taking AI output at face value and missing real threats.
- Data exposure. Feeding sensitive logs, code or incidents into public models without safeguards.
- Opaque decisions. Not being able to explain why an AI-supported decision was made when regulators or auditors ask.
So AI should not be treated as an autopilot for security. Its role is to suggest, not to decide. People still need to check the logic, question odd results, and set clear limits on where AI is allowed to act.
Building an AI-ready security culture
AI in security is not just about tools. It is about how people work. How do they think? And how services are set up around them.
In 2025, several studies show the same pattern. Most organisations are using AI somewhere, but very few feel ready for AI-related risk. One survey found that more than 70% of companies already use generative AI in daily work, while under 15% have formal policies or training for it. That gap is a culture problem, not a technology problem.
To close it, leaders need to focus on three areas.
1. Skills and mindset
Security teams do not all need to be AI researchers. But they do need AI literacy. That means they should understand:
- What the main AI models in use can and cannot do.
- Where AI is already embedded in their tools and services.
- How attackers might use the same technology against them.
This is not only for analysts. Incident responders, threat intelligence staff, risk and compliance teams, and even legal and HR, all have a role in how AI is used and governed.
2. Services and workflows
AI works best when it is part of clear services, not ad hoc experiments. For example:
- Using AI to help prepare and review incident reports, while experts sign off.
- Adding AI-based summarisation to post-incident reviews, so lessons are easier to share.
- Including AI support in playbooks for phishing response, fraud investigation, or executive protection.
Red teaming and cyber crisis exercises should also evolve. Many organisations now include AI driven phishing, deepfake calls, or automated fraud scenarios in their simulations to test how people respond under pressure. Keepnet Labs This keeps training aligned with how attacks are actually changing.
3. External partners
No team can cover every skill alone. This is even more true with AI. External services can help in areas such as:
- Incident response and digital forensics when an AI-enabled attack hits.
- Threat intelligence on how specific groups are using AI in the wild.
- Advisory work on AI governance, policy, and secure use of models.
Partners who have seen real incidents across many clients can bring patterns that a single organisation might miss. For a firm like ThreatScene, this means combining hands-on incident work, training, and advisory support so clients do not have to learn every AI lesson the hard way.
Governance. Setting guardrails for AI in security
AI in cybersecurity now sits in the same category as any high-impact system. It needs clear rules. Not just good intentions.
Regulators are moving in that direction. The EU AI Act, approved in 2024 and entering phases of application from 2025, treats many security and fraud use cases as “high risk”. It asks for risk assessments, human oversight, and clear documentation of how AI systems work and are monitored. Boards and auditors will expect to see this thinking in place, not just on paper.
For most organisations, practical guardrails for AI in security should cover at least:
- Where AI is allowed. Which use cases are approved today, and which are not.
- What data it can see. Clear limits on feeding sensitive logs, code, or customer data into external models.
- Who stays accountable. Named owners for each AI-supported process, with a human still signing off key decisions.
- How it is checked. Regular reviews of outputs, error cases, and any incidents linked to AI use.
Good governance does not slow teams down. It gives them confidence that they can use AI without crossing unseen lines.
Questions leaders should ask their teams today
AI in security is now a leadership topic. Not just a technical one. Simple questions can open the right conversations. For example:
- Where are we already using AI in security work. Is this mapped and approved.
- Have we seen AI used against us. Phishing, fraud, deepfakes, automated scanning. What did we learn.
- Which security tasks could AI safely support today. And which must stay fully human for now.
- What policies do we have for staff using public AI tools. Are people clear on what they can and cannot share.
- How are we training our teams for AI-enabled incidents. Do exercises include AI-assisted attackers.
These questions do not require technical depth. They require curiosity and a clear view of risk.
Wrapping up
AI is now part of how attacks start, spread, and hide. It is also part of how good teams detect and contain them. The gap between the two will come down to habits, not headlines. Map where AI already touches your security work. Decide where it should help, and where it must not decide alone. Set simple rules for using public and internal models. Train people to question output, not just read it.
Leaders shape how real this becomes. Ask for one clear page that shows where AI is used in defence today. Review a few AI assisted investigations or reports each month. Check that incident exercises include deepfakes, AI written phishing, and faster fraud. When gaps appear, close them with changes to process, training, or external support, not just new tools.
Book a consultation with our team to review your readiness for AI enabled threats, refine your incident playbooks, and align your people and services for the next phase of this AI vs AI arms race.
References
- BCG (2025). AI Creates Cyber Risks, but It Can Also Resolve Them.
https://www.bcg.com/publications/2025/ai-creates-cyber-risks-can-resolve-them - Tookitaki (2025). Deepfake CEO Scam in Singapore: How US$499,000 Was Stolen.
https://www.tookitaki.com/blog/deepfake-ceo-scam-singapore-2025 - TechTarget (2025). Microsoft and OpenAI Warn That Nation-State Hackers Are Abusing LLMs.
https://www.techtarget.com/searchsecurity/news/366569937/Microsoft-OpenAI-warn-nation-state-hackers-are-abusing-LLMs - SoSafe (2025). Global Businesses Face Escalating AI Risk as 87 Percent Are Hit by AI-Driven Cyberattacks.
https://sosafe-awareness.com/company/press/global-businesses-face-escalating-ai-risk-as-87-hit-by-ai-cyberattacks/ - Accenture (2025). State of Cybersecurity Resilience 2025.
https://www.accenture.com/content/dam/accenture/final/accenture-com/document-3/State-of-Cybersecurity-report.pdf - SandboxAQ (2025). AI Adoption Is Outpacing AI Security Readiness, New Report Finds.
https://www.sandboxaq.com/press/sandboxaq-report-ai-adoption-outpacing-ai-security - Cybersecurity Insiders (2025). Pulse of the AI SOC Report 2025: From Alert Fatigue to Actionable Intelligence.
https://www.cybersecurity-insiders.com/pulse-of-the-ai-soc-report-2025-from-alert-fatigue-to-actionable-intelligence-how-ai-is-reshaping-detection-response-and-analyst-confidence/ - Nudge Security (2025). Shadow AI: The Emerging Security Threat in IBM’s 2025 Cost of a Data Breach Report.
https://www.nudgesecurity.com/post/shadow-ai-the-emerging-security-threat-in-ibms-2025-cost-of-a-data-breach-report - Keepnet Labs (2025). Deepfake Statistics and Trends: How AI Is Transforming Social Engineering.
https://keepnetlabs.com/blog/deepfake-statistics-and-trends
