Artificial intelligence (AI) is now squarely on the frontlines of information security. However, as is often the case when the pace of technological innovation is very rapid, security often ends up being a secondary consideration. This is increasingly evident from the ad-hoc nature of many implementations, where organizations lack a clear strategy for responsible AI use.

Attack surfaces aren’t just expanding due to risks and vulnerabilities in AI models themselves but also in the underlying infrastructure that supports them. Many foundation models, as well as the data sets used to train them, are open-source and readily available to developers and adversaries alike.

Unique risks to AI models

According to Ruben Boonen, CNE Capability Development Lead at IBM: “One problem is that you have these models hosted on giant open-source data stores. You don’t know who created them or how they were modified, and there are a number of issues that can occur here. For example, let’s say you use PyTorch to load a model hosted on one of these data stores, but it has been changed in a way that’s undesirable. It can be very hard to tell because the model might behave normally in 99% of cases.”

Recently, researchers discovered thousands of malicious files hosted on Hugging Face, one of the largest repositories for open-source generative AI models and training data sets. These included around a hundred malicious models capable of injecting malicious code onto users’ machines. In one case, hackers set up a fake profile masquerading as genetic testing startup 23AndMe to deceive users into downloading a compromised model capable of stealing AWS passwords. It was downloaded thousands of times before finally being reported and removed.

In another recent case, red team researchers discovered vulnerabilities in ChatGPT’s API, in which a single HTTP request elicited two responses indicating an unusual code path that could theoretically be exploited if not addressed. This, in turn, could lead to data leakage, denial of service attacks and even escalation of privileges. The team also discovered vulnerabilities in plugins for ChatGPT, potentially resulting in account takeover.

While open-source licensing and cloud computing are key drivers of innovation in the AI space, they’re also a source of risk. On top of these AI-specific risk areas, general infrastructure security concerns also apply, such as vulnerabilities in cloud configurations or poor monitoring and logging processes.

AI models are the new frontier of intellectual property theft

Imagine pouring huge amounts of financial and human resources into building a proprietary AI model, only to have it stolen or reverse-engineered. Unfortunately, model theft is a growing problem, not least because AI models often contain sensitive information and can potentially reveal an organization’s secrets should they end up in the wrong hands.

One of the most common mechanisms for model theft is model extraction, whereby attackers access and exploit models through API vulnerabilities. This can potentially grant them access to black-box models — like ChatGPT — at which point they can strategically query the model to collect enough data to reverse engineer it.

In most cases, AI systems run on cloud architecture rather than local machines. After all, the cloud provides the scalable data storage and processing power required to run AI models easily and accessibly. However, that accessibility also increases the attack surface, allowing adversaries to exploit vulnerabilities like misconfigurations in access permissions.

“When companies provide these models, there are usually client-facing applications delivering services to end users, such as an AI chatbot. If there’s an API that tells it which model to use, attackers could attempt to exploit it to access an unreleased model,” says Boonen.

Red teams keep AI models secure

Protecting against model theft and reverse engineering requires a multifaceted approach that combines conventional security measures like secure containerization practices and access controls, as well as offensive security measures.

The latter is where red teaming comes in. Red teams can proactively address several aspects of AI model theft, such as:

• API attacks: By systematically querying black-box models in the same way adversaries would, red teams can identify vulnerabilities like suboptimal rate limiting or insufficient response filtering.
• Side-channel attacks: Red teams can also carry out side-channel analyses, in which they monitor metrics like CPU and memory usage in an attempt to glean information about the model size, architecture or parameters.
• Container and orchestration attacks: By assessing containerized AI dependencies like frameworks, libraries, models and applications, red teams can identify orchestration vulnerabilities, such as misconfigured permissions and unauthorized container access.
• Supply chain attacks: Red teams can probe entire AI supply chains spanning multiple dependencies hosted in different environments to ensure that only trusted components like plugins and third-party integrations are being used.

A thorough red teaming strategy can simulate the full scope of real-world attacks against AI infrastructure to reveal gaps in security and incident response plans that could lead to model theft.

Mitigating the problem of excessive agency in AI systems

Most AI systems have a degree of autonomy with regard to how they interface with different systems and respond to prompts. After all, that’s what makes them useful. However, if systems have too much autonomy, functionality or permissions — a concept OWASP calls “excessive agency” — they can end up triggering harmful or unpredictable outputs and processes or leaving gaps in security.

Boonen warns that components, such as optical character recognition (OCR) for PDF files and images which multimodal systems rely on to process inputs, “can introduce vulnerabilities if they’re not properly secured”.

Granting an AI system excessive agency also expands the attack surface unnecessarily, thus giving adversaries more potential entry points. Typically, AI systems designed for enterprise use are integrated into much broader environments spanning multiple infrastructures, plugins, data sources and APIs. Excessive agency is what happens when these integrations result in an unacceptable trade-off between security and functionality.

Let’s consider an example where an AI-powered personal assistant has direct access to an individual’s Microsoft Teams meeting recordings stored in OneDrive for Business, the purpose being to summarize content in those meetings in a readily accessible written format. However, let’s imagine that the plugin doesn’t only have the ability to read meeting recordings but also everything else stored in the user’s OneDrive account, in which many confidential information assets are also stored. Perhaps the plugin even has write capabilities, in which case a security flaw could potentially grant attackers an easy pathway for uploading malicious content.

Once again, red teaming can help identify flaws in AI integrations, especially in environments where many different plugins and APIs are in use. Their simulated attacks and comprehensive analyses will be able to identify vulnerabilities and inconsistencies in access permissions, as well as cases where access rights are unnecessarily lax. Even if they don’t identify any security vulnerabilities, they will still be able to provide insight into how to reduce the attack surface.

The post How red teaming helps safeguard the infrastructure behind AI models appeared first on Security Intelligence.

Artificial intelligence (AI) and machine learning (ML) have entered the enterprise environment.

According to the IBM AI in Action 2024 Report, two broad groups are onboarding AI: Leaders and learners. Leaders are seeing quantifiable results, with two-thirds reporting 25% (or greater) boosts to revenue growth. Learners, meanwhile, say they’re following an AI roadmap (72%), but just 40% say their C-suite fully understands the value of AI investment.

One thing they have in common? Challenges with data security. Despite their success with AI and ML, security remains the top concern. Here’s why.

Full steam ahead: How AI and ML get smarter

Historically, computers did what they were told. Thinking outside the box wasn’t an option — lines of code dictated what was possible and permissible.

AI and ML models take a different approach. Instead of rigid structures, AI and ML models are given general guidelines. Companies supply vast amounts of training data that help these models “learn,” in turn improving their output.

A simple example is an AI tool designed to identify images of dogs. The underlying ML structures provide basic guidance — dogs have four legs, two ears, a tail and fur. Thousands of images of both dogs and not-dogs are provided to AI. The more pictures it “sees,” the better it becomes at differentiating dogs.

Off the rails: The risks of unauthorized model modification

If attackers can gain access to AI models, they can modify model outputs. Consider the example above. Malicious actors compromise business networks and flood training models with unlabeled images of cats and images incorrectly labeled as dogs. Over time, model accuracy suffers and outputs are no longer reliable.

Forbes highlights a recent competition that saw hackers trying to “jailbreak” popular AI models and trick them into producing inaccurate or harmful content. The rise of generative tools makes this kind of protection a priority — in 2023, researchers discovered that by simply adding strings of random symbols to the end of queries, they could convince generative AI (gen AI) tools to provide answers that bypassed model safety filters.

And this concern isn’t just conceptual. As noted by The Hacker News, an attack technique known as “Sleepy Pickle” poses significant risks for ML models. By inserting a malicious payload into pickle files — used to serialize Python object structures — attackers can change how models weigh and compare data and alter model outputs. This could allow them to generate misinformation that causes harm to users, steal user data or generate content that contains malicious links.

Staying the course: Three components for better security

To reduce the risk of compromised AI and ML, three components are critical:

1) Securing the data

Accurate, timely and reliable data underpins usable model outputs. The process of centralizing and correlating this data, however, creates a tempting target for attackers. If they can infiltrate large-scale AI data storage, they can manipulate model outputs.

As a result, enterprises need solutions that automatically and continuously monitor AI infrastructure for signs of compromise.

2) Securing the model

Changes to AI and ML models can lead to outputs that look legitimate but have been modified by attackers. At best, these outputs inconvenience customers and slow down business processes. At worst, they could negatively impact both reputation and revenue.

To reduce the risk of model manipulation, organizations need tools capable of identifying security vulnerabilities and detecting misconfigurations.

3) Securing the usage

Who’s using models? With what data? And for what purpose? Even if data and models are secured, use by malicious actors may put companies at risk. Continuous compliance monitoring is critical to ensure legitimate use.

Making the most of models

AI and ML tools can help enterprises discover data insights and drive increased revenue. If compromised, however, models can be used to deliver inaccurate outputs or deploy malicious code.

With Guardium AI security, businesses are better equipped to manage the security risks of sensitive models. See how.

The post The straight and narrow — How to keep ML and AI training on track appeared first on Security Intelligence.

Cybersecurity requires creativity and thinking outside the box. It’s why more organizations are looking at people with soft skills and coming from outside the tech industry to address the cyber skills gap. As the threat landscape becomes more complex and nation-state actors launch innovative cyberattacks against critical infrastructure, there is a need for cybersecurity professionals who can anticipate these attacks and develop creative preventive solutions.

Of course, a lot of cybersecurity work is mundane and repetitive — monitoring logs, sniffing out false positive alerts, etc. Artificial intelligence (AI) has been a boon in filling the talent gaps when it comes to these types of tasks. But AI has also proven useful for many of the same things that creative thought brings to the threat table, such as addressing more sophisticated threat actors, the rapid increase of data and the hybrid infrastructure.

However, many companies are seeing the value of AI, especially generative AI (gen AI), in handling a greater share of creative work — not just in cybersecurity but also in areas like marketing and public relations, writing and research. But are these organizations using AI in a way that could threaten the importance of human creativity in threat detection?

Why creativity is important to cybersecurity

The very simple reason why cybersecurity requires innovative people is that threat actors are already coming up with novel approaches to how to get into your system. Are they using gen AI to launch their attacks? You bet they are; phishing emails have never been more grammatically constructed or realistic. But before AI was available, threat actors were designing social engineering attacks that attracted clicks. Now, they have advanced beyond “how can we lure in victims” to “how can we get more out of a single attack after we lure in the victims.”

Creativity isn’t just coming up with new ideas. It is also the ability to see things through a big-picture lens and discern historical data or where to find information you might not know you need to look for. For example, creative thought is required for the following security tasks:

• Threat hunting or predicting a threat actor’s move or finding their tracks in a system
• Finding buried evidence in a forensic search
• Understanding historical data in anomaly detection
• Ability to tell a real email or document versus a well-designed phishing attack
• Verifying new zero day attacks and other malware variants found in otherwise unknown vulnerabilities

AI can augment human creativity, but gen AI gets a lot of things wrong. Users have found themselves in situations where AI claimed plagiarism on original work or AI hallucinations offered false information that nullified the research of human analysts. AI algorithms are also susceptible to bias that could lead to false positives.

AI’s role in creative cybersecurity and beyond

While many creative people, cybersecurity professionals and beyond, see gen AI as a mixed blessing, many embrace the technology because it is a huge timesaver.

“Gen AI can help prototype much faster because the large language models can take over the refactoring and documentation of code,” wrote Aili McConnon in an IBM blog post. Also, the article pointed out, AI tools can help users create prototypes or visualize their ideas in minutes versus hours or days.

Creativity married to AI can help identify future leaders. According to research from IBM, two-thirds of company leaders found that AI is driving their growth, with four specific use cases — IT operations, user experience, virtual assistants and cybersecurity — most commonly favored by leaders.

“A Learner will typically copy predefined scenarios using out-of-the-box technologies,” Dr. Stephan Bloehdorn, Executive Partner and Practice Leader, AI, Analytics and Automation-IBM Consulting DACH, was quoted in the study. “But a Leader develops custom innovations.”

Over-reliance on AI?

As gen AI becomes more ubiquitous in the workplace and as more creative folks and leaders rely on it as a way to put their ideas in motion, are we also relying on the technology to the point that it could lead to a degradation of other important necessary skills, like the ability to analyze data and create viable solutions?

It is unclear if organizations are over-relying on gen AI, according to Stephen Kowski, Field CTO at SlashNext Email Security+, but it is becoming more of a designed feature due to unintended consequences related to resource allocation in organizations.

“While AI excels at processing massive volumes of threat data, real-world attacks constantly evolve beyond historical patterns, requiring human expertise to identify and respond to zero-day threats,” said Kowski in an email interview. “The key is achieving the right balance where AI handles high-volume routine detection while skilled analysts investigate novel attack patterns and determine strategic responses.”

Yet, Kris Bondi, CEO and Co-Founder of Mimoto, isn’t worried about AI leading to a degradation of skills — at least not for the foreseeable future.

“One of the biggest challenges for cybersecurity professionals is having too many alerts and too many false positives. AI is only able to automate a small percentage of responses. It’s more likely that AI will eventually automate additional requirements for someone deemed to be suspicious or the elevation of alert so that a human can analyze the situation,” Bondi said via email.

However, organizations should watch out for AI’s role in defining threat-hunting parameters. “If AI is the sole driver defining threat hunting parameters without spot-checks or audits, the threat intelligence approach could eventually be focused in the wrong area. The answer is more reliance on critical thinking and analytical skills,” said Bondi.

Embracing creativity in an AI-driven world

AI overall, and gen AI in particular, are going to be part of the business world going forward. It is going to play a vital role in how organizations and analysts approach cybersecurity defenses and mitigations. But the soft skills that creative thought depends on will still play an important and necessary role in cybersecurity.

“Rather than diminishing soft skills, AI integration has the opportunity to elevate the importance of communication, collaboration and strategic thinking, as security teams must effectively convey complex findings to stakeholders,” said Kowski. “The human elements of cybersecurity — leadership, adaptability and cross-functional partnership — become even more critical as AI handles the technical heavy lifting.”

The post Will AI threaten the role of human creativity in cyber threat detection? appeared first on Security Intelligence.

Human communication is multimodal. We receive information in many different ways, allowing our brains to see the world from various angles and turn these different “modes” of information into a consolidated picture of reality.

We’ve now reached the point where artificial intelligence (AI) can do the same, at least to a degree. Much like our brains, multimodal AI applications process different types — or modalities — of data. For example, OpenAI’s ChatGPT 4.0 can reason across text, vision and audio, granting it greater contextual awareness and more humanlike interaction.

However, while these applications are clearly valuable in a business environment that’s laser-focused on efficiency and adaptability, their inherent complexity also introduces some unique risks.

According to Ruben Boonen, CNE Capability Development Lead at IBM: “Attacks against multimodal AI systems are mostly about getting them to create malicious outcomes in end-user applications or bypass content moderation systems. Now imagine these systems in a high-risk environment, such as a computer vision model in a self-driving car. If you could fool a car into thinking it shouldn’t stop even though it should, that could be catastrophic.”

Multimodal AI risks: An example in finance

Here’s another possible real-world scenario:

An investment banking firm uses a multimodal AI application to inform its trading decisions, processing both textual and visual data. The system uses a sentiment analysis tool to analyze text data, such as earnings reports, analyst insights and news feeds, to determine how market participants feel about specific financial assets. Then, it conducts a technical analysis of visual data, such as stock charts and trend analysis graphs, to offer insights into stock performance.

An adversary, a fraudulent hedge fund manager, then targets vulnerabilities in the system to manipulate trading decisions. In this case, the attacker launches a data poisoning attack by flooding online news sources with fabricated stories about specific markets and financial assets. Next, they launch an adversarial attack by making pixel-level manipulations — known as perturbations — to stock performance charts that are imperceptible to the human eye but enough to exploit the AI’s visual analysis abilities.

The result? Due to the manipulated input data and false signals, the system recommends buying orders at artificially inflated stock prices. Unaware of the exploit, the company follows the AI’s recommendations, while the attacker, holding shares in the target assets, sells them for an ill-gotten profit.

Getting there before adversaries

Now, let’s imagine that the attack wasn’t really carried out by a fraudulent hedge fund manager but was instead a simulated attack by a red team specialist with the goal of discovering the vulnerability before a real-world adversary could.

By simulating these complex, multifaceted attacks in safe, sandboxed environments, red teams can reveal potential vulnerabilities that traditional security systems are almost certain to miss. This proactive approach is essential for fortifying multimodal AI applications before they end up in a production environment.

According to the IBM Institute of Business Value, 96% of executives agree that the adoption of generative AI will increase the chances of a security breach in their organizations within the next three years. The rapid proliferation of multimodal AI models will only be a force multiplier of that problem, hence the growing importance of AI-specialized red teaming. These specialists can proactively address the unique risk that comes with multimodal AI: cross-modal attacks.

Cross-modal attacks: Manipulating inputs to generate malicious outputs

A cross-modal attack involves inputting malicious data in one modality to produce malicious output in another. These can take the form of data poisoning attacks during the model training and development phase or adversarial attacks, which occur during the inference phase once the model has already been deployed.

“When you have multimodal systems, they’re obviously taking input, and there’s going to be some kind of parser that reads that input. For example, if you upload a PDF file or an image, there’s an image-parsing or OCR library that extracts data from it. However, those types of libraries have had issues,” says Boonen.

Cross-modal data poisoning attacks are arguably the most severe since a major vulnerability could necessitate the entire model being retrained on an updated data set. Generative AI uses encoders to transform input data into embeddings — numerical representations of the data that encode relationships and meanings. Multimodal systems use different encoders for each type of data, such as text, image, audio and video. On top of that, they use multimodal encoders to integrate and align data of different types.

In a cross-modal data poisoning attack, an adversary with access to training data and systems could manipulate input data to make encoders generate malicious embeddings. For example, they might deliberately add incorrect or misleading text captions to images so that the encoder misclassifies them, resulting in an undesirable output. In cases where the correct classification of data is crucial, as it is in AI systems used for medical diagnoses or autonomous vehicles, this can have dire consequences.

Red teaming is essential for simulating such scenarios before they can have real-world impact. “Let’s say you have an image classifier in a multimodal AI application,” says Boonen. “There are tools that you can use to generate images and have the classifier give you a score. Now, let’s imagine that a red team targets the scoring mechanism to gradually get it to classify an image incorrectly. For images, we don’t necessarily know how the classifier determines what each element of the image is, so you keep modifying it, such as by adding noise. Eventually, the classifier stops producing accurate results.”

Vulnerabilities in real-time machine learning models

Many multimodal models have real-time machine learning capabilities, learning continuously from new data, as is the case in the scenario we explored earlier. This is an example of a cross-modal adversarial attack. In these cases, an adversary could bombard an AI application that’s already in production with manipulated data to trick the system into misclassifying inputs. This can, of course, happen unintentionally, too, hence why it’s sometimes said that generative AI is getting “dumber.”

In any case, the result is that models that are trained and/or retrained by bad data inevitably end up degrading over time — a concept known as AI model drift. Multimodal AI systems only exacerbate this problem due to the added risk of inconsistencies between different data types. That’s why red teaming is essential for detecting vulnerabilities in the way different modalities interact with one another, both during the training and inference phases.

Red teams can also detect vulnerabilities in security protocols and how they’re applied across modalities. Different types of data require different security protocols, but they must be aligned to prevent gaps from forming. Consider, for example, an authentication system that lets users verify themselves either with voice or facial recognition. Let’s imagine that the voice verification element lacks sufficient anti-spoofing measures. Chances are, the attacker will target the less secure modality.

Multimodal AI systems used in surveillance and access control systems are also subject to data synchronization risks. Such a system might use video and audio data to detect suspicious activity in real-time by matching lip movements captured on video to a spoken passphrase or name. If an attacker were to tamper with the feeds, resulting in a slight delay between the two, they could mislead the system using pre-recorded video or audio to gain unauthorized access.

Getting started with multimodal AI red teaming

While it’s admittedly still early days for attacks targeting multimodal AI applications, it always pays to take a proactive stance.

As next-generation AI applications become deeply ingrained in routine business workflows and even security systems themselves, red teaming doesn’t just bring peace of mind — it can uncover vulnerabilities that will almost certainly go unnoticed by conventional, reactive security systems.

Multimodal AI applications present a new frontier for red teaming, and organizations need their expertise to ensure they learn about the vulnerabilities before their adversaries do.

The post Stress-testing multimodal AI applications is a new frontier for red teams appeared first on Security Intelligence.

The rising influence of artificial intelligence (AI) has many organizations scrambling to address the new cybersecurity and data privacy concerns created by the technology, especially as AI is used in cloud systems. Apple addresses AI’s security and privacy issues head-on with its Private Cloud Compute (PCC) system.

Apple seems to have solved the problem of offering cloud services without undermining user privacy or adding additional layers of insecurity. It had to do so, as Apple needed to create a cloud infrastructure on which to run generative AI (genAI) models that need more processing power than its devices could supply while also protecting user privacy, stated a ComputerWorld article.

Apple is opening the PCC system to security researchers to “learn more about PCC and perform their own independent verification of our claims,” the company announced. In addition, Apple is also expanding its Apple Security Bounty.

What does this mean for AI security going forward? Security Intelligence spoke with Ruben Boonen, CNE Capability Development Lead at IBM, to learn what researchers think about PCC and Apple’s approach.

SI: ComputerWorld reported this story, saying that Apple hopes that “the energy of the entire infosec community will combine to help build a moat to protect the future of AI.” What do you think of this move?

Boonen: I read the ComputerWorld article and reviewed Apple’s own statements about their private cloud. I think what Apple has done here is good. I think it goes beyond what other cloud providers do because Apple is providing an insight into some of the internal components they use and are basically telling the security community, you can have a look at this and see if it is secure or not.

Also good from the perspective that AI is constantly getting bigger as an industry. Bringing generative AI components into regular consumer devices and getting people to trust their data with AI services is a really good step.

SI: What do you see as the pros of Apple’s approach to securing AI in the cloud?

Boonen: Other cloud providers do provide high-security guarantees for data that’s stored on their cloud. Many businesses, including IBM, trust their corporate data to these cloud providers. But a lot of times, the processes to secure data aren’t visible to their customers; they don’t explain exactly what they do. The biggest difference here is that Apple is providing this transparent environment for users to test that plane.

SI: What are some of the downsides?

Boonen: Currently, the most capable AI models are very big, and that makes them very useful. But when we want AI on consumer devices, there’s a tendency for vendors to ship small models that can’t answer all questions, so it relies on the larger models in the cloud. That comes with additional risk. But I think it is inevitable that the whole industry will be moving to that cloud model for AI. Apple is implementing this now because they want to give consumers trust to the AI process.

SI: Apple’s system doesn’t play well with other systems and products. How will Apple’s efforts to secure AI in the cloud benefit other systems?

Boonen: They are providing a design template that other providers like Microsoft, Google and Amazon can then replicate. I think it is mostly effective as an example for other providers to say maybe we should implement something similar and provide similar testing capabilities for our customers. So I don’t think this directly impacts other providers except to push them to be more transparent in their processes.

It’s also important to mention Apple’s Bug Bounty as they invite researchers in to look at their system. Apple has a history of not doing very well with security, and there have been cases in the past where they’ve refused to pay out bounties for issues found by the security community. So I’m not sure they’re doing this entirely out of the interest of attracting researchers, but also in part of convincing their customers that they are doing things securely.

That being said, having read their design documentation, which is extensive, I think they’re doing a pretty good job in addressing security around AI in the cloud.

The post Cybersecurity awareness: Apple’s cloud-based AI security system appeared first on Security Intelligence.

Have you ever wished you had an assistant at your security operations centers (SOCs) — especially one who never calls in sick, has a bad day or takes a long lunch? Your wish may come true soon. Not surprisingly, AI-driven SOC “co-pilots” are topping the lists for cybersecurity predictions in 2025, which often describe these tools as game-changers.

“AI-driven SOC co-pilots will make a significant impact in 2025, helping security teams prioritize threats and turn overwhelming amounts of data into actionable intelligence,” says Brian Linder, Cybersecurity Evangelist at Check Point. “It’s a game-changer for SOC efficiency.”

What is an AI-driven SOC co-pilot?

AI-driven SOC co-pilots are generative AI tools that use machine learning to help security analysts run and manage the SOC. Common co-pilot tasks include detecting threats, managing incidents, triaging alerts, predicting new trends and patterns for attacks and breaches and automating responses to threats. Co-pilots may be proprietary tools built by the company for their specific needs or commercially available cybersecurity co-pilots such as Microsoft Copilot.

For example, a co-pilot can review alerts and use AI to predict which are most likely to be a high priority. This reduces a common issue in SOCs: false positives. The analysts can then focus on the alerts that are most likely to be a real threat. Because they are not chasing down noncritical alerts, analysts have more time to spend on actual threats and are more likely to be successful in containing the threat.

Co-pilots can take many different forms in a SOC. Analysts can use the co-pilot similarly to how many people use ChatGPT, assigning it a specific task such as incident response. The analyst enters information about a specific incident, and the co-pilot analyzes data to suggest possible causes as well as how the organizations should respond to the incident. However, you can also use co-pilots to automate parts of the workflow without human intervention, such as monitoring current firewalls and detecting vulnerabilities.

Benefits of using AI-driven SOC co-pilots

Businesses that turn to AI-driven co-pilots to help manage their SOC see a wide range of benefits. Common benefits include:

• Improved productivity: Because it can process a much higher volume of data than even the most efficient cybersecurity analyst, a co-pilot gets significantly more work done in less time. With humans and machines working together, co-pilots are able to more effectively monitor the SOC with fewer human resources.
• Additional time for cybersecurity professionals to complete high-level tasks: When co-pilots handle manual and repetitive tasks, analysts have more time for higher-level tasks such as strategy and analytics. Analysts are more likely to be fully engaged when their day is filled with more interesting work, which reduces burnout.
• Fewer errors: Humans make mistakes, especially with manual tasks such as reviewing logs. While AI tools are only as “smart” as the algorithm and the training data used for the algorithm, they are often able to spot patterns that may be undetectable to humans. This reduces errors and prevents issues that can lead to a breach or attack.
• Quicker response to threats: Whereas humans may not recognize an area of vulnerability or may be slower to respond, a co-pilot uses automation to respond and send a notification immediately. Co-pilots also don’t take bathroom or lunch breaks; they are always “at their desk,” leading to faster response times.
• Reduced impact of worker shortage and skills gaps: When cybersecurity positions are not filled or the analyst does not have the right skills for the job, the company’s risk increases. AI-driven co-pilots can help reduce open positions by taking on various manual tasks, which means greater coverage by the SOC.

Will AI-driven SOC co-pilots replace humans?

Like many AI tools, co-pilots can take over many manual and repetitive tasks currently done by humans. However, the fear of AI replacing the need for humans in the SOC is not likely to become reality. Setting up co-pilots to operate without human oversight or intervention would likely be a mistake. But businesses that have analysts and co-pilots work together can see a reduction in risk, better responses and higher employee satisfaction.

While co-pilots can be the first line of defense in the SOC, companies should set up gen AI tools so that humans remain the ultimate decision-makers. For example, an analyst may set up an automation with an AI-driven co-pilot to monitor and prioritize alerts based on set criteria. Yet, as threat actors begin using new tactics, the analyst may need to change the criteria to catch the latest threats. Once the co-pilot identifies a high-priority alert, the human can ask the tool to analyze the situation and provide recommended next steps. The analyst then uses human judgment to make the best decisions in the situation and instructs the tool to take the next action, such as shutting down systems or taking the network temporarily offline.

Putting AI-driven co-pilots into action in the SOC

When it comes to putting co-pilots in action, consider starting on a small scale with a limited use case. Many organizations use a commercial product to start, leaving open the option to create a proprietary tool in the future. Creating a list of time-consuming tasks in the SOC, especially those that are error-prone or frustrating for analysts, will help you determine which use case to start with. After launching the tool, a single analyst can gather feedback and make changes.

Upon seeing success, your team can begin expanding the use of co-pilots to additional analysts and use cases. By taking a measured approach to using co-pilots and continuously soliciting feedback from the analysts, businesses can create a partnership between analysts and co-pilots that improves human job satisfaction while also keeping the organization more secure.

The post How AI-driven SOC co-pilots will change security center operations appeared first on Security Intelligence.

“A computer can never be held accountable, therefore a computer must never make a management decision.”

– IBM Training Manual, 1979

Artificial intelligence (AI) adoption is on the rise. According to the IBM Global AI Adoption Index 2023, 42% of enterprises have actively deployed AI, and 40% are experimenting with the technology. Of those using or exploring AI, 59% have accelerated their investments and rollouts over the past two years. The result is an uptick in AI decision-making that leverages intelligent tools to arrive at (supposedly) accurate answers.

Rapid adoption, however, raises a question: Who’s responsible if AI makes a poor choice? Does the fault lie with IT teams? Executives? AI model builders? Device manufacturers?

In this piece, we’ll explore the evolving world of AI and reexamine the quote above in the context of current use cases: Do companies still need a human in the loop, or can AI make the call?

Getting it right: Where AI is improving business outcomes

Guy Pearce, principal consultant at DEGI and member of the ISACA working trends group, has been involved with AI for more than three decades. “First, it was symbolic,” he says, “and now it’s statistical. It’s algorithms and models that allow data processing and improve business performance over time.”

Data from IBM’s recent AI in Action report shows the impact of this shift. Two-thirds of leaders say that AI has driven more than a 25% improvement in revenue growth rates, and 72% say that the C-suite is fully aligned with IT leadership about what comes next on the path to AI maturity.

With confidence in AI growing, enterprises are implementing intelligent tools to improve business outcomes. For example, wealth management firm Consult Venture Partners deployed AIda AI, a conversational digital AI concierge that uses IBM watsonx assistant technology to answer potential clients’ questions without the need for human agents.

The results speak for themselves: Alda AI answered 92% of queries correctly, 47% of queries led to webinar registrations and 39% of inquiries turned into leads.

Missing the mark: What happens if AI makes mistakes?

92% is an impressive achievement for Alda AI. The caveat? It was still wrong 8% of the time. So, what happens when AI makes mistakes?

For Pearce, it depends on the stakes.

He uses the example of a financial firm leveraging AI to evaluate credit scores and issue loans. The outcomes of these decisions are relatively low stakes. In the best-case scenario, AI approves loans that are paid back on time and in full. In the worst case, borrowers default, and companies need to pursue legal action. While inconvenient, the negative outcomes are far outweighed by the potential positives.

“When it comes to high stakes,” says Pearce, “look at the medical industry. Let’s say we use AI to address the problem of wait times. Do we have sufficient data to ensure patients are seen in the right order? What if we get it wrong? The outcome could be death.”

As a result, how AI is used in decision-making depends largely on what it’s making decisions about and how these decisions impact both the company making the decisions and those the decision affects.

In some cases, even the worst-case scenario is a minor inconvenience. In others, the results could cause significant harm. 

Taking the blame: Who’s accountable if AI gets it wrong?

In April 2024, a Tesla operating in “full self-driving” mode struck and killed a motorcyclist. The driver of the vehicle admitted to looking at their phone prior to the crash despite active driver supervision being required.

So who takes the blame? The driver is the obvious choice and was arrested on charges of vehicular homicide.

But this isn’t the only path to accountability. There’s also a case to be made in which Tesla bears some responsibility since the company’s AI algorithm failed to spot the victim. Blame could also be placed on governing bodies such as the National Highway Traffic Safety Administration (NHTSA). Perhaps their testing wasn’t rigorous or complete enough.

One could even argue that the creator(s) of Tesla’s AI could be held liable for letting code that could kill someone go live.

This is the paradox of AI decision-making: Is someone at fault, or is everyone at fault? “If you bring all the stakeholders together who should be accountable, where does that accountability lie?” asks Pearce. “With the C-suite? With the whole team? If you have accountability that’s spread over the entire organization, everyone can’t end up in jail. Ultimately, shared accountability often leads to no accountability.”

Drawing the line: Where does AI end?

So, where do organizations draw the line? Where does AI insight give way to human decision-making?

Three considerations are key: Ethics, risk and trust.

“When it comes to ethical dilemmas,” says Pearce, “AI can’t do it.” This is because intelligent tools naturally seek the most efficient path, not the most ethical. As a result, any decision involving ethical questions or concerns should include human oversight.

Risk, meanwhile, is an AI specialty. “AI is good in risk,” Pearce says. “What statistical models do is give you something called a standard error, which lets you know if what AI is recommending has a high or low potential variability.” This makes AI great for risk-based decisions like those in finance or insurance.

Finally, enterprises need to prioritize trust. “There are declining levels of trust in institutions,” says Pearce. “Many citizens don’t feel confident that the data they share is being used in a trustworthy manner.”

For example, under GDPR, companies need to be transparent about data collection and handling and give citizens a chance to opt-out. To bolster trust in AI use, organizations should clearly communicate how and why they’re using AI and (where possible) allow customers and clients to opt out of AI-driven processes.

Decisions, decisions

Should AI be used for management decisions? Maybe. Will it be used to make some of these decisions? Almost certainly. The draw of AI — its ability to capture, correlate and analyze multiple data sets and deliver new insights — makes it a powerful tool for enterprises to streamline operations and reduce costs.

What’s less clear is how the shift to management-level decision-making will impact accountability. According to Pearce, current conditions create “blurry lines” in this area; legislation hasn’t kept pace with increasing AI usage.

To ensure alignment with ethical principles, reduce the risk of wrong choices and engender stakeholder and customer trust, businesses are best served by keeping humans in the loop. Maybe this means direct approval from staff is required before AI can act. Maybe it means the occasional review and evaluation of AI decision-making outcomes.

Whatever approach enterprises choose, however, the core message remains the same: When it comes to AI-driven decisions, there’s no hard-and-fast line. It’s a moving target, one defined by possible risk, potential reward and probable outcomes.

The post AI decision-making: Where do businesses draw the line? appeared first on Security Intelligence.

With cloud infrastructure and, more recently, artificial intelligence (AI) systems becoming prime targets for attackers, security leaders are laser-focused on defending these high-profile areas. They’re right to do so, too, as cyber criminals turn to new and emerging technologies to launch and scale ever more sophisticated attacks.

However, this heightened attention to emerging threats makes it easy to overlook traditional attack vectors, such as human-driven social engineering and vulnerabilities in physical security.

As adversaries exploit an ever-wider range of potential entry points — both new and old — security leaders must strike a balance to ensure that they’re capable of addressing all risks effectively.

Cyber crime is still a human problem

Despite overwhelming hype, technology is not a panacea. It can’t replace human expertise in every domain, and AI alone can’t match the innately human qualities of intuition and creative thinking. Adversaries know this too, which is why the smarter — and much more dangerous — ones use a blend of human- and technology-powered tactics.

While major technical vulnerabilities tend to make the headlines, the reality is that the weakest link is almost always the human element. Almost all attacks involve a social engineering element, and despite the buzz around generative AI and deepfakes helping scale such attacks, it’s human-to-human interaction where the greatest risks lie.

Synthetic content is now all around us, and people are getting better at telling it apart. Whether we get to the point when that’s no longer the case is a topic for another discussion. But for now, the most dangerous and effective social engineering attacks still depend primarily on human conversations, whether by phone, email or even in person. After all, a seasoned attacker can build trust and forge sham relationships in a way that no AI nor deepfake can match.

Cyber espionage remains a serious threat

Take state-sponsored cyber espionage, for example. Highly trained social engineers are a far cry from the typical rabble of independent cyber crime rackets operating off the dark web, who tend to rely more on scale than targeting specific enterprises and individuals. These attackers may target data systems, but when it comes to their own arsenals, their talents in manipulation and deception are by far their greatest weapons.

Technology still has a long way to go before it can come close to matching the age-old tactics of spycraft.

When facing an attacker who can pose effectively as an internal employee or any other trusted individual, someone relying solely on technology to mitigate the threat stands little chance of protecting themselves. That isn’t a technology failure. It’s a process failure, hence why the human element must always be a key factor in any cybersecurity strategy.

Of course, that’s not to say technology doesn’t have a vital role to play in bolstering your cyber defenses. It most certainly does, not least, because more and more routine threats are being automated or are carried out en-masse by attackers who are less skilled or experienced. The value of technology — especially AI-powered cybersecurity automation — exists primarily in its ability to free up time for security leaders to focus on the threats that technology alone can’t solve.

It’s not all about the cloud, either

The majority of business data is now stored in the cloud, and the percentage continues to rise. Many businesses, especially smaller organizations and startups, exclusively use the cloud for data storage and other IT operations. The rise of AI, given how computationally demanding it is, is further accelerating cloud adoption.

Nonetheless, cloud computing isn’t the best option in all situations. On-premises remains the preferred choice for high-performance workloads that require extremely low latencies. In some cases, on-premises computing is also the cheaper option, and that’s unlikely to change in the near future.

Even though more companies are migrating to the cloud, that doesn’t mean they don’t keep sensitive data on-site. For instance, edge computing, which brings data processing closer to where it’s needed, has become a critical enabler in certain use cases. Examples include smart energy grids, remote monitoring of industrial assets and autonomous vehicles. These include cases where you can’t always rely on internet connectivity.

The smarter and better-funded adversaries aren’t just targeting cloud-hosted infrastructure. They’re also setting their sights on local servers and cyber-physical systems, such as industrial control systems and hardware supply chains. The fact that there’s often minimal collaboration between logistics, production and cybersecurity departments makes these risks all the more serious.

Ransomware remains one of the biggest threats targeting on-premises systems despite the small reduction in attacks over the last year. While cloud systems aren’t inherently immune from ransomware attacks, the vast majority target bare-metal hypervisors and local servers. In one recent case, the Akira ransomware group reverted to its earlier double extortion tactics, experimenting with different code frameworks to target systems running ESXi and Linux.

Botnets are another growing concern as the number of IoT devices continues to soar. Used to launch distributed denial of service (DDoS) attacks spanning thousands of devices, these botnets primarily target unsecured IoT devices, like those that monitor and operate industrial machines and critical infrastructure. One recent report discovered that DDoS attacks against critical infrastructure have increased by 55% in the last four years. These attacks don’t directly involve the exfiltration of sensitive data, but given how they can cause widespread disruption, adversaries may rely on them to draw attention away from more serious threats.

Why physical security is still relevant

As security leaders focus on locking down their cloud-hosted assets, they cannot afford to lose sight of the risks facing their physical infrastructure. Sometimes, the easiest way into the cloud is from within.

Even thin clients and dumb terminals — both widely used in high-security environments like healthcare and finance — can potentially give attackers a foothold in wider systems, including cloud infrastructure and remote data centers. Edward Snowden proved that while working at the National Security Agency when he exfiltrated 20,000 government documents stored on the servers in NSA’s headquarters 5,000 miles away. He did so without using any advanced technology. While that happened way back in 2013, and the NSA has long since updated its physical security protocols, the risk is just as relevant today as it was then.

While most thin clients are now protected by multiple layers of security, including encryption and multifactor authentication, these solutions alone can’t fully protect against physical compromise. If an attacker gains access to a terminal — perhaps by way of social engineering — they may be able to compromise it using unauthorized peripherals or by directly manipulating the device’s firmware. This could give them access to the wider network, potentially allowing for the injection of customized malware that goes undetected by regular security scans.

IoT devices are another leading reason behind the expansion of attack surfaces. They often lack adequate security, also giving attackers a potential entry point into the broader computing infrastructures they’re connected to. The fact that these connected technologies are being rolled out en masse in areas like smart cities, critical infrastructure and transportation networks, greatly magnifies such vulnerabilities.

Ultimately, if an attacker is able to get past your physical safeguards, then these connected systems present far easier pathways to an organization’s so-called “crown jewels” than trying to break through multi-layered cloud defenses.

Cloud data is not always the true target

In other cases, data hosted in the cloud might not be the attacker’s end goal. Many companies, such as those subject to stringent data residency regulations or that require high performance for real-time applications, still store their data on on-premises servers.

Some of these systems are air-gapped, meaning they’re entirely disconnected from any other networks, including the Internet itself. While more secure than any cloud-hosted server, at least in theory, their security can’t be taken for granted. For instance, anyone with physical access to the servers may be able to compromise them, either maliciously or accidentally.

Physical security, such as CCTV and biometric security checkpoints, is as important as ever in such cases. But it’s not just about protecting against intentional physical tampering. Indirect attacks orchestrated by highly skilled social engineers can also dupe unsuspecting employees into taking a desired action — such as lending them a biometric security access card.

These are not the sort of adversaries that usually work by email or use AI to scale their attacks – they’re far likelier to deceive someone in person, a tactic as old as humanity itself. In fact, the attacker could be anyone, such as a disgruntled former employee, a hacker operating in the interests of a rival company or even a rogue state.

Bridging the gap between digital and human security

Technology alone can’t protect an organization from the myriad threats out there, and neither can humans keep up with ever-expanding system logs and security information feeds if they’re relying solely on manual processes.

The reality is that you need both, starting with people and using technology to broaden their capabilities. A layered security strategy should typically start with locking down physical access to any data-bearing system or system that is connected to another.

The next layer of defense is the human one. This revolves heavily around security awareness training. But the reality is that many programs are ineffective, either because they lack practical application, are overly reliant on generic content or focus too much on technical factors that are beyond the target audience’s understanding.

Phishing simulations are often similarly limited in their scope, focusing on common lures like trending news topics, a sense of urgency or even outright threats. However, more sophisticated attackers tend to use subtler ways to elicit a response. This could be something as simple as sending messages about a routine policy update regarding company dress code or remote work guidelines. These topics might seem trivial, but they can pique interest, especially when they concern changes to daily routines and work-life balance. Attackers could then use this to dupe unsuspecting victims into divulging sensitive information via a sham survey.

Like any other security measure, physical systems and awareness training will only ever be effective if they’re tested regularly. That’s where physical red teaming comes in. Whereas red teaming in the context of IT focuses on technical measures like penetration testing, physical red teaming is all about having teams try to gain entry to restricted areas and systems. To do so, they might use a blend of simulated social engineering attacks and technology to hack into physical security systems. By attempting to bypass physical security barriers or impersonate staff, red teams can reveal gaps that might otherwise go unnoticed. That’s what makes them a valuable part of any comprehensive information security program.

The post AI and cloud vulnerabilities aren’t the only threats facing CISOs today appeared first on Security Intelligence.

Many times a day worldwide, a boss asks one of their team members to perform a task during a video call. But is the person assigning tasks actually who they say they are? Or is it a deepfake? Instead of blindly following orders, employees must now ask themselves if they are becoming a victims of fraud.

Earlier this year, a finance worker found themselves talking on a video meeting with someone who looked and sounded just like their CFO. After the meeting was over, they then dutifully followed their boss’s instructions to send $200 million Hong Kong dollars, which equals $25 million.

But it wasn’t actually their boss — just an AI video representation called a deepfake. Later that day, the employee realized their terrible mistake after checking with the corporate offices of their multinational firm. They had been a victim of a deepfake scheme that defrauded the organization out of $25 million.

Businesses are often deepfake targets

The term deepfake refers to AI-created content — video, image, audio or text — that contains false or altered information, such as Taylor Swift promoting cookware and the infamous fake Tom Cruise. Even the recent hurricanes hitting the U.S. led to multiple deepfake images, including fake flooded Disney World photos and heartbreaking AI-generated pictures of people with their pets in floodwaters.

While deepfakes, also referred to as synthetic media, targeted at individuals typically serve to manipulate people, cyber criminals targeting businesses are looking for monetary gain. According to the CISA Contextualizing Deepfake Threats to Organizations information sheet, threats targeting businesses tend to fall into one of three categories: executive impersonation for brand manipulation, impersonation for financial gain or impersonation to gain access.

But the recent incident in Hong Kong wasn’t just one employee making a mistake. Deepfake schemes are becoming increasingly common for businesses. A recent Medus survey found that the majority (53%) of finance professionals have been targeted by attempted deepfake schemes. Even more concerning is the fact that more than 43% admitted to ultimately falling victim to the attack.

Watch Unmask the Deepfake

Are deepfake attacks underreported?

The key word from the Medus research is “admitted.”  And it raises a big question. Do people fail to report being a victim of a deepfake attack because they are embarrassed? The answer is probably.  After the fact, it seems obvious it was a fake to other people. And it’s tough to admit that you fell for an AI-generated image.  But the underreporting only adds to the shame and makes it easier for cyber criminals to get away with it.

Most people assume that they could spot a deepfake. But that’s not the case. The Center for Humans and Machines and CREED found a wide gap between people’s confidence in identifying a deepfake and their actual performance. Because many people overestimate their ability to identify a deepfake, it adds to the shame when someone falls victim, which likely leads to underreporting.

Why people fall for deepfake schemes

The employee who was tricked by the deepfake of the CFO to the tune of $25 million later admitted that when they first got the email supposedly from his CFO, the mention of a secret transaction made them wonder if the email was actually a phishing email. But once he got on the video, they recognized other members of his department in the video and decided it was authentic. However, the employee later learned that the video images of his department members were also deepfakes.

Many people who are victims overlook their concerns, questions and doubts. But what makes people, even those educated on deepfakes, push their concerns to the side and choose to believe an image is real? That’s the $1 million — or $25 million — question that we need to answer to prevent costly and damaging deepfake schemes in the future.

Sage Journals asked the question about who was more likely to fall for deepfakes and didn’t find any pattern around age or gender. However, older individuals may be more vulnerable to the scheme and have a hard time detecting it. Additionally, the researchers found that while awareness is a good starting point, it appears to have limited effectiveness in preventing people from falling for deepfakes.

However, computational neuroscientist Tijl Grootswagers of Western Sydney University likely hit the nail on the head as to the challenge of spotting a deepfake: it’s a brand new skill for each of us. We’ve learned to be skeptical of news stories and bias, but questioning the authenticity of an image we can see goes against our thought processes. Grootswagers told Science Magazine “In our lives, we never have to think about who is a real or a fake person. It’s not a task we’ve been trained on.”

Interestingly, Grootswagers discovered that our brains are better at detection without our intervention. He discovered that when people looked at a picture of a deepfake, the image resulted in a different electrical signal to the brain’s visual cortex than a legitimate image or video. When asked why, he wasn’t quite sure — maybe the signal never reached our consciousness due to interference from other brain regions, or maybe humans don’t recognize the signals that an image is fake because it’s a new task.

This means that each of us must begin to train our brain to consider that any image or video that we view could possibly be a deepfake. By asking this question each and every time we begin to act on content, we may be able to begin detecting our brain signals that are spotting the fakes before we can. And most importantly, if we do fall victim to a deepfake, especially at work, it’s key that each of us reports all instances. Only then can experts and authorities begin to curb the creation and proliferation.

The post Are successful deepfake scams more common than we realize? appeared first on Security Intelligence.

Imagine this scenario: A sophisticated, malicious phishing campaign targets a large financial institution. The attackers use emails generated by artificial intelligence (AI) that closely mimic the company’s internal communications. The emails contain malicious links designed to steal employee credentials, which the attackers could use to gain access to company assets and data for unknown purposes.

The organization’s AI-powered cybersecurity solution, which continuously monitors network traffic and user behavior, detects several anomalies associated with the attack, blocks access to the suspicious domains across the network, quarantines the phishing emails, resets passwords for all potentially compromised accounts and sends real-time alerts to the security operations center, providing detailed information about the attack vector and affected systems.

Using predictive analytics, the AI suggests potential next steps the attackers might take, allowing the security team to strengthen defenses in those areas proactively.

The good guys won. But was the AI solution worth the price? What’s the value in dollars of that victory? It’s easy to measure the investment in AI. But how do you measure the return on that investment? Specifically, how do you measure the value of data never stolen, unknown reputational damage that never happened, customer trust never lost or reduced operational risks never incurred?

The rise of AI cybersecurity

To be sure, cybersecurity AI spending is set to increase dramatically. Organizations spent $24 billion in 2023, with an expected rise to $133 billion by 2030. Cybersecurity professionals and the companies they work for will increasingly rely on advanced AI solutions as threats grow and the cost of data breaches also rises.

The challenging nature of cybersecurity ROI is compounded by many other factors — dozens, hundreds or thousands of attempted cyberattacks per year per organization; the lack of universally accepted metrics or calculations for cybersecurity ROI; the long payback period for investments in cybersecurity AI; the fast-changing nature of the threat landscape; the fact that cybersecurity investments also touch areas like operational efficiency, regulatory compliance and others.

Historically, organizations calculated ROI in cybersecurity investments by estimating money saved in the absence of security incidents. But that fails to account for proactive security measures, efficiency gains in operations and the overall security posture. With the integration of AI, cybersecurity has fundamentally changed, offering enhanced threat detection and prevention capabilities beyond simply measuring the absence of incidents.

A proactive approach and improved operational efficiency through task automation provide tangible benefits not captured in traditional ROI calculations.

New metrics for ROI calculation

The use of AI tools has transformed the typical cybersecurity ROI calculation, introducing several quantifiable metrics:

• Reduction in false positives
• Time saved on routine tasks
• Faster incident response times
• Improved threat intelligence accuracy
• Cost savings from prevented breaches

These metrics offer a more comprehensive view of the value derived from AI-powered cybersecurity investments, enabling organizations to make more informed decisions about resource allocation and strategic planning.

Cost savings can also be measured in the aggregate. According to the IBM 2024 Cost of a Data Breach report, organizations extensively using security AI and automation in prevention workflows saved an average of $2.2 million in breach costs compared to those without such technologies.

Still, measuring AI cybersecurity ROI comes with challenges, including difficulty attributing prevented incidents directly to AI, the constantly evolving threat landscape and balancing initial investment costs with long-term benefits.

Taking a holistic approach to cybersecurity AI ROI

Organizations can leverage established frameworks, such as the NIST Cybersecurity Framework, to effectively measure and communicate AI’s ROI in cybersecurity. By aligning AI initiatives with these functions, organizations can more accurately measure their impact on overall cybersecurity performance.

To effectively measure the impact of AI on cybersecurity ROI, organizations should focus on specific Key Performance Indicators (KPIs):

• Mean time to detect
• Mean time to respond
• Security operational efficiency
• Threat intelligence accuracy
• Compliance adherence rate

The best approach is to adopt a more comprehensive approach that uses risk assessment frameworks, measures risk reduction, considers and estimates intangible benefits and regularly reviews and updates calculations.

Organizations must adopt a holistic approach that considers the proactive capabilities, efficiency gains and quantifiable metrics provided by AI-powered solutions. This comprehensive evaluation allows a more accurate assessment of cybersecurity investments’ true value and impact in today’s complex threat landscape.

Of course, cyberattacks don’t happen randomly or in a vacuum. Take the follow-on consequences of the ongoing cybersecurity skills gap, which can be self-enlarging, according to Sam Hector, senior strategy leader of IBM Security.

“When you don’t have enough skilled experts in monitoring and defending your infrastructure, a few things happen,” Hector said. “The time to triage alerts grows as the queue of incidents to review becomes longer, meaning you’re more likely to be breached, and attackers dwell times increase (when they are in your environment undetected) as you’re less likely to find the needle in the haystack. The time to detect increasing directly leads to higher breach costs on average.”

And the problem keeps growing: “Teams that are stretched too thin don’t have the time to devote to improving cybersecurity processes, integration and efficiency,” Hector said. “They’re unable to drill exercises and embark on further training as they’re too focused on keeping the lights on. This means over time, they’re less effective comparable to the threat landscape, and misconfigurations and gaps develop that attackers can exploit.”

Hector said persistent attackers are unlikely to go unnoticed by these weakening defenses: “If there’s a specific industry, region or even organization that is known to be struggling to acquire cybersecurity skills, this puts them at increased risk of being targeted by attackers who will be anticipating weaker defenses.”

An ongoing shift in cybersecurity investment

The integration of AI in cybersecurity has fundamentally changed how organizations approach and measure their security investments. By providing more tangible and comprehensive ROI metrics, AI enables organizations to make data-driven decisions about their cybersecurity strategies. As cyber threats continue to evolve, the role of AI in cybersecurity will only grow more critical, making it essential for organizations to invest in — and effectively measure — the impact of these technologies.

The post How to calculate your AI-powered cybersecurity’s ROI appeared first on Security Intelligence.

AI has made an impact everywhere else across the tech world, so it should surprise no one that the 2024 ISC2 Cybersecurity Workforce Study saw artificial intelligence (AI) jump into the top five list of security skills.

It’s not just the need for workers with security-related AI skills. The Workforce Study also takes a deep dive into how the 16,000 respondents think AI will impact cybersecurity and job roles overall, from changing skills approaches to creating generative AI (gen AI) strategies.

Budgets and the skills gap

According to the study, two-thirds of respondents think that their expertise in cybersecurity will augment AI technology; on the flip side, a third are concerned their jobs could be eliminated in an AI-focused world.

That, of course, is not going to happen immediately. Not even half the respondents have implemented gen AI into their tools. The more immediate concern for cybersecurity professionals is budgets.

“In 2024, 25% of respondents reported layoffs in their cybersecurity departments, a 3% rise from 2023, while 37% faced budget cuts, a 7% rise from 2023,” the report stated.

These budget cuts have impacted the skills gap, as two-thirds of the respondents said not only have the budget cuts led to current staffing shortages but they are expected to make closing the skills gap even more difficult in the next few years.

Many of the respondents pointed out that the skills gap has had a more negative effect on organizational security than the decrease in on-site staff. In part because the funding isn’t available for training and because those with skills in high demand are moving on to better-paying positions, many security teams struggle to address the threats and risks in today’s cybersecurity landscape.

The role of AI in the skills gap

Two years ago, AI wasn’t even considered a required skill set for cybersecurity jobs, but now it is a top five skill, said Jon France, CISO with ISC2.

“And we suspect that probably next year, it will be the number one in-demand skill set around security,” France said in a conversation at ISC2’s Security Congress in Las Vegas.

(If you’re wondering, the other skills in the top five are cloud, zero trust architecture, forensics, incident response and application security — all areas that have been at the top of the skills need list for a long time.)

AI’s role in cybersecurity is changing because of the exponential increase in data and the need to gather good intelligence on the data being generated.

“AI is one of the tools that can obviously consider large data sets very quickly,” said France. Still, human eyes are necessary to validate the results generated from AI models. This is where AI security skills will be most needed to advance the changes in how analysts and incident responders analyze data.

France also believes that AI will change the scope of entry-level security positions. “I think if you’re coming into the profession, and if you’ve got to pick up one thing to learn, you’ll get the most favorable opportunities if you have experience of using generative AI coding.”

Right now, however, there is a bit of a disconnect between the technical skills that hiring managers think are needed and what non-hiring managers want. Both types of managers list cloud computing security skills at the top of the list, but when asked about AI/ML skills, only 24% of hiring managers said it was a skill they want right now, ranking last on the skills-need list. When non-hiring managers are asked about the skills most in demand to advance careers, 37% said AI/ML, higher than every other listed skill but cloud security.

AI is reinventing cybersecurity skills

In its study AI in Cyber 2024, ISC2 found that 82% of respondents are optimistic that AI will improve work efficiency, and 88% thought it would impact their job role in some way. Relying more on AI in the cyber world has a lot of positive points, but there are also issues around the technology causing stress. Four in ten respondents said they aren’t prepared for the explosion of AI, according to the AI study, and 65% said their organization needs more regulations around the safe use of gen AI, according to the Workforce study.

But there are also a lot of question marks surrounding what skills will be needed. “While study participants speculated on what skills may be automated or streamlined, they cannot yet predict what activities, if any, AI will replace,” the study reported. Perhaps this is why hiring managers are showing some reluctance to hire cybersecurity professionals who have AI technical expertise.

With AI, many anticipate an uptick in the need for non-technical skills. Cybersecurity has been more open to finding potential professionals outside of the traditional technical areas and training them for their new roles, so it isn’t too surprising that, because hiring managers aren’t certain of the type of skills that will be required for using gen AI as a security tool (or for securing gen AI, for that matter), there is a greater willingness to default to non-tech skills that are seen as more transferable as the technology evolves. Overall, strong communication skills were listed as the most in-demand skill set across all of cybersecurity, followed closely by strong problem-solving skills and teamwork/collaboration skills.

The cyber workforce in the world of AI

Looking at the overall picture of how AI skills will fit into the cybersecurity workforce going forward, it is likely that the issues that hamper hiring today will have a similar impact on AI expertise. Budget cuts will decrease the workforce, as already mentioned. France pointed to the human resources gap as well, where entry-level positions are posted with requirements such as certifications that require five years of work experience.

“We also need to blow this myth: New entrance into the cybersecurity workforce doesn’t mean young. It can be a career change. In fact, career changes bring a lot of different viewpoints and experiences,” said France.

Hire for the skills the employee is bringing to the table, even if they aren’t what you need right now. “The rest,” said France, “can be taught.”

The post ISC2 Cybersecurity Workforce Study: Shortage of AI skilled workers appeared first on Security Intelligence.

Cybersecurity concerns in 2024 can be summed up in two letters: AI (or five letters if you narrow it down to gen AI). Organizations are still in the early stages of understanding the risks and rewards of this technology. For all the good it can do to improve data protection, keep up with compliance regulations and enable faster threat detection, threat actors are also using AI to accelerate their social engineering attacks and sabotage AI models with malware.

AI might have gotten the lion’s share of attention in 2024, but it wasn’t the only cyber threat organizations had to deal with. Credential theft continues to be problematic, with a 71% year-over-year increase in attacks using compromised credentials. The skills shortage continues, costing companies an additional $1.76 million in a data breach aftermath. And as more companies rely on the cloud, it shouldn’t be surprising that there has been a spike in cloud intrusions.

But there have been positive steps in cybersecurity over the past year. CISA’s Secure by Design program signed on more than 250 software manufacturers to improve their cybersecurity hygiene. CISA also introduced its Cyber Incident Reporting Portal to improve the way organizations share cyber information.

Last year’s cybersecurity predictions focused heavily on AI and its impact on how security teams will operate in the future. This year’s predictions also emphasize AI, showing that cybersecurity may have reached a point where security and AI are interdependent on each other, for both good and bad.

Here are this year’s predictions.

Shadow AI is everywhere (Akiba Saeedi, Vice President, IBM Security Product Management)

Shadow AI will prove to be more common — and risky — than we thought. Businesses have more and more generative AI models deployed across their systems each day, sometimes without their knowledge. In 2025, enterprises will truly see the scope of “shadow AI” – unsanctioned AI models used by staff that aren’t properly governed. Shadow AI presents a major risk to data security, and businesses that successfully confront this issue in 2025 will use a mix of clear governance policies, comprehensive workforce training and diligent detection and response.

Identity’s transformation (Wes Gyure, Executive Director, IBM Security Product Management)

How enterprises think about identity will continue to transform in the wake of hybrid cloud and app modernization initiatives. Recognizing that identity has become the new security perimeter, enterprises will continue their shift to an Identity-First strategy, managing and securing access to applications and critical data, including gen AI models. In 2025, a fundamental component of this strategy is to build an effective identity fabric, a product-agnostic integrated set of identity tools and services. When done right, this will be a welcome relief to security professionals, taming the chaos and risk caused by a proliferation of multicloud environments and scattered identity solutions.

Everyone must work together to manage threats (Sam Hector, Global Strategy Leader, IBM Security)

Cybersecurity teams will no longer be able to effectively manage threats in isolation. Threats from generative AI and hybrid cloud adoption are rapidly evolving. Meanwhile, the risk quantum computing poses to modern standards of public-key encryption will become unavoidable. Given the maturation of new quantum-safe cryptography standards, there will be a drive to discover encrypted assets and accelerate the modernization of cryptography management. Next year, successful organizations will be those where executives and diverse teams jointly develop and enforce cybersecurity strategies, embedding security into the organizational culture.

Prepare for post-quantum cryptography standards (Ray Harishankar, IBM Fellow, IBM Quantum Safe)

As organizations begin the transition to post-quantum cryptography over the next year, agility will be crucial to ensure systems are prepared for continued transformation, particularly as the U.S. National Institute of Standards and Technology (NIST) continues to expand its toolbox of post-quantum cryptography standards. NIST’s initial post-quantum cryptography standards were a signal to the world that the time is now to start the journey to becoming quantum-safe. But equally important is the need for crypto agility, ensuring that systems can rapidly adapt to new cryptographic mechanisms and algorithms in response to changing threats, technological advances and vulnerabilities. Ideally, automation will streamline and accelerate the process.

Data will become a vital part of AI security (Suja Viswesan, vice president of Security Software Development, IBM)

Data and AI security will become an essential ingredient of trustworthy AI. “Trustworthy AI” is often interpreted as AI that is transparent, fair and privacy-protecting. These are critical characteristics. But if AI and the data powering it aren’t also secure, then all other characteristics are compromised. In 2025, as businesses, governments and individuals interact with AI more often and with higher stakes, data and AI security will be viewed as an even more important part of the trustworthy AI recipe.

Organizations will continue learning the juxtaposition of AI’s benefits and threats (Mark Hughes, Global Managing Partner, Cybersecurity Services, IBM)

As AI matures from proof-of-concept to wide-scale deployment, enterprises reap the benefits of productivity and efficiency gains, including automating security and compliance tasks to protect their data and assets. But organizations need to be aware of AI being used as a new tool or conduit for threat actors to breach long-standing security processes and protocols. Businesses need to adopt security frameworks, best practice recommendations and guardrails for AI and adapt quickly — to address both the benefits and risks associated with rapid AI advancements.

Greater understanding of AI-assisted versus AI-powered threats (Troy Bettencourt, Global Partner and Head of IBM X-Force)

Protect against AI-assisted threats; plan for AI-powered threats. There is a distinction between AI-powered and AI-assisted threats, including how organizations should think about their proactive security posture. AI-powered attacks, like deepfake video scams, have been limited to date; today’s threats remain primarily AI-assisted — meaning AI can help threat actors create variants of existing malware or a better phishing email lure. To address current AI-assisted threats, organizations should prioritize implementing end-to-end security for their own AI solutions, including protecting user interfaces, APIs, language models and machine learning operations, while remaining mindful of strategies to defend against future AI-powered attacks.

There’s a very clear message from these predictions that understanding how AI can help and hurt an organization is vital to ensuring your company and its assets are protected in 2025 and beyond.

The post Cybersecurity trends: IBM’s predictions for 2025 appeared first on Security Intelligence.

The focus on data privacy started to quickly shift beyond compliance in recent years and is expected to move even faster in the near future. Not surprisingly, the Thomson Reuters Risk & Compliance Survey Report found that 82% of respondents cited data and cybersecurity concerns as their organization’s greatest risk. However, the majority of organizations noticed a recent shift: that their organization has been moving from compliance as a “check the box” task to a strategic function.

With this evolution in data privacy, many organizations find that they need to proactively make changes to their approach to set themselves up for the future. Here are five key considerations to get ready for the future of data privacy.

1. Create a process for staying up to date on new and evolving regulations

While data privacy is more than simply compliance, your organization must comply with all regulations first and foremost — or else risk fines and reputational damage. However, regulations are constantly being passed and changed, making it exceptionally challenging to stay up to date. As of September 2024, 20 states had consumer data privacy laws, with legislation pending in numerous other states. While the U.S. does not currently have a federal data privacy law, the American Privacy Rights Act is in the first stage of legislation.

As the data privacy regulation landscape continues to change, organizations must create a process to manage all pertinent regulations, which can be challenging for global companies. Because organizations must comply with the regulations of their customer locations, not the company’s locations, global businesses often find themselves bound by many different regulations. Organizations are increasingly turning to artificial intelligence (AI) with tools that monitor all relevant regulations and ensure compliance, which saves time and reduces fines.

2. Focus on balancing data privacy with analytics and AI goals

AI at the University of Pennsylvania’s Wharton School found that the percentage of employees who used AI weekly increased from 37% in 2023 to 73% in 2024. However, this significant and rapid increase in AI adoption has created significant data privacy issues. Top concerns include a lack of data transparency, new endpoints for vulnerabilities, third-party vendors and potential regulatory gaps. At the same time, businesses not using AI will likely quickly fall behind competitors in productivity and personalization.

Because not using AI is rarely the right business decision, organizations must take a strategic approach to creating a balance between business value and data security. While technology is part of the solution, platforms and systems cannot solve the challenges without a balanced approach. By creating processes and a framework that helps organizations evaluate risks and benefits, businesses can make smart business decisions with regard to data privacy. For example, a company may adopt automation throughout their organization using AI except in use cases that involve sensitive customer and employee data.

3. Consider privacy-preserving machine learning (PPML)

By using specific techniques in AI and analytics, organizations can reduce data privacy risks. Many organizations are turning to PPML, which is an initiative started by Microsoft to protect data privacy when training large-capacity language models. Here are the three components of PPML defined by Microsoft:

1. Understand: Organizations should conduct threat modeling and attack research while also identifying properties and guarantees. Additionally, leaders need to understand regulatory requirements.
2. Measure: To determine the current status of data privacy, leaders should capture vulnerabilities quantitatively. Next, teams should develop and apply frameworks to monitor risks and mitigation success.
3. Mitigate: After gaining a full picture of data privacy, teams must develop and apply techniques to reduce privacy risks. Lastly, leaders must meet all legal and compliance regulations.

4. Focus on data minimization

In the past, many businesses defaulted to keeping all — or at least most of — their data for a lengthy period of time. However, all data stored and saved must follow compliance regulations, causing many organizations to use a strategy referred to as data minimization.

Deloitte defines data minimization as taking steps to determine what information is needed, how it’s protected and used and how long to keep it. By taking this measured approach and determining which data to keep, organizations can reduce costs, make it easier to find the right data and improve compliance. Additionally, it’s easier and takes fewer resources to secure a smaller volume of data.

5. Create a culture of data privacy

Just like cybersecurity, data privacy is not simply the job of specific employees. Instead, organizations need to instill the mindset that every employee is responsible for data privacy. Creating a data privacy culture doesn’t happen overnight or with a single meeting. Instead, leaders must work to instill the values and focus over time. The first step is for leaders to become champions, express the shift in responsibility and “walk the walk” in terms of data privacy.

Because data privacy depends on team members following the processes and requirements specified, organizations must not simply dictate the rules but instead must explain the importance of data privacy. When employees understand the risks of not following the processes as well as the consequences to the organization and its consumers, they are more likely to comply.

Additionally, leaders should measure compliance with the processes to determine the current state and then the goal. By then offering incentives, organizations can help encourage compliance as well as stress its overall importance.

Start crafting your data privacy approach now

As your team focuses on planning for 2025 and beyond, now is the time to pause to make sure that your approach and goals align with where the industry is moving. Organizations that understand where data privacy is likely headed and take the steps needed to align their goals with the future of data privacy can be better prepared to more effectively gain business value from their data while still ensuring compliance.

The post Preparing for the future of data privacy appeared first on Security Intelligence.

For the last couple of years, a lot of attention has been placed on the evolutionary state of artificial intelligence (AI) technology and its impact on cybersecurity. In many industries, the risks associated with AI-generated attacks are still present and concerning, especially with the global average of data breach costs increasing by 10% from last year.

However, according to the most recent Cloud Threat Landscape Report released by IBM’s X-Force team, the near-term threat of an AI-generated attack targeting cloud computing environments is actually moderately low. Still, projections from X-Force reveal that an increase in these sophisticated attack methods could be on the horizon.

Current status of the cloud computing market

The cloud computing market continues to grow exponentially, with experts expecting its value to reach more than $675 billion by the end of 2024. As more organizations expand their operational capabilities beyond on-premise restrictions and leverage public and private cloud infrastructure and services, adoption of AI technology is steadily increasing across multiple industry sectors.

Generative AI’s rapid integration into cloud computing platforms has created many opportunities for businesses, especially when enabling better automation and efficiency in the deployment, provisioning and scalability of IT services and SaaS applications.

However, as more businesses rely on new disruptive technologies to help them maximize the value of their cloud investments, the potential security danger that generative AI poses is something closely monitored by various cybersecurity organizations.

Why are AI-generated attacks in the cloud currently considered lower risk?

Although AI-generated attacks are still among the top emerging risks for senior risk and assurance executives, according to a recent Gartner report, the current threat of AI technologies being exploited and leveraged in cloud infrastructure attacks is still moderately low, according to X-Force’s research.

This isn’t to say that AI technology isn’t still being regularly used in the development and distribution of highly sophisticated phishing schemes at scale. This behavior has already been observed with active malware distributors like Hive0137, who make use of large language models (LLMs) when scripting new dark web tools. Rather, the current lower risk projections are relevant to the likelihood of AI platforms being directly targeted in both cloud and on-premise environments.

One of the primary reasons for this lower risk has to do with the complex undertaking it will take for cyber criminals to breach and manipulate the underlying infrastructure of AI deployments successfully. Even if attackers put considerable resources into this effort, the still relatively low market saturation of cloud-based AI tools and solutions would likely lead to a low return on investment in time, resources and risks associated with carrying out these attacks.

Preparing for an inevitable increase in AI-driven cloud threats

While the immediate risks of AI-driven cloud threats may be lower today, this isn’t to say that organizations shouldn’t prepare for this to change in the near future.

IBM’s X-Force team has recognized correlations between the percentage of market share new technologies have across various markets and the trigger points related to their associated cybersecurity risks. According to the recent X-Force analysis, once generative AI matures and approaches 50% market saturation, it’s likely that its attack surface will become a larger target for cyber criminals.

For organizations currently utilizing AI technologies and proceeding with cloud adoption, designing more secure AI strategies is essential. This includes developing stronger identity security postures, integrating security throughout their cloud development processes and safeguarding the integrity of their data and quantum computation models.

The post Cloud Threat Landscape Report: AI-generated attacks low for the cloud appeared first on Security Intelligence.

With generative artificial intelligence (gen AI) on the frontlines of information security, red teams play an essential role in identifying vulnerabilities that others can overlook.

With the average cost of a data breach reaching an all-time high of $4.88 million in 2024, businesses need to know exactly where their vulnerabilities lie. Given the remarkable pace at which they’re adopting gen AI, there’s a good chance that some of those vulnerabilities lie in AI models themselves — or the data used to train them.

That’s where AI-specific red teaming comes in. It’s a way to test the resilience of AI systems against dynamic threat scenarios. This involves simulating real-world attack scenarios to stress-test AI systems before and after they’re deployed in a production environment. Red teaming has become vitally important in ensuring that organizations can enjoy the benefits of gen AI without adding risk.

IBM’s X-Force Red Offensive Security service follows an iterative process with continuous testing to address vulnerabilities across four key areas:

1. Model safety and security testing
2. Gen AI application testing
3. AI platform security testing
4. MLSecOps pipeline security testing

In this article, we’ll focus on three types of adversarial attacks that target AI models and training data.

Prompt injection

Most mainstream gen AI models have safeguards built in to mitigate the risk of them producing harmful content. For example, under normal circumstances, you can’t ask ChatGPT or Copilot to write malicious code. However, methods such as prompt injection attacks and jailbreaking can make it possible to work around these safeguards.

One of the goals of AI red teaming is to deliberately make AI “misbehave” — just as attackers do. Jailbreaking is one such method that involves creative prompting to get a model to subvert its safety filters. However, while jailbreaking can theoretically help a user carry out an actual crime, most malicious actors use other attack vectors — simply because they’re far more effective.

Prompt injection attacks are much more severe. Rather than targeting the models themselves, they target the entire software supply chain by obfuscating malicious instructions in prompts that otherwise appear harmless. For instance, an attacker might use prompt injection to get an AI model to reveal sensitive information like an API key, potentially giving them back-door access to any other systems that are connected to it.

Red teams can also simulate evasion attacks, a type of adversarial attack whereby an attacker subtly modifies inputs to trick a model into classifying or misinterpreting an instruction. These modifications are usually imperceptible to humans. However, they can still manipulate an AI model into taking an undesired action. For example, this might include changing a single pixel in an input image to fool the classifier of a computer vision model, such as one intended for use in a self-driving vehicle.

Data poisoning

Attackers also target AI models during training and development, hence it’s essential that red teams simulate the same attacks to identify risks that could compromise the whole project. A data poisoning attack happens when an adversary introduces malicious data into the training set, thereby corrupting the learning process and embedding vulnerabilities into the model itself. The result is that the entire model becomes a potential entry point for further attacks. If training data is compromised, it’s usually necessary to retrain the model from scratch. That’s a highly resource-intensive and time-consuming operation.

Red team involvement is vital from the very beginning of the AI model development process to mitigate the risk of data poisoning. Red teams simulate real-world data poisoning attacks in a secure sandbox environment air-gapped from existing production systems. Doing so provides insights into how vulnerable the model is to data poisoning and how real threat actors might infiltrate or compromise the training process.

AI red teams can proactively identify weaknesses in data collection pipelines, too. Large language models (LLMs) often draw data from a huge number of different sources. ChatGPT, for example, was trained on a vast corpus of text data from millions of websites, books and other sources. When building a proprietary LLM, it’s crucial that organizations know exactly where they’re getting their training data from and how it’s vetted for quality. While that’s more of a job for security auditors and process reviewers, red teams can use penetration testing to assess a model’s ability to resist flaws in its data collection pipeline.

Model inversion

Proprietary AI models are usually trained, at least partially, on the organization’s own data. For instance, an LLM deployed in customer service might use the company’s customer data for training so that it can provide the most relevant outputs. Ideally, models should only be trained based on anonymized data that everyone is allowed to see. Even then, however, privacy breaches may still be a risk due to model inversion attacks and membership inference attacks.

Even after deployment, gen AI models can retain traces of the data that they were trained on. For instance, the team at Google’s DeepMind AI research laboratory successfully managed to trick ChatGPT into leaking training data using a simple prompt. Model inversion attacks can, therefore, allow malicious actors to reconstruct training data, potentially revealing confidential information in the process.

Membership inference attacks work in a similar way. In this case, an adversary tries to predict whether a particular data point was used to train the model through inference with the help of another model. This is a more sophisticated method in which an attacker first trains a separate model – known as a membership inference model — based on the output of the model they’re attacking.

For example, let’s say a model has been trained on customer purchase histories to provide personalized product recommendations. An attacker may then create a membership inference model and compare its outputs with those of the target model to infer potentially sensitive information that they might use in a targeted attack.

In either case, red teams can evaluate AI models for their ability to inadvertently leak sensitive information directly or indirectly through inference. This can help identify vulnerabilities in training data workflows themselves, such as data that hasn’t been sufficiently anonymized in accordance with the organization’s privacy policies.

Building trust in AI

Building trust in AI requires a proactive strategy, and AI red teaming plays a fundamental role. By using methods like adversarial training and simulated model inversion attacks, red teams can identify vulnerabilities that other security analysts are likely to miss.

These findings can then help AI developers prioritize and implement proactive safeguards to prevent real threat actors from exploiting the very same vulnerabilities. For businesses, the result is reduced security risk and increased trust in AI models, which are fast becoming deeply ingrained across many business-critical systems.

The post Testing the limits of generative AI: How red teaming exposes vulnerabilities in AI models appeared first on Security Intelligence.