AI-Driven Malware Reverse Engineering Pipeline

# Malware Reverse Engineering is no longer a human problem!

I recently presented a keynote on the state of malware reverse engineering and analysis in the AI era. But instead of starting with slides, I started with silence.

I shared my screen, I launched my personal threat research pipeline, and submitted an unknown malware sample I had never analyzed before. Then I continued the talk as if nothing had happened.

While I was speaking, my team of AI agents was working in the background. They were performing static analysis, reverse engineering, enrichment, pivoting, YARA testing, and report generation autonomously.

By the time I returned to the demo about 30 minutes later, the analysis was finished. The screenshot below shows my malware analysis board and my team of agents. 

This article explains how I built a system like this over years of experimentation, and why I believe malware reverse engineering is becoming an obsolete skill.

Disclaimer: The systems and tools described in this article are part of my personal research and were developed independently. They do not represent the work, views, or positions of my employer.

## Where This Journey Started

First, a bit of context. I am Thomas Roccia aka @fr0gger_. I have been working in the security industry for more than 15 years now (exactly 18). I started my career doing IT support, while spending my free time learning hacking and reverse engineering since I was younger.

Back in the day, the resources to learn hacking and reverse engineering were very limited. No YouTube videos, no online courseware, no lectures, only IRC, small forums, crackme, and the early hacking communities. 

To get access to knowledge, you had to earn it.

I grew up getting what I wanted by searching, because asking without doing your own research was not acceptable. You know, the RTFM culture.

A couple of years later, I moved into pentesting and red teaming in the early days, coding my own offensive tools and scripts.

After 5 years and a Master's degree in computer science, I moved to McAfee, working on incident response across the world and investigating high stakes incidents in very sensitive environments.

After 3 years of travelling the world, I moved to McAfee Labs, focusing more on threat intelligence and malware reversing, and collaborating with law enforcement.

I have been on the frontline of some of the biggest incidents, to name a few: NotPetya, Triton, Olympic Destroyer, or SolarWinds.

In 2021, I joined Microsoft as a Malware Analyst, in the Microsoft Defender team. My role focused on security research, improving detections, and tracking threat actors and their operating methods.

To me, a good CTI researcher must understand all facets of the field: how to exploit a vulnerability and pivot, how to code your own tools, how to respond to an incident and deep dive into logs, how to reverse a sample and understand the code and operating method of an entire breach.

Reverse engineering used to be the hardest part of the work: reversing obfuscated code, identifying data structures, patterns, and tricks. I enjoyed it because I liked pulling things apart to understand how they worked.

I learned it the hard way over many years.

## The Moment AI Became Useful for Malware Analysis

During my presentation, I explained the work I have done over the past 4 to 5 years, which focused on leveraging AI for tasks I used to perform manually. I experimented very early with LLMs.

Fun fact: the first time I used GPT was in 2018 with GPT-1. I used it for YARA rule generation, but back then it was not what it is today. After a week of testing, I decided to give up because the technology was not there yet.

I have always been skeptical of AI, but I also always had an interest in what could be achieved with machine learning. So I learned it for malware similarity and malware classification. I never called machine learning or deep learning "AI". In fact, I intentionally stayed away from the term AI because I associated it with marketing hype for years.

Then in 2022, when ChatGPT came out, I saw something different. After experimenting with it for jokes or poems, like everyone else, I asked myself: how can I use this for security?

A week later, I created a tool called IATelligence. 

It was very simple: it extracted the IAT of a PE sample and automatically mapped the imported functions to the MITRE ATT&CK matrix.

It may sound trivial today, but it was a real problem to solve back then. This tool went viral online because it showed an early glimpse of what LLMs could do for malware analysis.

And it was just the beginning.

## Adding Memory and Context to Analysis

A couple months later I experimented with disassembly code and GPT explanation. I used it in my workflow with Capstone and Unicorn, specifically for malware analysis and code analysis.

I also leveraged GPT with Frida for binary instrumentation. Fun fact: I presented this work to the same conference in Feb 2023 where I did this keynote.

I then started to experiment with RAG by embedding malware analysis documentation and later by embedding disassembly code.

RAG, or retrieval augmented generation, is a way to add external data to the model, but not only. The retrieval mechanism is what makes RAG more interesting. 

The retrieval process allows you to retrieve the most similar or relevant data based on the information you provide. 

This becomes very useful when studying malware. For example, you may want to identify a specific section related to an algorithm such as encryption, a tactic or technique such as persistence or evasion, or a pattern such as IOCs.

## When Analysis Became a System Instead of a Task

Around mid 2023, I started experimenting with agents specifically for malware analysis and threat intelligence.

I created multiple agents for this: an agent connected to the Unprotect Project the open database I created for malware evasion technique identification, an agent for static analysis with parsing features, agents for IOC extraction and enrichment, and agents dedicated to reverse engineering using Capstone. (https://unprotect.it/)

I also demonstrated some of this work at the SANS CTI Summit, where I presented my MSTICPy agent.

At some point, I decided to merge all my agents into a single system. By the end of 2023, I had created my first multi agent system for malware analysis, significantly speeding up my pipeline.

But when I started talking about it, the reaction was always the same: it would not work, it would never work, and it was not going to replace reverse engineering. The industry was not even talking about AI Agents yet.

Even if it was not perfect, I knew I was onto something.

## The Missing Piece for Shared Agent Tooling

During 2024, I continued experimenting with AI and agents, improving my malware analysis system by using the latest models and testing different frameworks: AutoGen, CrewAI, LangChain, or LlamaIndex.

One of the main issues I faced was reusing my tooling across the different AI systems I built. I eventually created a plugin system, but it was a very technical approach that still required writing your own tools if you wanted to use it.

In November 2024, Anthropic released MCP, the Model Context Protocol. I immediately saw the value: the ability to share tooling for AI so the community could reuse the tools I had built.

I created my first MCP tool a couple of weeks after the release for CTI, followed by another for static malware analysis and another for YARA creation. 

Fun fact: MCP gained widespread attention about four months later, in March 2025, but by then I had already implemented several MCP servers in my own pipelines.

## Coding Assistants Are Actually Autonomous Agents

The year 2025 was the year of AI coding assistants: Claude Code, Codex, OpenCode, and others. But something many people missed is that coding assistants are not just coding assistants.

They are general AI agents.

Yes, they are designed to create code, but they can also execute that code.

I quickly saw the opportunity to connect them to my pipelines. I forked Codex and created my own version for malware analysis called MalCodex. The difference was that MalCodex was specifically designed for malware analysis. It was connected to my reversing tools, my CTI APIs, and my virtual machine. I coded MalCodex using Codex.

It was built for malware analysis, but fundamentally it had the same capabilities as coding agents. It could access my VM terminal and execute the tools I configured in the environment. 

The only things I added were design work around system prompts and access to the malware analysis tools and scripts I was already using.

It worked amazingly well. 

I could reverse engineer a sample, ask about specific sections, validate the output, ensure the results were accurate, and generate the report without spending a week writing it.

## Turning Prompts Into Capabilities

A little later in 2025, agent skills were introduced.

Agent skills are a simpler way to load tools or capabilities that an AI agent can use. The main difference compared to MCP is that a skill is loaded on the fly when the agent needs it, saving context window space. Another difference is that MCP can be accessed remotely, while skills need to be downloaded locally on your machine.

Agent skills are markdown files, essentially well crafted prompts designed to guide an LLM for a specific task.

But skills are much more than markdown files. You can attach scripts, resources, and assets.

This matters because that is where the real value of skills lies. Otherwise, you are just loading a prompt.

For example, if you create a skill for YARA rule creation using only markdown, you provide guidelines for your LLM to generate rules. However, if you do not validate or test the generated rules, there is a chance the LLM produces a rule that does not work. If you instead add scripts that test and compile the rule, the agent can automatically correct itself when the generated rule fails, thanks to the embedded scripts attached to the skill.

By the way, this skill exists. I added my YARA validation tool to improve the original version from @cyb3rops.

Agent skills are powerful because you can add capabilities to your agent with a well crafted prompt and a few scripts. Skills are tuned and optimized for agent execution.

## When AI Performed the Analysis Alone

At this point, I think you may have realized that fully autonomous malware reverse engineering is now within reach.

So let me explain the demo I showed.

I started my talk by saying this:

"Hi everyone, I am super happy to be here today. Before starting my talk, I will share my screen and do something without talking. I want you to really pay attention to what I am doing."

I then shared my screen, woke up my team of agents, and gave them an unknown sample that I had never analyzed before.

Then I continued my talk. I spoke about my journey and all the research I had done up to that point.

By the end of my talk (about 30 minutes later), I returned to the demo. My team of agents, specialized in malware analysis, had completed a full investigation of a sample I had no prior knowledge of.

My team performed the entire process: the static analysis I used to conduct manually to extract information about the binary, OSINT enrichment through multiple CTI tools, the complete reverse engineering process with a focus on the most interesting functions, detection of evasion techniques and persistence mechanisms, analysis of network connections, pivoting through specific artifacts to identify other campaigns or related families, IOC extraction, and the creation of a complete diagram drawing graph, as I like to make in my reports, to provide a single clear view of what the sample is doing.

The system also generated and tested YARA rules, uploaded them to external services for automated hunting, mapped activity to MITRE ATT&CK, produced recommendations, and assembled everything into a detailed report grounded in the original data.

(The full report is too long and detailed to reproduce here.)

All of that in about 30 minutes.

30 minutes.

I could spend a full week reversing a single sample, and another week writing the report. I could also spend multiple days crafting the diagram.

30 minutes.

## How my Multi Agent Malware Analysis System Works

The system I used is a combination of my own arsenal, tooling I crafted over the years, a set of skills I created for malware analysis, different models for optimization, a multi agent architecture designed to distribute context without losing information along the way, and an @openclaw gateway.

When I trigger a malware analysis board by submitting a sample, the lead agent creates multiple tasks and distributes the work across other agents I have created.

All tasks are clearly defined by the lead agent and assigned to specialized agents. Tasks depend on one another, and some cannot start until previous ones are completed.

When a task is finished, the assigned agent leaves a comment containing analysis and observations that are later used to generate the final report. 

Once all tasks are completed, a full threat report is automatically produced with all the details. I can also review individual tasks if I want to validate specific parts of the analysis.

To be clear, this system was already possible two years ago.

The difference today is that models are far more reliable. Building the same system now requires significantly less engineering effort and complexity, less design work around output verification, and a simpler overall architecture and optimization process.

## Can Autonomous Malware Analysis Be Trusted?

But let's talk about the results themselves, because I can already hear people saying: yes, but "there are hallucinations, we cannot trust it, or it might miss important things".

For the purpose of my demo, I configured my agent team to work autonomously without me in the loop. However, if you look at my threat research pipeline, I have an "Review" column.

If I enable human in the loop (me), every task moves to that column before being marked as completed so I can review it. If I think the result is not sufficient or not accurate, I can leave a comment with my observations, and the agent will refine the analysis based on my feedback until I am satisfied with the outcome.

Okay, okay, I can still hear some skeptical people at the back of the room. What if the malware is packed or heavily obfuscated, or something very unusual? What about prompt injection or LLM exploitation? What if my agent goes rogue?

For obfuscation or packing, my system is designed to detect it and work around it. If the case is completely unknown or unsolvable, that is where human in the loop becomes valuable. But in the majority of cases, the agents are capable of finding a solution on their own.

For particularly tricky situations, which let's be honest are not common, the same approach applies. 

The system is also designed to evolve. You can add your own tooling and your own knowledge, through skills.

Regarding hallucinations, first, models are far less prone to hallucinations than they were two years ago. Second, this is where understanding how an AI system works becomes important. 

Running an autonomous system without visibility is like flying blind and hoping you are still on course. That is not how this works.

You need visibility into exactly what your agents are doing: which skills they use, which tools they call, what resources they fetch, which prompts are executed, and what code is produced.

And you also need guardrails.

@HackingLZ mentioned as an example the magic string for Claude that can break a pipeline, and this is one of a valid concern, as malware already leverages these types of evasion. 

I use my own system with Nova rules to detect this behavior. I can flag it and raise an alert to identify where my pipeline breaks.

Nova is versatile and customizable to also detect prompt injection attempts, bypass techniques and more.

At this stage, I hope you can see what I see.

## Is This the End of Malware Reverse Engineering?

Yes and no.

First, I could not have built a reliable system like this one without a deep understanding of threat actors, malware analysis, and reverse engineering.

Second, for now, human in the loop still remains necessary for probably 10 percent of cases. Until when? Probably not for long.

Third, without a clear system to monitor your agents, secure them, and maintain visibility into everything they do, your system remains vulnerable and cannot be trusted. What we discuss here is exactly what I teach in my training.

I love reverse engineering.

I worked hard to earn this skill through years of effort and sleepless nights. It gave me a career.

The intellectual challenge of understanding assembly code is deeply rewarding. The thrill of breaking an obfuscation mechanism, dumping an unpacked binary from memory, or uncovering what a sample is truly doing is incredible. There is something deeply satisfying about peeling back the layers of a malicious piece of code until you understand its intent.

But this skill I crafted over more than 15 years is now becoming obsolete. And if it is not already, it will be very soon.

If software development becomes cheap, reverse engineering will be too. 

## What Malware Analysts Should Do Now

What should you do now? Is it the end of your career?

No.

First, you can still enjoy traditional malware reversing for a couple more years. The industry is still catching up, and the system I demonstrated will realistically become widely available within a year or two. (I have not tried Codex for malware analysis yet, though it could be sooner.) OpenAI, if you are reading this, I would love a preview :)

But this is a transition period.

If you have not started yet, you should start learning these technologies now. Fast. Faster than before. I still see people discovering what MCP or RAG are today. The pace is accelerating, and if you wait too long, you will be left behind.

The opportunities now lie in orchestration, in the architecture of these systems, and in their security and monitoring, as we discussed.

This is a new field and we are laying the security foundation.

## Stop Waiting. Start Building.

Of course, I could not talk about this topic without mentioning my training "Practical GenAI for CTI". (https://securitybreak.io/training-genai)

Everything I described in this article is exactly the type of system we build during the training.

In my @BlackHatEvents training, one of the core labs focuses on building a multi agent malware analysis system from scratch. You learn how agents collaborate, how tasks are orchestrated, how analysis pipelines are structured, and how to add monitoring and security around autonomous workflows.

The goal is to understand how these systems actually work so you can build and adapt them yourself.

If you want to catch up with where the field is going, this training will give you the tools and knowledge to start building today.

So now you have a choice: You can wait until these systems become standard, or you can start building them today. 🤓

Thanks for reading, I am sure this article will spark discussions. That is the point.