Abstract
The proliferation of malicious programs for the Windows operating system has highlighted the need for detection frameworks that protect these vital systems. Traditional malware detection approaches struggle to generalize across the diverse and evolving behavioral patterns that are demonstrated by modern malware. In order to gain an upper hand against malware developers, researchers have experimented with varying representations of programs that can be used in conjunction with machine learning models to automatically detect and mitigate malware threats.This work presents Graphene, a graph-based malware detection framework that leverages a RoBERTa-based classification model for malware detection. Graphene encodes program behavior through graph representations that are subsequently linearized into function-call sequences, enabling the use of advanced language-model architectures for malware detection. We evaluate Graphene on a dataset of approximately 150,000 Windows Portable Execubable (PE) files and benchmark its performance against two conventional machine learning models: a deep neural network (DNN) and a recurrent neural network (RNN). Experimental results demonstrate that Graphene significantly outperforms these baselines, achieving 94% accuracy with a false positive rate below 4%. These findings show the effectiveness of transformer-based architectures, when informed by graph representations, in advancing the state of the art automated malware detection.