CSE: A Novel Dynamic Obfuscation Based on Control Flow, Signals and Encryption

Abstract

This paper introduces a novel dynamic obfuscation method named CSE, which stands for Control flow, Signals, and Encryption. Obfuscation, often used invasively by malware authors, is repositioned here as a defensive strategy to protect software and sensitive information from security threats. The proposed CSE method innovatively combines three techniques: control flow obfuscation, signal-based obfuscation, and encryption of a management table. First, the hybridization of control flow and signal techniques alters and hides the program's control flow graph, replacing traditional call and jump commands with signals sent via the operating system. This transformation morphs the program's structure into an ambiguous star-like graph, complicating static analysis. Subsequently, a Management Table (MT) containing communication addresses is generated and encrypted using the Spritz stream cipher, a symmetric encryption algorithm, to safeguard this critical component from attackers.

A key achievement of this research is the comprehensive evaluation framework developed to assess the obfuscation's effectiveness. The authors propose a multi-objective evaluation based on six implementation metrics—such as program length, cyclomatic complexity, and nesting complexity—which are used to formulate two primary objective functions: ability and resiliency. The ability function measures the increase in program complexity after obfuscation, while resiliency evaluates the resistance to de-obfuscation attempts. Additionally, classic cost and Mishra similarity criteria are employed for comparative analysis. This rigorous evaluation approach allows for a nuanced understanding of the trade-offs between obfuscation strength and computational overhead.

Experimental validation demonstrates the competitive advantages of the CSE method. Tests were conducted on a small sample program and a benchmark dataset of 30 viruses from the VX Heaven repository. Compared to the BUZATU obfuscation method, CSE showed a significantly higher ability score (1.01 vs. 0.56) and a lower, more desirable resiliency score (-0.34 vs. 4.3), indicating stronger and more efficient obfuscation. Furthermore, when compared against nine state-of-the-art obfuscation techniques using the Mishra similarity criterion, CSE performed exceptionally well, particularly on the NGVCK virus dataset, achieving the lowest average similarity score (0.10906). This result signifies that programs obfuscated with CSE bear minimal resemblance to their original forms, a crucial metric for evasion and protection.

The paper makes a valuable contribution to cybersecurity by presenting a dynamic obfuscation technique that effectively resists static program analysis, a common method for malware detection. By intelligently combining control flow manipulation, signal-based execution, and robust encryption, CSE successfully increases the complexity of the obfuscated code while managing operational costs. The proposed evaluation methodology itself is a notable contribution, providing a structured way to quantify obfuscation quality. The findings suggest that CSE is a promising defensive tool for software protection, capable of "raising the bar" for attackers by making reverse engineering and analysis substantially more difficult without imposing prohibitive runtime penalties.