Abstract
Network defense has moved beyond simple perimeter filtering because modern attacks are faster, stealthier, and more adaptive than traditional rule sets can handle. This paper presents a layered security framework built around pfSense firewall, Snort intrusion detection, and the Isolation Forest anomaly detection algorithm. The design combines three complementary capabilities: access control at the network edge, signaturebased detection for known attacks, and machine learning-based anomaly detection for previously unseen behavior. The result is a practical architecture that can identify both documented threats and suspicious traffic patterns that do not match existing signatures. pfSense acts as the gateway and enforces segmentation between trusted and untrusted zones. Snort inspects live packets, compares them against rule sets, and generates alerts for malicious payloads, scans, and suspicious protocol behavior. Isolation Forest processes log-derived features such as packet size, source and destination addresses, connection rate, and timing patterns to detect deviations from normal activity. Because it is unsupervised, the model can be applied even when labeled attack data is limited. The full system is implemented in a virtualized environment so that it remains cost-effective, repeatable, and suitable for academic use. The proposed framework improves visibility, reduces response time, and strengthens the ability to defend against unknown threats. It also provides a basis for automated containment, where suspicious hosts can be blocked through firewall rule updates after detection and validation.