Instructor: Wu-chang Feng
Contact and discussion: Office hours: here
Required textbook


Week Topic Slides Labs Homework
1 Introduction
Basic Analysis
Chapter 1: Basic Static Techniques
Chapter 2: Malware Analysis in VMs
Chapter 3: Basic Dynamic Analysis


Install VM (EB 325)
1-1, 1-2
Ch01, Ch03
2 Advanced Static Analysis
Chapter 4: x86 Assembly
Chapter 5: IDA Pro
Chapter 6: C code in Assembly
Ch04, Ch06
Chapter 7: Malicious Windows Programs
Chapter 8: Debugging
Chapter 9: OllyDbg
6-1, 6-2
4 Advanced Dynamic Analysis
Malware Functionality
Chapter 11: Malware Behavior
Chapter 12: Covert Launching
12-1, 12-3
Ch11, Ch12
Chapter 13: Data Encoding
Chapter 14: Network Signatures
Chapter 15: Anti-Disassembly
  Advanced Software Armoring talk

13-1, 14-1
15-1, 15-2
Ch13, Ch15
Chapter 16: Anti-Debugging
Chapter 17: Anti-VM Techniques
  BluePill talk #1
  BluePill talk #2

talk #1 | slides
talk #2 | slides
16-1, 17-1 Ch16
Chapter 18: Packers and Unpacking
Special Topics
Chapter 19: Shellcode Analysis
Chapter 20: C++ Analysis
Chapter 21: 64-bit Malware
Final project: MetaCTF walkthrough


slides | src
Ch18, Ch 21
     Symbolic execution/analysis

AFL labs
angr labs/CTF
9      Symbolic execution/analysis
  angr labs/CTF
     Symbolic execution/analysis
angr labs/CTF

Sat. 3/17
Final project due in D2L      
Mon. 3/19
Final CTF     Download
Fri. 3/23
Lab notebook due in D2L, Homeworks due      


Lab notebook
You will pair up with a lab partner to perform the in-class lab. As part of the lab work, you and your partner will maintain a shared lab notebook (a single Google Doc, Microsoft Office, or LibreOffice file) that will contain your write-ups of each lab. The write-up should include answers to questions asked and screenshots of the completed work (via gnome-screenshot, gimp, Print Screen, etc.). The notebook will be graded based upon thoroughness and clarity of the write-ups. You may also choose to work on labs individually.

Homework (MetaCTF)
For homework, we will be applying the concepts learned in the labs to Linux binaries. Assignments are to be done individually at the following site: Binaries are unique to each student and no collaboration is allowed. The binaries implement a set of capture-the-flag challenges that require you to reverse engineer a set of binary executables. Each binary asks for a password that will unlock it and print "Good Job". While you will be running binaries on your own machine, answers to each should be submitted at the above site. The homework is intended to give you practice for the final exam CTF. To download binaries on a linuxlab machine, do the following:
mkdir metactf; cd metactf; virtualenv env
source env/bin/activate
pip install requests bs4
python username password
If you wish, instead, to run the binaries on a Ubuntu VM of your own, directions for doing so are here. Note that you must ensure that radare2 and the 32-bit libraries are installed on the VM by performing
sudo apt-get install gcc-multilib radare2

Final project (Option #1: MetaCTF level)
"See one, Do one, Teach one". The goal of this project will be to develop your own metamorphic challenge that can be used to help someone learn a topic covered in this course that is not currently addressed in your homework assignment. Source code for several of the early MetaCTF challenges will be given as a template. We will solve each others' projects after the final exam. At a minimum, levels should:
  • Be tied to a single topic or technique. Levels that span multiple techniques are not recommended.
  • Be generated metamorphically using methods equivalent to the current MetaCTF challenges.
  • Have the same format as the current MetaCTF challenges as described in the following instructions.
The rubric can be found here.

Final project (Option #2: Malware RCE)
The goal of this final project is to reverse-engineer a piece of malware of your choice using everything you have learned in this course. After doing so, you will create a narrated screencast that walks-through your process of obtaining the malware, running the analysis on it, and analyzing its behavior. Screencast software and submission are to be done via PSU's MediaSpace on the course's channel. Resources for malware: The rubric can be found here.

Final CTF
A final CTF will be run on the last day of class consisting of two parts. The first, graded part will be the exam CTF consisting of several CTF levels that are similar to the homework CTF. This is to ensure that you have mastered the knowledge and skills the course is attempting to provide. When students complete the first part, they will go on to the second part, in which students will attempt to solve the final project CTF levels submitted by others.

Course objectives

  • Understand the underlying mechanisms used by malware on compromised systems.
  • Understand counter-measures that detect malware.
  • Understand techniques malware uses to circumvent and evade detection and analysis
  • Develop skills to monitor the behavior of malware safely.
  • Develop skills to reverse-engineer malware


Attendance 5%
Homeworks 40%
Lab notebook 30%
Final project 10%
Final CTF 15%
Academic misconduct
  • Includes allowing another student to copy your work unless specifically allowed by the instructor.
  • Results in a grade of 0 for the assignment or exam.
  • Results in the initiation of disciplinary action at the university level.