Malware Reverse Engineering

Instructor: Wu-chang Feng
Contact and discussion: Office hours: here
TA: Jeff De Lamare @dejef
TA Office hours: Mondays 10:30am-12:30pm (Fishbowl)
Required textbook


Week Topic Slides Labs Homework
1 Introduction
Basic Analysis
Chapter 1: Basic Static Techniques
Chapter 2: Malware Analysis in VMs
Chapter 3: Basic Dynamic Analysis


Install VM (EB 325)
1-1, 1-2
3-2, 3-4
Ch01, Ch03
2 Advanced Static Analysis
Chapter 4: x86 Assembly
Chapter 5: IDA Pro
Chapter 6: C code in Assembly
6-1, 6-2
Ch04, Ch06
Chapter 7: Malicious Windows Programs
Chapter 8: Debugging
Chapter 9: OllyDbg
4 Advanced Dynamic Analysis
Malware Functionality
Chapter 11: Malware Behavior
Chapter 12: Covert Launching
12-1, 12-3
Ch11, Ch12
Chapter 13: Data Encoding
Chapter 14: Network Signatures
Chapter 15: Anti-Disassembly
  Software Armoring

13-1, 14-1
15-1, 15-2
Ch13, Ch15
Chapter 16: Anti-Debugging
Chapter 17: Anti-VM Techniques

Talks #1,#2 | Slides #1,#2
16-1, 17-1 Ch16
Chapter 18: Packers and Unpacking
Special Topics
Chapter 19: Shellcode Analysis
Chapter 20: C++ Analysis
Chapter 21: 64-bit Malware
Final project: MetaCTF source walkthrough


MetaCTF level src
Ch18, Ch 21
     Symbolic execution/analysis

AFL labs
angr labs/CTF
00, 01, 02 
9      Symbolic execution/analysis
  angr labs/CTF 03, 05, 06
08, 10, 11, 13
     Symbolic execution/analysis
angr labs/CTF
15, 16, 17
Sat. 3/16
Final project due in D2L      
Tue. 3/19
Final CTF     Download
Sun. 3/24
Late homework deadline (for 80% credit)      


Lab notebook
As part of the lab work, you will maintain a lab notebook (a single Google Doc, Microsoft Office, or LibreOffice file) that will contain your write-ups of each lab. The write-up should include answers to questions asked and screenshots of the completed work (via gnome-screenshot, gimp, Print Screen, etc.). The notebook will be graded based upon thoroughness and clarity of the write-ups. While you are encouraged to work together on labs, each student should submit an individual notebook each week. Notebooks must be submitted in the associated D2L dropbox on Monday at 11:30pm the week after they are assigned. For example, 1-1 and 1-2 are assigned for the first week of class and are due on Monday the following week. Ensure your notebook is properly uploaded each week.

Homework (MetaCTF)
For homework, we will be applying the concepts learned in the labs to Linux binaries. Assignments are to be done individually at the following site: Binaries are unique to each student and no collaboration is allowed. The binaries implement a set of capture-the-flag challenges that require you to reverse engineer a set of binary executables. Each binary asks for a password that will unlock it and print "Good Job". While you will be running binaries on your own machine, answers to each should be submitted at the above site. The homework is intended to give you practice for the final exam CTF. To obtain full credit, levels associated with a particular chapter must be completed by Monday at 11:30pm the week after they are assigned. For example, Ch01 and Ch03 levels are assigned for the first week of class and are due on Monday the following week. Partial credit will be given at the end of the course for levels that are turned in late.

To download binaries on a linuxlab machine, do the following:
mkdir metactf; cd metactf; virtualenv env
source env/bin/activate
pip install requests bs4
python username password
If you wish, instead, to run the binaries on a Ubuntu VM of your own, directions for doing so are here. Note that you must ensure that radare2 and the 32-bit libraries are installed on the VM by performing
sudo apt-get install gcc-multilib radare2

Final project (Option #1: MetaCTF level)
(Undergraduates only, Individual projects) "See one, Do one, Teach one". The goal of this project will be to develop your own metamorphic challenge that can be used to help someone learn a topic covered in this course that is not currently addressed in your homework assignment. Source code for several of the early MetaCTF challenges will be given as a template. After doing so, you will create a narrated screencast that walks-through the source code of your level and a demo of how you would go about solving the level using an actual binary. Screencast software and submission are to be done via PSU's MediaSpace on the course's channel. After uploading your screencast to MediaSpace, ensure that the screencast is published onto the course channel. We will solve each others' projects after the final exam. At a minimum, levels should:
  • Be tied to a single topic or technique. Levels that span multiple techniques are not recommended.
  • Be generated metamorphically using methods equivalent to the current MetaCTF challenges.
  • Have the same format as the current MetaCTF challenges as described in the following instructions.
The rubric can be found here.

Final project (Option #2: Malware RCE)
(Individual or Group projects) The goal of this final project is to reverse-engineer a piece of malware of your choice using everything you have learned in this course. After doing so, you will create a narrated screencast that walks-through your process of obtaining the malware, running the analysis on it, and analyzing its behavior. Properly edit the screencast so that your analysis is under 20 minutes long per person. Screencast software and submission are to be done via PSU's MediaSpace on the course's channel. Resources for malware: If done as a group, each student will narrate the part of the malware he/she has reverse-engineered. The rubric can be found here.

Final CTF
A final CTF will be run on during finals week consisting of two parts. The first, graded part will be the exam CTF consisting of several CTF levels that are similar to the homework CTF. This is to ensure that you have mastered the knowledge and skills the course is attempting to provide. When students complete the first part, they will go on to the second part, in which students will attempt to solve the final project CTF levels submitted as final projects by others.

Course objectives

  • Understand the underlying mechanisms used by malware on compromised systems.
  • Understand counter-measures that detect malware.
  • Understand techniques malware uses to circumvent and evade detection and analysis
  • Develop skills to monitor the behavior of malware safely.
  • Develop skills to reverse-engineer malware


Attendance 10%
Homeworks 30%
Lab notebook 35%
Final project 15%
Final CTF 10%
Academic misconduct
  • Includes allowing another student to copy your work unless specifically allowed by the instructor.
  • Results in a grade of 0 for the assignment or exam.
  • Results in the initiation of disciplinary action at the university level.