back to write-ups
PMA March 24, 2025

PMA Lab 16-2: Anti-Debugging

Analyze the malware found in Lab16-02.exe using a debugger. The goal of this lab is to figure out the correct password. The malware does not drop a malicious payload.

pmachapter-16

Lab 16-2

Analyze the malware found in Lab16-02.exe using a debugger. The goal of this lab is to figure out the correct password. The malware does not drop a malicious payload.

Questions:

  1. What happens when you run Lab16-02.exe from the command line?

    When trying to run the file from the command line it writes an usage message explaining how to run it. The file requires a parameter:

    image-20250322220929868

  2. What happens when you run Lab16-02.exe and guess the command-line parameter?

    While trying to provide a password argument, the program seems to validate the string and print if the password is correct or not.

  3. What is the command-line password?

    In order to find out the password x32dbg has been used. Two strings are pushed to the stack prior to function sub_402110 (abcd is the dummy argument used in order to test the code):

    image-20250322221802040

    It seems pretty obvious that sub_402110 may perform some sort of string compare in order to check if the two arguments are equal. This explains why after the call to the function eax is compared with 0, and depending on the value of eax, the code will either jump to 0040125A or continue to 0040124B. Depending on the path the code takes, two strings are pushed into the stack, and sub_402148 is called. This function is likely to be the print function:

    image-20250322222054548

    Let’s investigate:

    image-20250322222517117

    Here it is possible to see a function that at some point calls repe cmpsb which will compare two operands byte by byte, and continue until it finds a difference between them. This function doesn’t store any results, it just sets the ZF.

    Thus, it is possible to find out that the password is byrrp@ss:

    image-20250322223714885

    EDIT: After question 4 it is possible to see that sub_402110 is actually a strncmp function which takes 3 parameters, two strings and an integer. We could see the number 4 being pushed to the stack as well and in fact only 4 characters are being checked, therefore the password is anything that begins with byrr.

  4. Load Lab16-02.exe into IDA Pro. Where in the main function is strncmp found?

    strncmp can be found at 0040123A, which is the same sub_402110 function we’ve seen before in x32dbg. Thus, it is not a custom function.

    Looking at the syntax and what strncmp does, this function compares two strings up to the number of count characters.

    int strncmp(
       const char *string1,
       const char *string2,
       size_t count
    );
    

    Since this is being used in the context of the password, only the first 4 characters are being compared, meaning that in reality the password is actually byrr.

  5. What happens when you load this malware into OllyDbg using the default settings?

    Immediately after loading the file in Ollydbg, the process exits. The Malware is validating if there’s a window opened named OLLYDBG, using a TLS Callback:

    image-20250322231623393

    Using IDA Pro is easier to analyze the Callback:

    image-20250323021540848

    As it is possible to see in the Code, there’s a call to FindWindowA, with the parameter OLLYDBG.

    But before that, there’s a cmp and jnz logic. Here the program is validating if arg_4 is 1. This argument can have 4 values, and is related to when the callback is executed:

    • 0 - When the process exits / unloads
    • 1 - When the process starts
    • 2 - When a new thread starts
    • 3 - When a thread exits

    So here, the Callback is checking if the argument is 1, meaning that if the Process is starting it will take one branch, otherwise it will take the other. If the program is starting, FindWindowA is called. This function retrieves a handle to the top-level window whose class name and window name match the specified string. If this function succeeds the return value is a handler, otherwise it will return Null. As it possible to see the code is using test eax, eax, and a jump zero instruction to validate if a Window has been found.

    If found, the code will exit, taking the left branch, otherwise the right branch is taken:

    In this branch the program is checking again the context in which the TLS Callback is running, in this case it is comparing the value 2 (when a new thread starts). If the value is 2 sub_401020 is executed, otherwise it will be skipped.

    Here it is possible to see that sub_401020 is employing another Anti-Debugging technique:

    image-20250322225551729

    This function alters the value of the last error to 0x3039 using SetLastError. Then it calls OutputDebugStringA, which is a Windows API that sends a string to the debugger output window (only if a debugger is present). Basically if a debugger is being used the LastError will be changed to the content of OutputString, otherwise it will remain the same as before (which in this case is 0x3039).

    The code is then using cmp eax, [ebp+dwErrCode], which was previously set to 0x3039. If both operands are different the jump will occur, if not byte_40A968 is changed to 1 (previously was 0):

    image-20250323012240741

    In order to understand the impact that this flag may have on code execution we can use Cross-References:

    image-20250323015758520

    It is possible to see that this flag is being used in the StartAddress of the created Thread (00401090), which means that the flag is also used to alter the (possible fake) password:

    image-20250323015857714

  6. What is unique about the PE structure of Lab16-02.exe?

    The Sample contains a .tls section:

    image-20250322224528619

  7. Where is the callback located? (Hint: Use CTRL-E in IDA Pro.)

    The Call back is located at 00401060:

    image-20250322224616843

  8. Which anti-debugging technique is the program using to terminate immediately in the debugger and how can you avoid this check?

    The program is being terminated if FindWindowA finds a window called Ollydbg. (explained in 5)

    This can be avoided by changing the Debugger settings and setting it to break at the System Breakpoint and not in Main, otherwise the TLS Call back will execute before we have the chance to bypass it and the process will be terminated.

    This Callback is executed on Process Start and Thread Start, however, the part that may exit the process only runs if the Callback is executed on Process Start.

    Here’s an example of a successful breakpoint at the start of the TLS Callback, meaning that we’re able to step over the instructions and alter the course of the function, and thus bypassing it:

    image-20250322230633364

  9. What is the command-line password you see in the debugger after you disable the anti-debugging technique?

    The command-line password seen in the debugger is: byrr

    image-20250323003249735

    EDIT: After seeing question 11, I understood this question. (more details on 11)

    The answer depends on whether or not the program managed to identify it is running in a debugger.

  10. Does the password found in the debugger work on the command line?

    EDIT: After seeing question 11, I understood this question. (more details on 11)

    The correct password is byrr. Assuming that the debugger was found by the program and an incorrect password was found, this wouldn’t work on the command line since when the program was executing via the command line it wouldn’t generate the same password (because this time the value of the BeingDebugged flag would be different).

    image-20250323005534809

  11. Which anti-debugging techniques account for the different passwords in the debugger and on the command line, and how can you protect against them?

    When I started the exercise, I was using x32dbg with ScyllaHide. I found the FindWindowA call, looking for Ollydbg, which in a way I was bypassing already since I wasn’t using it:

    image-20250323003510413

    However, because of ScyllaHide, I completely missed out on what was going on.

    Starting all over again, and looking at question 11, its known that the program may have multiple passwords. This can be confirmed by using Ollydbg:

    image-20250323003623919

    In order to understand what is going on, we need to look at the buffer of the string (byte_408030):

    image-20250323003809369

    If we use the Cross-References in IDA, it is possible to see that the buffer is being used a lot, specially in string operation instructions:

    image-20250323003830634

    The majority of these instructions are in 00401090 which is the startAddress of the Thread that is created prior to the string compare. Looking closely to the instructions, one stand out: mov ebx, large fs:30h:

    image-20250323003936676

    Here the program is using the BeingDebugged flag:

    image-20250323004123756

    This means that depending on the BeingDebugged flag, the password will be different because the byte of the flag is used within the whole algorithm that decodes the password.

    By disabling ScyllaHide, the password changed, and it is now byqr, (previously byrr).

    Parameters of sub_402110 with BeingDebugged flag active:

    image-20250323005322227

    Parameters of sub_402110 with BeingDebugged flag disabled:

    image-20250323005422502

    For some reason the password is different when seen inside Ollydbg:

    image-20250323010528607