SU-CTF Quals 2014 - Reverse Me (200pts) writeup

The challenge description was: Reverse me and find a valid serial number!

We had to download a Windows binary and reverse it to get the proper serial. After opening the file with PEiD I noticed it was packed with ASPack 2.12:

Unpacking these with Ollydbg is pretty straight forward and covered by a lot of articles online. You can read this one on ring-zero.com which is well explained: Finding OEP and unpacking Malware packed with ASPACK 2.12

Now that I have my unpacked binary it's time to open it in Ollydbg and see what is going on. I first run it to get an idea of its behavior to understand what I'm looking for:

We need to provide an email address and a serial number to register. Let's see what is going on when we provide random input:

We get a message box when the serial is not correct. This is good it makes it easier to spot where in the code the serial validation is done. And it's especially easier if it's making Win32 API calls. In Ollydbg after getting all intermodular calls I found a call to MessageBoxA and set a breakpoint on it. Running the application and providing random input again should get us right before the Registration Failure message pops up.

004012EB   > 6A 40          PUSH 40                                  ; /Style = MB_OK|MB_ICONASTERISK|MB_APPLMODAL
004012ED   . 68 38FD4000    PUSH ReverseM.0040FD38                   ; |Title = "Registeration"
004012F2   . 8D85 C0FEFFFF  LEA EAX,DWORD PTR SS:[EBP-140]           ; |
004012F8   . 50             PUSH EAX                                 ; |Text
004012F9   . 53             PUSH EBX                                 ; |hOwner
004012FA   . FF15 00C14000  CALL DWORD PTR DS:[<&user32.MessageBoxA>>; \MessageBox

Before this point there is obviously a check for the validity of our serial, let's get higher in the disassembly and see what is going on before that point:

00401199   . 2BCA           SUB ECX,EDX
0040119B   . 83F9 10        CMP ECX,10
0040119E   . 0F85 FD000000  JNZ ReverseM.004012A1
004011A4   . 80BD C0FDFFFF >CMP BYTE PTR SS:[EBP-240],42    
004011AB   . 0F85 F0000000  JNZ ReverseM.004012A1
004011B1   . 0FBE85 CFFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-231]
004011B8   . 83C0 42        ADD EAX,42                      
004011BB   . 3D 9B000000    CMP EAX,9B
004011C0   . 0F85 DB000000  JNZ ReverseM.004012A1
004011C6   . 0FBE8D C1FDFFF>MOVSX ECX,BYTE PTR SS:[EBP-23F]
004011CD   . 8D41 FD        LEA EAX,DWORD PTR DS:[ECX-3]
004011D0   . 83F8 57        CMP EAX,57                      
004011D3   . 0F85 C8000000  JNZ ReverseM.004012A1
004011D9   . 0FBE85 CEFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-232]
004011E0   . 03C1           ADD EAX,ECX
004011E2   . 3D 9B000000    CMP EAX,9B                      
004011E7   . 0F85 B4000000  JNZ ReverseM.004012A1
004011ED   . 0FBE8D C2FDFFF>MOVSX ECX,BYTE PTR SS:[EBP-23E]
004011F4   . 8D41 01        LEA EAX,DWORD PTR DS:[ECX+1]
004011F7   . 83F8 3A        CMP EAX,3A                      
004011FA   . 0F85 A1000000  JNZ ReverseM.004012A1
00401200   . 0FBE85 CDFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-233]
00401207   . 03C1           ADD EAX,ECX
00401209   . 3D 9B000000    CMP EAX,9B                      
0040120E   . 0F85 8D000000  JNZ ReverseM.004012A1
00401214   . 80BD C3FDFFFF >CMP BYTE PTR SS:[EBP-23D],64    
0040121B   . 0F85 80000000  JNZ ReverseM.004012A1
00401221   . 0FBE85 CCFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-234]
00401228   . 83C0 64        ADD EAX,64
0040122B   . 3D 9B000000    CMP EAX,9B                      
00401230   . 75 6F          JNZ SHORT ReverseM.004012A1
00401232   . 80BD C4FDFFFF >CMP BYTE PTR SS:[EBP-23C],6D
00401239   . 75 66          JNZ SHORT ReverseM.004012A1
0040123B   . 0FBE85 CBFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-235]
00401242   . 05 81000000    ADD EAX,81
00401247   . 3D C8000000    CMP EAX,0C8                     
0040124C   . 75 53          JNZ SHORT ReverseM.004012A1
0040124E   . 0FBE8D C5FDFFF>MOVSX ECX,BYTE PTR SS:[EBP-23B]
00401255   . 8D41 D3        LEA EAX,DWORD PTR DS:[ECX-2D]
00401258   . 83F8 44        CMP EAX,44                      
0040125B   . 75 44          JNZ SHORT ReverseM.004012A1
0040125D   . 0FBE85 CAFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-236]
00401264   . 03C1           ADD EAX,ECX
00401266   . 3D AA000000    CMP EAX,0AA                     
0040126B   . 75 34          JNZ SHORT ReverseM.004012A1
0040126D   . 80BD C6FDFFFF >CMP BYTE PTR SS:[EBP-23A],34    
00401274   . 75 2B          JNZ SHORT ReverseM.004012A1
00401276   . 0FBE85 C9FDFFF>MOVSX EAX,BYTE PTR SS:[EBP-237]
0040127D   . 83C0 34        ADD EAX,34
00401280   . 3D 9B000000    CMP EAX,9B                      
00401285   . 75 1A          JNZ SHORT ReverseM.004012A1
00401287   . 80BD C7FDFFFF >CMP BYTE PTR SS:[EBP-239],63
0040128E   . 75 11          JNZ SHORT ReverseM.004012A1
00401290   . 0FBE85 C8FDFFF>MOVSX EAX,BYTE PTR SS:[EBP-238]
00401297   . 83C0 63        ADD EAX,63
0040129A   . 3D 9B000000    CMP EAX,9B                      
0040129F   . 74 1A          JE SHORT ReverseM.004012BB

And there it is, the part where the serial is checked and generated. You can see that it first checks if the serial is 0x10 (16) characters long, check for the validity of each byte in a different order and always jump to the same address when both values are not equal: JNZ SHORT ReverseM.004012A1. Now this is the fun part, stepping through each check and understand which value I should have input. The fact that they are not checking the serial from the first character to the last makes it a bit more challenging, but it's nothing really hard when you specificy a pattern in your own input. You can also understand which letter in your serial EBP is pointing to by looking at the value it is substracted with.

Time to reverse the serial, let's place a breakpoint at the beginning of this validation at address 004011A4 and input a serial like: 0123456789ABCDEF and try to register again. Remember our serial must be 16 characters long to pass the first validation step.

004011A4   . 80BD C0FDFFFF >CMP BYTE PTR SS:[EBP-240],42             ;  pass[0]: B

First character of our serial should be 0x42, which is the letter B.

004011B1   . 0FBE85 CFFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-231]
004011B8   . 83C0 42        ADD EAX,42                               ;  pass[15]: Y
004011BB   . 3D 9B000000    CMP EAX,9B

The program jumped to the last letter of our own serial, adds 0x42 and compare the result with 0x9B. To get the valid character we just have to substract 0x9B with 0x42 and the result is 0x59, which is the letter Y.

004011C6   . 0FBE8D C1FDFFF>MOVSX ECX,BYTE PTR SS:[EBP-23F]
004011CD   . 8D41 FD        LEA EAX,DWORD PTR DS:[ECX-3]
004011D0   . 83F8 57        CMP EAX,57                               ;  pass[1]: Z

Here it stores the value pointing to the second character of our serial in ECX, substract by 0x3 and store the result in EAX which is then compared to 0x57. To get the proper value we just have to add 0x3 to 0x57, which is 0x5A. The letter Z.

004011D9   . 0FBE85 CEFDFFF>MOVSX EAX,BYTE PTR SS:[EBP-232]
004011E0   . 03C1           ADD EAX,ECX
004011E2   . 3D 9B000000    CMP EAX,9B                               ;  pass[14]: A

Moving to the 15th letter of our input and storing it in EAX, adding the value of ECX which is the value of the second correct character in the serial previously checked ( 0x5A ) and comparing the result with 0x9B. To get the proper value we just have to substract 0x9B with the value of ECX ( 0x5A ) giving us 0x41, which is the letter A.

The rest of the serial is calculated in a similar way and after stepping through each validation step I ended with the following serial: BZ9dmq4c8g9G7bAY

After entering this serial and clicking the OK button to register we are greeted with this message:

We got our flag:

BZ9dmq4c8g9G7bAY