• Set DYLD_INSERT_LIBRARIES to /usr/lib/libMallocDebug.A.dylib. This makes you link against libMallocDebug, which prints errors on bad accesses. Set a breakpoint on MDprintf to see a backtrace when this triggers.
• set MallocScribble and MallocPreScribble to YES to initialize new memory to 0xAA and freed memory to 0×55, which will also help finding out when you’re accessing bad memory.
• set MallocGuardEdges to YES to make it harder to smash memory and instead get a bad access exception.

Any other memory debug tricks you guys have?

Anyway, my recent posting on Intel Assembly Language Programming on Mac OS X kinda left you hanging in the air on this one. I didn’t say anything about debugging. Why? Because, honestly, I hadn’t got that far yet. Of course, the first bug didn’t leave me waiting for long, so here’s some handy tools if you want to debug your assembler program.

First, you need to compile your assembler source files with GCC using the -g option. That will give you debug symbols, which means the debugger will show you each line. Once you’ve done that, you just launch GDB, on the command line, as usual:

% gdb path/to/your/executable

You’ll get the GDB prompt you may have already seen in Xcode’s debugger console. Type in

(gdb) start

and it will jump to the start of your main function. GDB will always print the next line, and you can use the step command to execute it and see the next one. Of course, you may want to see what is in a particular register or at a particular memory address. Easy:

(gdb) print/x $eax

will print register eax as hexadecimal (that’s what the ‘/x’ means – there’s also ‘/d’ for decimal, ‘/c’ for character, ‘/s’ for string and ‘/t’ for binary). If you want to view a memory location, you use the ‘x’ command instead.

(gdb) x/1xb $eax

will take the address in $eax, and print 1 byte at that address in hexadecimal. The parts after the slash decode into /<count><displayFormat><type>. displayFormat is the same as the thing after the slash when you print, count is a number indicating how many to print, and type is ‘b’ for byte, ‘h’ for halfword (2 bytes) or ‘w’ for word (4 bytes).

Oh yeah and to get out of gdb again, the command is quit. Happy debugging!

Update: I recently realized I’d omitted two important little tricks from this description: If you don’t have debug symbols, you can still step through code. The relevant commands are

(gdb) si

which steps through code by one instruction (this even works with system functions etc.) and

(gdb) p/i

which disassembles and prints the current instruction.