Well, I'm a little confused as to just how the N64's DENC actually works in terms of signalling. This is what I think I've figured out so far. The video data pins are a little bit of an assumption, given that disconnecting them results in a change in the image that's consistant with disconnecting a bit. However, disconnecting pins 1 and 2 result in a signal that my display considers to be invalid. This might indicate that there's sync data being passed as well.
My theory at the moment is that the first 7 pins are actually all data. This data is clocked in four phases (hence the constant 12.5MHz signal on pin 9 vs the 50MHz shared clock). As there does not appear to be any seperate vsync/hsync, I'm assuming that something in one of the sets of data clocked in is used to indicate v and h sync.
pin 1 - d0?video data
pin 2 - d1?video data
pin 3 - d2?video data
pin 4 - d3?video data
pin 5 - d4?video data
pin 6 - d5?video data
pin 7 - d6?video data (MSB)
pin 8 - +3v
pin 9 - ? 12.5MHz signal
pin 10 - GND
pin 11 - shared clock with RCP (50MHz)
pin 12 - GND
pin 13 - GND
pin 14 - ?
pin 15 - ?
pin 16 - GND
pin 17 - +5v
pin 18 - C out
pin 19 - Y out
pin 20 - Composite out
pin 21 - ?
pin 22 - ?
pin 23 - ?
pin 24 - GND
Unfortunately, in this process I've managed to kill the DENC. However, I might see if I can validate my theories by trying to extract the four phases. Don't hold your breath on this though!