Funny, I experienced this problem a couple of times, too.
Actually, I never managed to get recording running on my machine. Either it halts shortly before 100% or I get an “engine crashed” error message.
Currently, I cannot reproduce this (I just get crahses instead).
Don't know too much about fraps, but from recording video under linux, I found that things that can improve video recording framerate is to use a smaller window size (kind of obvious, but, might not occur to people - can zoom out to catch more of the scene in the game) and to turn off compositing in the window manager which can help depending on your graphics card/driver...
The fact that recording is failing kinda sucks, but I've had mixed success with it myself :-/ I think the lib we are using is a bit unreliable.
--
Oh, what the heck. 1PLXzL1CBUD1kdEWqMrwNUfGrGiirV1WpH <= tip a hedgewars dev