The overflow may occur if you have a large amount of code ifdef'd out (i.e. #ifdef / #endif).
You have to keep the sections that are 'commented' out to under 4k and you have to put a tokenizable entry between the sections.
One way to see the large gaps in processable code is to look in the .i file (preprocessor output).
The tokenizer replaces every character that is to be removed (comments and #ifdef’d out lines) with a space to easily keep track of what line it is on. When the tokenizer tries to process these pre-processed, large strings of spaces, it overflows the internal buffer and you get an error.
You must either comment out less stuff or break the removed area with a tokenizable line. (i.e. not a comment.)
Note that version 5 tokenizer has a much smaller buffer than the 6, 8 and 10 tokenizers.