Eh!!!! A large part of nsHTMLEditorStyle.cpp, nsHTMLEditRules.cpp and even nsHTMLEditor.cpp are dedicated to that problem : style or remove the selection. It's a *very* complex problem.

I found in the news.mozilla.mathml group that there is another project which is related to abacus. See openmathedit in sourceforge. (in case you don't know yet)