You can hide some of the default gvim ToolBar buttons by adding lines like the following to your .vimrc:
aunmenu ToolBar.FindNext
aunmenu ToolBar.FindPrev
Type ":h builtin-tools" in normal mode to get a list of available buttons. The names may be different if your language is not English. To find the actual names, type in normal mode:
:aunmenu ToolBar.<Tab>