You can hide some of the default gvim ToolBar buttons by adding lines like the following to your .vimrc:
aunmenu ToolBar.FindNext aunmenu ToolBar.FindPrev
Type :h builtin-tools in normal mode to get a list of available buttons. The names may be different if your language is not English. To find the actual names, type (in normal mode) :aunmenu ToolBar.<Ctrl-D>