It's possible to use 'vim' as a replacement for 'less', to set it as your $PAGER.
Gentoo provides the scripts 'vimpager' and 'vimmanpager'.
less.vim is included in newer versions of Vim distros
See "vimmanpager" from Gentoo. Possible source here:
actually hyperlinking man pages
Is there a way to actually browse man pages in a hyperlinked fashion?