It's possible to use 'vim' as a replacement for 'less', to set it as your $PAGER.

Gentoo provides the scripts 'vimpager' and 'vimmanpager'.

normal pager

less.vim is included in newer versions of Vim distros

man pager

See "vimmanpager" from Gentoo. Possible source here: [1] [2]

plugins

actually hyperlinking man pages

Is there a way to actually browse man pages in a hyperlinked fashion?

Maybe? [1] [2]