To fulfill my current requirements (that I be able to add TODO items by category, but choose which ones to work on in order of priority), I do these two things:

insert by category: use Vim's foldmethod of marker. The todo list is separated by category, and each category has markers that indicate a fold should start there.
remove by category: use Vim's foldmethod of expr. This plugin is used to make it easier.

However, you don't HAVE to use that plugin. Something like these work well:

marker: !!          :set foldexpr=getline(v:lnum)!~'^.*!!.*$'
marker: ##          :set foldexpr=getline(v:lnum)!~'^.*##.*$'