This is because you rarely use it to click+drag. It's only used for a quick-click-and-immediate-release. That means we can easily change it into a "act only on release" button, which means we can check what happens between press and release, and if anything else happens, act on that instead.
Some scripts that do this: