This dude is also trying a similar thing.
This page describes the low-level protocol, and this page describes the higher level protocol.
It is definitely possible to "remotely control and/or view the screen on a TI-86.
Various VT-terminals exist. Some of them are Telnet 83, FTerm, RTerm, RTerm 83, ZTerm. Apparently they have semi-specific requirements (grey graphlink cable, 9600 baud), but other than that, you plug in the graphlink cable and it's a standard serial port.
If I can hook directly to the output port, it doesn't have to be 9600 baud, it can be any arbitrary rate, and the whole setup would be smaller/cheaper. This document gives a very good idea of what the Graphlink cable does.
Not being terribly popular, a brief scan didn't reveal anything about the IR protocol or possible terminal software, other than that the IR supports IrDA which may have some pertinent standards.
And there's at least 4 VT100 programs out there, if not more.