I often cross a railroad track on foot near me. From what I've read, this is trespassing because most track in the US is owned by some freight company. It's considered company property, and I guess there are right of way laws that apply to it.
(note: here's how to tell if a rail line is unused)