One of the interesting things I came across in my AI classes was a formalization of the exploration/exploitation problem. That is: when your goal is to maximize reward, how much of your time do you spend EXPLOITING already-known reward opportunities, versus how much time do you spend EXPLORING the unknown terrain for additional possible reward opportunities?

Related wikipedia articles: