Knowing What to Look For