Watch What Distinguishes the Past