Problem is: “Are either of us going” sounds right too.

Either is not always singular - Either the Red Sox or the Yankees are going to win tonight, not Either the Red Sox or the Yankees is going to win.