C.Lee would have done all of his in front of green screen, by himself, non-stop. In contrast Weaving would have done his scenes on set with a whole bunch of other cast members. Time to set up, and to film scenes (with retakes) would've taken longer for Weaving, than Lee.
But I generally agree with your order anyway:
Gandalf - Most active member
Elrond - We know he also enters Dol Guldur
Galadriel - Most likely enters Dol Guldur
Saruman - Unknown