Sometime I joined in second, or third (but never fourth) raka'ah in duhr or asar. What I and other Shafa'i brothers always do in this situation just follow Imam. But then when Imam say salam in the last raka'ah, we raise up and continue our salah.