We deal with the problem of spatially and temporally registering multiple video sequences of a non-rigid dynamical scene. For example, registering multiple videos of a fountain taken from different vantage points. Our approach is not based on frame-by-frame or volume-by-volume registration. Instead, we use the dynamic texture framework, which models the non-rigidity of the scene with linear dynamical systems encoding both the dynamics and the appearance of the scene. Our key contribution is to observe that a certain appearance matrix extracted from the dynamic texture model is invariant with respect to rigid motions of the camera, thus it can be directly used to register the video sequences. Our framework is applicable to both synchronized videos as well as videos that contain a temporal lag between them. The final result is a simple and flexible method that achieves state-of the- art performance with a significant reduction in computational complexity.