I would build on that saying, you should look at double buffering or maybe triple buffering.

Frame A is copied to the GPU. Frame B is being decoded into

Frame B is copied to the GPU. Frame C is being decoded into

Frame C is copied to the GPU. Frame A is being decoded into