Why doesn’t the Video SEO plugin index all my videos?
Especially on websites with a large number of videos, the plugin can have problems finding all the videos, due to (a shortage of) memory.
You should try going to SEO-Video and then re-indexing a number of times. You may need to scroll down to find this feature. However, be sure to leave the ‘Force Re-Index’ box unchecked. Once the number in the sitemap is static, it has found all the videos.