If anyone ever stumbles upon this and has a similar issue I found a decent workaround. Essentially I was already on the right track with the vertex weights. What I ended up doing was to assign all the vertices in the eyes to a vertex group connected to a static bone in the armature, in my case I just used Head_M. Then by adjusting the weights you have full control over the animation strength e.g. if they are assigned as 0.9 for eye_l and 0.1 for Head_M you get almost the full movement, but if you go for 0.1 eye_l and 0.9 Head_M there's barely any movement at all. Having them at 0.5 nd 0.5 caused some issues with normilization upon exporting the meshes, so I thought this wasn't a viable solution for some time and I was lucky to try some different weights, which made me realize that the problem wasn't the solution itself but just some exporting weirdness.