I've never messed with rendering in Maya, but I can tell you that when I use a displacement map to deform geometry in Mudbox I set the intensity to .01; the render that you posted is exactly what happens when you have the intensity set way too high. Also, judging by the size of some of the spikes, it's possible that your geometry doesn't have enough subdivisions to handle the level of detail in the displacement map. It would help if you were to post your displacement map and a screenshot of your mesh with the "Wireframe on Shaded" box ticked.