I have the Nidhogg as being found in the Sunken Caves. I don't know why killing them in Mt. Gagazet wouldn't count but if it doesn't try the sunken caves and see if you can capture them there.
Also (I figure you may know this by now though), you must KO the fiend with the weapon that has capture on it. If you hit a fiend with capture weapon, it doesn't mean you will capture it. It must be the death blow. Magic will not capture the fiend even if the caster is holding a capture weapon.