The fear of madness is generally known as dementophobia.
People who suffer from this fear are afraid that they are going insane or losing touch with reality. The fear may be triggered by a family history of mental illness or periods of severe stress.