Is infertility the worst thing that could happen to you?

Since all of my friends seem to get pregnant, some the natural way, some with help from doctors, and since I got the baby fever as well, I can't help it wondering... is it really?