The Fifth Amendment of the Constitution concerns the rules of trial. It states, for example, that you cannot be tried twice for the same crime (the "double jeopardy" rule), and the self-incrimination rule. This last one is the one usually referenced when people "plead the Fifth" or "take the Fifth." This rule states that a person cannot be required to testify against himself or to make statements that might incriminate him. This is why a person may choose not to take the stand in his own trial, for example.

When a person uses this expression in a casual context, he is saying that he refuses to answer because his answer might get him in trouble. For example, I might ask my fiance how I look in a new outfit. If he takes the fifth, it's because he doesn't like it but doesn't want me mad at him. Ditto when he asks me how much I spent on a haircut. :-)

You don't have to incrimiate youself.

Simply put, the Fifth Amendment means that an accused cannot be forced to testify against himself in any court of law.

They refuse to answer on the grounds that what they say may incriminate themselves or others.

Lots of fun.

Very simply, it usually means that you do not have to say something that will get yourself in trouble.

So if you were a witness in a trial and were asked by the prosecution "did you steal the car?", you are not legally required to answer the question if the answer will incriminate yourself. You can therefore "take the 5th" and legally refuse to answer the question.

There are other implciations of the 5th ammendment, but that is the jest of it in terms of general speech.