It actually comes from an older expression "give somebody the bag to hold" from the mid 18th century Britain. It meant to give someone the bag and then leave them holding the evidence, like when a robber would give someone the stolen goods and that person is left responsible for the troubles that may follow. They get the blame. SO the newer expression "Left holding the bag" is more from the victim's point of view.

What exactly is inside of that "bag"?

Many splendid questions and answers posed by a giddy, eclectic group of inquiring minds.

Responsibility/Accountability

Aggravation, bullshit, and the evidence.

the cat ... right?

Shit...I'm not kidding, that actually what it is supposed to be. Metaphorical shit...

I like to think it's a big bag full of sh*t, but usually I visualize the bag being full of all the things that would be necessary to fix/take care of the problem at hand. They left ME with the bag of stuff.

The bills that politicians are expecting you to pay for their frivolous spending and corruption.

I checked out several dictionaries, etc., and although they all gave me the definition, none gave me the origin.

So here is my take on it. Since the meaning is to be left with the onus of something that is supposed to be a joint responsibility, I am going to guess that it's from the underworld - the bag was supposed to carry the loot and your partners in crime have deserted you, who are now holding the evidence all by yourself, when you are arrested and charged with the theft.

Remember, this is just a guess on my part, and I am more than willing to be proven wrong.

Richard Simmons.