It actually comes from an older expression "give somebody the bag to hold" from the mid 18th century Britain. It meant to give someone the bag and then leave them holding the evidence, like when a robber would give someone the stolen goods and that person is left responsible for the troubles that may follow. They get the blame. SO the newer expression "Left holding the bag" is more from the victim's point of view.