The smaller shops can't win.
To accept cards costs money, usually between £5 and £20 per month for the terminal, the cost of the phone line (although the newest ones are always-on broadband) plus a fixed fee for debit cards and a percentage of the transaction for credit cards.
Like everything else in a business the customer pays for it, because in a business that's how it works. If you're a huge company it's a small amount, and because of the value you process you can command better deals. For the smaller retailer you accept what you can get, and, unlike cash, know that it'll be a few days before you can get to the money as it'll take that long to hit your account.
So, what is the answer, to increase the cost of goods so the everybody pays it, well then people go elsewhere. To swallow the cost, which doesn't really happen, as if you make less you have to build it up some other way. If your profit margin on a Sale
is 10%, then on a £5 Sale
the transaction fee of 50p has eaten everything away, so by the time you've then added in collection of the goods, pricing, wastage, rent, rates, insurance, wages.....it's cheaper to refuse to sell the item, or not accept cards, and that's before you get into charge backs when someone disputes a transaction or the card has been stolen/cloned.
I can see why people get upset by this, and cash has problems of its own, counterfeit notes, theft, security collection costs, bank fees for paying it in. However as long as the place makes it clear they charge for card use I don't see what the issue is, and if they don't, I'm sure lots of people walking away from the till leaving their goods on the counter will give them a good incentive to make it clear.