There is a game they play in the City called "show all". It works like this. One banker goes to another and dares him with a request to "show all". Then they both get their wallets out and the one who has the most cash takes all the other person's money. That simple. Winner takes all. Pity the poor trainee who took out a few thousand pounds in cash to buy a used car. He was absolutely convinced he was going to beat his boss when offered the "show all" challenge. He lost.

This mine-is-bigger-than-yours mentality says much about the mentality of bankers. Size matters. You don't need to go and see Martin Scorsese's new film The Wolf of Wall Street to know that excess is a part of the warp and weft of city life. Reports of city boys (and girls) whooping their appreciation of DiCaprio's Jordan Belfort chucking $100 bills in the bin, sniffing coke off a hooker's behind and throwing Velcro-covered dwarves at a huge dartboard do little to endear them to anyone outside their magic circle.

But speaking of magic, there is a subtle piece of misdirection going on in our being persuaded to get emotionally riled by overpaid Armani louts. For one of the most ingenious strategies used by the City establishment to maintain the status quo is the few bad apples argument. We don't need systemic change is the unspoken subtext of all this. We just need more moral bankers. Get rid of the Jordan Belforts of this world and all will be well. What we need is a return to respectability and personal morality. Evangelical bankers say this sort of thing all the time. Which is why they got so especially exercised by the failures of poor crystal Methodist minister, Rev Paul Flowers. He let the side down.

But the problem is not a few bad apples. The problem is systemic. Here is where size matters. For what was revealed for all to see back in the financial crisis of 2008 was that those banks that are too big too fail were able to get away with pretty much whatever they liked. Moral hazard begins with immunity from going bust. The fact that the state bailed out various banks created the absurd reality of socialism for the rich, with profits counting as private gain and losses as a public responsibility. Heads I win, tails you lose. Ed Miliband is dead right to recognise that it is the hegemonic grip that some banks have over our economy that is at the root of many of our problems. But his new announcement simply concerns the size of high street banks, which is important enough, but still leaves hanging the larger question of how high street banking, and the measly few quid in my current account, might be protected from the sort of wolfish casino capitalism of Jordan Belfort. Current arrangements don't go nearly far enough.

All this boring but important stuff is what we really ought to be worried about. For if we make the issue about individual hedonistic excess and Caligula-like orgies, we lose sight of the real target: size. Big banks need to be broken up. Even red-in-tooth-and-claw capitalists recognise this. Inordinate size confers an unfair competitive advantage in the form of a get-out-of-jail-free card. This squeezes out precisely the sort of legitimate competition that drives down prices. The size of a bank is the more dangerous form of "show all", where the smaller players get gobbled up by the bigger ones. Individual immorality is the more eye-catching. But the dull, difficult-to-spot, structural and systemic immorality is the thing that does the real damage. The trouble is, it's much more difficult to make a sexy film about all of this.

Twitter: @giles_fraser