Let's concede your argument about deregulation for the moment. Do you then say that the bankers bear absolutely no blame for the actions that followed?
Absolutely not. The bankers caused the crisis - deregulation gave them the tools to do it with. Both parties are equally culpable - financial institutions for engaging in practices that were virtually guaranteed to cause eventual systemic failure, and the political move by giving them the ability to do so for ideological purposes in the face of economic realities.
If there are two ways to take advantage of deregulation -- one positive way, with integrity, and one selfish way, with greed and short-sightedness -- and if a banker chooses greed over integrity, is he completely innocent? Even if he willfully chooses to enrich himself, his colleagues, and his profession at the expense of the economy and the general population?
Here's the trouble - financial leaders in the private sector are expected to make the most money in the fastest way possible for their shareholders. This nonsensical position of proponents of a completely de-regulated free market ignores one fundamental truth about humanity: people are not primarily motivated, but by Darwinian competition. It's written into our behaviour as a species. Yes, altruistic qualities are present and enable us to function as a society, but our behaviour is oriented toward individual success; more so in a free-market society than any other.
Market forces are not always stabilizing and self-regulating, just as people are not always self-regulating in their behaviour. It takes social links to keep us behaving within socially-defined limits. Without those links, our default state is self-oriented. The same is true of corporations - as large entities driven by a particular set of fundamental motivations and lacking in a collective consciousness to impose morality, they require regulation in order to establish those socially-defined limits. Where individuals are bound by unwritten rules within a society because of their ability to perceive social rules, large collective entities lack this ability (the formal term for this is 'groupthink,' which is defined slightly differently in social psychology than in lay terms. I wrote a paper on it a few years back.)
Individual corporations make buck this trend on the individual merits of their leaders from time to time, but even that is rare. When a corporation with a motto of "Don't be evil" still manages to do so purely by accident because of the lack of a collective consciousness, it's pretty clear that regulation in some form is always going to be required. When that's missing, we can hardly expect corporations without that collective consciousness or morality to abide by rules that are not defined and lack consequences when their motivating purpose is to achieve an end in the most expedient way possible.
TL;DR version:
The financial sector and politicians who de-regulated it are equally as culpable in the recession. However, as the public has a very short collective memory and a poor understanding of the interaction between politics and economics in general (Presidents have virtually no impact on economics as a whole), that was promptly forgotten this election, as is quite evident from the results.