Gallup has produced a series of remarkable maps that show which religious groups cluster in what states. 
Highly Catholic states are found in the mid-Atlantic and New England area, while Protestants and other non-Catholic Christians are heavily concentrated in the South. Specifically, states that have a lower percentage of non-Catholic Christians have proportionally much larger percentages of followers from other religions.
Jews are found mostly in the mid-Atlantic states too. Obviously, Mormons reside mainly in Utah and the surrounding states. And those with no religious identity tend to live in the Northeast and Northwest regions of the country (plus Hawaii).
As a write-up of the analysis explains:

A good deal of the religious dispersion across the states is explainable by historical immigration patterns—particularly the impact of the large waves of European Catholics and Jews who came through ports of entry in the Middle Atlantic states in the 19th and early 20th centuries.
The geographic concentration of Mormons in and around Utah reflects the cross-country migration of that group in the mid-1800s from Illinois and other Eastern states to their new home. The fact that certain states like Oregon and Vermont consist disproportionately of residents with no religious identity is more difficult to explain, with hypotheses focusing on the particular and idiosyncratic cultures of those states and/or the migration of certain types of Americans to those states over the decades.

Here’s a state-by-state breakdown of the results: