I don't think there's any evidence in the novels, except one ambiguous sentence in what Odin says about the Lords of Outer Night, to suggest belief in a god = power of that god, or even necessarily worship in general = power (sacrifices specifically are a source of power, yes, but that's not specifically 'religious' - works fine for warlocks ).
The power of belief we've seen is power for the believer, not for the object of belief.
When Odin explains why he can't just smack down the Lords of Outer Night, he talks about advantage of location (Chichen Itza is the LotON's power center, someplace in Scandinavia is his) and that the Norse pantheon has fallen apart. Never mentions lack of worship. He does say that the LotON were once worshiped by many, but he also says the power of their blood is divided among thousands of descendants, whatever that means .... at the very least, it doesn't seem to be a straightforward belief/worship = power thing. Advantage-of-location comes up with the naagloshii in Turn Coat too. We have much more evidence for gods and god-like entities having power based on where they are and something to do with their relationship with their 'pantheon' or others of their kind (Odin and the Norse gods, the LotON and 'dividing' their power among their descendants...) than based on worship.