Since a Stealth roll is always an opposed roll against an opponent's Alertness or Investigation roll you give the environmental difficulty to whomever would get the benefit of the environmental difficulty modifier on their roll and therefore make it more difficult for his opponent.
So hiding in a dark environment with lots of cover has an environmental bonus of +3 to hide. Hider rolls his Stealth skill +4DF +3 points environmental bonus vs the Looker who rolls his Alertness or Investigation skill +4DF and see who is the winner.
That's not how the rules work. You're actually repeating one of my fixes.
Environmental modifiers affect the difficulty of rolls, not the rolls. I've given page references in this thread.
The way that works by the book: Joe wants to hide in the dark. His difficulty is modified by -3 because of lighting. He rolls. Whatever he gets becomes the difficulty for someone to spot him. Since the modifier is to the difficulty, and not to the Effort, all that the darkness modifier does is make lower Efforts successes, which means that it's actually easier to spot someone in the dark.
Let me put this more concretely.
Let's say that the base difficulty would be 0 to hide. Actually, you know what? I'm gonna use rules without naming Skills. People get caught up on the Skills.
It's difficulty 0 to do a thing. Roll Skill A, and an Effort of 0 or higher succeeds. If someone wants to contest that, they roll Skill B. Because the difficulty is 0, the Effort necessary to contest is 0 or higher.
Now something good happens for the person rolling Skill A. They get a -2 difficulty! Now they need only roll a -2 or higher Effort to succeed. Great! Wait... While they succeed more often, the opposition still only needs to match the Skill A Effort, which now begins at -2 and goes up. So while Skill A succeeds more of the time, Skill B beats it more of the time.
That's my problem. Your mistaken impression of how the rules work is actually one of my proposed fixes, and it does indeed seem to solve the problem. But it's not how the rules in the book actually work.