When the histogram starts to look like a comb you are losing data. Each bar represents the amount of a particular tone in the image. Once you start seeing gaps you are losing some tones therefore losing the smooth gradation from one tone to another.
The comb occurs when data is stretched to have more contrast than it can handle. For instance, if the data is only 10 values (of the 256 values of grey) wide and is stretched over 50 values -- increasing contrast -- there is now a value every 5 units, and the histogram looks like a comb.
Yes that causes banding in any low contrast area.
The best way to prevent the comb look is to work in layers with less than 100% opacity. By blending the result back with the original the original data helps to fill in the vacant areas created by the comb.
...The best way to prevent the comb look is to work in layers with less than 100% opacity. By blending the result back with the original the original data helps to fill in the vacant areas created by the comb.
David
Sorry, I was rushing out the door on my way to work when I wrote the above. It didn't get far before I realized the solution I posted as 'the best' doesn't really address the issue at all. :(
The reason it doesn't solve the problem is each pixel has a value on each of the two layers (top and bottom). Blending with less than 100% opacity just moves the values closer together. But, each pixel with the same values will remain the same -- this is the banding. So the above may reduce the effect, but no more than applying the effect to a less degree to begin with.
To remove the pixelation, the bands must blend into each other better. This means the pixels near the edges of the bands must change differently than those farther from it. To do this, select the area that is banding and blur it after applying the effect (levels, curves, whatever). The strength of the blur will depend on the image, but probably no more than is needed to just remove the obvious banding would work well without making the area overly smooth.
Obviously the above will remove detail from the area as well as smooth the banding, so it is only of practical use for areas with no detail (clear skies for example).