I'm not a prolific modder by any stretch. In fact, I've only ever posted one map, and it's extremely primitive. One reason is that I'm never satisfied with the result of importing a heightmap. With only 256 possible height values, a map with any significant elevation changes imports as a terraced mess that has to be smoothed in the editor, destroying the features I've created elsewhere. Is there any possibility of enabling the editor to import a heightmap in a 12 or 16-bit format like scRGB? Having a finer elevation resolution would preserve features I've painstakingly baked into my terrains. Alternatively, enabling the editor to import certain 3D formats would help, but there would be a lot of debate over which ones to use, and possible licensing issues.
Hopefully I have understood your question correctly. So, before I import a heightmap, I edit the image with a 16-bit capable image editor. I always export the image as GRAY and in 16 bit. This way the terrain in the game is always smooth and a lot of details are preserved. As far as I know you don't have to set anything in the editor. I hope I could answer your question.
If you're in RGB color space, you only have 256 gray values, no matter what you use to edit the image. You can smooth the heightmap by stretching it out or adding noise, but you lose definition. Make a 1024x1024 heightmap that consists of gradient, say 25% gray to 75% gray across its width. Import it with 256 m max elevation. Instead of a continuous slope, you end up with 128 distinct linear elevation changes, whether you save it as 8, 16, or 24 BPP because there are 128 shades of gray in the image. The main reason most maps are smooth is that they're edited in-game. You end up with a smooth, usable map, but it lacks realism. A lot of geographic detail is added later as objects, like every cliff in Utah. Increasing vertical resolution would allow hi-res maps to retain all of their detail, resulting in much more immersive gameplay. The main question is how to do it. You'd need to use a format that modeling programs could export, like maybe a Blender plugin that would export a large number of shades of gray by making them gray-ish; instead of {0,0,0}, {1,1,1}, {2,2,2}... you could use {0,0,0}, {0,0,1}. {0,1,0}, {1,0,0}, {1,1,0}, {1,1,1}, {1,1,2}, {1,2,1}, {1,2,2}, {2,1,1}, {2,1,2}, {2,2,1}, {2,2,2}. Clamping the color difference to 1 would multiply the number of elevation levels by 6 while producing a heightmap that's visually close to pure gray. You could open up the whole color space, but 16.7 million increments would be overkill. (You could produce a map 16.7 km high with millimeter precision, far beyond the capability of the rendering engine.) Or you could select scRGB interpretation. Gaea (for one) can export heighmaps in this format.
