Custom River Stamps

[EvilPolygons]

So the maps I had been generating to test the new stamps were set to spawn no towns and no wilderness POIs because I wanted to maximize the number of rivers. For testing purposes, you know.

But when I started generating test maps with towns and wilderness POIs, I noticed a potential issue arising when using giant 1024x1024 stamps: the worldgen simply won't place any river stamps that it thinks will interfere with its placement of towns or where it wants to draw roads.

The huge river stamps do still have a chance of spawning, but it's very low because it's just a matter of luck that a river might spawn in an area that doesn't happen to have any towns or roads nearby. It all comes down to dice rolls, and the dice don't favor 1024x1024 sized stamps.

My next experiment will be to create 512x512 sized copies of those large stamps. BUT I'll keep the original large stamps in the mix. I want to know if the worldgen algorithms are "smart" enough to pick and choose river stamps for placement based on suitability (size in this case), or if the algorithm is purely based on dumb luck.

 

[zztong]

Interesting. That suggests to me RWG isn't scaling the stamp, or only scaling it to a limited amount.

Your observation leads me to wonder if 256x256 and 128x128 stamps are possible.

 

[EvilPolygons]

Interesting. That suggests to me RWG isn't scaling the stamp, or only scaling it to a limited amount.

Your observation leads me to wonder if 256x256 and 128x128 stamps are possible.

I wouldn't bet my life on it, but I'd bet a nickel that the game would indeed use smaller stamp sizes.

 

[EvilPolygons]

And here we go. Now using a total set of 12 custom stamps (six 1024x1024 and six 512x512 copies), here are the results on a 4k map:



1 vanilla river and 3 custom rivers. One of the rivers is kinda big, but I'm not sure if it's an undersized 1024 stamp or an oversized 512. Either way, the game used the stamps it needed to use. It would take a lot more test runs to draw any real conclusions, but I'm super lazy and it looks good enough to me!

One thing to note is that I currently have a lot of extra 1-tile settlements spawning (old oversized wilderness POI converted to tiles). They are set to force spawn for testing purposes. So it may be that all those extra settlements and dirt roads are overcrowding the map and preventing rivers from spawning where they otherwise would. Just something to keep in mind.

 

[zztong]

I resized one of my existing canyon stamps to 512x512. It works. This shows the image file on the right and the resulting canyon in RWG on the left. Below that is a view in-game.






I fired up Photoshop to make a test canyon by hand.

New 512x512, RGB color mode, 8 bit, background transparent, 72 DPI. Inserted a black (0,0,0 so that Red is zero) "ZZ" in 300 point font. See next image from RWG. It looks nasty on the edges for probably reasons that Cpt Krunch mentioned but that I don't understand.

BTW, in RWG "Cracks" are "Canyons"... (e.g. canyon_zztong_01.png in a Stamps folder at the top level of a modlet.)

 

[zztong]

256x256 seems to work. Err, wait, I may not have restarted since I changed.
Yeh, pretty sure 256x256 and 128x128 both work.

 

[EvilPolygons]

FYI, if you look in the original vanilla rwgmixer.xml, you'll find the xml settings for rivers and canyons, which includes the scale that they can spawn in at.

 

[EvilPolygons]

Unfortunately I got sidetracked by something mentioned earlier. Namely, the custom settlements (old oversized wilderness POIs) and how they are crowding my maps. I've been using the methods worked out by @stallionsden and @zztong to make my settlements, and they spawn perfectly. Problem is, they spawn TOO perfectly.; every settlement now has a 100% chance to spawn on every map -- which is great for testing the POIs -- but bad for everything else.

So I've been trying to figure out how to give a random chance for settlements to spawn. But 1) city/town spawn probabilities don't appear to be exposed to xml anymore in the rwgmixer, and 2) the limited xpath v1.0 supported by 7DTD doesn't allow any sort of random() or math.random() function calls.

I think I've figured out a work-around, though. If you pair a "fake" district & tiny 5x5 streettile with your REAL district for each custom settlement, and set the spawn_weights appropriately, you can sort of hack a random roll into the settlement spawn. When the fake district gets selected by rwg, all that spawns it its little 5x5 prefab instead of a full-size tile. That's important because it prevents a giant blank 150x150 tile imprint from being stamped into the terrain, which is what will happen with a regular blank tile.

So that seems to work, but the $!@#% dirt road is STILL being drawn to the fake district location, even though there's no RoadExit in the little 5x5 prefab! Not sure what to do about that...

 

[zztong]

@EvilPolygons , do you end up with biome decorations under the water of your rivers? I'm messing with craters and end up with crater lakes that have things like cars and other stuff in them.

 

[EvilPolygons]

@EvilPolygons , do you end up with biome decorations under the water of your rivers? I'm messing with craters and end up with crater lakes that have things like cars and other stuff in them.

Yep, but that happens in vanilla a lot, too, from what I've seen. At some point (I think around A20?) they added tires and cars as deco to water biome areas.

However! Inspecting a bunch of my own river stamps on a whoooole bunch of test maps, it seems as though something might be amiss because I'm occasionally seeing ground decos that I don't think should be on the river bottoms. It may be because the gradient colors aren't quite right (going from river bottom to riverbank), or it might be because my alpha channel blending is off.

I'm still banging my head on the wilderness tile/POI issue though. Once I have that figured out I'll come back to the river stamps.

I really like your canyon stamps, btw. I hope you'll have the fixed 512x512 versions in your next ZZTong prefabs update!

 

[zztong]

I really like your canyon stamps, btw. I hope you'll have the fixed 512x512 versions in your next ZZTong prefabs update!

I did make all four of those stamps into sizes RWG can deal with. It will be in the 2.2-ZZ030 release along with a few POI bug fixes.

I was kicking around craters earlier to day, mostly having ChatGPT write a python script to make them. It kinda worked; kinda didn't. I wasn't entirely happy with the results, but it was close.

 

[EvilPolygons]

Eureka! I did it! Stallionsden said it wouldn't work, but I did it anyway! Because I'm stubborn!

A one-tile township/district superclass that can spawn multiple completely individual wilderness tile/POIs, allowing you to control spawn probabilities!

I'll make a separate post about it since it doesn't really belong here.

 

[EvilPolygons]

After a lot of trial and error, I figured out how to convert the original .raw stamps to 16-bit png and back to .raw again (avoiding 8-bit!) using a couple of python scripts. That's important because you cannot override a .raw stamp with a .png stamp. It's also important for understanding how the game engine handles these files.

What I learned is that the .raw stamps don't have any transparency information encoded in them at all. The game engine masks off bits that shouldn't be rendered (such as the blank background in the stamp file) purely based on "elevation" in the heightmap data. Png files, however, don't get that treatment for whatever reason, and do require alpha channel masks to blend them properly into the terrain.

I'm currently experimenting with some "stepped" raw stamp files in order to determine the exact height at which water levels and terrain blending are applied to raw stamps. Each step in the stamp increases by 1000 currently but I'll lower the step size once I've zero'd in on the approximate number ranges.

 

[zztong]

So using a RAW file is superior to using a PNG then?

The python conversion scripts sound handy.

 

[EvilPolygons]

So using a RAW file is superior to using a PNG then?

The python conversion scripts sound handy.

I wouldn't say that raw files are superior, per se. The problem with png files is that hardly any software can import/export them without converting them to 8-bit at some point during the process. So png stamps end up looking not great because their precision gets squished from a 16-bit unsigned integer (~65k max) down to an 8-bit number (255 max). That's a massive loss of detail level.

I personally don't have any image editing software that can natively handle 16-bit images without 8-bit downsampling. That's why I'm using python to convert back and forth, because Pillow & numpy can do it through pure math. I'm using python to generate the raw "stepped" stamp for that reason, as well.

Anyway, I'll post the python scripts here later. I have an appointment to get to.

 

[Qnut]

I wouldn't say that raw files are superior, per se. The problem with png files is that hardly any software can import/export them without converting them to 8-bit at some point during the process. So png stamps end up looking not great because their precision gets squished from a 16-bit unsigned integer (~65k max) down to an 8-bit number (255 max). That's a massive loss of detail level.

I personally don't have any image editing software that can natively handle 16-bit images without 8-bit downsampling. That's why I'm using python to convert back and forth, because Pillow & numpy can do it through pure math. I'm using python to generate the raw "stepped" stamp for that reason, as well.

Anyway, I'll post the python scripts here later. I have an appointment to get to.

Sounds similar to what I have worked out in my thread Editing heightmap with correct smoothing. Maybe your Python scripts also would work for heightmap conversion as I was doing it back to RAW with the right Teragon preset.

 

[Qnut]

Png files, however, don't get that treatment for whatever reason, and do require alpha channel masks to blend them properly into the terrain.

If you have a look in the original Navezgane splat3 there also alpha blending is used for smooth roads, also in water masks. I don't even figured it out to correctly use those mask for splats.

 

[EvilPolygons]

Here are the python scripts. Pillow is required!

------------------------------------------------------
Raw2PNG.py:

import numpy as np
from PIL import Image
import os

# Init
path = "canyon_01.raw" # File to convert
W, H = 512, 512
dtype = np.uint16
channel_index = 1 # Only channel 1 has data!!!

# Load raw data and reshape
data = np.fromfile(path, dtype=dtype)
arr = data.reshape((H, W, 3)) # We know it's 3 channels because the math works

# Extract channel 1 (heightmap)
height = arr[:, :, channel_index]
min_val, max_val = height.min(), height.max()

print(f"Heightmap range: min={min_val}, max={max_val}") # Just in case...

if min_val == max_val:
raise ValueError("Heightmap is flat! Something went wrong!") # Also just in case...

# Invert heightmap data -- REQUIRED!!!
inverted = max_val - height


# Save as 16-bit png
Image.fromarray(inverted, mode="I;16").save("test.png") # Output filename. Yes, this is lazy
print("RAW file converted to 16-bit test.PNG")

------------------------------------------------------
PNG2Raw.py:

from PIL import Image
import numpy as np

def png16_to_raw_channels(png_path, raw_path):
# Load the 16-bit greyscale png...
img = Image.open(png_path)
if img.mode != "I;16":
img = img.convert("I;16")

height_data = np.array(img, dtype=np.uint16)
h, w = height_data.shape

# Invert the heightmap -- this is required!!
max_val = np.max(height_data)
inverted = max_val - height_data # Invert within actual data range, not just max uint16!!

# Create 3-channel array with zeros
arr3 = np.zeros((h, w, 3), dtype=np.uint16)

# Put inverted heightmap data into channel 1. Channel 0 and 2 are unused!
arr3[:, :, 1] = inverted

# Just in case...
print(f"Image shape: {arr3.shape}, dtype: {arr3.dtype}")
print(f"Original Min: {height_data.min()}, Max: {height_data.max()}")
print(f"Inverted Min: {inverted.min()}, Max: {inverted.max()}")

# Save as raw binary (little-endian)
arr3.tofile(raw_path)
print(f"Saved RAW file with inverted heightmap in channel 1 to: {raw_path}")

# Filename in and out. Change to cmd line arguments or something because this is lazy and terrible
png16_to_raw_channels("test.png", "test.raw")


Yeah, you have to edit the scripts to change in/out filenames. Because I'm lazy.
If you want to add cmd line arguments or a fancy gui, go right ahead!

 

[EvilPolygons]

Important thing to note about those scripts is that, again, there is no transparency data at all in the raw canyon stamp. So when you convert it to a 16-bit png, it will have no alpha channel info. If you want to use raw files converted to png (for whatever reason) you'll need to add your own alpha mask.

On a related note, I used a visualization tool to load the raw canyon stamp into a 3D isometric window so that I could get a better idea of how the game blends this stamp into the terrain:


This was VERY helpful. The game engine considers "ground level" to be an elevation of 59643. Anything over that (up to the uint16 limit) causes the entire stamp (including the square background space) to be pushed downward into the terrain. So that's the magic number that the game uses to determine where to blend the topmost part of the canyon into the surrounding terrain. Bear in mind heightmaps are always inverted, so you have to take that into account when dealing with these things. If your canyon looks like a mountain ingame, it's because you didn't invert the heightmap data before export!

The blue layer towards the bottom is approximately where I suspect the water table sits at. I'm currently trying to determine the maximum depth a stamp can go without ANY chance of filling with water, given that there is a bit of random variance in stamp elevation during world generation. So far, water consistently appears at the bottom of canyons at an elevation of 7453 - 7710 (which scales to 29 - 30 in 8bit). That perfectly matches up with the RGB xx,xx,29 value seen in the vanilla PNG river stamps.

 

[EvilPolygons]

After running many, many tests with stamps of varying depths, I've found that the water level doesn't just "sit at" 7453 or thereabouts. It basically ranges from 0 all the way up to an elevation of approximately 50,000. At 50k, you will still occasionally get water in the bottom of a terrain feature, but it's pretty uncommon. That's because the worldgen adds a bit of random variance to stamps when placing them. They can spawn a bit larger or smaller, and therefore higher or lower.

I didn't expect the water level applied to stamps to be so high, but I guess I *should* have because it basically corresponds to sea level, which you can see anytime you generate a map with a coastline.

Anyway, I'm sure there's an elevation at which point stamps could spawn and never have any amount of water in them, regardless of RNG. But 50k is good enough for me and I don't feel like trying to narrow it down any further.

So the lesson here is:
If you want to make a stamp and have it rarely (if ever) flood with water, your raw heightmap needs to be generated inside the range of 50,000 to 59,643 elevation. That's not very deep. Certainly not canyon-deep! If you want a really deep canyon, then you can either deal with it spawning flooded most of the time, or you can edit the splat4.png after the map is generated and remove water from those areas by hand.

Of course none of that matters if you're trying to make river stamps, which you obviously want to be flooded. But that's not always the case with canyons, hollows, draws, crevasses and other such terrain features.