Preheat oven to 400°F.
Peel and thinly slice all four potatoes, keeping separated by color.
In a 10-inch round (or similar-sized) deep casserole, place half the white potatoes on the bottom in a thin layer, overlapping slightly. Sprinkle with salt, pepper, 1 1/2 teaspoons chopped rosemary, and a grating of nutmeg.
Next, cover with a layer of half the sweet potatoes; sprinkle with salt, pepper, 1 teaspoon chopped rosemary, and a grating of nutmeg.
Repeat layering using up all remaining potatoes, sprinkling each layer with salt, pepper and rosemary.
Pour cream over potatoes and cover casserole with lid (or foil). Bake for 1 hour to 1 hour 15 minutes (see notes).
Meanwhile, mix the grated cheeses. Remove the top and check the potatoes for doneness (if not tender, return to the oven for a bit). Sprinkle the cheese mixture evenly over the top, then return to the oven, uncovered, for 3 minutes, or until cheeses are nicely melted and bubbly.