Request Update Earth night light texture to latest NASA Black Marble version

[lccrane]

Alright, some new development again.
Actually I was looking for another source of daylight maps because I was unhappy with the fact that I have to heavily edit the ocean of google maps images because it has this strangely coloured bathymetry map. But while searching for a better daylight map I found something very interesting: The "OSM dark" map. I did not find any internet page displaying it, but it is accessible through qgis. It seems to be a derivative of the well-known version of OpenStreetMap, where the density and type of road is translated into a brightness value, giving an impression of night lights. It's pretty amazing, giving a much more high resolution impression, which is a huge advantage over VIIRS imagery. But of course (because of the lack of actual image data at this resolution) it is computer generated, not a real image. And the amount of illumination only depends on the density and type of road, so even every dirt road and forest track mapped in OSM is illuminated here, which of course (fortunately!!!) is not the case in the real world. This would literally be the end of the night if come true. Goodbye insects, goodbye healthy sleep.
But it looks awesome anyway.

This is what it looks like:


Even though this is not how the real world actually looks like, for populated areas it gives quite a realistic nighttime look, so it's a huge opportunity to improve our new night lights map if we combine it with the VIIRS data.

This for example is a view of Paris in the OSM dark map:


Compare it with an actual image of Paris I took a few years ago. Yeah, there is of course some differences, but in general it's quite similar. Be aware, that the overall brightness of the image is completely arbitrary and only depends on camera exposure or the sensitivity of your eyes. From space and in the absence of any other light sources you would probably perceive it much brighter.



So now I took this data to make a new version of night map for Orbiter and here's a comparison of different versions.

1) The (already shown earlier) Google maps + VIIRS blend version


2) A more aggressive blend of OSM dark and VIIRS


3) A more nuanced blend of OSM dark and VIIRS


4) Only OSM dark with some brightness adjustments



5) Only VIIRS for comparison


In my opinion number 3 is the best one as a compromise between VIIRS actual recorded brightness and some higher resolution features.

And here is a view of version 3 in full resolution as the above images lost some detail to resizing and compression:


Now I'm very interested, what your opinion is. Which one is the best version?

 

[lccrane]

Here's a side-by-side comparison of the old and (potential) new night textures:



I think both of them have something to them, I don't think one of them is better than the other. The right one is certainly more realistic from farther away but at that zoom level it also seems a bit too bright. But if I reduce the brightness, other details in regions that are not so bright will disappear. Well, we've got to make some compromises here unless we could make an actual hi res model of the entire urban infrastructure. But at the moment our biggest constraints are the architecture of the Orbiter textures, data availability and probably most importantly computational constraints of making a new global night time scenery. On the other hand it is tempting to throw away the VIIRS data entirely and just use the very high resolution OSM dark data. But again, both datasets have advantages and disadvantages themselves:
The OSM data does not contain light sources not included in the general OSM data (such as offshore activity) and it displays too much brightness in areas where there are roads but no actual street lights, but it has an extremely high resolution. The VIIRS data on the other hand is very low resolution and is overexposed in urban centres but it displays the general brightness of human light sources from orbit well.

 

[Boxx]

I guess #4 is probably what we should expect from space, as compared with an image taken by astronauts

 

[lccrane]

Thanks for your opinion!
I kind of disagree, because the image you're referring to was taken in full moonlight. But besides that you always have to be careful about camera images because they depend on exposure and contrast settings. The best DSLR cameras out there can natively capture 16 bit of brightness information (while our eyes can capture way more than that, and we are talking about an exponential relationship which people always underestimate), but most screens can only display 8 bit of brightness information (it has gotten better with hdr screens but their dynamic range is still vastly less than actual life), so the 16 bit has to be mapped by some function to the 8 bit output. What we are looking for is some kind of best way of translating the raw data into an 8-bit output here - from rural areas to city centres.

And then, in order to get the full picture of light distribution on Earth's surface, we have to blend in some VIIRS information because as mentioned the other dataset does not contain offshore activity and other light sources while other areas that appear light in the OSM dataset are actually completely dark in real life. The only question is how the blending should be done in the best way possible. To be fair, it is one of the most difficult - and controversial - aspects in image processing. Just take the discussion about the nighttime shot of Earth by the Artemis II crew as a good example, but really these discussions about exposure and dynamic range have been ongoing for as long as I can think.

 

[Boxx]

hehe, you're right about opinions (but why did you include #4, then?! was it a test? then, I failed )... the same happens with the Moon surface: what is its color? Probably the reason why the Artemis II astronauts were asked to describe the colors they saw (and they saw braun-ish and green-ish features).

 

[lccrane]

no, don't get me wrong, I really value your opinion! The fact that I somewhat disagree doesn't mean I don't hear you and you don't have a point. I probably asked the wrong question though and your answer helped me see that.
The question should not have been, which one is the best version but: how we can optimise the blending process such that most can agree it is the best solution or at least close to the best solution.

I found that there is actually a VIIRS dataset with a little bit higher resolution of 15 arcsec or 500 m at the equator, which can be achieved by long term observations (such as annual means), so now I started working with that. It also comes in an unprocessed 32-bit version such that I can create an entirely new brightness map that isn't overexposed like the Blue Marble at night image. I will use that as a brightness mask for the OSM data, which is probably the most accurate way we can do it. Since it is greyscale I have to find a new method to achieve the right colouration, but I think it will be a bit better. I'll share a new version once I get to it. One more advantage: It is from 2016 and so it is slightly less outdated than the 2012 version. You can actually see some differences.

 

[misha.physics]

Amazing!

So now I took this data to make a new version of night map for Orbiter and here's a comparison of different versions.

Does "2" and "3" contain Google Maps (namely it's OSM + VIIRS + Google Maps), or it's only OSM + VIIRS without Google Maps?

I think "2" and "3" look the best, whereas "4" is too contrast.

In my opinion number 3 is the best one as a compromise between VIIRS actual recorded brightness and some higher resolution features.

Now I'm very interested, what your opinion is. Which one is the best version?

For me the "2" seems to be better, since it contains more details (more contribution of OSM), although it may be not so realistic as "3". I think it would be a shame to lose this high resolution that OSM provides.

I would propose to try even more aggressive blending of OSM and VIIRS to see how it looks. By the way, it would be right to make global night tiles of low levels like 9, 8, 7 and see at these night lights from the low Earth orbit and compare it with real pictures (from orbit). It make sence since most of the time we look at the Earth from orbit.

Here's a side-by-side comparison of the old and (potential) new night textures:

I think both of them have something to them, I don't think one of them is better than the other

I think we should combine (blend) them for the best result. But it would be nice to compare it at global view, as I described a little higher. It's interesting how the only OSM night textures will look when seeing at the whole Earth.

 

[gamer19]

Iccrane, buddy, you doing great great work
please keep up
I really looking forward to this!

 

[lccrane]

Well, well...

I've been experimenting a bit more with the more recent and higher resolution VIIRS data. This time I used QGIS to blend it with the OSM data because 1) it lacks colour information 2) it also had to be reprojected to the right projection and 3) it is much less work and more efficient to export it as a final product from QGIS rather than to export all the individual layers and then blend them in Photoshop.
There is actually even more recent (annual mean) high resolution VIIRS data (2021 is the most recent) but unfortunately it has some issues with pixels from the auroral ovals that have not entirely been filtered out. Even though I wanted to insert auroral ovals as well, the data just doesn't look as clean as the 2016 dataset so I'm sticking with that. In terms of adding auroral ovals I don't really have an idea what data I should use for that because despite VIIRS generally has beautiful imagery of it, it is also very messy and since it lacks colour information it is nearly impossible to separate it from other light sources. It would probably be waaaay easier to just paint it in with the good old brush and just use the satellite views as a reference location-wise.

Anyway, this latest method for the city lights again has some advantages and also some disadvantages.
I tried to use a few more helpful images taken from the ISS of various regions, as a reference to get a general idea of what should be visible and what shouldn't and the approximate brightness. I think for some regions this works really well, for others the results are probably not ideal and it very likely mostly has to do with the limitations of the OSM data. Again, when not all existing roads or buildings are represented in the OSM data, there won't be as much illumination as the VIIRS data shows. So there are some regions that appear a bit darker than they actually are, for example it is noticeable in Africa, India and South America. For Europe and North America I think it worked very well, which is not really surprising as the usage and data quality of OSM is probably the highest in the world.
Despite these limitations, in general I think the results are pretty convincing and both more realistic and appealing than the old textures.

What I could do as another step is after the export from QGIS to brighten up at least some of the regions appearing too dark a bit in Photoshop without affecting the others. And as a last step I think I would manually add any offshore lights that pretty much vanished in the blended product because of the lack of offshore data in the OSM dataset.

Here's some images of the latest results (be aware that unfortunately image compression reduces the amount of visible detail):

Buenos Aires (probably a bit too dark)


Southeastern Australia (slightly too dark)


Paris & BeNeLux (Belgium in particular seems slightly to dark, otherwise ok)


Denver (pretty good; if you see the background with some greenish blocks, that's compression artifacts from the forum image compression)


Great Britain & Ireland (looks ok to me)


Great Lakes (pretty realistic imo)


India (clearly too dark)


Central Alps (perfect)


London (perfect)


Kairo & Nile Delta (slightly too dark)

 

[misha.physics]

I like all these pictures. They look fantastic. You’re right, some look too dark like India, Buenos Aires, South Australia and maybe Great Britain & Ireland.

 

[lccrane]

I think the simple post-export brightness adjustment works out fine. That looks acceptable to me.



It is not the final version though, as you can see some greyish patches. That is where the VIIRS imagery (that has no colour information, hence the greyish tones) goes beyond the OSM coverage. That should be fixed in one more step of using it once more as a colour mask.
So you see it is quite a complicated process to get to the most realistic version and it means processing dozens to hundreds of GB of data. Just the non-brightness-adjusted blended night lights dataset has a total size of 128 GB for the whole world at level 12 resolution. The hi res VIIRS dataset (the 32 bit version) is another 54 GB. On some things I have to work manually, so it's probably gonna take a couple of weeks to complete this, because it is still a lot of steps to do and I won't be able to work on it as frequently as I have done in the past days. But @misha.physics, I think I'm gonna make a low resolution version for testing as suggested by you in the very near future.

 

[misha.physics]

I think the simple post-export brightness adjustment works out fine. That looks acceptable to me.

Yes, India looks very good here.

 

[lccrane]

Alright, I think I figured it out. I added another iteration of colourised VIIRS data on top of the earlier version and I think that seals it. It looks perfect. It adds missing brightness in places where OSM dark has no data while keeping dark areas dark that are dark in reality as well.
I did the processing entirely in QGIS and what's interesting is that India is still not as bright compared to other areas like Europe and North America. So it is in fact a real brightness difference because everything is processed with the same settings. So I'll probably just leave it like that and I can now move on to building the land-sea-mask and the final product. I will make a downscaled version without a sea-land mask to test it because building that from the daylight maps is going to take the longest time.

Central Europe


Southern India & Sri Lanka (note the offshore lights in the upper left corner)


Northern India with Delhi


Here's a direct comparison of an area where there is a big difference between VIIRS and OSM in North Dakota and the Pacific Northwest (the right one is the final addition of both datasets):


I looked at the area in google maps to find out what the source of the bright lights in the area in the lower right portion is and it seems to be agricultural activities. The lights in the upper left portion are large mines and oil drilling/processing sites in Alberta.

Here's two examples of the final version. I think this looks perfect and it is at least very close to the best possible outcome.




Once the export of the global product is finished (each tile of 16 in total is taking about one hour to export, so tomorrow ) I'm going to make a level 11 preview version (without land-sea mask). That's the best I can get to in a reasonable amount of time. Making level 12 will be a lot of work and probably a couple of hours of computing time, so before I do that I have to finish the land-sea mask, which requires me to finish the daylight textures before that. But I'm very happy we got here and actually faster than I had thought possible. Thank you all for your valuable inputs!

 

[lccrane]

Alright, I have good news and bad news...

It kind of worked and it kind of didn't work at all.
Take a look for yourself:













I'm not really sure what happened here.
There shouldn't be any missing files in the texture files, but for some reason it only loads parts of them regardless the zoom level and if I zoom in too much, the night lights textures disappear entirely. Maybe it has to do with the missing land-sea mask, maybe it has to do with the fact that I used the night lights as a daylight texture as well while compiling the files because the plsplit64-tool always asks for daylight textures even if I only want to compile the mask texture (but they are not written into the "Mask" folder and .tree-file, so theoretically it should be fine (??)). It could also have to do with the fact that the "black" in the night light texture is not black but a very dark grey. At this point I don't know what it is but unfortunately I don't have the time to investigate it in the coming days. But I'm pretty sure that whatever it is, it will be a lot of work to fix it.

What is surprising to me though is, that the resulting mask.tree only has a size of about 78 MB although the Mask-folder containing all the dds-tiles has a size of almost 1.8 GB. So it has been compressed to just over 4% the original size. With my cloud layer there was only a compression to about 39% of the original size. But of course it could still be possible because the night lights textures contain a lot of empty space.
In case you want to have a look at it: I attached the mask.tree that I compiled at level 11 through level 1.

To close on a positive note - the textures themselves look pretty awesome.

 

[misha.physics]

I canʼt try your Mask.tree now (I’m in another city right now), but try to check (compare) DDS tiles (open them in Photoshop) that are displayed and not. Maybe they differ from each other. I used to have similar issue and it was due to lack of the alpha channel for tiles that contain only land or only water. Although it’s other thing.

 

[misha.physics]

I tried your Mask.tree and I notice the same issues. I don't know why some tiles are displayed, whereas other are not. I suppose making mask tiles of all levels for which surface tiles exist should give the correct behaviour, but I'm not entirely sure that this is the reason.

 

[lccrane]

Thanks for looking into it!
I think I will almost have to start at zero again because it seems I also used the wrong projection that just slightly differs from the projection used in Orbiter as the tiles don’t fit together perfectly (see e.g. Northern Italy). But now at least I know how to do it and what to expect and I think there is some potential of making the whole process a bit faster. I will try to make new surface tiles first and then another level 11 version before I go for level 12 because that’s so awfully more work to complete.

 

[lccrane]

A little update:
I successfully exported a new version of the night lights in the correct coordinate system with no issues.
However, the surface and land mask tile making (on a global scale) is a slightly bigger challenge as I'm trying different approaches to get nice land cover imagery and good-looking water coverage. So far nothing has worked 100% without issues and caveats. Getting both the land and ocean surface looking good as well as a very high resolution land-sea mask is not a small feat. So it's gonna take some time, but I'm getting there. Can't give a good estimate on when it's gonna be finished because just as it happened recently, there might be unforeseeable issues that could make me have to start all over again. Almost every time I open some newly exported files there are (usually bad) surprises. Sometimes things are working out 80% of the time but the remaining 20% turn out to be a huge headache and I have to decide whether finding some improvised solutions or just different datasets is the better way forward.

In the meantime here's a few scenes from the new and hopefully final night light textures. I really like them and can't wait to see a flawlessly working version in Orbiter. I wish I could make them at level 13 or even higher but as I mentioned earlier that unfortunately is just not doable in any reasonable amount of time. But I still think it's mostly gonna be a good improvement over the old textures.

 

[lccrane]

After two weeks of working a bit on cloud textures, daylight textures and sea mask tiles a little update.
While the main goal still is to get better night light textures, first I need to make new daylight textures that match the resolution of the new night light textures, so I'm still working on that and fortunately I have made good progress so far. But it's tiresome and lots of work that unfortunately cannot be automated because of the necessary removal of lots of unwanted artifacts. Here's a few tiles of the new daylight texture to give you an idea how it's going to look like.
I was able to find a nice high resolution Antarctic sea ice dataset as well (though in order to access it I had to download almost 50 GB worth of raw data but it was definitely worth it ).

 

[lccrane]

I think I'm almost there now. There's still a few steps to do before I can compile a new test version but the results so far look very promising.
Here's the whole global map at day and night:





Of course, at this scale there's no big difference between my version and the official black marble version. The only big difference is that mine has sea ice in the southern hemisphere and a little more sea ice in the northern hemisphere.