Sorry, there isn't much documentation (yet).
Essentially, select 'Open' from the file menu, then pick the directory root of the texture tree you want to inspect (e.g. Textures\Earth).
This should then load the lowest resolution Earth tile (global coverage, 128x128 pixels for the surface texture). In addition, other layers of the same tile can be displayed beside it (elevation, water mask, night lights, if applicable).
Make sure that "Pan and zoom" is selected for mouse action. Move the mouse into one of the tile displays. The tile should then be surrounded by a red square. Click on it, and this should load the child of the root tile (level 2, still global coverage, 256x256).
Keep clicking, and at some point you'll see that the rectangle only covers half of the map (western or eastern hemisphere). This is the start of the quadtree. At the deeper levels, each tile will have 4 children. You can descend into the tree by moving the mouse into one of the quadrants (will be indicated by a red square) and click.
You can move up to the parent level by moving the mouse to the centre of the image until a red X appears, and click.
You can shift the view to a neighbour tile by moving the mouse towards the centre of one of the edges, until an arrow appears, and click.
You can visit any part of the quadtree even if no texture or elevation file exists for that tile. tileedit then interpolates from a subrange of an ancestor tile, just like Orbiter.
You can see the 'logical' designation of the tile in the info block at the top left. The 'physical' address of a tile is shown below the tile display. When these differ, you know that you see an interpolated, synthesized tile.
By picking a different mouse action, you can edit elevation data (no other layers currently allow editing). If you edit an elevation tile, the modifications will not be written to the original elevation tile, but to a new file under 'Elev_mod' in the directory tree. The mod tiles will contain the differences to the original tile. By deleting a mod tile, you undo all modifications to that tile.
That's all I can think of at the moment. Most should be fairly self-explanatory. Let me know if anything doesn't make sense.
Note that currently there is no support for creating a new elevation tile that isn't yet present in the tree. I have a matlab script that generates a new tile by interpolating the parent, but that hasn't been incorporated into tileedit yet. You could always just copy an arbitrary tile file, and then use the mod function to completely replace it.
Edit: I should also mention that the elevation mod tiles only get written to disk when you navigate away from the edited tile.
Also, tileedit automatically propagates the edits to all the ancestor tiles (and the neighbour siblings for edits in the overlap areas), but not to the children. You should always make the edits on the tiles in the leaf nodes of the quadtree.