This tutorial explains how to create a very simple visibility mesh.
Why is visibility mesh important?
Serious Engine uses visibility mesh (sectors+portals) to speed up rendering. It is the first and most important step in deducing what part of the level should get rendered. For example, when you are inside an undeground temple, visibility algorithm will probably extract only current room and neighbouring corridor as seen sectors, never ever touching or knowing about sectors/models that are above ground or behind corner. That is crucial step in determining what gets rendered in a level. After list of seen sectors is assembled, all models that are dip into these sectors are added to further testing (distant rejection, frustum calling, occlusion culling). Only models that pass all the testing gets rendered. Knowing that you should start seeing an level in another way, as a soup of models that are dip into sectors.
Note: when you are outside any of the existing sectors, engine will render all the models in the level.
Important: Engine uses this sector/portal traversing algorithm to render all types of scene rendering (shadows for distant light, buve background view, reflection etc). So, it is important to see that good visibility geometry will help distant light to determine less models that cast shadows and thus improve performance. It is very important to create separate visibility sector (cube room) around background scene (cube background entity). Othervise, engine couldn't determine starting sector when rendering background scene and would add all models from the level to the rendering.
How sector recursion works?
First of all, engine finds a current sector (where camera-player eyes is located). Then it uses polygons and portals from this sector to deduce if any part of the portal that leads to another sector is seen. If slightest part of the portal is seen, sector that is behind the seen portal is added to the test and algorithm seeks if any portal from newly added secor is visible. The process is repeated untill all sectors are collected and there are no new portals to add.
How to create an visibility mesh?
I will try to explain this in simplest case of having a room and an corridor exiting this room.
Insert an model (key "Insert") at 0,0,0 ("Q" and ajdust coordinates), name it "Visibility" (so it could easily be found using find tool-"F"). Enter mesh edit mode ("E") and create a box ("Shift+X", then "Ctrl+Shift+LMB" to create a cube). Center it using align to origin tool (Ctrl+A). Flip the polygons ("Shift+F").
Use create window tool ("Alt+Shift+W") to draw a door on one side of the room (hold "Ctrl" to use edge discrete marks, roll mouse wheel to change no of marks). Select door polygon and use bevel tool ("B) to create an corridor (hold "Ctrl+LMB" and move mouse down to bevel in direction of the normal).
Note: Visibility mesh for doesn't support concave polygons so concave n-gons get triangulated during the baking process.
Creating "door" polygons
Ok, now we have two sectors, room and corridor. We will now create portals. Enter the vertex editing mode ("V"). Select room's door 4 vertices in anticlockwise direction and press "P" to create "door" polygon. Do the same to create corridor's door polygon on the other side (you can also copy first polygon and flip it). Now we have 6 room polygons, 5 corridor polygon, 1 "door" polygon looking into room and 1 "door" polygon looking into cooridor.
Assigning visibility maps and properties
We have to classify these polygons so that editor could bake valid visibility mesh. All room polygons (including the "door" polygon that looks into room) must be assigned to the first sector (e.g. "Room") and all corridor polygons into second sector (e.g. "Corridor"). So, select room polygons and add new polygon map (using bottom-right combo action Create new map or just press "Ctrl+Shift+G"). Select Map type visibility sector name it "Room" and accept. Then select corridor polygons (e.g. invert selection), press "Ctrl+Shift+G", and name sector "Corridor". Now we have to mark "door" polygons as portals. Select both door polygons (one facing room and one facing corridor). Select visibility properties button ("VP"), then click onto map name combo and select Actions->Create new polygon map (or "Ctrl+Shft+G"). Map type must be "Visibility properties" and name must be "VisPortal" (name is case sensitive!).
Note: You can check if polygons are properly assigned into visibility maps by turining on "Selected map" rendering option (change render flags combo value using mouse or shortcuts "F" then "M" or just "Shift+Z"). Make sure that visibility sectors polygon map is active ("PM" and "VS" buttons) and change current map using the map combo in bottom-right corner of the view to examine polygon-map relations (or cycle using "Shift+mouse wheel").
Use visibility instead of rendering
All that is left to do is to assign our layer as one that will form visibility. When user starts editing new mesh geometry, mesh editor automatically creates an rendering layer (we automated this because that is the most common operation). But we want our layer to be used for visibility, not for rendering. So, select rendering layer map (buttons "LM" and "R"), select map named "Rendering 1" and delete it (RMB, "Delete layer map"). Then we assign our geometry layer into visibility map by select visibilty map (buttons "LM" and "V") and creating new one (RMB, "Create new layer map"). Name of the visibility map is not important.
Baking and testing
Exit mesh editor ("E"). Don't be confused with geometry dissapearing (model's rendering layer is deleted). Bake visibility ("B") and switch to the sectors mode ("V"). What you should see is a valid visibility mesh, with gray walls and green portals. This is a shell that we will dip our models into.
Example 1: A level showing results of this how-to tutorial.File:Visibility01.wld
Example 2: A level showing two rooms and coridor between File:Visibility02.wld. Use this to clarify and completly understand how visibility works. It has one visibility mesh, separate model for each room and corridor. You will be able to see how corridor/oposite room dissapear as you don't see portals any more. Walls are translucent so that you can see the contents of the other room/corridor.
Note: You can manually adjust list of sectors that certain entity belongs to (model, fog). This should generally be avoided but can optimize performance by having acceptable visual glitches in some cases (e.g. palms don't get classified into sector above yard walls). When you press "F3" over the entity in entity mode, info page will display a list of sectors that entity is classified in.
Hint: When in sector mode, turn on "Sectors" property page ("Shift+Alt+2") and click on the available tree items and fiters (bottom of the page) to examine sectors, entities and their relationships.