SceneQuads are computed 2D primitives that describe a flat surface and has an API designed to be used to help placement of the objects.

When using the triangle mesh from Spatial Map to perform placement, one had to scan all areas of the quad and perform hole filling/post-processing to identify good locations for object placement. This is not always necessary with Quads, as the Scene understanding runtime is capable of inferring which areas of the quad that were not scanned, and invalidate areas of the quad that are not part of the surface.

If your application intends to place 2D or 3D holograms on rigid structures of your environment, the simplicity and convenience of SceneQuads for placement is preferable to computing this information from the spatial mapping mesh.

Short link:

Overview

The Spatial Awareness system provides real-world environmental awareness in mixed reality applications.

Scene understanding provides Mixed Reality developers with a structured, high-level environment representation designed to make developing for environmentally aware applications intuitive.

Spatial Awareness is one of the core features of HoloLens. It allows the device to understand the real world around you.

The Spatial Awareness system builds a collection of meshes that represent the shapes in your environment, allowing developers to make some very cool interactions between holograms and real-world objects.

The most common use case is for placing Holograms on surfaces. If we ignore the room surfaces we can get some strange results where Holograms are behind a wall or go through physical objects. This is bad for a few reasons.

• This is very uncomfortable for users because their eyes are constantly fighting to focus between the real surface and the Hologram.

• And second, it breaks the illusion of our 3D objects being placed in the real world.

What we recommend is to use spatial awareness to place holograms on top of a surface, or attached to a surface. If we enable spatial mapping in our apps we can place holograms in a more natural way, but if we use scene understanding you can achieve even better results, but I'll dive into that in a minute.

Even though HoloLens is constantly scanning your environment, make sure that your app has enough spatial information in order for it to be a great experience. So incorporate scanning moments into the flow of your app. These moments can be disguised by asking the user to move around an object or to look in certain a direction, but always remember to keep it short and fun. Now back to Scene Understanding. If you use Scene Understanding in your apps, you can create magical moments where holograms are automatically placed in your room. For example, you could place a holographic screen on a wall, or even better, sit a virtual character on a user's real sofa! On HoloLens you can download room data from the device to your PC. This allows you to import the room data as a 3d model into Unity and other development tools, and simulate the user's environments. But most importantly, you can test your spatially aware app in different settings and make tweaks to the model inside the editor. Whenever you spawn objects in your apps, make sure that their positions respect the room's bounds. Spawning visible objects behind a real wall is a bad idea, especially if you are using spatial mapping with an occlusion shader! The Occlusion shader will hide the object since it is behind a real one. Always check that spawned objects stay within the bounds of your room. Constantly showing the spatial mesh in your app, despite looking cool, can be very noisy and distracting for the user. That's why we recommend showing the mesh only if the user needs to know that it's there, and even in those cases, pulses should be used so that their appearance is subtle and elegant. Our research has taught us that users expect holograms to behave like real objects. For example, users stay away from holograms with sharp edges, and they avoid walking through holograms. Makes sure that your holograms respect their surroundings because your users will expect them to do so.

In this chapter we've explained some fundamental recommendations around Spatially Aware apps:

• Use Scene Understanding to place objects on specific surfaces. (Use SU for hologram placement)

• Make scanning a part of your user experience ("Scanning part of UX")

• Test your app in different environments by importing mesh data into your development tool.

• Spawn objects within the room bounds.

• Only show the spatial mesh when needed.

• Make holograms respect their surroundings.

Spatially aware apps can be incredibly powerful, and if they're designed properly they can truly blend holograms with reality.

Scene Understanding is an advanced feature of Spatial Awareness that gives HoloLens 2 developers a structured, high-level environment representation.

Your mixed reality device is constantly integrating information about what it sees in your environment. Scene Understanding funnels all of these data sources and produces one single cohesive abstraction.

With Scene Understanding, you can rely on the system to tell you when it has detected certain surfaces and identify the surfaces as walls, floors, ceilings, and platforms. Scene Understanding creates a watertight mesh of your environment, even if you haven't scanned it completely.

The goal of Scene understanding is to transform the un-structured environment sensor data that your Mixed Reality device captures and to convert it into a powerful, but abstracted, representation that is intuitive and easy to develop for. The Scene Understanding SDK acts as the communication layer between your application and the Scene Understanding runtime. It's aimed to mimic existing standard constructs such as 3D scene graphs for 3D representations and 2D rectangles and panels for 2D applications. In general, Scene Understanding is framework agnostic, allowing for interop between varied frameworks that interact with it.

Many of the core scenarios for environmentally aware applications, for example placement of objects, creating occlusion for a virtual object and running physics simulations that takes into account of the virtual objects and your environment are addressable by both Spatial Mapping and Scene Understanding. There are few differences you need to consider before deciding which one to use for your application. A core difference between Scene understanding and Spatial mapping is a trade-off of maximal accuracy and latency to structure and simplicity. If your application requires the lowest-latency possible and mesh triangles that only you will want to access, use Spatial Mapping directly. If you are performing higher level processing, you may consider switching to the Scene understanding model as it should provide you with a super-set of spatial mapping functionality. Also note that because Scene understanding provides a snapshot of the spatial mapping mesh as part of its representation, you will always have access to the most complete and accurate spatial mapping data possible.

• Case Study Gallery: aka.ms/CaseStudyGallery

• MRTK Spatial Awareness Documentation: aka.ms/MRTKSpatialAwareness

• Scene Understanding Documentation: https://aka.ms/SceneUnderstandingDocs

• Scene Understandging SDK: aka.ms/UnitySceneUnderstandingSDK

• Nuget for Unity Github Repo: https://github.com/GlitchEnzo/NuGetForUnity