unity混合树_针对混合现实进行设计：从Unity MARS HMD配套应用中学习
unity混合树
The spatial design team, part of the authoring tools group in Unity Labs, is currently working on a head-mounted display (HMD) companion app for Unity MARS, a Unity extension that gives creators the power to build mixed and augmented reality experiences that intelligently interact with the real world. The HMD companion app acts as a bridge between the Editor and the real world.
空间设计团队是Unity Labs创作工具小组的一部分，目前正在开发 Unity MARS 的头戴式显示器(HMD)配套应用程序 ，这是Unity扩展，可让创作者智能地构建混合和增强现实体验与现实世界互动。 HMD配套应用程序充当编辑器与现实世界之间的桥梁。
While the Editor experience is the primary authoring environment for Unity MARS, both the mobile and HMD companion apps provide on-device solutions that extend authoring beyond just a mouse and keyboard. In the HMD companion app, you can quickly capture environments from the real world to later import into the Editor for simulation, perform content editing, greyboxing, author conditions, and queries. It leverages the work of both Editor XR and the XR Interaction Toolkit, which provide reusable components that enable you to build interactive and immersive experiences quickly and easily. This post shares insight into the design process to show the thinking behind our work – creating meaningful connections between bits and atoms to help produce better interfaces for augmented reality (AR).
编辑器体验是Unity MARS的主要创作环境，而移动和HMD配套应用程序均提供了设备上的解决方案，使创作范围不仅限于鼠标和键盘。 在HMD配套应用中，您可以快速捕获现实世界中的环境，然后将其导入到Editor中进行仿真，执行内容编辑，灰盒，作者条件和查询。 它利用了 Editor XR 和 XR Interaction Toolkit的工作 ，它们提供了可重用的组件，使 您能够快速，轻松地构建交互式和身临其境的体验。 这篇文章分享了对设计过程的深刻见解，以展示我们工作背后的想法–在位和原子之间创建有意义的连接，以帮助为增强现实(AR)产生更好的接口。
空间设计原则 (Spatial design principles)
When designing for screens, digital representations are often closely coupled to the design tool. When designing for AR, however, the gap between the actual thing and its representation can be deceptive. We live in an unpredictable world, and our virtual spaces need to adapt and respond to the contexts they inhabit. Because of this, we favor working in 3D, at 1:1 scale, and on-device throughout the design process. In our minds, this is the only way to truly know how users are going to experience the completed app.
在设计屏幕时，数字表示通常与设计工具紧密耦合。 但是，在为AR设计时，实际事物及其表示之间的差距可能是欺骗性的。 我们生活在一个无法预测的世界中，我们的虚拟空间需要适应并响应他们所居住的环境。 因此，我们赞成以1：1比例在3D中工作，并希望在整个设计过程中在设备上工作。 在我们看来，这是真正了解用户如何体验完整应用程序的唯一方法。
Constraints such as a limited field of view, the near clipping plane, and brightness/contrast levels can drastically influence the decision-making process, so gaining feedback quickly is of the utmost importance. AR HMDs’ limited field of view imposes many design challenges that we have had to address directly. For example, in Unity MARS, when you are authoring for particular conditions in the environment, matches can populate all around the user, even in another room. Only by prototyping in the device with a large spatial map did we understand how critical it would be to overcome this large range and have a “miniworld” version of the space. Being able to see all the objects in your scene from multiple angles through the miniworld gives you a better sense of the space and reduces moments of occlusion.
诸如有限的视野，接近裁剪平面以及亮度/对比度级别之类的约束会极大地影响决策过程，因此，快速获得反馈至关重要。 AR HMD的视野有限，给我们带来了许多必须直接解决的设计挑战。 例如，在Unity MARS中，当您为环境中的特定条件进行创作时，匹配项可能会散布在用户周围，甚至是在另一个房间中。 仅通过在具有大型空间地图的设备中进行原型制作，我们才能了解克服如此大的范围并拥有“微型世界”版本的空间有多么关键。 能够通过微型世界从多个角度查看场景中的所有对象，从而使您更好地感知空间并减少遮挡的时间。
Miniworld prototype showing a 3d map of your surroundings
Miniworld原型展示了您周围环境的3D地图
The creator is at the center of how we design for spatial computing, and we explore practices of structuring space as a series of layers surrounding the human body. Our clothing, objects, and interior/exterior boundaries are all layers that influence how we think about the potential organization of space. These layers translate to how we perceive space, which is largely based on our relationship with scale.
创作者是我们进行空间计算设计的核心，我们探索将空间构造为围绕人体的一系列层次的实践。 我们的衣服，物品以及内部/外部边界都是影响我们如何思考潜在空间组织的所有层次。 这些层次转化为我们对空间的感知方式，这很大程度上取决于我们与规模的关系。
Understanding scale within the design process is a dense topic. How do spatial UI elements respond and adapt to these varying layers surrounding the user? How much space does it take to perform a given task? What is the right method of manipulation at a given scale? The HMD companion app will favor Unity MARS functionality that is inherently spatial and requires 3D manipulation or benefits from the juxtaposition of real-world contextual information.
在设计过程中了解规模是一个密集的话题。 空间UI元素如何响应并适应用户周围的这些不同层？ 执行给定任务需要多少空间？ 在给定规模下正确的操纵方法是什么？ HMD配套应用将支持Unity MARS功能，该功能本质上是空间性的，需要3D操作或从并置的真实上下文信息中受益。
Prototype for distance manipulation
距离操纵的原型
设计工作流程 (Designing workflows)
The traditional workflows for design, whether for screens or physical objects, don’t meet the needs of current XR development. More than other digital media, XR demands accurate visualizations, faster iteration, and some way to account for ever-changing environments. The spatial team looked to fill in the gaps of traditional processes by customizing and designing our own workflows in order to better understand the complexities of creating XR experiences. Here are some of the workflows that we tried – and what we learned as a result.
无论是屏幕还是物理对象，传统的设计工作流程都无法满足当前XR开发的需求。 与其他数字媒体相比，XR需要更精确的可视化效果，更快的迭代速度以及某种方式应对不断变化的环境。 空间团队希望通过自定义和设计自己的工作流程来填补传统流程的空白，以便更好地了解创建XR体验的复杂性。 这是我们尝试过的一些工作流程，以及我们从中学到的知识。
纸样机 (Paper prototyping)
Spatial design requires an understanding of scale, not just for objects and spaces, but also for bodily movement in order to avoid fatigue. One-to-one paper prototypes shine as a means of testing movement and ergonomics.
为了避免疲劳，空间设计不仅需要了解对象和空间的比例，还需要了解身体的比例。 一对一的纸制原型可以作为测试运动和人体工程学的一种方式。
For example, we designed and tested a virtual tray that would fall or rise based on the user’s head tilt movement. This physical prototype of a paper model tray was hung around the neck, directly below the chin. During a normal work day, we put frequently used tools in the tray, such as a phone, glasses, pens, and so on. After wearing the paper prototype for an hour, the neck movement associated with the summoning interaction was quite tiring, as were the frequent arm movements to access tools near the neck region. Therefore, we chose not to move forward with this design proposal. When designing for mixed reality, building quick-and-dirty paper prototypes can save you time and help you to think spatially.
例如，我们设计并测试了一个虚拟托盘，该托盘会根据用户的头部倾斜运动而下降或上升。 纸模型托盘的这个物理原型悬挂在下巴正下方的脖子上。 在正常工作期间，我们将常用工具放在托盘中，例如电话，眼镜，笔等。 佩戴纸质原型一小时后，与召唤互动相关的颈部运动非常累人，频繁的手臂运动也接近脖子区域的工具。 因此，我们选择不继续执行该设计建议。 在为混合现实设计时，构建快速脏纸原型可以节省您的时间并帮助您进行空间思考。
Paper Prototype for a design proposal for a chin tray for tools
用于工具下巴托盘的设计方案的纸张原型
在VR中模拟AR (Simulating AR in VR)
User flows for mixed reality are difficult to review effectively using traditional 2D tools like presentations and animated mockups. Instead of creating 2D storyboards, we built prototypes in VR using Tvori. Tvori enabled fast iteration while editing and animating interactions. We used the timeline feature to keyframe cameras, potential UI elements, virtual objects, and input devices.
使用传统2D工具(例如演示文稿和动画模型)，很难有效地评估用于混合现实的用户流。 我们没有创建2D故事板，而是使用Tvori在VR中构建了原型。 Tvori在编辑和设置交互动画时启用了快速迭代。 我们将时间轴功能用于关键帧相机，潜在的UI元素，虚拟对象和输入设备。
Flow prototyped in Tvori VR with AR FOV view
使用AR FOV视图在Tvori VR中原型化的流程
Using VR, we were able to change the scale of our world constantly. We could experience the world firsthand at human scale while simultaneously miniaturizing the scene to enable easier editing. When we were presenting flows to others on the team, they could simply put on the VR headset, set the scale to 1:1, and scrub the timeline.
使用VR，我们能够不断改变世界的规模。 我们可以亲身体验人类的世界，同时将场景缩小到最小，以便于编辑。 当我们向团队中的其他人介绍流程时，他们可以简单地戴上VR头盔，将比例设置为1：1，并清理时间线。
In order to get a better understanding of the appropriate scale and proportion of virtual elements, a typical AR headset field of view “safe area” was introduced as an efficient way to quickly simulate AR in VR. The rectangle obstructing the human viewing angle is very practical: as spatial designers, this constraint influences how immersive our experiences are, not only in terms of what we see but also how we move (i.e., hand gestures).
为了更好地了解虚拟元素的适当比例和比例，引入了典型的AR头戴式耳机视场“安全区域”，作为在VR中快速模拟AR的有效方法。 遮挡人类视角的矩形非常实用：作为空间设计师，这种限制不仅影响我们看到的内容，而且影响我们的移动方式(即手势)，从而影响我们的体验身临其境。
用户流和输入映射 (User flow and input mapping )
Input mapping for palette menu
调色板菜单的输入映射
Throughout the design process, we kept two living documents, one for input mapping and the other for user flow. The user flow document was continuously being updated as the source of truth. Here, we determined both how and where a user would accomplish a certain task. Is a task accomplished within a particular workspace – our word for individual pieces of XR UI – or in the real world, or can the task be performed in both?
在整个设计过程中，我们保留了两个活动文档，一个用于输入映射，另一个用于用户流程。 用户流程文档一直在不断更新，以作为事实的来源。 在这里，我们确定了用户完成某项任务的方式和地点。 任务是在特定的工作空间中完成的(我们的意思是XR UI的各个部分)还是在现实世界中完成，或者可以同时在两者中执行任务吗？
We also employed an input mapping exercise to better understand actions being performed in the app and their corresponding frequency. It can be simply described as taking the following steps:
我们还进行了输入映射练习，以更好地了解应用程序中正在执行的动作及其相应的频率。 可以简单地描述为采取以下步骤：
-
List all known actions and rank them in terms of their priority.
列出所有已知的动作，并根据其优先级对其进行排名。
-
Write down how often each action will be performed.
写下每个动作执行的频率。
-
Decide on the type of interaction.
确定交互的类型。
As complexity increased, we had to enhance our prototype fidelity. By using the “mind map” view, we could better understand overlapping inputs and how to adjust accordingly. This understanding is increasingly important with input devices such as the Magic Leap One controller, where the number of inputs can be limited. By requiring hover states for particular actions, we were able to distribute inputs while avoiding conflicts. For generic actions, we decided to use the bumper for a contextual menu and the home tap to summon the main palette menu. Also, in order to address fatigue, we avoided any input that required a trigger or bumper press in combination with the touchpad.
随着复杂性的增加，我们不得不提高原型保真度。 通过使用“思维导图”视图，我们可以更好地理解重叠的输入以及如何进行相应的调整。 对于诸如Magic Leap One控制器之类的输入设备(输入数量可能受到限制)，这种理解变得越来越重要。 通过要求将悬停状态用于特定操作，我们能够分配输入，同时避免冲突。 对于一般动作，我们决定将保险杠用于上下文菜单，同时使用主屏幕水龙头来调用主调色板菜单。 此外，为了解决疲劳问题，我们避免了任何需要与扳机结合使用扳机或保险杠的输入。
快速的设备上迭代 (Rapid on-device iteration )
Using Unity, we assembled 3D models, UI designs, and controller scripts to quickly get designs onto the Magic Leap. The goal was to get feedback from quick iterations of interaction prototypes, while not being concerned with writing perfect code early in the process. In addition to building our own prototypes, we were inspired by the many apps developed by the broader XR community, and we often used them as references to convey particular interaction behaviors.
使用Unity，我们组装了3D模型，UI设计和控制器脚本，以将设计快速引入Magic Leap。 目的是从交互原型的快速迭代中获得反馈，而不必担心在过程的早期编写完美的代码。 除了构建自己的原型外，我们还受到更广泛的XR社区开发的许多应用程序的启发，并且我们经常将它们用作传达特定交互行为的参考。
Most interactions rely on “magic” numbers such as timings, thresholds, and distances. The first step after building a prototype was to tune those values to match the feeling that we imagined for the interaction. In the Unity Editor, we could tweak values while in Play Mode, but we also wanted to be able to adjust values in a runtime build. To help with this, we built a simple reusable settings menu within the app. Any prototype script could use a single line of code to add a value slider or checkbox to tweak those magic numbers.
大多数交互依赖于“魔术”数字，例如时间，阈值和距离。 构建原型后的第一步是调整这些值以匹配我们为交互所想象的感觉。 在Unity Editor中，我们可以在“播放模式”中调整值，但是我们还希望能够在运行时构建中调整值。 为了解决这个问题，我们在应用程序中构建了一个简单的可重用设置菜单。 任何原型脚本都可以使用一行代码来添加值滑块或复选框，以调整这些幻数。
Tuning the “magic” numbers for interaction settings
调整交互设置的“魔术”数字
The next step was to get feedback and iterate. Here are some tips that we found useful when someone else is testing an interaction:
下一步是获取反馈并进行迭代。 以下是一些我们在其他人测试交互时发现有用的提示：
-
Say as little as possible at first. If it’s truly necessary, you can tell a user what they should be trying to do, and maybe briefly explain how. Remember, you don’t come with the software, so you need to observe how they interact without your influence.
开始时尽量少说。 如果确实有必要，您可以告诉用户他们应该尝试做什么，并且可以简要地解释如何做。 请记住， 您 不是随软件一起提供的，因此您需要观察它们如何相互作用 而不受 您的影响。
-
Look for red flags such as confusion, frustration, or surprise, then ask the user what they expected to happen (versus what actually happened).
查找混乱，沮丧或惊讶之类的危险信号，然后询问用户期望发生的事情(相对于实际发生的事情)。
-
If the user is having trouble explaining their feedback, you can stream or record the headset view so they can demonstrate and gesture towards virtual objects.
如果用户在解释他们的反馈时遇到问题，则可以流式传输或记录耳机视图，以便他们可以演示并指向虚拟对象。
UI缩放 (UI scaling)
A virtual object’s scale is hard to measure in the real world, and it should be responsive to users’ viewing distance. We cannot use real-world scale as the only reference, since the user can only sense that the UI is physically larger because of stereoscopic vision and other depth cues such as the ray’s translational movement. If a user closes one eye or looks at a UI without any other context, then they can’t tell how large or far away it is.
虚拟对象的比例在现实世界中很难测量，并且应该响应用户的观看距离。 我们不能将现实世界的比例用作唯一参考，因为 由于立体视觉和其他深度提示(例如射线的平移运动)，用户只能感觉到UI在物理上更大。 如果用户在没有任何其他上下文的情况下闭上眼睛或看着UI，那么他们将无法分辨出它的大小或距离。
We used the distance-independent millimeter (DMM) system created by Google’s Daydream team for VR design. The idea is to scale distance and size proportionately. For example, 1 dmm can refer to the size of a 1*1 mm button placed 1 meter away; it can also refer to the size of a 3*3 mm button placed 3 meters away. Using the dmm system, Google provided a set of suggested font and minimum ray-based hit sizes, such as 20 dmm as the smallest font size and 64 dmm with 16 dmm padding as the minimum hit size.
我们使用 Google的Daydream团队创建 的 距离无关的毫米(DMM)系统 进行VR设计。 这个想法是按比例缩放距离和大小。 例如，1 dmm表示放置在1米外的1 * 1 mm按钮的大小； 它也可以指放在3米外的3 * 3毫米按钮的尺寸。 Google使用dmm系统提供了一组建议的字体和基于射线的最小命中尺寸，例如最小字体为20 dmm，最小命中尺寸为64 dmm，填充为16 dmm。
When we tested the suggested sizes with a Magic Leap, we felt that they were too large, and we ended up making things a bit smaller. This is likely because of the Magic Leap’s smaller field of view and 6DoF controller, compared to Daydream’s higher FOV and 3DoF controller. When the UI scales proportionally with distance, it renders at the same pixel size and covers the same angular region for a raycast, but we noticed that it feels very different. We’d be curious to hear from the community if they have similar findings.
当我们使用Magic Leap测试建议的尺寸时，我们觉得它们太大了，最终使尺寸缩小了一些。 与Daydream较高的FOV和3DoF控制器相比，这可能是因为Magic Leap的视场和6DoF控制器更小。 当UI与距离成比例缩放时，它 会以相同的像素大小进行渲染，并为光线投射覆盖相同的角度区域， 但是我们注意到它的感觉非常不同。 我们很想听听社区是否有类似的发现。
视觉设计 (Visual design)
The spatial UI design for the companion app is centered around the concept of using light as a material. The layering of light and shadow helps to create a more tactile experience, and it is meant to enhance the user’s perception of depth in an AR HMD. Ultimately, light is relational, in that we only perceive it by how it’s shaped by other materials and surfaces. All of the spatial UI elements in the companion app are treated as if they were a physical construction and carefully considers the relationship between the light source and the plane of light.
配套应用程序的空间UI设计以使用 光作为材料 的概念为中心 。 光线和阴影的分层有助于创建更多的触觉体验，并且旨在增强用户对AR HMD的深度感知。 最终，光是相关的，因为我们只能通过其他材料和表面的形状来感知它。 伴随应用程序中的所有空间UI元素都被视为物理结构，并仔细考虑了光源和光平面之间的关系。
This effect is created by employing a negative shadow technique. AR head-mounted displays use a light-based additive color space. Black renders transparent, as it is the absence of light. If a black object is positioned in front of other virtual objects, it acts as a mask. In order to create the illusion of shadows, we have to surround the darker shadow area with light. Human perceptual systems rely heavily on the cues provided by shadows to infer the motion and depth of 3D objects. If AR apps built for these devices neglect to incorporate shadows, we deprive the user of important depth cues.
通过使用负阴影技术可以创建此效果。 AR头戴式显示器使用基于光的附加色彩空间。 黑色使透明，因为它没有光线。 如果黑色对象位于其他虚拟对象的前面，则它将充当遮罩。 为了产生阴影的错觉，我们必须用光围绕较暗的阴影区域。 人类的感知系统严重依赖于阴影提供的提示来推断3D对象的运动和深度。 如果为这些设备构建的AR应用程序忽略了合并阴影，那么我们将使用户失去重要的深度提示。
Depth cues provided to the user, button hover shadow (Top) and textured legibility mask (Bottom)
提供给用户的深度提示，按钮悬停阴影(上)和带纹理的易读性面具(下)
In addition to shadows, the use of texture gives users cues to understand scale and the relative position of objects. Banding is a problem when dealing with color in light-based HMDs and often occurs when using high-contrast gradients. Using this as a constraint, we decided to incorporate the use of texture into the design of the spatial UI elements, not only to expand the range of colors we could use for gradients, but also as another way to improve the perception of depth within the authoring experience.
除阴影外，纹理的使用还为用户提供了提示，以了解比例和对象的相对位置。 在基于光的HMD中处理颜色时，条带化是一个问题，当使用高对比度渐变时，经常会出现条带化。 以此为约束，我们决定将纹理的使用纳入空间UI元素的设计中，这不仅是为了扩展我们可以用于渐变的颜色范围，而且是提高对内部深度的感知的另一种方法。创作经验。
Baked shadow texture maps with noise for palette menu (Left) and context menu (Right)
调色板菜单(左)和上下文菜单(右)的带有阴影的烘焙阴影纹理贴图
The relationship of a spatial UI element and a light source is designed to highlight the potential behavior of the interface. These visual (alongside audio and physical) modalities help to communicate affordances or provide input feedback to the user. Much like in our everyday spaces, light accents create visual interest and highlight objects or specific features of a space. Borrowing from these affordances found in the physical world, we created various relationships between the light source and spatial UI elements that resemble embedded, cove, and recessed lighting conditions.
空间UI元素和光源的关系旨在突出显示界面的潜在行为。 这些视觉(以及音频和物理方式)形式有助于传达能力或向用户提供输入反馈。 就像在我们的日常空间中一样，浅色点缀可以营造视觉趣味并突出显示空间中的物体或特定特征。 借用现实世界中的这些能力，我们在光源和空间UI元素之间创建了各种关系，这些关系类似于嵌入式，凹形和凹入式照明条件。
Input feedback visual/motion studies for spatial UI elements
输入用于空间UI元素的反馈视觉/运动研究
下一步是什么 (What’s next)
We are always searching for new workflows that can help us to build better experiences. We hope the learnings we’ve shared in this post help you with your own spatial designs, and please don’t hesitate to reach out with techniques that you have discovered. As a community of creators, sharing knowledge is crucial to shaping this new medium and enabling us to create things we never thought were possible.
我们一直在寻找可以帮助我们建立更好体验的新工作流程。 希望我们在这篇文章中分享的学习知识对您自己的空间设计有所帮助，并且请不要犹豫，与您发现的技术取得联系。 作为一个创作者社区，共享知识对于塑造这种新媒体以及使我们能够创造出我们认为不可能的事情至关重要。
If you are interested in using the Unity MARS Magic Leap companion app in your own project, please help us to better understand your needs by taking this brief survey. We’re looking forward to hearing from you.
如果您有兴趣在自己的项目中使用Unity MARS Magic Leap配套应用程序，请通过这份 简短的调查 帮助我们更好地了解您的需求 。 我们期待着您的回音。
unity混合树