A browser plugin for Unity

Unity's development tools and engine are far and away the most common way to build applications for VR and AR today. Previously, we've made it possible to export web-based experiences from Unity. Today, we're excited to show some early work addressing the other way that Unity developers want to use the web: as a component in their Unity-based virtual environments.

Building on our work porting a browser engine to many platforms and embedding scenarios, including as Firefox Reality AR for HoloLens 2, we have built a new Unity component based on Servo, a modern web engine written in the Rust language.

The Unity engine has a very adaptable multi-platform plugin system with a healthy ecosystem of third-party plugins, both open-source and proprietary. The plugin system allows us to run OS-native modules and connect them directly to components executing in the Unity scripting environment.

The goals of the experiments were to build a Unity native plugin and a set of Unity C# script components that would allow third parties to incorporate Servo browser windows into Unity scenes, and optionally, provide support for using the browser surface in VR and AR apps built in Unity.

Today, we’re releasing a fully-functional prototype of the Servo web browser running inside a Unity plugin. This is an early-stage look into our work, but we know excitement is high for this kind of solution, so we hope you’ll try out this prototype, provide your feedback, and join us in building things with it. The version released today targets the macOS platform, but we will add some of the other platforms supported by Servo very soon.

Getting started

We’ve open-sourced the plugin, at https://github.com/MozillaReality/servo-unity. Head on over, click the star and fork the code, check it out to your local machine, and then open the project inside Unity.

Developer instructions are in the README file in the repository.

What it does

You can work directly with the browser window and controls inside the Unity Editor. Top-level config is on the ServoUnityController object. Other important objects in the scene include the ServoUnityWindow, ServoUnityNavbarController, and ServoUnityMousePointer.

The ServoUnityWindow can be positioned anywhere in a Unity scene. Here, we’ve dropped it into the Mozilla mushroom cave (familiar to users of Firefox Reality, by the amazing artist Jasmin Habezai-Fekri), and provided a camera manipulator that allows us to move around the scene and see that it is a 3D view of the browser content.

Servo has high-quality media playback via the GStreamer framework, including audio support. Here we’re viewing sample MPEG4 video, running inside a deployed Unity player build.

Customizable search is included in the plugin. A wide variety of web content is viewable with the current version of Servo, with greater web compatibility being actively worked on (more on that below). WebGL content works too.

How it works

Architecture

Development in Unity uses a component-based architecture, where Unity executes user code attached to GameObjects, organised into scenes. Users customise GameObjects by attaching scripts which execute in a C# environment, either using the Mono runtime or the IL2CPP ahead-of-time compiler. The Unity event lifecycle is accessible to user scripts inheriting from the Unity C# class MonoBehaviour. User scripts can invoke native code in plugins (which are just OS-native dynamic shared objects) via the C# runtime’s P/Invoke mechanism. In fact, Unity’s core itself is implemented in C++ and provides native code in plugins with a second set of C/C++-accessible interfaces to assist in some low-level plugin tasks.

Servo is itself a complex piece of software. By design, most of its non user-facing functionality is compiled into a Rust library, libservo. For this first phase of the project, we make use of a simplified C-compatible interface in another Rust library named libsimpleservo2. This library exposes C-callable functions and callback hooks to control the browser and view its output. Around libsimpleservo2, we put in place native C++ abstractions that encapsulate the Unity model of threads and rendering, and expose a Unity-callable set of interfaces that are in turn operated by our C# script components.

Getting the browser content into Unity

We create an object in Unity, an instance of ServoUnityWindow, to wrap an instance of Unity’s Texture2D class and treat it as a browser content pane. When using Unity’s OpenGL renderer, the Texture2D class is backed by a native OpenGL texture, and we pass the OpenGL texture “name” (i.e. an ID) to the plugin, which binds the texture to a framebuffer object which receives the final composited texture from Servo.

As we do not have control over the binding of the texture and the Unity context, the current design for updating this texture uses a blit (copy) via Servo’s surfman-chains API. Essentially, Servo’s WebRender writes to an OS-specific surface buffer on one thread, and then this surface buffer is bound read-only to Unity’s render thread and a texture copy is made using OpenGL APIs. In the initial macOS implementation for example, the surface buffer is an IOSurface which can be zero-cost moved between threads, allowing an efficient implementation where the browser compositor can write in a different thread to the thread displaying the texture in Unity.

Control and page meta-data is communicated separately, via a set of APIs that allow search and navigation to URLs, updating of page titles, and the usual back/forward/stop/home button set.

Because the browser content and controls are all ultimately Unity objects, the Unity application you’re building can position, style, or programmatically control these in any way you like.

Challenges

Getting the project to this stage has not been without its challenges, some of which we are still addressing. Unity’s scripting environment runs largely single-threaded, with the exception of rendering operations which take place on a separate thread on a different cadence. Servo, however, spawns potentially dozens of lightweight threads for a variety of tasks. We have taken care to marshal returning work items from Servo back to the correct threads in Unity. There are some remaining optimizations to be made in deciding when to refresh the Unity texture. Currently, it just refreshes every frame, but we are adding an API to the embedding interface to allow finer-grained control.

As an incubator for browser technology, Servo is focused on developing new technologies. Notable tech that has moved from Servo to the Gecko engine powering Firefox include the GPU-based rendering engine WebRender, and the CSS engine Stylo. Those successes aside, full web compatibility is still an area where Servo has a significant gap, as we have focused primarily on big improvements for the user and specific experiences over the long tail of the web. A recent effort by the Servo community has seen great advances in Servo’s webcompat, so we expect the subset of the web browsable by Servo to continue to grow rapidly.

Development plans

Supporting the full range of platforms currently supported by Servo is our first follow-up development priority, with Windows Win32 and Windows UWP support at the top of the list. Many of you have seen our Firefox Reality AR for HoloLens 2 app, and UWP support will allow you to build Servo into a your own AR apps for the HoloLens platform using the same underlying browser engine.

We’d also like to support a greater subset of the full browser capability. High on the list is multiple-window support. We’re currently working on graduating the plugin from the libsimpleservo2 interface to a new interface that will allow applications to instantiate multiple windows, tabs, and implement features like history, bookmarks and more.

This first release is focused on the web browsable through 2D web pages. Servo also supports the immersive web through the WebXR API, and we’re exploring connecting WebXR to Unity’s XR hardware support through the plugin interface. We’ll be starting with support for viewing 360° video, which we know from our Firefox Reality user base is a prime use case for the browser.

Finally…

Whether it’s a media player, an in-game interface to the open web, browser-as-UI, bringing in specific web experiences, or the myriad of other possibilities, we can’t wait to see some of the imaginative ways developers will exploit Servo’s power and performance inside Unity-built apps.