Developing for HoloLens: Tips, tricks, and technical challenges

Common challenges of working with the HoloLens, and some ways to get around them

The HoloLens does a lot of things very well. If used correctly, holograms really look as if they’ve been placed in the real world. Spatial sound is accurate; you can easily pinpoint where a sound is coming from.  Colours are vibrant, spatial anchoring is reliable, and the device doesn’t weigh too heavily on your head.

There are some things, though, that can cause you some grief if you don’t think about them from the get-go.

Testing

Testing on the HoloLens can take a little while, as you’ll need to build and deploy your application to the device. A huge time saver here is to make good use of the emulator, as well as the holographic remoting capabilities of the device.

There are some things that are difficult to test without an actual build, but most things can be efficiently tested by simply opening the Holographic Remoting application in the HoloLens, connecting through Unity, and running the application. This brings the testing workflow much closer to that of a normal Unity app.

Logistics

Wearing the device on and off for 8 hours can be tiring.  Of course, this won’t be a problem for the majority of users.  But when developing for the HoloLens I spend a lot of time taking the device on and off my head and testing the same features repeatedly.  If you’re going to be using the device all day, my advice is to make sure you fit it correctly.  Especially if you’ve got glasses.  The HoloLens is very adjustable, and can be altered to fit almost any head shape without pressing, squeezing or digging in to your head.

Input

Input on the HoloLens is limited to a couple of gestures, centered around the ‘Air Tap’. This is a gesture where you raise your hand up in front of you, hold a finger up in the air, and tap it down and up, so that the device sees your finger disappear and then reappear.  This is a ‘click’ on the HoloLens. Dragging is an extension of this, where you lower your finger, move your hand, and then raise it back up.  This gesture can be tiring to perform repeatedly, and people can find it a little awkward to get to grips with, so make sure that:

You remember that there’s a clicker that comes with the device that you can use (keep it charged!)

Show people the gesture tutorial that comes loaded on the device before you set them loose

In a perfect world, you could record and train your own custom gestures. Personally, I really like the ‘Bloom’ gesture that is used for opening the main menu, and would like a few more gestures like that to be available for developers to use.

Field of view

‍

Another area that people commonly complain about is the field of view (FOV). The current FOV measures approximately 32x17.5 degrees. This is not a limitation of processing power in the device, but a physical limitation, because of the way the device works. This is not just a limitation of the HoloLens, either, but a limitation of all mixed reality headsets. The HoloLens is actually getting close to best field of view possible with current technology. Light cannot be easily bent to arbitrary angles, and so, depending on the materials used for the lenses, the FOV of all mixed reality headsets will have a limit until there is a new breakthrough in the display technology.

Whether the field of view on these devices is a problem depends on the use-case. If developed correctly, applications can use things such as sound cues and fading to avoid the user noticing at all.  A very impressive example of this is RoboRaid, a game that comes with the device – you wouldn’t even know there was a limitation when playing this. The FOV is also hardly noticeable when viewing normal objects or images, as you tend to turn to face objects when you’re looking at them.

Getting files to and from the HoloLens

We encountered a few technical challenges when developing for the HoloLens, as is to be expected when working with such a new piece of technology. One of the more significant challenges was how we wanted to communicate with the device.

In our Holo Exhibition app, we needed to allow a user of our app to put their files on the device. These could be anything from text files to images, videos or PDFs. In a regular application, this would be trivial – they could simply choose via a folder picker where their files were located. This gets a little more complicated when you need to be able to send files to a remote device. In this section, I’ll go into some detail on how we achieved this, what we tried, and what was deceptively difficult.

Attempt 1: Azure API

Our first attempt at getting files to the device was to get them from an API. Our application requested JSON meta-data describing what files were available, and some information about them, and the application could then choose the specific files to download.

This was not only a bit awkward to update, but took a long time to load. A video could take minutes, which isn’t an acceptable load time.  This load time had a real effect on the responsiveness of the application. The downloading also used a lot of bandwidth. Even with caching, this wasn’t a sustainable option.

Initially, we requested the image, text, video etc as we needed them – as the user placed the object in the scene.  We then cached this, so that it would be instant that next time they placed that item. When we realised how slow this was, we decided to begin downloading all of the objects as soon as you run the application, which helped to speed things up. This still meant you had to wait for some objects to become available, however, and still meant issues with bandwidth, especially with large collections.

Attempt 2: OneDrive

We thought we’d found our solution when we decided to use OneDrive to transfer the files across. The plan was to be able to simply throw the files we wanted into OneDrive from a PC, and to then automatically have them sync to the HoloLens for use in our application.

Unfortunately, this wasn’t quite as easy as we thought it would be: syncing the files worked fine, connecting to OneDrive on the device worked fine, but we couldn’t manage to load the files into the device. There is no ‘File Explorer’ on the HoloLens.  You don’t have permission, as a developer, to access the C drive directly. The functionality provided by Microsoft to load files is called the FileOpenPicker – a custom utility for opening single files. Unfortunately for us, you can’t open a whole folder using the file picker; this seemingly small difference meant that the user would have to hand pick every individual file that they might want to use in their exhibition, which simply wouldn’t be practical. We needed to be able to load the entire folder, with potentially dozens of items, into the application.

Attempt 3: The Bluetooth Companion App

And so we continued our search, until we stumbled across a great article by Mike Taulty: Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library.

In this article, Mike talks about a way to connect and interact with the HoloLens through another PC.  The use case he discusses and develops is to change the colour of an object in the HoloLens by pressing a button on the other device. We realised that this could be just what we needed to transfer files over. It would also allow us to send any other commands to the device in the future. So, we built a Bluetooth companion application for our Holo Exhibition app.

Connecting the two devices is simple.You run the companion app, it begins to advertise its connection, and you press connect in the HoloLens menu. No passwords, no file pickers.  Once paired, folders can be sent to the device. Because it’s over a local network, this is very fast.

You can do two important things with this app: Sync folders from your PC directly onto the HoloLens, and clear all of the folders that you’ve already synced off of the HoloLens. Having a companion app also means that we can, in the future, add functionality to interact with the HoloLens from a computer or a mobile phone. Applications for this include manipulating things using your smartphone as a controller, capturing images from the device using the companion app, managing save data and so on.

We were quite happy with the outcome of this exploration. We ended up with an efficient solution, with no overhead server costs, and with room to add any other multi-device interaction features in the future.

In conclusion...

Each of the solutions we attempted has its own use case and benefits – here’s a wrap-up.

If you want to allow the user to upload a file to your application, OneDrive is a great way to go. The user only needs to sign in to their Microsoft account once on the HoloLens, and can then sync their files by just putting their files on OneDrive. For many projects, this would be the ideal solution.

If you’ve got a data source that updates regularly, with many smaller files, or if you’ve got a bunch of files that you want to distribute to your users, an Azure API is ideal.

If you want a more bespoke solution, a low-level socket connection over Bluetooth or Wifi is a good bet. This allows you to communicate directly to your application, giving you full control.

Developing for HoloLens and the Universal Windows Platform can seem restrictive, but these restrictions are often there for the sake of security. They mean that you can trust applications not to completely break your device, or delete your files. If you need to give yourself full control of an application, you can, but oftentimes the built-in solutions will be perfectly suitable.