SOLVED: What is Holoportation & How Is It Different From Augmented Reality, Mixed Reality and Microsoft Mesh

Holoporation is something that blows people away when they see that it is no longer science fiction, but to explain it we need some brief background on Virtual Reality (VR),  Augmented Reality (AR) and Mixed Reality (MR).

Here is the video version of this article:

What is Virtual Reality (VR)?

We think everyone understands VR now, so let us make this short by simply saying VR is just an immersive viewing of a digital world that you can move through.  So you can turn your head left and see what is on the left.  Very cool, but not new.

What is Augmented Reality (AR)?

Augmented Reality is the overlaying of digital objects (animals, products, maps, information bubbles, people…) into your real world.  Here are some examples of Augmented Reality (AR)

1. if you are walking down the street and your glasses are overlaying the names of the stores, the times they are open and maybe the menus of the restaurants in your line of site, that is AR.
2. if you are in your living room and you are looking at your mobile phone which has its camera turned on (so you can see ‘through’ the phone), and you see a cute digital character like Pikachu sitting in a chair, that is AR
3. if you are driving your car and a tiny projector in the dashboard is showing the map / driving directions of where you need to go, right on the windshield in front of you, that is AR.

What is Mixed Reality (MR)?

Mixed reality is just Augmented Reality that you can interact with.  So lets look at some examples:

1. Perhaps you are training to repair jet engines.  They are expensive so the airline that owns that engine might not want a new mechanic to start pulling wires and disconnecting fluid lines on a real engine.  Emirates Airlines has their mechanics train on digital jet engines.  The student mechanics can open little doors, pull wires, unscrew bolts… all on an engine that does not exist but the Mechanic can both see and interact with it using Microsoft Hololens ‘glasses’.  That is Mixed Reality.
2. Perhaps you are a sick child who is allergic to dogs but all you want is a dog.  If you can put on a pair of glasses and see what looks like a real dog, and you can throw a digital ball that the dog will fetch, or you can give the dog commands and have him try to ‘shake paw’ or you push him over to pet him, that is Mixed Reality.
3. What if you could interact with movies you were watching?  What if the main character is walking ahead and YOU decide that you want to see what is in the closet, so you just turn your head, reach your hand out, grab the handle and pull it open.  Remember, this is a movie and that door 🚪 is really not in the middle of your living room, but you can see it and you can interact with it; that is Mixed Reality.

What is Microsoft Mesh?

Microsoft Mesh is a new infrastructure that allows developers to make Mixed Reality work on most platforms and with more than just one person.  I know that contains some jargon, so I will break that down here.

Using Microsoft Mesh you could not only interact with digital things (aka Mixed Reality) but you AND OTHERS AT THE SAME TIME use a mobile phone, or tablet, or PC, or HTC Vive headset, or Hololens, or another computer that has a camera on it to see what is in front of you and a screen to let that computer overlay digital things that you can interact with.

Here are some examples of Microsoft Mesh:

1. If you are in a virtual meeting (Zoom/Teams/Skype/House Party/Instagram…) and on the screen of your cell / tablet you can see the prototype of a new car that you want to walk around and open the doors on by ‘pinching and zooming’ your screen with your fingers.  In the same meeting is someone else wearing digital glasses (Hololens, Google Glasses…) and they see you opening that door as you do it, and you see them walking around the other side of the car; that is what Microsoft Mesh makes possible.
2. If you are walking down the street and you hold up your phone so the camera can see the buildings and the screen also displays a menu of the restaurants which you can flip through.  When you find what you want you call the restaurant and instead of just getting a voice on the phone, you see a representation of the person taking your order and they can see a digital representation of you.  That is what Microsoft Mesh makes possible.
3. If you are wearing digital glasses (Google Glass, Microsoft Hololens, HTC Vive…) and you see someone you know waaaay across the room, wouldn’t it be great if you could just ‘touch’ them and start a video call?  But what if they are in a hurry and can’t stay so they turn down a side street.  With Microsoft Mesh it would be possible for the two of you to continue to see avatars of each other and interact.  You could then also bring in another person (or two or more) into the meeting.  That is what Microsoft Mesh makes possible.

What is Holoporation?

The year 2020 was a mess for schools, families and businesses alike but one of the things that made it more bearable was video meetings.  In the end, society was pushed forward about 5 years and we realized that face to face physical meetings are not as critical to success as we thought.  However, I think everyone agrees that a Zoom meeting is never going to be as impactful as a physical encounter.  What if there was something between a video chat and a physical meeting?  That is what Microsoft has invented.

Do you remember how Star Trek was able to “beam” people from one place to another?  Well, that isn’t going to happen anytime soon, but what is real today is holoportation enabled by Microsoft Mesh.  Holoportation puts a digital avatar of you (likely with a photo of your real face with eyes that blink and mouths that move to the words!), in a space with others that you can fulling interact with.

Take a look at this old Microsoft demonstration video from 2016 and you can imagine how good it is today:

Here is the lead developer of Microsoft Mesh to explain: