Hey guys, it’s me, António again. 👋 Last time we interacted I was writing about how we used an external sensor to turn iPhones into breathing rate monitors for Brythm. This time however, I’m going to tell you all about the amazing world of Augmented Reality (or AR for short).

This sudden thematic shift didn’t come out of nowhere but rather from Apple’s WWDC 2018 event, where they showcased many projects they are working on, how far they’ve come, and how there is still so much to improve upon. Among their many keynotes, one of them touched on their latest AR endeavor: ARKit 2. This in turn peaked my interest in the subject and, since I had some free time on my hands and wanted a new pet project, well… here we are 😅 so without further ado, let’s dive right in.

# Definition of AR

What is Augmented Reality after all? Plenty of people mistake it for Virtual Reality but I hope that by the end of this post you’ll be able to spot their differences as clearly as night 🌙 🌃 and day ☀️ 🏙.

As you might have noticed, your physical body and mind exist in a mix of space and time which you perceive as Reality, and nowadays you can even “exist” inside a computer-generated mix of space and time known as Virtual Reality. **However, in between these two extremes there can exist other types of reality where “normal” Reality and Virtual Reality intertwine to varying degrees, one of which being **Augmented Reality.

Types of reality. No, really…

As we can see, Augmented Reality as a concept implies Reality with a bit of Virtual Reality added for extra flavour, but not so much as to overwhelm it, or put into another perspective, the emphasis should be on bringing virtual elements into the physical world rather than the other way around. This all seems fine as a concept, but it brings us to the following question (or any less esoteric version of it):

# What properties and rules should an Augmented Reality experience abide by?

As it turns out, there are three key components to this:

  • 1) it must combine virtual and real information, with the real world as the primary place of action;
  • 2) it must be interactive with real-time updates;
  • 3) it must have virtual information registered in 3D space, in the physical environment.

In essence, while moving in a physical environment, the user’s actions (movement/interaction) have direct consequences on the rendered virtual elements, and all of this happens in real-time. If you pay close attention, these rules do not discriminate a specific output device or interaction medium, meaning that, in theory, AR is not limited to displays and visual stimuli. 🤔However, audio, haptic, olfactory and gustatory AR is more difficult to achieve.

# A brief history

Now that we know what AR is, you might be inclined to believe that it hasn’t been around for long, and you’d be wrong. 😶 Augmented Reality as a concept has been around for decades, and practical examples have been diverse and plentiful, so let’s take a quick look at how far it all has gotten throughout the years. I promise to not bore you to death with gruesome mathematical details, my role in all of this is showcasing how awesome AR has become! 😜

Sparing any previous intentions of using simple overlays to add information to existing tasks, it all started when Ivan Sutherland first proposed the “ultimate display” in 1965 and realized it in 1968. This head-mounted display (HMD) was nicknamed Sword of Damocles, since it had to be hanged from the ceiling and over the user’s head. It included head tracking, three-dimensional movement freedom and see-through optics.

Sword of Damocles, Ivan Sutherland, 1968

And thus, VR and AR were born! As theoretical concepts… It took a couple of decades for technology to catch up, and in the 1990’s AR found itself real-life applications for assisting workers with indoors assembly and maintenance.

KARMA was the first knowledge-driven AR application. A user with an HMD could see instructions on printer maintenance. Courtesy of Steve Feiner, Blair MacIntyre, and Doreé Seligmann, Columbia University.

Up until 1999, AR software and hardware was only used for academic research and development, or in very specific factory settings, but this changed with the release of ARToolKit. It was the first open-source software platform for AR and featured a 3D tracking library using black-and-white fiducials (also known as fiducial markers), which could readily be printed on a regular laser printer.

# Fiducial what?

Wait a minute, let’s revisit that idea. If you’re familiar with barcodes and QR Codes, you won’t be shocked to know that fiducial markers aren’t that different. All of them are easy to produce, uniquely identifiable, machine-readable visual representations of data; but whereas barcodes and QR codes are typically used to store information about the item they are placed on (ID number, source of origin, a relevant URL, etc.), fiducials are used to establish position, orientation, and scale in relation to them.

Examples of fiducials used in AR software

As you’ve probably already guessed, fiducial markers are heralded as lifesavers for AR. 🙌 They make it extremely easy to attach virtual elements to physical world objects, relying only on “simple” computer vision algorithms and minimal real-world intrusion.

How a fiducial marker is used to position virtual elements in the physical space

# Back to the history lesson

By the end of the 1990’s reliable webcams were a given, and in the early 2000’s we witnessed the surge of mobile devices, so it was only a matter of time until the first handheld devices capable of AR were developed. We’re not even talking about smartphones at this point! Remember PDAs? Boy, was that a long time ago…

Anyway, handheld devices suddenly became reliable alternatives to head-mounted displays, and it should come as no surprise that in recent years they have become the most cost-effective way of developing Augmented Reality experiences, and more importantly, making them available to a wider user base. Nowadays, almost any smartphone is capable of supporting AR. All it needs is a camera, a display, and enough computing power to assure “real-time” interactions with the rendered virtual elements. Check, check, and triple check. ✅

So as we’ve seen, AR has come a long way from its origins, and nowadays it’s already performing feats of wonder, but if you think this is it you better hold on to your socks, because you’re in for a ride! 🎢 Let me show you a bit of what AR is already capable of and what you can expect in the future. 🔮

# Showcasing AR

Chances are, you’ve probably benefited from AR in your day-to-day life without even noticing it. For instance, have you ever seen or used one of these parking assistants?

AR parking assistant

Yup, by definition, they are a perfectly good and simple example of augmented reality: the computer-generated elements help you gauge distances and trajectories in real-time! I know, I know, self-driving and self-parking cars are all the rage now, and they’ll pretty much make this example useless in the coming years, but I bet you never once thought of these displays as an awesome piece of AR.

Not a fan of cars? That’s fine, what are your thoughts on sports? Using fixed position cameras, it becomes easier to overlay additional information on some of the most popular sports, such as football 🏈, soccer ⚽️, and even pool 🎱.

AR visualization of an ongoing soccer match

Augmented reality has also seeped into gaming, and unless you have been living under a rock, you probably heard of last year’s sensation: Pokemon Go. Each year, more and more AR and VR games are added to app stores; especially AR-based games, as they don’t usually require additional expensive hardware to be purchased.

# What’s next?

Currently, Augmented and Virtual Reality’s progress is still being halted by the same challenges as decades ago:

  • Portable power supplies
  • High-definition portable displays
  • Processing power

If you think your modern smartphone doesn’t last long enough on a single charge, imagine the toll a fully fledged AR or VR app can have your phone’s battery. Spoiler alert: it doesn’t stand a chance. ⚡ ️🔋 This means that you can’t stroll around in your augmented world for too long, at least not without compromising your ability to further use your smartphone.

But wait, that’s not all! There’s still the issue of current portable displays not having enough resolution or not being portable enough to carry around with us. Once again looking at our most faithful companions, smartphone screens have grown since their conception, but they seem to have reached our pocket-size limit for the time being. This is because we are attempting to preserve their portability whilst wanting to achieve better screen resolutions. Tablets, on the other hand, discard some of their portability for the added screen real estate.

Last but certainly not least, even though a single mobile device now has more computing power than NASA when sending men to the moon (yup, let that sink in for a second), the most prevalent devices still can’t keep up with complex AR and VR scenes as smoothly as a reasonably priced gaming computer.

# Existing Tools and Frameworks

So by now, you’re probably wondering “How can I get started with my amazing AR app?”, and before I leave you to fend for yourselves in the amazing world of AR, I thought I’d list the current industry standards for building and testing out Augmented Reality apps.

[ARToolKit](https://github.com/artoolkit/artoolkit5) — As we’ve mentioned before, this was the first widely available and open-source library focused on AR. In order to create strong augmented reality, it uses video tracking capabilities that calculate the real camera position and orientation relative to square physical markers or natural feature markers in real time. Once the real camera position is known, a virtual camera can be positioned at the same point and 3D computer graphics models can be drawn, exactly overlaid on the real marker. So ARToolKit solves two of the key problems in Augmented Reality: viewpoint tracking and virtual object interaction. Moreover, ARToolKit is available for the most common platforms: Windows, macOS, Linux, iOS, and Android.

[ARKit](https://developer.apple.com/arkit/) — With their latest A11 chips and depth-perceiving cameras, Apple has taken a leap forward by releasing ARKit to iOS developers. It provides persistent AR experiences that persist between sessions and can be resumed at a later time. Moreover, these sessions can be shared by multiple users, each with their iOS device. Of course ARKit also provides Image and Object Detection and Tracking: ARKit 1.5 added support for 2D image detection, letting you trigger an AR experience based on 2D images like posters, artwork, or signs; ARKit 2 extends this support to offer full 2D image tracking, so you can incorporate movable objects like product boxes or magazines into your AR experiences; ARKit 2 also adds the ability to detect known 3D objects like sculptures, toys, or furniture.

[ARCore](https://developers.google.com/ar/discover/) — Google has also developed a platform for building augmented reality experiences for Android and iOS. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS to enable shared AR experiences. ARCore uses your phone’s camera for motion tracking, allowing it to understand and track its position relative to the world. Afterwards, it uses environmental understanding to detect the size and location of all type of surfaces: horizontal, vertical and angled surfaces like the ground, a coffee table or walls. Finally, it performs light estimation to estimate the environment’s current lighting conditions.

Believe me, we’re only scratching the surface regarding what these libraries are capable of, and now is the perfect time to pick one up and push them to their limit. With such an arsenal at our disposal, the future of AR sure is looking bright! 😎 💡Honorable mentions go out to Vuforia and Wikitude as sensible alternatives to ARToolKit, but be mindful that I’m no expert on the subject (Yet!), so don’t take my word too seriously. There are plenty of alternatives out there, so take a look for yourself 😅

# Wrapping up 🏁

I hope this post has inspired you to look into AR or has taught you something new. I’m going to leave a bunch of materials at the end of the page, but feel free to comment or send me any other interesting/helpful videos or links you might have found. There’s so much more that I’d love to discuss and share regarding the current and future applications of AR, but by now I’ve probably exhausted your attention, so let’s leave it for another time.

As for me, I’ve decided to take ARKit 2 for a test drive, so who knows if I’ll come back in the following days/weeks/months to show off what I’ve made or showcase what I learnt. See you soon! 👋



# History of AR

# ARKit

# ARKit at WWDC 2018

# ARCore

# ARToolKit

# AR Use Cases

Originally published on Medium