Project 1 | CS 428 | Saksham Khatod

Introduction

This project was originally created for CS 428 : Virtual, Augmented, Mixed Reality. It is an AR application built using Unity and the Vuphoria Engine. It allows users to see two monuments : Eiffel Tower & Statue of Liberty on top of paper cubes. It basically augments physical objects by turning them into live interactive computer-generated objects which show monuments, show the time & the weather at the place of the monument as well as an interesting fact about the monument. These information are overlayed on the sides of the cubes and the main monument model is put on the top of it. If you get close enough to the cube, you can hear relevant music w.r.t the location and you can even change the lighting by rotating the cube on the z-axis until it’s upside down, waiting for a few seconds, and then turning it back up. This application also creates a dynamic laptop sticker for the astronaut image. This dynamic sticker is basically a spiderman fight sequence. It includes Spiderman along with two evil spiders on the wall of a building. The below image demonstrates what this application can do.

To download & run the project on your own computers or smartphones, follow the instructions in this article.

Downloading the project

You can find the project here

Click the Green “Code” Button, to reveal the cloning window.

If you’re not familiar with the command line, i recommend just clicking on the “Download ZIP” option to download the project into your downloads folder.

If you’re familiar with the command line, then you can clone using the HTTPS option. Make sure you click on the “HTTPS” tab. There you will see a text field with a link inside and a copy button on the right. Click that button to copy the link onto your clipboard.

If you’re on mac or linux, open terminal on your computer. Make sure you have git installed. Run git —version to see if it outputs a version no.

Go to the directory where you want to download this project using “cd” command. Once there, run
git clone paste_link_from__your_clipboard
This should download a zip file containing the project right in your chosen directory.

If you’re on windows, open Powershell. Make sure you have git installed. Run git —version to see if it outputs a version no. If it’s not installed, check the next section on installing git. Enter the command
git clone paste_link_from_your_clipboard
and hit Enter key. This should download a zip file containing the project right in your chosen directory.

Installing Git

If you’re on Mac, follow these steps :-

Download the latest Git for Mac installer
Follow the prompts to install Git.
Open a terminal and verify the installation was successful by executing this command : git --version

If you’re on Linux, follow these steps :-

From your terminal, install Git using the following two commands
sudo apt-get update
sudo apt-get install git
Verify the installation was successful by executing
git —version
in the terminal.

If you’re on Windows, follow these steps :-

Download the latest Git for Windows installer
When you’ve successfully started the installer, you should see the Git Setup wizard screen. Follow the Next and Finish prompts to complete the installation. This should install git onto your computer.

Running the Project

This project is built on Unity and the Vuphoria Engine. The Unity version used for this project is LTS Release 2021.3.6f1. I recommend you to use the same version for optimal results.

If you don’t have unity, you can download it by following these instructions :-

Go to https://unity.com/ and click the Get Started button. You will be redirected to another page
Click on the Student and hobbyist tab
Click on the Get Started button in the personal tab. This will redirect you to another webpage
Click the Download for Windows button if you’re on Windows.
Otherwise, click on Download other versions and click on the link with your current OS written on it. This will download the installation file.
Click on the Installation File. Follow the prompts to install Unity Hub on your computer.
Once you Unity Hub running, you should click on the Installs tab and then Install Editor to add a new Unity Version.
Click the Archive button near the top and click Long-Term Support to visit the download archive.
Scroll down a little. Click on LTS Release 2021.3.6f1 . This will download the recommended version of Unity to run this application.
- Unity
- Documentation
- Android Build Support (for running on the Oculus Quest, or Android Smartphone)
  Click on the little arrow and make sure Android SDK & NDK Tools and OpenJDK are both selected
- iOS Build Support if you want to deploy to an iPhone

I used Vuphoria version 10.9 and it’s built-in AR tools to create this application. You’ll also need to download its unity package to be able to run this application.

Go to this link : Vuforia Developer Portal
Register for a Vuphoria Account. Fill in the necessary information. Then, click on the Downloads to head back to the previous page
Make sure the release version is 10.10. This code was originally written with 10.9. However, Vuphoria no longer distributes that version. So, we’ll pick 10.10. Click on Add Vuforia Engine to a Unity Project or upgrade to the latest version. This will download the Vuphoria unity package.

You are gonna need paper cubes to be able to view the monuments, interact with the lighting, hear the sound effects, etc. The application looks for specific images in the cubes to overlay knickknacks on top of it. You can download these cube files from here.

Once downloaded, you can print out the files named “paper-merge-cube.pdf” and “paper ClassCube.png”. You can cut out the printed paper, fold and re-assemble to create these cubes. These files contain instructions on how to assemble them. After doing all this, you should end up two cubes that look like this :-

You're also gonna have to print out this astronaut image to see the dynamic laptop sticker

Now, Let's run it

Now, open the Unity Hub Application, Go to the projects tab. Click on the open button, find the directory & select the downloaded project. Once that is done, click on the project to open it.
Intially, the project will open with a lot of errors since we have not included the vuphoria package yet. Click on continue. It'll prompt you to open Safe mode. Click Ignore.
Once, you have the project open, go to the Assets menu, click on Import Package and select Custom Package. Now, go to the directory where you downloaded the Vuphoria package and select it.
Now, you're almost done. Select the scenes tab in the assets folder at the bottom. Drag the Main Scene to the Hierarchy
The Application will start your webcam. Bring your cube in front of the webcam and it'll show the knickknack.

First Knickknack

The first knickkack is of the Eiffel Tower. The Eiffel Tower was one the tallest buildings in the world until 1930. The Eiffel Tower is so beloved to the french that during WW2, the government left paris for the most part to let germany have paris without that much of a fight to protect paris from german stukas. This history combined with the architecture made me add this model to the project. The AR app also includes classis french music with this knickknack which starts playing when the cube comes close enough to the camera. It includes the 3d model of the Eiffel Tower, a bunch of trees, a bench, a girl standing close to the eiffel tower, a man standing near the lamp & 4 lamps (one on every corner). The Lighting changes if you hold the cube upside down for a few seconds. You can change it back the same way.

Citations

Models that were made by me

Bench
Girl standing next to the Eiffel Tower

Second Knickknack

The second knickknack is of the Statue of Liberty. The Statue of Liberty is one of the most iconic buildings in New York. It was gift from France to the US. It became a beacon of freedom. The Tower is so tall that lightning reaches it 600 times every year. The AR app includes Statue of Liberty, it’s base, a bunch of trees, a table, a man, a chair & 2 lamps. The song “Statue of Liberty” starts playing once you get close to the cube with your camera. The Lighting changes if you hold the cube upside down for a few seconds. You can change it back the same way.

Citations

Statue of Liberty -https://sketchfab.com/3d-models/statue-of-liberty-3d-scan-6807cf4a3c8748aeb34acebd5debe649
Sitting Man -https://sketchfab.com/3d-models/man-sitting-d8ca320050c6494581c34292356a280b
Trees -https://sketchfab.com/3d-models/tree-7016d1d32fe748f0a8b3f5eb39374bc4
Lamps -https://sketchfab.com/3d-models/lantern-f0b0ea89f20b4f10bb583c449ae04d9c
Music - Statue of Liberty "The New Colossus"

Models that were made by me

Base of Statue of Liberty
Table
Chair that the man is sitting on

Dynamic Sticker

As a big fan of Spiderman, i decided to recreate a spiderman fight scene in the form a dynamic sticker. The idea is that spiderman is fighting a mechanical spider and a mutant spider at the same time on the wall of a building. The Sticker shows that the building has windows as well as water pipes.

Citations

Models that were made by me

Windows
Water Pipe

Video

You can view the project video here

Discussion

Augmented Reality is far from its nascent stages as a piece of technology. Today, we have several AR headsets, glasses & even smartphones with SDKs supporting AR functionality. The pro model of the iPhone ships with a LiDAR scanner for quick, easy & precise AR overlays onto the camera output. AR applications which augment the physical object by overlaying a virtual one on top of it are bound to get more in-demand as cameras on our devices get better, custom neural processing units ease the task of running CNNs and hardware sensors like LiDAR become more popular.

AR already has various useful applications in real life today e.g. measuring any surface with a virtually overlayed ruler, getting directions when you’re going somewhere, AR games, etc. Some of the more interesting games I’ve tried were the Angry Birds games in AR. The structure falls to the surface visible on the camera as the birds hit it. It also allows you to roam freely inside such an environment, aiming wherever you please, and figure out the most vulnerable point for any structure by seeing it from every angle.

However, this project was based on enhancing physical objects to create interactive computer-generated objects. The example shown in this project is very basic compared to the applications that can be created using this piece of technology. This project only focuses on Monuments being overlayed on top of cubes with lighting & sound effects. Just imagine being able to see how you look in a suit before you buy it online. This can be achieved by the same technology. All we need is a cloth that acts as a physical object and the suit can be virtually placed on top of it. This will allow customers to try on clothes before they buy them online. Imagine being able to play sports like table tennis in AR.

We just need a handle which we can manoeuvre with our hand and a pair of AR glasses. AR Glasses can overlay an actual table tennis racket onto our handle and create a virtual table for people to play on. This way, you can enjoy table tennis sessions wherever, and whenever you want as long as you have the space. Since it’s augmenting an actual physical object in your hand, this method still retains the physical feel of an actual racket which most players consider to be essential to play effectively and enjoy the game. There are already toys available where the actual physical object becomes interactive when viewed through the AR app made by the manufacturer. I’m sure more toy manufacturers will get on board as time goes on. This can help create more immersive experiences for any physical product.

The best way to view these types of AR objects would be a pair of AR glasses. While our smartphones have tech inside of them which enables them to provide immersive AR experiences, we still have to hold & move our phones around to navigate in the AR environment. AR glasses would eliminate this need as they would go right over your head. While AR glasses exist today, they’re either too expensive, not marketed towards consumers but primarily business customers or too bulky for everyday use. If we want AR glasses to penetrate the consumer market, we’d have to figure out how to manufacture them more economically, create a reliable and easy-to-code-to SDK for developers to build awesome AR apps & use power efficient system on a chip design based on the ARM architecture with built-in neural engine for faster computation while having amazing performance-per-watt.

Introduction

Downloading the project

Installing Git

Running the Project

First Knickknack

Second Knickknack

Dynamic Sticker

Video

Discussion

Made by Saksham Khatod