This portion of the incubator is focused on 3D and immersive technologies, with a focus on 3D object creation using photogrammetry (photographs) and laser scanning (lasers) as well as the creation of basic alternate-reality (AR) scenes with Apple's RealityComposer platform.
Having gained familiarity with the basic functionality of RealityComposer, challenge yourself to create a simple AR scene with behaviors using prebuilt models or some imported from Sketchfab. Some possible scenarios with the basic models:
A pool table/bowling lane where the ball flies into the other balls when tapped
"kicking" a football through the uprights
A rocket ship flies into space with sounds (and then slowly comes back to its landing spot)
Whatever else you want!
In the Digital Studio, we have access to equipment for both 3D modeling and laser scanning. The aim of this hands-on experiment is to explore and try to create 3D models, perhaps for use in the Alternate Reality (AR) scene to be created in the next part of the incubator.
Photogrammetry setup with Lightbox and basic DSLR camera, good for objects down to about the size of your fist
Hand-held Laser Scanner (borrowed from Schiller), good for quick scanning of larger models (and people!)
Hand-held laser Scanner with roundtable setup, good for objects down to 3cm
Reality Composer (RC) for iOS, iPadOS, and macOS makes it easy to build, test, tune, and simulate AR experiences for iPhone or iPad. With live linking, you can rapidly move between Mac and iPhone or Mac and iPad to create stunning AR experiences, then export them to AR Quick Look or integrate them into apps with Xcode. This tutorial will introduce the basic functionality of the tool and review the interface. Please note that screenshots for this introduction were created using Reality Composer on a MacBook. The screen layout is slightly different when working directly on an iPad, but the steps are the same.
Reality Composer is free to download from the app store for iPad or iphone, or downloaded as part of the free XCode for Mac iOS (Apple Account Required). A warning that XCode is quite a larger program for developing applications, so may not be feasible for older MacBooks or those without much storage space. Running Reality Composer on a MacBook also requires you to send your project to an appropriate iPad/iPhone for testing (see below) or open it in an emulator using xCode.One important fact to note is that Reality Composer projects (i.e. what opens in Reality Composer) are saved as .rcprojects while sharable experiences that open directly onto an iPhone or iPad with the Quick Look tool are .reality files.
When you first open up a new Reality Composer project, your first decision to make is what kind of anchor you want for the project. The Anchor type will determine what requirements are needed to open your scene in AR. When you start up RC, you are asked to select an Anchor typer for your project. There are 5 relatively-straightforward anchor types to chose from in Reality Composer:
Horizontal - will look for any horizontal surface (e.g. a table or the floor) to open your experience on
Vertical - will look for any vertical surface (e.g. a wall) to open your experience
Image - will look for a specific defined image that you determine (e.g. a painting or business card) to open your experience around
Face - will look for a person's face (as recognized by Apple's camera) to open your experience around
Object - will look for a previously scanned physical object (see details at the end of this workshop) to open your experience around
Once you have chosen an anchor, the main window of Reality Composer will open up, which changes slightly based on the anchor you have chosen (above, the horizontal anchor). You can also change your anchor if needed in the right window Properties pane (as well as change the overall physics of the scene if desired, a la allow things to fall through the floor. This can also be set object-by-object later).
The window also opens with two objects already present, a cube and a little placard showing some text. Before checking those out, let's look at the other options on the main toolbar. The most important ones for us will be the Add button, the Behaviors button, and the Properties button, but we can quickly review them all.
The Scenes option opens up a sidebar that allows you to name the "scene," or AR view, you are currently working in, as well as create new scenes that can be linked by certain actions (see below).
The Frame button just lets you navigate more easily around your scene by allowing you to zoom in on certain objects or on the scene as a whole.
The Snap button allows you to snap objects to other objects or to certain horizontal or vertical lines within the scene (similar to the Photoshop snap tool)
The Modify tool allows you to swap between adjusting an objects position/rotation and adjusting its length/width/size (this can also be done in the properties pane, as we will see)
The Space button swaps between "global" and "object-oriented" coordinates, allowing you to move an object along various axes within your scene
The Add button adds more models to your scene, of lots of different types!
The Play button (also AR button when working on an iPad/iPhone) allows you to test our your scene on a device. The Send to button (Mac only) allows you to send your scene directly to a linked iPad for testing
The Info button checks for updates to your models/scenes
The Behavior button allows you to assign different behaviors to your object based on specific prompts (e..g. the object it tapped on, it flies in the air)
The Properties button allows you to edit the properties of a specific model or of the scene as a whole.
Let's play with some objects, starting with what comes in our default scene.
Clicking on the cube will automatically open the properties pane, which allows you to directly edit its various properties like width, height, color, etc. You can also name the object, transform it in space (if you want to set something exactly) as well as decide if it will obey the laws of physics (e.g. fall to the surface if it appears up in the air). You can also edit directly in the viewing window once an object is selected; Arrows pointing along axes allow you to move an object, which clicking the tip of an arrow will allow you to rotate an object around that axis. Give it a try!
Clicking on the placard will yield a similar pane, though this one also allows you to edit the text appearing on the sign. Each object you add will have its own individual properties that you can edit here. For now, edit these two items however you wish!
If you want to add a new object, just click the + Add button. There are many pre-installed 3D objects to work from, as well as signs that can hold text and "Frames" that can hold attached Images. You can also introduce your own objects.
For now, I will drop a plane in our scene, by clicking and dragging it in or by double-clicking. I then rotated it around to be pointing away from us.
Let's do one last thing before trying out our model in AR....adding a behavior! I'm going to select the plane, and then click the behaviors button on the toolbar, and a new pane will open up along the bottom of the page
Clicking the + button will open the new behavior pane, where there are several pre-set behaviors you can choose from, or if you scroll down to the bottom you can create a custom behavior. For now, I will choose Tap and Add Force so we can give our plane some energy
Now, our behavior pane has its first behavior! You can rename the behavior (currently called Behavior) in the left pane; the Trigger pane allows you to determine what action will make the behavior start (in this case, tapping) as well as which affected objects have the trigger attached (in this case, the plane, but you could add more!)
Finally, the action sequence allows you to create a series of effects that happen, either simultaneously or one after another, when the trigger is...triggered. In this case, we are going to have the plan start moving forward at a certain velocity.
So we can make our plan move forward, but that's not really taking off. Adding a second force in an upward trajectory after a moment will make this takeoff look a bit more realistic. To add further actions to the sequence, simply tap the + sign next to the words Action Sequence in the Behaviours pane. This will then pop up a bunch of different pre-coded behaviors you can choose from.
In the image above, I added two new behaviors, a Wait Behavior and a second Add Force behavior in a 45 degree upward angle. Importantly, I direclty attached the Wait behavior to the first Add Force behavior (just a drag and drop) which means that these two actions will begin simultaneously, and the 2nd Add Force will not start until the first set is complete. This means our plane will move forward a certain amount before briefly "taking off".
Now that we have an experience, we need to test it out to see how it is functioning. There are a few ways to do this.
If you are working on an iPad or iPhone, its easy! Just hit the AR button on the top toolbar to open the experience in AR, and then the Play button to start the experience from the beginning.
If you are working on a Mac, it's a bit more difficult. On the one hand, if you hit the Play button on the top toolbar, the experience will start, but will obviously not be in AR, making testing a little bit difficult (though you could certainly test the functionality, as in the images below where our plane has flown far off into the distance.
There are other options for testing from a Mac, however. If you have an iPhone or iPad handy that has Reality Composer installed, you could connect it via a USB/USB-C cord to your computer. If you then open Reality Composer on both devices and hit the Send To button on your Mac, the file will open in Reality Composer and be editable/testable on your iPad!
Note that if you are using special imported models, they may not be available on your second device, unless they have been imported there as well.
Another option is to simply export the file and share it as a fully .reality file. To do this, go to File --> Export, and pick either to export the entire project or just the current scene. After saving it on your computer, you can navigate to that folder, select the .reality file, again go to File --> Share in the Finder menu, and choose how you want to share the file (text, AirDrop, etc) to your iPad or iPhone. Opening it on your iPhone or iPad in this way does not require Reality Composer, as it is using the built in Apple Quick Look platform (you can also share your experience with other people in this way!
We are going to add one last action that will make our action "replayable": returning the plan to its original location after a certain amount of time. Otherwise, we tap the plane once and it is gone!
This can be done by adding one more action to our action sequence, a Move, Rotate, Scale To action that will move our object back home. Adding this action to the end of our action list, and then selecting the Plane as our affected object will allow you to choose where you want the plane to return to. In this case, I will adjust it so it ends up back where it started (by moving the plane in the image back to the left to the starting place). Also noticed I added one more Wait action so that the plane will wait one second after it stops being impacted by the Force to return home.
And that's it! Now the plane will return to its original location (and might even let you see the ingrained physics on the way as the plane hits the block as it returns). Project testing videos and the files created with this tutorial can be found below. There is obviously a ton of other behaviors and models to play with, so give it a try!
As a bonus, you could use the Hide and Show behaviors to make the plane seem to magically "appear" back in its home location at a certain moment. See if you can make it work!
