Build iOS augmented reality and 3D experiences with RealityKit and ARKit. Use when adding RealityView content, loading entities or USDZ models, anchoring objects to planes or world positions, distinguishing entity hit tests from ARKit real-world raycasts, handling AR camera availability, world tracking, scene updates, or RealityKit entity gestures and interactions.
SKILL.md
RealityKit
Build AR experiences on iOS using RealityKit for rendering and ARKit for world
tracking. Covers RealityView, entity management, raycasting, scene
understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.
Content displayed through an AR camera view on iOS, non-AR on macOS
Entity
All
Base class for all scene objects
ModelEntity
All
Entity with a visible 3D model
AnchorEntity
All
Tethers entities to a real-world anchor
RealityView Basics
RealityView is the SwiftUI entry point for RealityKit.
RealityViewCameraContent is the iOS/macOS content type. On iOS, it uses an AR
camera view by default and can use content.camera = .virtual for non-AR mode
when requested or when AR/camera access is unavailable.
import ARKit
import SwiftUI
import RealityKit
struct ARExperienceView: View {
var body: some View {
RealityView { (content: RealityViewCameraContent) in
if !ARWorldTrackingConfiguration.isSupported {
content.camera = .virtual
}
let sphere = ModelEntity(
mesh: .generateSphere(radius: 0.05),
materials: [SimpleMaterial(
color: .blue,
isMetallic: true
)]
)
sphere.position = [0, 0, -0.5] // 50cm in front of camera
content.add(sphere)
}
}
}
Make and Update Pattern
Use the update closure to respond to SwiftUI state changes:
Load 3D models asynchronously to avoid blocking the main thread:
RealityView { content in
if let robot = try? await ModelEntity(named: "robot") {
robot.position = [0, -0.2, -0.8]
robot.scale = [0.01, 0.01, 0.01]
content.add(robot)
}
}
Adding Components
Entities use an ECS (Entity Component System) architecture. Add components
to give entities behavior:
let box = ModelEntity(
mesh: .generateBox(size: 0.1),
materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
// Make it respond to physics
box.components.set(PhysicsBodyComponent(
massProperties: .default,
material: .default,
mode: .dynamic
))
// Add collision shape for interaction
box.components.set(CollisionComponent(
shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
// Enable input targeting for gestures
box.components.set(InputTargetComponent())
Anchoring and Placement
AnchorEntity
Use AnchorEntity to anchor content to detected surfaces or world positions:
RealityView { content in
// Anchor to a horizontal surface
let floorAnchor = AnchorEntity(.plane(
.horizontal,
classification: .floor,
minimumBounds: [0.2, 0.2]
))
let model = ModelEntity(
mesh: .generateBox(size: 0.1),
materials: [SimpleMaterial(color: .orange, isMetallic: false)]
)
floorAnchor.addChild(model)
content.add(floorAnchor)
}
Anchor Targets
Target
Description
.plane(.horizontal, ...)
Horizontal surfaces (floors, tables)
.plane(.vertical, ...)
Vertical surfaces (walls)
.plane(.any, ...)
Any detected plane
.world(transform:)
Fixed world-space position
Raycasting
Keep RealityKit scene queries separate from ARKit real-world raycasts:
RealityViewCameraContent.ray(through:in:to:) returns a camera ray in
RealityKit coordinate spaces. It projects a screen point into the virtual
scene; it is not proof of a detected physical surface.
RealityViewCameraContent.hitTest(point:in:query:mask:) hits virtual
entities made hittable by CollisionComponent shapes. Use those shapes for
entity picking and targeted gestures, not ARKit plane detection.
Use AnchorEntity(.plane(...)) for simple placement on detected planes.
Use ARKit ARRaycastQuery plus ARSession.raycast(_:) when the task needs
a one-shot intersection with real-world surfaces, then anchor with
AnchorEntity(raycastResult:).
let results = session.raycast(query)
if let result = results.first {
let anchor = AnchorEntity(raycastResult: result)
anchor.addChild(model)
content.add(anchor)
}
Do not treat entity hit tests as substitutes for ARKit surface raycasts.
Gestures and Interaction
For gesture-based entity interaction, add CollisionComponent for the hittable
shape and InputTargetComponent for input targeting.
Drag Gesture on Entities
struct DraggableARView: View {
var body: some View {
RealityView { content in
let box = ModelEntity(
mesh: .generateBox(size: 0.1),
materials: [SimpleMaterial(color: .blue, isMetallic: true)]
)
box.position = [0, 0, -0.5]
box.components.set(CollisionComponent(
shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
box.components.set(InputTargetComponent())
box.name = "draggable"
content.add(box)
}
.gesture(
DragGesture()
.targetedToAnyEntity()
.onChanged { value in
let entity = value.entity
guard let parent = entity.parent else { return }
entity.position = value.convert(
value.location3D,
from: .local,
to: parent
)
}
)
}
}
Scene Understanding
Per-Frame Updates
Subscribe to SceneEvents.Update for continuous scene work rather than driving
RealityKit with a SwiftUI timer. Keep the subscription alive and use
event.deltaTime; see Entity Animations.
Platform Boundaries
On visionOS, ARKit provides a different API surface with ARKitSession,
WorldTrackingProvider, and PlaneDetectionProvider. These visionOS-specific
types are not available on iOS. On iOS, RealityKit handles world tracking
automatically through RealityViewCameraContent.
For iOS architecture or migration notes, gate AR with
ARWorldTrackingConfiguration.isSupported, host content with
RealityViewCameraContent, and build scenes from Entity/ModelEntity placed
with AnchorEntity.
Handoffs:CollisionComponent + InputTargetComponent handle RealityKit
interaction; AccessibilityComponent handles entity accessibility metadata;
detailed SwiftUI gestures and VoiceOver/Switch Control policy belong to siblings.
Treat existing SCNView/SCNNode work as either a separate SceneKit path or an
explicit migration to RealityKit, not a mixed scene graph.
Common Mistakes
DON'T: Skip AR capability checks
Use the configuration-specific support check from Setup before presenting AR.
Show non-AR content or an explicit unavailable state when it fails.
DON'T: Load heavy models synchronously
Loading large USDZ files on the main thread causes frame drops and hangs.
The make closure of RealityView is async -- use it.
// WRONG -- synchronous load blocks the main thread
RealityView { content in
let model = try! Entity.load(named: "large-scene")
content.add(model)
}
// CORRECT -- async load
RealityView { content in
if let model = try? await ModelEntity(named: "large-scene") {
content.add(model)
}
}
DON'T: Forget collision and input target components for interactive entities
Interactive entities need both components shown in
Gestures and Interaction; without them, taps and
drags pass through.
DON'T: Create new entities in the update closure
The update closure runs on every SwiftUI state change. Creating entities
there duplicates content on each render pass.
// WRONG -- duplicates entities on every state change
RealityView { content in
// empty
} update: { content in
let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
content.add(sphere) // Added again on every update
}
// CORRECT -- create in make, modify in update
RealityView { content in
let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
sphere.name = "mySphere"
content.add(sphere)
} update: { content in
if let sphere = content.entities.first(
where: { $0.name == "mySphere" }
) as? ModelEntity {
// Modify existing entity
sphere.position.y = newYPosition
}
}
DON'T: Ignore camera permission
RealityKit on iOS needs camera access. If the user denies permission, the
view shows a black screen with no explanation.
// WRONG -- no permission handling
RealityView { content in
// Black screen if camera denied
}
// CORRECT -- check and request permission
struct ARContainerView: View {
@State private var cameraAuthorized = false
var body: some View {
Group {
if cameraAuthorized {
RealityView { content in
// AR content
}
} else {
ContentUnavailableView(
"Camera Access Required",
systemImage: "camera.fill",
description: Text("Enable camera in Settings to use AR.")
)
}
}
.task {
let status = AVCaptureDevice.authorizationStatus(for: .video)
if status == .authorized {
cameraAuthorized = true
} else if status == .notDetermined {
cameraAuthorized = await AVCaptureDevice
.requestAccess(for: .video)
}
}
}
}
Review Checklist
NSCameraUsageDescription set in Info.plist
AR device capability checked before presenting AR views
Camera permission requested and denial handled with a fallback UI
arkit required-device capability added when AR is the app's core purpose
3D models loaded asynchronously in the make closure
Entities created in make, modified in update (not created in update)
Interaction prerequisites and sibling handoffs follow the Gestures section
Entity hit tests, camera rays, and ARKit surface raycasts follow the Raycasting distinctions
SceneEvents.Update subscriptions used for per-frame logic (not SwiftUI timers)
Large scenes use ModelEntity(named:) async loading, not Entity.load(named:)
Anchor entities target appropriate surface types for the use case
