Zappar for React Three Fiber

• Zappar for A-Frame (website, NPM)
• Zappar for ThreeJS (website, NPM)
• Zappar for Unity (website)
• Zappar for JavaScript (website, NPM), if you'd like to build content with a different 3D rendering platform
• ZapWorks Studio (website), a full 3D development environment built for AR, VR and MR

Table Of Contents Click to expand table of contents

• Zappar for React Three Fiber
  • Table Of Contents
  • Getting Started
    • Bootstrap Projects
      • GitHub
      • create-react-app
    • Example Projects
  • Starting Development
    • NPM
  • Overview
  • Local Preview and Testing
  • Compatibility and Browser Support
    • Detecting Browser Compatibility
  • Publishing and Hosting Content
    • Licensing
  • Setting up the Canvas
  • Setting up the Camera
  • Advanced Usage
    • Custom Video Device
    • First Frame
    • Setting the default camera
    • User Facing Camera
    • Mirroring the Camera
    • Realtime Camera-based Reflections
    • Camera Pose
  • Tracking
    • Image Tracking
      • Target File
      • Events
    • Face Tracking
      • Events
    • Face Landmarks
    • Face Mesh
    • Head Masking
    • Instant World Tracking
      • Default Placement UI
  • Loading UI
  • Usage with react-router-dom
  • Integrating into an existing create-react-app project
  • Links and Resources

Getting Started

Bootstrap Projects

GitHub

Tracking Type	JavaScript	TypeScript
Image Tracking	GitHub (JSX)	GitHub (TSX)
Face Tracking	GitHub (JSX)	GitHub (TSX)
Instant Tracking	GitHub (JSX)	GitHub (TSX)

create-react-app

npx create-react-app uar-app --template @zappar/r3f-face-tracking-typescript

Click to expand available templates

• @zappar/r3f-face-tracking
• @zappar/r3f-face-tracking-typescript
• @zappar/r3f-image-tracking
• @zappar/r3f-image-tracking-typescript
• @zappar/r3f-instant-tracking
• @zappar/r3f-instant-tracking-typescript

Example Projects

Starting Development

NPM

npm install --save @zappar/zappar-react-three-fiber

import { ZapparCamera /* ... */ } from "@zappar/zappar-react-three-fiber";

module.exports = {
  //...
  module: {
    rules: [
      //...
      {
        test: /zcv\.wasm$/,
        type: "javascript/auto",
        loader: "file-loader",
      },
      //...
    ],
  },
};

Overview

import { render } from "react-dom";
import React from "react";
import {
  ZapparCamera,
  ImageTracker,
  ZapparCanvas,
} from "@zappar/zappar-react-three-fiber";

const App = () => {
  // Use Webpack to load in target file
  const targetFile = "example-tracking-image.zpt";
    .default;
  return (
    <ZapparCanvas>
      {/* Setup Zappar Camera*/}
      <ZapparCamera />
      {/* Setup Image Tracker, passing our target file */}
      <ImageTracker targetImage={targetFile}>
        {/* Create a normal pink sphere to be tracked to the target */}
        <mesh>
          <sphereBufferGeometry />
          <meshStandardMaterial color="hotpink" />
        </mesh>
      </ImageTracker>
      {/* Normal directional light */}
      <directionalLight />
    </ZapparCanvas>
  );
}
render(<App />, document.getElementById("root"));

Local Preview and Testing

zapworks serve .

zapworks serve . --lan

Compatibility and Browser Support

• Safari for iOS (version 11.3 and later)
• Chrome for Android (versions from at least the last year)

• Most Webkit/Blink-based web browsers for Android, including Brave (good)
• Most third-party web browsers for iOS from iOS 14.3 and later (good)
• iOS in-app web views implemented with SFSafariViewController (good)
• iOS in-app web views implemented with WKWebView from iOS 14.3 (good)
• Firefox for Android (good, however performance may be lower than other browsers)
• Chrome for Mac/Windows (*)
• Firefox for Mac/Windows (*)
• Safari for Mac (*)

• iOS in-app web views implemented with WKWebView prior to iOS 14.3 - this iOS technology do not support camera access at all and thus we’re unable to support it. Apple has rectified this issue in iOS 14.3.
• iOS in-app web views implemented with the deprecated UIWebView component - this iOS technology do not support camera access at all and thus we’re unable to support it.
• Non-Safari web browsers on iOS, including Chrome, Firefox and Brave, before iOS 14.3 - these browsers use WKWebView due to App Store restrictions and thus do not support camera access.

Detecting Browser Compatibility

<>
   <BrowserCompatibility />
   <Canvas>
     {/* Content */}
   </Canvas>
</>

<BrowserCompatibility fallback={<div>Sorry!</div>} />

const incompatible = browserIncompatible(); // true / false

Publishing and Hosting Content

Licensing

Setting up the Canvas

import { ZapparCanvas /* ... */ } from "@zappar/zappar-react-three-fiber";

<ZapparCanvas>{/** YOUR CONTENT HERE **/}</ZapparCanvas>;

<Canvas colorManagement={false}>{/** YOUR CONTENT HERE **/}</Canvas>;

<ZapparCanvas gl={{ preserveDrawingBuffer: true }}>

<Canvas
  colorManagement={false}
  gl={{ preserveDrawingBuffer: true }
>

Setting up the Camera

import { ZapparCamera /* ... */  } from "@zappar/zappar-react-three-fiber";

// ...
return (
  <ZapparCanvas>
    <ZapparCamera/>
  </ZapparCanvas>
);

Advanced Usage

Custom Video Device

<ZapparCamera
  sources={{
    userCamera:
      "csO9c0YpAf274OuCPUA53CNE0YHlIr2yXCi+SqfBZZ8=",
    rearCamera:
      "RKxXByjnabbADGQNNZqLVLdmXlS0YkETYCIbg+XxnvM=",
  }}
/>

First Frame

 <ZapparCamera
  onFirstFrame={() => {
    console.log("first frame");
  }}
/>

Setting the default camera

<ZapparCamera
  makeDefault={false} // default: true
  renderPriority={0} //  default: 1
/>

const set = useThree(state => state.set)
const cameraRef = useRef()

useLayoutEffect(() => {
  set(() => ({ camera: cameraRef.current }))
}, [])


// ...
<ZapparCamera makeDefault={false} ref={ref}/>
// ...

User Facing Camera

<ZapparCamera userFacing />

Mirroring the Camera

<ZapparCamera userCameraMirrorMode="poses" />

• poses: this option mirrors the camera view and makes sure your content aligns correctly with what you're tracking on screen. Your content itself is not mirrored - so text, for example, is readable. This option is the default.
• css: this option mirrors the entire canvas. With this mode selected, both the camera and your content appear mirrored.
• none: no mirroring of content or camera view is performed

Realtime Camera-based Reflections

<ZapparCamera environmentMap />

const App = () => {
  const [envMap, setEnvMap] = useState<THREE.Texture>();
  return (
    <ZapparCanvas>
      <ZapparCamera useEnvironmentMap={setEnvMap} />
      <mesh position={[0, 0, -5]}>
        <sphereBufferGeometry />
        <meshStandardMaterial metalness={1} roughness={0} envMap={envMap} />
      </mesh>
      <directionalLight position={[2.5, 8, 5]} intensity={1.5} />
    </ZapparCanvas>
  );
};

Camera Pose

• default: in this mode the camera stays at the origin of the scene, pointing down the negative Z axis. Any tracked groups will move around in your scene as the user moves the physical camera and real-world tracked objects.
• attitude: the camera stays at the origin of the scene, but rotates as the user rotates the physical device. When the Zappar library initializes, the negative Z axis of world space points forward in front of the user.
• anchor-origin: the origin of the scene is the center of the group specified by the camera's poseAnchorOrigin prop. In this case the camera moves and rotates in world space around the group at the origin.

• To have a light that always shines down from above the user, regardless of the angle of the device or anchors, use attitude and place a light shining down the negative Y axis in world space.
• In an application with a physics simulation of stacked blocks, and with gravity pointing down the negative Y axis of world space, using anchor-origin would allow the blocks to rest on a tracked image regardless of how the image is held by the user, while using attitude would allow the user to tip the blocks off the image by tilting it.

Tracking

• Image Tracking can detect and track a flat image in 3D space. This is great for building content that's augmented onto business cards, posters, magazine pages, etc.
• Face Tracking detects and tracks the user's face. You can attach 3D objects to the face itself, or render a 3D mesh that's fit to (and deforms with) the face as the user moves and changes their expression. You could build face-filter experiences to allow users to try on different virtual sunglasses, for example, or to simulate face paint.
• Instant World Tracking lets you tracking 3D content to a point chosen by the user in the room or immediate environment around them. With this tracking type you could build a 3D model viewer that lets users walk around to view the model from different angles, or an experience that places an animated character in their room.

import {
  InstantTracker,
  ImageTracker,
  FaceTracker,
} from "@zappar/zappar-react-three-fiber";

Image Tracking

<ImageTracker targetImage={targetFile} >
  {/*PLACE CONTENT TO APPEAR ON THE IMAGE HERE*/}
</ImageTracker>

Target File

zapworks train myImage.png

const App = () => {
  const targetFile = 'target.zpt';
  return (
    <ZapparCanvas>
      <ZapparCamera userCameraMirrorMode="css" />
      <ImageTracker targetImage={targetFile} >
        {/*PLACE CONTENT TO APPEAR ON THE IMAGE HERE*/}
      </ImageTracker>
    </ZapparCanvas>
  );
}

Events

• onVisible - emitted when the image appears in the camera view
• onNotVisible - emitted when the image is no longer visible in the camera view
• onNewAnchor - emitted when a a non-previously seen before anchor appears in the camera view.

<ZapparCanvas>
  <ZapparCamera />
  <ImageTracker
    onVisible={(anchor) => console.log(`Visible ${anchor.id}`)}
    onNotVisible={(anchor) => console.log(`Not visible ${anchor.id}`)}
    onNewAnchor={(anchor) => console.log(`New anchor ${anchor.id}`)}
    targetImage={targetFile}
  >
    {/*PLACE CONTENT TO APPEAR ON THE IMAGE HERE*/}
  </ImageTracker>
</ZapparCanvas>

Face Tracking

<ZapparCamera/>
<FaceTracker >
  {/*PLACE CONTENT TO APPEAR ON THE FACE HERE*/}
</FaceTracker>

Events

• onVisible - emitted when the face appears in the camera view
• onNotVisible - emitted when the face is no longer visible in the camera view

<ZapparCamera />
<FaceTracker
  onNotVisible={(anchor) => console.log(`Not visible ${anchor.id}`)}
  onVisible={(anchor) => console.log(`Visible ${anchor.id}`)}
>
  {/*PLACE CONTENT TO APPEAR ON THE FACE HERE*/}
</FaceTracker>

Face Landmarks

const App = () => {
  return (
    <ZapparCanvas>
      <ZapparCamera  />
      <FaceTracker />
      <FaceLandmark
        target="nose-bridge"
      >
        {/*PLACE CONTENT TO APPEAR ON THE NOSE BRIDGE HERE*/}
      </FaceLandmark>
      <directionalLight />
    </ZapparCanvas>
  );
}

Face Mesh

import { FaceBufferGeometry /* ... */ } from "@zappar/zappar-react-three-fiber";

// ...
const App = () => {
  const faceTrackerGroup = useRef();
  return (
      <ZapparCanvas>
        <ZapparCamera />
        <FaceTracker ref={faceTrackerGroup}>
            <mesh>
              <FaceBufferGeometry trackerGroup={faceTrackerGroup} />
            </mesh>
        </FaceTracker>
        <directionalLight />
      </ZapparCanvas>
  );
};

<FaceBufferGeometry
  fillEyeLeft
  fillEyeRight
  fillMouth
  fillNeck
/>

<FaceBufferGeometry fullHead  />

Head Masking

import { HeadMaskMesh /* ... */ } from "@zappar/zappar-react-three-fiber";

// ...
<ZapparCamera />
<FaceTracker>
  <HeadMaskMesh  />
  {/*OTHER 3D CONTENT GOES HERE*/}
</FaceTracker>

Instant World Tracking

<InstantTracker placementMode >
  {/*PLACE CONTENT TO APPEAR IN THE WORLD HERE*/}
</InstantTracker>

const [placementMode, setPlacementMode] = useState(true);
return (
  <>
    <ZapparCanvas>
      <ZapparCamera />
      <InstantTracker placementMode={placementMode} >
        <mesh>
          <sphereBufferGeometry />
          <meshStandardMaterial color="hotpink" />
        </mesh>
      </InstantTracker>
      <directionalLight />
    </ZapparCanvas>
    <div
      id="zappar-placement-ui"
      onClick={() => {
        setPlacementMode((currentPlacementMode) => !currentPlacementMode);
      }}
    >
      Tap here to
      {placementMode ? " place " : " pick up "}
      the object
    </div>
  </>
);

Default Placement UI

export default function App() {
  return (
    <ZapparCanvas>
      <ZapparCamera />
      <InstantTracker
        placementUI="placement-only"
        placementCameraOffset={[0, 0, -10]}
      >
        <mesh>
          <sphereBufferGeometry />
          <meshStandardMaterial color="hotpink" />
        </mesh>
      </InstantTracker>
      <directionalLight position={[2.5, 8, 5]} intensity={1.5} />
    </ZapparCanvas>
  );
}

Loading UI

import {Loader, /*...*/ } from '@zappar/zappar-react-three-fiber'
// ....
return (
  <ZapparCanvas>
    <ZapparCamera rearCameraMirrorMode="css"  />
    <FaceTracker>
      <Suspense fallback={null}>
        <HeadMaskMesh  />
        <Model />
      </Suspense>
    </FaceTracker>
    <directionalLight />
    <Loader />
  </ZapparCanvas>
)

Usage with react-router-dom

import {
  ZapparCanvas,
  ZapparCamera,
  FaceTracker,
  Pipeline
} from "@zappar/zappar-react-three-fiber";

const App = () => {
  const pipeline = new Pipeline();

  return (
    <ZapparCanvas>
      <ZapparCamera pipeline={pipeline} />
      <FaceTracker pipeline={pipeline}>
        {/* PLACE CONTENT HERE*/}
      </FaceTracker>
      <directionalLight/>
    </ZapparCanvas>
  );
};

Integrating into an existing create-react-app project

module.exports = {
  webpack: function (config, env) {
    config.module.rules = config.module.rules.map(rule => {
      if (rule.oneOf instanceof Array) {
        rule.oneOf[rule.oneOf.length - 1].exclude = [/\.(js|mjs|zbin|jsx|ts|tsx)$/, /\.html$/, /\.json$/];
        return {
          ...rule,
          oneOf: [{
              test: /zcv\.wasm$/,
              type: "javascript/auto",
              loader: "file-loader",
              options: {
                outputPath: 'static/js',
                publicPath: '.',
                name: '[name].[ext]',
              },
            },
            ...rule.oneOf
          ]
        };
      }
      return rule;
    });

    config.resolve.extensions.push(".wasm")
    return config;
  },
  jest: function (config) {
    return config;
  },
  devServer: function (configFunction) {
    return function (proxy, allowedHost) {
      const config = configFunction(proxy, allowedHost);
      return config;
    };
  },
  paths: function (paths, env) {
    return paths;
  },
}

GitHub