Tag Archives: WebGL

Browser based AR and VR

OK, so “WebAR and WebVR” may have been a snappier title but also technically incorrect. “WebVR” is an actual standard still under development and “WebAR” doesn’t exist as such (oh look, I spoke too soon! It looks like it may be a pending standard, after all).

The terms VR¹ and AR², along with MR³, RR⁴, ER⁵, XR⁶ and god knows what else can be used to mean a surprising variety of things (check Wikipedia for more info).
¹ Virtual Reality
² Augmented Reality
³ Mixed Reality
⁴ Real Reality (WTF?)
⁵ Extended Reality
⁶ X Reality (I don’t think they’re even trying anymore)

Milgram’s Continuum of Mixed Reality

Google’s version of Milgram’s Continuum

These are supposedly convenient labels for different things on a spectrum, but I think they largely serve to muddy the water and are little more than an exercise in marketing bullocks. To this end, I prefer to just use the terms AR and VR.

Definition of terms
I claim no authority, but I’ll be defining AR and VR as follows (hopefully to provide clarity for the length of this article only):
AR – adding digital content to the existing environment
VR – (near) total immersion in an artificial environment

A “quick” note about hardware
We’ll be browser based, so there’s no bulky, expensive or high-end equipment (although we might burn out your GPU or drain your battery). I’m using my Nexus 4 (ancient my modern standards). A tablet can be used for AR, but might be a bit awkward for VR 🙂 .

While researching this article, I used a SketchFab cardboard and a pair of Go4D glasses. Neither was ideal, the cardboard provides an easy touch button input, while the Go4D has adjustable lenses. Both were needed for a proper experience, so I was sadly disappointed.

Researching for this article is the first time I’ve used used VR for any length of time on a mobile, and it’s both literally headache inducing and arm tiring. I’ve found dedicated systems/headsets far more comfortable to use (although they do make my head hot).
I’m not sure what caused the headaches, possibly poor resolution or poor placement causing eye-strain, adjusting the inter-ocular distance seemed to ease the problem.
It’s evident that for prolonged use some kind of strap is required along with some kind of cushion for the nose.

APIs

Jerome Etiene’s AR.js
argon.js
Aruco

VR Demos

https://airtable.com/embed/shr2Lc7pmlJis02R4/tblZbV2S0W0T5DDth?viewControls=on

AR Demos

Hatsune Miku Dancing in Augmented Reality
WebAR Playground: AR in a Few Clicks
(more info)

Browser Based VR

Requirements
Phone
Headset
Orientation/Positioning/Tracking
Input method

Implementation

VR is based on old-fashioned stereoscopic viewing (and I do mean old-fashioned). The basic requirement is to provide the viewer with two slightly offset stereoscopic images giving the impression of depth. True “VR” is achieved by adding positional awareness — as the viewer moves the images change to reflect the viewer’s new point of view.

Stereoscopy is OLD!

Without positional tracking the VR experience is severely limited, only allowing the user to look around, resulting in an experience akin to 360° video. Teleporting is a practical alternative to moving around as this is already a widely used method of moving around in a virtual environment.

While the WebVR standard makes VR easier to implement, it’s not widely supported (at the time of writing only Edge, Firefox and Chrome for Android have limited support). Polyfills and various other hacks are used to achieve the necessary functionality. Ultimately, happily, all the required legwork has already been done by many brave, hearty souls, leaving us to simply find the API we like the most.

A number of headsets are available, most notably the cardboard. The cardboard become uncomfortable after a short time.
Recent iterations have moved away from the neodymium magnets, which was the most expensive part of the cardboard and could interfere with the magnetometer/compass functionality of the phone.

The “big three” Three.js, Babylon.js and PlayCanvas make it relatively easy to generate VR content

Polyfill
WebVR
Early browser VR

Browser Based AR

Requirements
Positioning (markers or visual positioning)
Camera
Image processing
Input

AR has significantly more requirements as images additionally interact with the local visual environment (to some extent).

Editors

ARKit – Apple only and can’t find actual demos only movies (Apple restricted?)

A-Frame

Code it yourself or use on-line editor

APIs and Code

VR

A-Frame
Three.js
Babylon.js
PlayCanvas

AR

A-Frame

1
2
3
4
<script src="vendor/aframe/build/aframe.min.js"></script>
 
<script src='../build/aframe-ar.js'></script>
<script>ARjs.Context.baseURL = '../../three.js/'</script>
1
2
3
4
5
6
7
8
9
10
11
12
<body style='margin : 0px; overflow: hidden;'>
  <a-scene embedded arjs='trackingMethod: best;'>
    <a-anchor hit-testing-enabled='true'>
      <a-box position='0 0.5 0' material='opacity: 0.5; side:double; color:red;'>
        <a-torus-knot radius='0.26' radius-tubular='0.05'>
          <a-animation attribute="rotation" to="360 0 0" dur="5000" easing='linear' repeat="indefinite"></a-animation>
        </a-torus-knot>
      </a-box>
    </a-anchor>
    <a-camera-static/>
    </a-scene>
</body>

Three.js

1
2
3
<script src='vendor/three.js/build/three.min.js'></script>
<script src="../build/ar.js"></script>
<script>THREEx.ArToolkitContext.baseURL = '../'</script>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
var renderer = new THREE.WebGLRenderer({
  antialias: true,
  alpha: true
});
 
renderer.setClearColor(new THREE.Color('lightgrey'), 0)
renderer.setSize( 640, 480 );
renderer.domElement.style.position = 'absolute'
renderer.domElement.style.top = '0px'
renderer.domElement.style.left = '0px'
document.body.appendChild( renderer.domElement );
 
var onRenderFcts= [];
 
var scene = new THREE.Scene();
 
var camera = new THREE.Camera();
scene.add(camera);
 
var arToolkitSource = new THREEx.ArToolkitSource({ sourceType : 'webcam' });
 
arToolkitSource.init(function onReady() {
  onResize()
});
 
window.addEventListener('resize', function() {
  onResize()
});
 
function onResize() {
  arToolkitSource.onResize()
    arToolkitSource.copySizeTo(renderer.domElement)
    if( arToolkitContext.arController !== null ) {
    arToolkitSource.copySizeTo(arToolkitContext.arController.canvas)
  }
}
 
var arToolkitContext = new THREEx.ArToolkitContext({
  cameraParametersUrl: THREEx.ArToolkitContext.baseURL + '../data/data/camera_para.dat',
  detectionMode: 'mono',
});
 
arToolkitContext.init(function onCompleted() {
    camera.projectionMatrix.copy( arToolkitContext.getProjectionMatrix() );
});
 
onRenderFcts.push(function() {
  if( arToolkitSource.ready === false )  return
 
  arToolkitContext.update( arToolkitSource.domElement )
      scene.visible = camera.visible
  });
 
  var markerControls = new THREEx.ArMarkerControls(arToolkitContext, camera, {
    type : 'pattern',
    patternUrl : THREEx.ArToolkitContext.baseURL + '../data/data/patt.hiro',
    changeMatrixMode: 'cameraTransformMatrix'
});
 
scene.visible = false
 
var geometry = new THREE.CubeGeometry(1,1,1);
var material = new THREE.MeshNormalMaterial({
  transparent : true,
  opacity: 0.5,
  side: THREE.DoubleSide
}); 
 
var mesh = new THREE.Mesh( geometry, material );
mesh.position.y = geometry.parameters.height/2
scene.add( mesh );
 
var geometry = new THREE.TorusKnotGeometry(0.3,0.1,64,16);
var material = new THREE.MeshNormalMaterial(); 
var mesh = new THREE.Mesh( geometry, material );
mesh.position.y = 0.5
scene.add( mesh );
onRenderFcts.push(function(delta) {
  mesh.rotation.x += Math.PI*delta
});
 
onRenderFcts.push(function() {
  renderer.render( scene, camera );
});
 
var lastTimeMsec= null
requestAnimationFrame(function animate(nowMsec) {
  requestAnimationFrame( animate );
  lastTimeMsec = lastTimeMsec || nowMsec-1000/60
  var deltaMsec = Math.min(200, nowMsec - lastTimeMsec)
  lastTimeMsec = nowMsec
  onRenderFcts.forEach(function(onRenderFct) {
    onRenderFct(deltaMsec/1000, nowMsec/1000)
  });
});

Babylon.js

1
2
3
4
5
6
7
<script src="vendor/babylon.max.js"></script> 
<script src="../build/babylon-ar.js"></script>
<script>ARjs.Context.baseURL = '../../three.js/'</script> 
 
<div id="canvasZone">
  <canvas id="renderCanvas"></canvas>
</div>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
var canvas = document.getElementById("renderCanvas");
var engine = new BABYLON.Engine(canvas, true);
 
function createScene(){
  var scene = new BABYLON.Scene(engine);
  scene.clearColor = new BABYLON.Color4(0, 0, 0, 0);
 
  var camera = new BABYLON.ArcRotateCamera("Camera", 1.0, 1.0, 12, BABYLON.Vector3.Zero(), scene);
 
  var light = new BABYLON.HemisphericLight("hemi", new BABYLON.Vector3(0, 1, 0), scene);
 
  light.groundColor = new BABYLON.Color3(0.5, 0, 0.5);
 
  var box = BABYLON.Mesh.CreateBox("mesh", 1, scene);
  box.position.y = 0.5;
  box.showBoundingBox = true;
 
  var material = new BABYLON.StandardMaterial("std", scene);
  material.diffuseColor = new BABYLON.Color3(0.5, 0, 0.5);
 
  box.material = material;
 
  return scene;
}
 
var scene = createScene();
scene.clearColor = new BABYLON.Color4(0, 0, 0, 0);
 
ARjs.Babylon.init(engine, scene, scene.cameras[0])
 
engine.runRenderLoop(function () {
  scene.render();
});
 
window.addEventListener("resize", function () {
  engine.resize();
});

Down the rabbit hole with FBX

PlayCanvas

PlayCanvas does an admirable job of importing FBX models and animations, but animation control seems limited. In particular, there seem to be no events for animation start, end, etc.

Animated Sackboy

This immediately prompted me to stop and reassess all my requirements (and not just animation).

Workflow

  1. I felt that I would almost certainly need more control over animation events than PlayCanvas provides.
  2. PlayCanvas only has version control (very important) for legacy scripts.
  3. A fall back is needed if PlayCanvas fails for some reason (e.g. it’s cloud based, so poor internet or even just high contention would be problematic).

Using an API would immediately address issues 2 and 3. This leaves FBX support as the main issue.

It’s apparent that game creation requires a smooth workflow. Easily getting assets into the game is very important, regardless of their source. Be it pre-existing from a store or newly created by an on-site 3D artist, assets have to come from somewhere, and FBX is one of the more common formats particularly for animation. FBX support is very important.

Checking out the main alternatives:

Babylon.js

Babylon.js doesn’t support FBX directly. I was advised to use the Babylon exporter for Unity or Blender. Alas, the results were not impressive.

Source model (Pearl.fbx) as should be

Model after exporting to Babylon (from Unity and Blender)

Clara.io has an “export to Babylon” option, unfortunately its own FBX import is lacking, and I’m still trying to work out how to add animation to a model.

Model after import into Clara.io

Three.js

Three.js currently only supports ASCII FBX not binary (Takahiro is doing some sterling work to add binary FBX).

Current state of FBX import to Three.js (not released yet)

So, it may be a contender if support is available in time, but for now it’s back to PlayCanvas.

Conclusion (if you can call it that)

Converting 3D assets seems inevitable, to reduce file size and minimise load time (even PlayCanvas converts FBX to JSON).

I will look into FBX – glTF or JSON converters, but for now I’m returning to PlayCanvas (in fact, PlayCanvas engine is also available, so it’s its own fallback).

Choosing a framework/engine

Naturally, I want to work on a WebGL based game. This automatically makes it web based with all the attendant benefits and disadvantages.

My first choice is PlayCanvas (free version), which is like a (simplified) on-line Unity. The only alternative that I know of, GooCreate, sadly declared bankruptcy a few months ago.

Pros

  • on-line editor (drag ‘n’ drop)
  • six years old (so relatively mature and stable)
  • support is through forums for free accounts
  • FBX support
  • collaboration

Cons

  • no download for off line storage
  • no version control
  • limited animation control

Develop Game Jam

I’ll be attending this year’s (2017) Develop Game Jam in July.

I usually just ruck up on the day and use the opportunity to learn a bit about some game development technology or techniques.

While fun and educational, this is not the ideal way to compete. Frequently, I achieve little by way of a final result.

This year, I thought I’d actually do some real preparation and take a real stab at completing a “proper” game.

I plan to use WebGL (naturally), so I’m focusing on PlayCanvas, along with Three.js and Babylon.js as possible fall backs.

If things really go pear shaped there’s always Unity (or possibly Unreal or CryEngine).

I intend to blog the process for reference purposes and I’ll add links to blog/articles as I progress.

Develop 2017 Game Jam Preparation

OK, so, the Develop 2017 game jam “looms” and this time around I thought I’d do some prep to give me a chance at finishing a game. (At past jams I’ve been up against some seriously talent jammers, while I try to not be competitive I do feel I should try harder.)

Past experience has shown me that using tools is enlightening – highlighting possible problems and their solutions ahead of time, so reducing friction development during the jam. Something that this exercise has reminded me of.

I want to work on a WebGL based game. This automatically makes it web based with all the attendant benefits and disadvantages.