Up-dimensioning for VR

This note is actually an email I wrote in a thread where someone pointed out that:

When you search for “spatial web browsing,” you’ll find there are whole engineering teams trying to stick web browsers on walls within 3D AR/VR environments.

I wrote a long, rambling response (as you do on a lazy sunday), but it has some ideas that I think may be useful.

(mail headers removed)

If you think about geometry, it is always easy to imagine projections from higher dimensionality down to lower. Like putting a light behind a 3D shape and imagining what the shadow will look like. Going the other way is harder. Imagining what shape a 3D object must have to produce a given shadow is harder. And not just for the obvious reason that projection removes information, but because it probably isn’t something you do very often in everyday life. So your brain isn’t going to be any good at it. (Picture the cover of “Gödel, Escher, Bach: an Eternal Golden Braid”. Even though the two 3D objects are given 3 projections, you can easily imagine how many more degrees of freedom you have in how you craft the two shapes).

About a decade ago I started doing a lot of 3D CAD design. I started drawing objects which I would later 3D print or machine on a CNC machine. Typically functional objects that have to fulfill some function. Often in concert with other parts to result in assemblies with moving parts. Initially this was a struggle because I had no mental “language” for solving problems in three dimensions. If you can draw things in two dimensions you will be able to do a lot of things without thinking about what you are doing. For instance you might grab a round object to draw a circle or a rounded edge. This requires almost no cognitive horsepower. Go up one dimension and suddenly you have more problems and fewer cognitive tools from your everyday experience.

And this is easy to see. Let someone who has no experience with 3D drawing come up with a strategy for drawing a solid that has an oddly shaped cavity. This too can be done with very little cognitive horsepower, but initially you will find this very hard. Because your mind is learning what is essentially a new language for expressing shape.

The point isn’t really that 3D graphics is hard, but that the simple, basic bits of intuition are surprisingly hard at first.

In the above example a common strategy is to make a positive shape which you then subtract from another positive shape to create the negative space (the cavity). Or in other words: you make a “tool” shaped like the hole you wish to make. You do this in two dimensions as well (if you do 2D vector designs), but it isn’t how most people think because it doesn’t have much basis in your everyday experience. Most things around you do not respond like that. You can’t take a chisel and create a sealed cavity inside a log without opening the log in some way. Today we have 3D printers (additive manufacturing) that allow us to ignore those constraints. You can make solid objects with inside structures that are impossible to manufacture by removing material (subtractive manufacturing).

Getting a 3D printer and learning how to design functional objects for it is a great way to experience what it is like for your brain to grow an entirely new repertoire of cognitive skills.

Personally I think focusing on how you up-dimension things that are strongly perceived as flat (pages of text, code, web pages) is thinking about the problem the wrong way. It is limiting. You have to start with a perspective of higher dimensionality and in order to up-dimension something you perhaps need to look at it in a more abstract way. How can an idea be helped by up-dimensioning? Does it provide entirely new ways of thinking about it?

Let’s look at another example to explain what I mean.

Consider this game: you have two players. They take turns picking a number between 1 and 9. When a number has been picked it can’t be used again. The goal is to pick numbers that sum to 15, or failing that, to prevent the opponent to get to 15. You can do this with pen and paper and cross out numbers:

1 2 3 4 5 6 7 8 9

Try to mentally play this game a few times before scrolling down.

Now, if I lay out the numbers in a 3x3 grid in a certain manner (a magic square), does something jump out at you?

2 7 6
9 5 1
4 3 8

You’ll notice the rows and diagonals all sum to 15. So now you have reduced the problem to picking numbers that lie along a line or a diagonal. You know how to do this. This is tic-tac-toe, and most kids can play this game hours on end without losing (either getting a draw or a win). I doubt most adults can play the first version of the game as well as a child can play the second. It is exactly the same problem as before, but now in a representation that makes it available to much faster cognitive processes. Processes that are much closer to pure instinct.

This is where the promise of VR is: in representing things in way that significantly lower the cognitive load and almost manages to mainline information to your brain. Preferably to cognitive processes that have a high degree of intuition and instinct.

Doing so is surprisingly hard, though. As to why, let’s take a step in the other direction.

Imagine we are sitting around a campfire along with our neolithic family. It is story-time. Storytelling and stories can be seen as single-dimensional sequences of words or ideas(1). It is a vector of symbols that provide meaning arranged in a certain order. In our case words or symbols. The concept of a “page” comes from our way of representing these symbols practically. Imagine how our written traditions would have changed if we had started representing stories as differently shaped knots along a string. That our libraries consistend of spools of strings that are read by running them between our fingers.

The idea isn’t too foreign as we actually do this: prayer beads. No, they don’t carry a representation of the actual words, but they are nevertheless a linear mnemonic device (or rather, a circular, although I am not well enough versed in prayer practices to know to what degree this is important. I’d assume sometimes it is, and sometimes it isn’t).

Now, imagine you are explaining the concept of a page of text to a civilization that stores their information in spools of knotted strings. Imagine explaining the use of spatial relationships between headings, paragraphs, the separation of paragraphs, or even illustrations. Just imagining how you would talk about this to someone who lacks all of these concepts is hard to imagine. You’d have to show them. They’d have to develop a visual language for it. If we look back on our own conquest of the page, that has been a rather bumpy road.

People who try to come up with ways of representing web pages in 3D are not doing much better than string-people going “aha”, curling up paper in a tube, and wrapping their string around it while declaring “I got it now, I can see that paper is definitively the future”.

We’re string-people. And we’re wrapping our string around paper while our descendants are watching us from far in the future, facepalming and mumbling “and these people felt superior to monkeys”.

Okay, enough rambling. I have an office to rewire with IKEAs delightfully modular Åskväder extension cables. If you haven’t seen them: run to IKEA and buy their entire stock before anyone else realizes just how brilliant they are :-)

1. ok, I’m making the assumption that all storytelling is linear. It doesn’t have to be. In much the same fashion as the timeline in “Pulp Fiction” is jumbled, the retelling of when Ugh-Ugh The Unfortunate got trampled by Meat That Has Large Pointy Bits In Front, might start with retelling the bit where he goes from 3D to largely being a 2D entity, and then skip back in time to describe how he ingested Berries That Makes You See The Big Grandfather Yak for breakfast, before going forward in time to where Dingbat The Confused noticed Ugh-Ugh’s glassy stare right before confronting tonight’s dinner. But we can imagine that skilfull retelling of this cautionary tale is linear in representation.

-Bjørn