This is the follow-up article to Animation and 3D: the web is doing it wrong. In this article, I argue that the web is not helping us much with advanced storytelling. Based on our experience with Tao Presentations and interactive storytelling in 3D, we can offer a few suggestions. We can explain how to integrate 3D models, how to take advantage of our depth perception, how to organize group interactivity, how to create stronger visual impact for lasting memories. Can we bring these ideas to the web?
TL;DR: Storytelling is constrained by the tools you use. With Tao Presentations, Taodyne has discovered a large number of new ways to express ideas. We’d like to bring these to the web.
Good storytelling takes inspiration from the real world
Ultimately, we write documents and create presentations to share a story, to show things. And I hope that you noticed how the Tao Presentations code I showed in my previous article focused on the contents, on the storytelling, not on tags or low-level animations. Your main_title_slide contains a title and a subtitle.
import Slides main_title_slide "Main title", title text "Seasons Greetings theme" subtitle text "A theme for the Holidays"
With this approach, Tao Presentations presents the right level of abstraction, like LaTeX did for 2D pages, displacing the original TeX. You talk about titles and subtitles. You don’t have to learn that title should really be spelled <h1> as in HTML.
Better yet, words such as title are defined from lower-level primitives using the normal evaluation rules of a programming language. This means that all aspects of the document, from styling to animation to document structures, are all covered with a single language, with a single tool. The Tao Presentations platform presents many useful features that further enhance your storytelling capabilities.
Conclusion:Tao Presentations’ programing model is rooted in the real world and real speech.
We understand the world in 3D 4D.
Taking inspiration from reality extends to the drawing model of Tao Presentations. Did you ever notice that you have two eyes? Yet if you look at the average web page with one eye closed, it does not make much of a difference, does it? If you think this is normal, you spent way too much time in cyberspace.
Showing 2D contents on a screen is a technical limitation, much like black and white remained for decades a technical limitation for printers (OK, I’m showing my age again here…) For HTML, it derives from the fact that documents were initially thought of as “printed stuff”, not as “real stuff”. So the early design of HTML never took the third and fourth dimensions of space-time into much consideration. It’s time to fix that.
In Tao, the basic scene is set entirely in 3D space. Initially, the coordinates system is setup so that one unit matches one pixel on the screen for 2D objects. So if you draw a 400×300 red rectangle with the following code, it will cover a surface of 400×300 pixels:
color "red" rectangle 400, 300
2D objects and operations take arguments in the following order: X, Y, W, H. 3D objects and operations take them the following order: X, Y, Z, W, H, D. The X, Y, Z coordinates indicate the center of the object. The W, H, D indicate the width, heigh and depth of the object. For most shapes, it is the actual size. There are a few exceptions such as image or movie where the sizes are relative to the original picture.
You can rotate, translate or scale anything along any axis. In Tao, X is the horizontal axis, Y is the vertical axis, and Z is the axis crossing the screen. The center of the screen is coordinates (0, 0, 0). Positive X goes towards the right, positive Y towards the top of the screen, positive Z towards you. For example, we can rotate the rectangle along the Y axis and move it back and forth as follows:
translate mouse_x, mouse_y, 1000 * sin time rotate_y 60 * sin (2.17 * time) color "red" rectangle 400, 300
The same rules and transformations apply to anything, not just rectangle. You can change the position of text, images or videos using the exact same code. We can build a document containing four pages showing exactly that:
import VLCAudioVideo translate mouse_x, mouse_y, 1000 * sin time rotate_y 60 * sin (2.17 * time) color "red" page "Rectangle", rectangle 0, 0, 400, 300 page "Text", text_box 0, 0, 400, 300, font "Arial", 60 align 50% vertical_align 50% text "Hello World" page "Image", image 0, 0, 100%, 100%, "http://bit.ly/15poR40" page "Movie", movie_texture "http://www.youtube.com/watch?v=OK7pfLlsUQM" rectangle 400, 300
So now we have 3D and time in the same document, with very little effort. And we learned one single rule to be able to position any object in 3D space. We also learned how to colorize any object, for that matter.
Conclusion: Like us, Tao Presentations works in 4D, three dimensions of space and one of time. It’s easy.
Modeling the world around us
Of course, our 3D space would not be well populated without the ability to place 3D objects in it. So we can insert cubes, cones, spheres, tores, or complete 3D objects (e.g. modeled in your favorite Blender). This is barely more complicated than before. You need to add a bit of lighting, because otherwise the scene would look very bland. So I added two lights (named light 0 and light 1), one of them with a bluish tint:
import ObjectLoader light 0 light_position 1000, 1000, 1000 light 1 light_position -1000, 1000, 1000 light_diffuse 0%, 50%, 100%, 100% translate mouse_x, mouse_y, 1000 * sin time rotate_y 60 * sin (2.17 * time) color "#F44" cube 100, 100, 0, 50, 100, 200 cone 100, -100, 0, 50, 100, 200 sphere -100, -100, 0, 50, 100, 200, 45, 45 torus -100, 100, 0, 300, 400, 500, 45, 45, 20% object 0, 0, 200, 300, 300, 300, "Monkey.obj" text_box 400, 0, 400, 300, font "Arial", 60 align 50% vertical_align 50% extrude_depth 40 extrude_radius 5 text "Hello World"
The result looks like this:
It would have taken sweat and tears of frustration to do the same thing in five minutes with existing web tools. Moreover, what is not necessarily visible on the picture is that the scene can now be rendered on a 3D screen and show real depth, including on glasses-free 3D displays. Again, no additional effort on your part other than clicking a menu entry. The gory technical details are all taken care of for you.
Conclusion: Adding 3D space and 3D objects to your document description language opens up new and exciting possibilities in storytelling.
Enhance the visual impact
Why do 3D visuals or dynamic visual effects matter for storytelling? One reason is that the human brain dedicates a large fraction of its processing power to visual information. In short, we care about what we see. That saved our ancestors from being eaten by the stripey fangy beasts with long nails. It’s been hardwired into our brains for much longer than HTML or Facebook. The wise man will remember this important fact.
Make it shine, make it matter
Showing objects in 3D is only part of the visual impact. To make things matter, one way is to make them shine, to make them stand out. Programming the graphic card directly to show outstanding visual effects is one of the exciting possibilities offered by Tao Presentations.
It does not need to be complicated either. You can apply sophisticated materials on your objects with pre-defined modules. If I want to write some text on “marble”, for example, all it takes is using the marble word, supplied by the Materials module.
import Materials light 0 light_position 1000, 1000, 1000 light 1 light_position -1000, 1000, 1000 light_diffuse 0%, 50%, 100%, 100% translate mouse_x, mouse_y, 1000 * sin time rotate_y 60 * sin (2.17 * time) color "#F44" marble 0.5 text_box 0, 0, 400, 300, font "Arial", 60 align 50% vertical_align 50% extrude_depth 40 extrude_radius 5 text "Hello World"
The code above gives the following result:
The extrude_depth and extrude_radius are an easy way to add depth to your text. They also work on 2D shapes, i.e. you can extrude a circle or a rectangle the same way.
Of course, a similar effect can most likely be achieved on the web using WebGL, although it would probably take more time. I have not found any materials library that can be directly exploited on text. Actually, I’m not really sure how difficult it is to create extruded text like this. Anybody care to comment on that topic?
On the Tao Presentations side, more examples can be found on the web, notably in the Materials example.
Conclusion: Tao Presentations helps you make your visuals stand out.
Programming the graphic card with shaders
Under the hood, dynamic materials and effects like this are made possible by programming the graphic card directly using a language called GLSL. You can also use that language to program a variety of effects, which are demonstrated in the Shaders example of Tao Presentations (many shaders in that example courtesy of ShaderToy). Possible effects range from sophisticated visual effects to complete scenes ray-traced in real-time.
Shaders have been the topic of a number of dedicated blog entries, so I will not elaborate much more here. However, it’s worth noting that creating a fragment shader only takes a single word, fragment_shader, as in the following code, which converts a color movie to black and white on the fly:
import VLCAudioVideo
shader_program
fragment_shader <<
uniform sampler2D tex;
uniform float amount;
void main()
{
vec4 texcolor = texture2D(tex, gl_TexCoord[0].st);
vec4 blackAndWhite = vec4(vec3(dot(vec3(0.3333), texcolor.rgb)),1);
gl_FragColor = mix(texcolor, blackAndWhite, amount);
}
>>
shader_set amount := 2*mouse_x / window_width
movie 0, 0, 100%, 100%, "http://www.youtube.com/watch?v=TKAofTbxJWc"
The shader_set function lets us directly control the shader uniform variables, i.e. pass arguments to the program executing on the graphic card. Here, you can switch from color to black and white progressively by moving the mouse from left to right of the screen. Here are some screen captures:
Conclusion: By offering direct access to the power of 3D graphic cards, Tao Presentations enables a kind of highly visual and dynamic storytelling never seen before.
Tell your story in a productive way
It’s not enough to tell a beautiful story. You also need to be able to do it quickly enough. With Tao Presentations, you write this:
import SeasonsGreetingsTheme
theme "SeasonsGreetings"
main_title_slide "Main title",
title
text "Seasons Greetings theme"
subtitle
text "A theme for the Holidays"
With these 7 lines of code, you get this (minus the music ;-):
It’s hard to be more productive than that. Being able to build things on the shoulders of giants is a recurring theme when you use Tao Presentations.
Don’t reinvent the wheel, but add a warp drive if you need to
HTML has ways to import contents. You can integrate complete pages with frames. You can link to scripts, styles, images or movies. When you start writing a web page, you rarely build it from scratch.
Tao Presentations’ import statement plays a similar role. We used it in many examples. With import, you can leverage work that has already been done, either by you or by others. You can package work that you did, including the assets it needs (pictures, shaders, etc) into a module, and then reuse that module.
The module system in Tao Presentations can extend the core language very significantly with high-performance code written in C++. This is used for example to display 3D objects or videos. The video rendering is based on the well-known VideoLan Client library (VLC). For the technically inclined, the complete source code of the VLCAudioVideo module is available here.
Visualizing one billion data points in Tao Presentations was made possible thanks to this extensibility. Taodyne offers a PointCloud module capable of displaying large amounts of data. Loading that data in a specific binary format used by our customers, however, was not directly possible. So this particular customer wrote a module doing just that. Interestingly, being able to visualize this amount of data in real-time easily highlighted the presence of filaments appearing during asteroid collisions, as shown in the video below:
Conclusion: The extension mechanism in Tao Presentations makes it possible to create and reuse even the most sophisticated and specific storytelling tools.
It’s about time!
Seasoned developers will have noticed that the various examples we used refered to the mouse or to time, but totally lacked things like event handlers or callbacks. Remember, these little onload="dorotate()" and other timer = setTimeout(...) we had to use all over the place in Javascript?
It’s because with Tao, the computer works more so that you don’t have to. Under the hood, Tao Presentation listens to events, whenever you use things like time or mouse_x. Actually, the execution and drawing model for Tao Presentations requires a relatively complex piece of code, that automatically analyzes your document to identify which portions depend on events, and only redraw these when a particular event is received. The code we wrote may have been complicated, but the benefit is that the code you write is so much simpler.
This is good for the planet too, as it helps Tao use energy conservatively. For example, consider the following code, that draws a small clock:
locally
color "lightgrey"
line_color "darkgrey"
circle 0, 0, 250
locally
rotatez -6 * minutes - 0.1 * seconds
rectangle 0, 100, 15, 250
locally
rotatez -30 * hours - 0.5 * minutes
rectangle 0, 50, 15, 150
locally
color "red"
rotatez -6 * seconds
rectangle 0, 80, 10, 200
circle 0, 0, 12
circle 0, 180, 25
If you run this code, you will see Tao Presentations consume very little CPU. On my machine, it’s about 2%. Now, replace seconds with time in the last block, i.e. you should have the following line:
rotatez -6 * time
You should immediately see the CPU usage spike. On my machine, I now get about 6% of CPU. The reason is that by replacing seconds with time, we told Tao Presentations that this section of code now refreshes 60 times per second. The boundary for the changes in that case is the surrounding locally block. The locally construct restricts the rotation we make (or any attribute change, for that matter) to the enclosed block, so Tao can automatically figure out that there is no impact on the rest of the document.
Conclusion: Tao conserves energy by automatically detecing how much work needs to be done.
Don’t repeat yourself. Really
When you tell a story, you often need to repeat the same thing a number of times. Consider the clock we drew earlier. It lacks tick marks. What are tick marks? Well, they are described using a very simple algorithm: there are 60 tick marks around the clock. Every fifth is different, because it marks the hour. So we need to repeat the same thing 60 times, with two different cases. In a traditional presentation program or graphics tool, you’d have to copy-paste things quite a few times to build the 60 tick marks.
Why not let the computer do the work for you instead?
tick_mark with 0 .. 59
tick_mark N ->
locally
rotatez -6 * N
if N mod 5 = 0 then
rectangle 0, 270, 15, 40
else
color "darkgrey"
circle 0, 270, 6
The notation tick_mark with 0..59 is a map, i.e. it is a shortcut for the following code:
for X in 0..59 loop
tick_mark X
Conclusion: Copy-pasting is a good thing. Automatic generation of graphic elements is even better.
Automating complex tasks
When drawing the clock tick marks, we simply let the computer repeat an operation that we could do manually relatively easily. But what if we can’t? Consider processing news from a live source, for example. You can’t do that ahead of time. With Tao Presentations, you can automate all sorts of tasks.
There is one thing that programmers like to do: colorizing code. You can do that with JavaScript, but since the particular example I found is about 1650 lines of code, I won’t list it here. You can also get a pretty good colorization with a small enough Tao Presentations code that I can actually show it here:
import RegExp
colorize T:text ->
regexp_parse_all T,
"[a-z]w*" -> format "keyword"
"[A-Z]w*" -> format "name"
"->" -> format "rewrite"
"//[^n]*" -> format "comment"
"W++" -> format "operator"
"""[^""]*""" -> format "text"
"d+.d+" -> format "number"
"d+" -> format "number"
"[a-zA-Z][A-Za-z0-9_]+|[ ]+|." -> text regexp_captured 0
format Fmt -> text_span { color Fmt; text regexp_captured 0 }
color "name" -> color "darkblue"
color "keyword" -> color "darkgreen"
color "comment" -> color "red"
color "rewrite" -> color "orange"
color "operator" -> color "green"
color "text" -> color "brown"
color "number" -> color "pink"
// Test it
font "Menlo"
text_box 0, 0, 1200, 600,
align 0
colorize load_text "example.ddd"
Granted, it’s probably not as extensive, powerful and remarkably configurable as the JavaScript library linked above. But it’s good enough for many cases, and it’s a tiny bit shorter. Plus you can really customize it the way you want, which is good, because no two programmers on earth ever agreed on what the right color scheme is for code. Here is what you get out of it :
The same approach can be used to do more zany things:
import RegExp
girly T ->
regexp_parse_all T,
"i" -> text "i"; anchor { i_heart }
"j" -> text "j"; anchor { j_heart }
"o" -> text "o"; anchor { o_flower }
"[^ijo]+" -> text regexp_captured 0
i_heart -> color "pink"; move_to -14, 19; scale 60%; text "♥"
j_heart -> color "red"; move_to -12, 19; scale 70%; text "♥"
o_flower -> color "green"; move_to -24, 0; scale 80%; text "✿"
text_box 0, 0, 600, 600,
color "darkgrey"
font "Arial", 40
align 0.5
vertical_align 0.5
girly "Going all girly with Tao is so easy! Enjoy decorated text with very little effort."
The anchor lets you place graphics relative to the current text position, even in text boxes with complex layout. Here, we use it to decorate all the i, o and j characters. This is the result:
Conclusion: Good programmers are happy programmers are lazy programmers. Tao will make you so lazy you’ll have a hard time returning to HTML afterwards.
Interactivity and reactivity
Drawing simple scenes like we did with the monkey face is only scratching the surface of what you can do using Tao Presentations’ 3D rendering engine. We have put more examples on the web for you to get inspiration from.
Responding to presenter’s actions
Storytelling benefits from interactivity and reactivity. For example, if you want to demonstrate a new car, the best way is to be able to show it from a variety of angles:
This example is a bit longer (375 lines of code with comments), so it cannot be shown entirely in line in this article. But if you installed Tao Presentations, you can try it out by yourself.
Collective storytelling
When presenting, there is no interactivity for the audience if they don’t participate. One way to engage them more is asking them for input. For example, your presentation can present a quiz, i.e. let the many people watching your talk give their advice or answer questions. You can even organize small games, e.g. a collective scratching game to reveal the face of a person or a new product.
To offer this kind of large-scale interactivity, Tao Presentations features a NodeJS module, which can for example instantiate a web server that visitors can connect to with their smartphone. This can also be used for example to build a video-player that you remote control with a smartphone.
Bringing this to the web?
So far, we talked about Tao Presentations quite a bit. But why should one install a specific proprietary application to get these benefits? Well, what you can do with Tao Presentations as it exists today is insanely great. Obviously, one way to encourage us is to buy our product. It’s quite inexpensive as far as graphic tools are concerned, and you can do so many things with it you’ll certainly get your money’s worth.
Nonetheless, wouldn’t it be even better to do the same thing directly from a web browser? To easily get amazing interactive 3D right in your web page? At Taodyne, we strongly believe that the technology we developed for Tao Presentations ultimately belongs to the web. We think that it would make the web browser a much friendlier development platform for interactive 3D contents, that it would facilitate storytelling. In short, we think it would be a Good Thing.
Great talk. But how do we do that? Quite frankly, bringing Tao Presentations to the web looks a lot like fitting dozens of square pegs in a few thousand round holes. And there are only four developers for the moment.
Possible strategies to bring that to the web?
To bring this kind of technology to the web, we at least need:
- A way to insert XL scripts (documents) in a web page
- A 3D rendering engine that is roughly equivalent to Tao Presentations
Taodyne’s current technology is written in C++. It uses Qt for the user interface, and OpenGL for the drawing code. It also uses other components that might be a bit difficult to adapt to a web environment, notably LLVM.
We can think of a few possible strategies to adapt this technology to the web:
- Integrating the C++ code in a browser plug-in, a bit like Adobe Flash. We think the web is past this kind of technology now. Plug-ins are so 1990’s…
- Integrating the C++ code directly into an open-source browser like Firefox, WebKit or Blink. Well, Google removed 8M lines of code from Blink. So we are talking about severely large and bloated projects. Probably just a tad bit out of our league.
- Rewriting an implementation of the XL language in JavaScript, and our rendering engine on top of WebGL. Given that the existing engine uses OpenGL, that might work and be relatively fast (to develop as well as at runtime).
Of these three options, only the third one seems to make much sense with the limited resources we have. That being said, if I had infinite resources, I’d go for the second one.
Open-sourcing the technology is technologically and legally feasible
We are a very small company, and even if our four developers are really talented, there’s only so much we can do. Crowdsourcing the development might be a good way to solve the problem.
We are actually very favorable to open-source. We already publish the core XL language as well as a few add-ons for our software. So we walk the talk.
But rewriting our engine in JavaScript? That’s another thing entirely. It is a very serious effort, with a very serious impact on our business.
The bottom line is this: if there is strong interest, then we want to do it. But if people don’t care, it simply makes no sense for us to spend the time. And the best way to test if people are interested is…
Money money money. Developers developers developers.
If you want to help, you can also contribute your skills, whether you are a graphic designer (we badly need help in that area) or whether you can code your way out of a Linux kernel crash while it happens using only cat|as. Simply chime in, post on our newly-created forum, and let’s talk about what you want to do. We need, as Ballmer would say…
So if you think that being able to add 3D animations to your web pages easily is worth a bit of your money, your time, whether we can pick your brain or your fundraising talent, come and help us. And if you have any other idea on the topic, please drop me a line.
Christophe de Dinechin
PS: Out of nostalgia, the very first video we created to explain what Tao Presentations is all about. That was over two years ago. Time flies. So do pigs.
Grenouille Bouillie 