Projection Mapping

In previous videos we have looked at how to do both 2D and 3D projection mapping. However, in all of those videos there was one caveat, we had to either find or create projection content beforehand, and once we loaded it into our system we could not change it. Today let's look at some ways we can generate our projection content in real-time which also allows us to add some interactibility to it.

The Project

For this project we are going to use a physical joystick to control the dynamic lighting in a 3d projection design, by moving the joystick around we also move the corresponding light around which will cause the shadows and reflections to change on the object. As always you can find links to the code used and all parts in the description below if you want to try this for yourself.

We can divide this up into 4 different sections. The first section is the Joystick that gets our input. The second section is how do we convert an electrical signal from the joystick into something the computer can understand. The third section is how to create a 3d environment to hold our 3d object and lights. The fourth and last section is the dynamic control and 3d projection mapping. Let's start with the Joystick.

Joystick

Joysticks are pretty simple input devices. We are probably most familiar with seeing them on electronic devices such as game consoles. The only thing that the joysticks do is it get the x and y coordinates from the user. You can push it to the left or right, up and down, or, a combination of the two axis to complete the coordinate input. Joysticks usually have 4 connections that are needed to get the x and y values. First, we have to connect it to power and ground. Next, there will be two pins that correspond to the x and y axis.

Internally a Joystick is a combination of two potentiometers, one for each axis. As we move the joystick the voltage level will change. For example if we were to move the joystick all the way to the left we would see 0v, and moving it to the right would result in 5v.

Arduino Microcontroller

We now need to make sense of these raw voltage values and convert them into something that the computer can understand. To do this we are going to be using a device called a Microcontroller, which is essentially a very little computer that has many inputs and outputs. The simplest microcontroller to get started with is the Arduino platform. This provides us with a very easy way to get into the world of embedded programming.

Let's start by connecting our joystick to our Arduino. we want to make the connections show below.

There are two main types of inputs, analog and digital. A digital input only has two states, either on or off just like a light switch. Since we need to measure the precise voltage from the pins on the joystick we will need to use what is called an analog input, otherwise known as an adc, analog to digital converter. What this does is takes a voltage level and maps it to a number. The lower the voltage the lower the number, and the higher the voltage the higher of the number. We simply need to write a few lines of code that configure the pins as inputs, then read the analog value of the x and y axis.

Now that we have the joystick values we need to find a way to get them from our Arduino device to a computer. Thankfully there is a protocol called serial that allows us to send data over a USB connection. To initialize a serial connection on our Arduino we need to call one line of code. First, we need to create a string to send over the serial connection. This will consist of the x value and y value separated by a comma and a newline character at the end. Now that the string is ready to go we just need to call a command to send it. Now we can upload this code to our board and test to see if it is working.

Processing

Now let's move on to the 3D section. I decided to use Processing, an open source framework for visual arts. It has both a 2D and a 3D rendering engine built on top of OpenGL which will make this a perfect choice for creating our own 3D projection mapping sketch. If you are new to 3D projection mapping check out my previous video that explains the concepts. Just like last time we need a three dimensional file that matches the object that we are going to be projection mapping onto in the real world. I found a pitcher design and 3D printed it out using white PETG. Since the 3D object and model are the exact same the mapping process will be fairly straightforward.

First we need to create a 3D environment within processing. Next we need to import our .obj file. From the last video we know that all we need to do is align the camera in the scene to the same location where the projecto is in the real world. In Processing we can accomplish this by translating, rotating, and scaling the object with a few lines of code. Since we will need to do this live when setting up our I wrote some code that listens to key presses and does it for us.

Now, we need to receive the serial communication from the arduino. Thankfully processing has a serial library built right in that we can start using without too much extra configuration. We just need to find the port that the Arduino is connected to, and we can do that by using the Arduino IDE and hovering over tools then ports. Simply put that in processing and we are ready to go!

Now in our draw loop we can listen for the incoming serial packets and parse them by splitting the string at the comma and parsing the resulting values into an integer variable. And that integer variable corresponds to the x and y coordinates of the point light we added earlier.

Arduino Code:

#define joyY A0
#define joyX A1

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  pinMode(joyX, INPUT);
  pinMode(joyY, INPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
  int xVal = analogRead(joyX);
  int yVal = analogRead(joyY);
  String data = String(xVal) + "," + String(yVal) + ",\n";
  Serial.print(data);
  delay(40);
}

Processing Code:

import processing.serial.*;
Serial myPort;
int currentX = 900;
int currentY = 700;
int currentZ = 600;
float rotX = 3.14159f;
float rotY = 0;
float rotZ = 0;
int lightX = 80;
int lightY = 20;

int moveMult = 10;
PImage img;

PShape s;
String data;
public void setup() {
  size(1920, 1080, P3D);
  frameRate(30);
  s = loadShape("vase2.obj");
  myPort = new Serial(this, "/dev/tty.usbmodem1101", 9600);
}
    

public void draw() {
  background(0);
  pointLight(50, 200, 255, lightX+300, lightY+100, 1700);
  //spotLight(255, 255, 255, lightX, lightY, 40, -1, 0, 0, PI / 2, 2);
  translate(currentX, currentY, currentZ);
  scale(50);
  rotateX(rotX);
  rotateY(rotY);
  rotateZ(rotZ);
  ambient(51, 26, 0); 
  shape(s);
  while (myPort.available() > 0) {
    data = myPort.readStringUntil(10);
    if (data != null) {
      String[]splitData = split(data, ',');
      lightX = int(splitData[0]);
      lightY = int(splitData[1]);
    }
  }
}

   
void keyPressed() {
        if (key == 'w') {
            currentY -= moveMult;
        }
        if (key == 'a') {
            currentX -= moveMult;
        }
        if (key == 's') {
            currentY += moveMult;
        }
        if (key == 'd') {
            currentX += moveMult;
        }
        if (key == '1') {
            currentZ += moveMult;
        }
        if (key == '2') {
            currentZ -= moveMult;
        }
        if (key == '3') {
            rotX += moveMult / 57.29;
        }
        if (key == '4') {
            rotX -= moveMult / 57.29;
        }
        if (key == 'k') {
            moveMult = 10;
        }
        if (key == 'l') {
            moveMult = 1;
        }

    }

Time to Test it Out!

I placed my projector in front of the object and used the arrow keys that we programmed in earlier to make sure the camera in the scene matches the projector's output exactly. Now, we can move the joystick around and see the lighting change. Neat!

Github Repo:

https://github.com/tmkinteractive/real-time-projection-mapping/blob/master/processing/processing.pde

Real-Time Projection Mapping Basics