Process

Before coding, I sourced free-licensed assets from Freepik and refined them in Adobe Illustrator to rescale and simplify the illustrations to work smoothly in the browser.

From there, I:

1. Uploaded and configured PoseNet and HandPose models in p5.js.

   poseNet = ml5.poseNet(video, modelReady);
   poseNet.on("pose", function (results) {
     poses = results;
   });

   function loadHandPose() {
     // Load the model
     handpose = ml5.handpose(video, modelLoaded);
   
     // Listen to new 'predict' events
     handpose.on("predict", gotPose);
   
     // Create an ml5 neural network
     let options = {
       task: "classification",
       debug: true,
     };
     brain = ml5.neuralNetwork(options);
   }

2. Defined coordinates for the nose and finger to control filter placement.

   let nosex = poses[0].pose.nose.x;
   let nosey = poses[0].pose.nose.y;

   indexX = int(results[0].annotations.indexFinger[3][0]);
   indexY = int(results[0].annotations.indexFinger[3][1]);

3. Coded logic for five filter options, each triggered by the position of the index finger.

   // Elf filter
   if (20 <= indexX && indexX <= 115 && 320 <= indexY && indexY <= 430) {
     image(elfLeftEar, nosex - 120, nosey - 110, 70, 70);
     image(elfRightEar, nosex + 120, nosey - 110, 70, 70);
     image(elfHat, nosex, nosey - 155, 190, 150);
   
   // Snowman filter
   } else if (145 <= indexX && indexX <= 240 && 320 <= indexY && indexY <= 430) {
     image(snowmanHat, nosex, nosey - 170, 140, 100);
     image(snowmanNose, nosex + 40, nosey, 130, 40);
     image(snowmanScarf, nosex, nosey + 200, 180, 180);
   
   // Santa filter
   } else if (270 <= indexX && indexX <= 365 && 320 <= indexY && indexY <= 430) {
     image(santaHat, nosex, nosey - 170, 180, 150);
     image(santaBeard, nosex, nosey + 80, 150, 200);
   
   // Polar Bear filter
   } else if (395 <= indexX && indexX <= 490 && 320 <= indexY && indexY <= 430) {
     image(polarbearLeftEar, nosex - 80, nosey - 150, 100, 100);
     image(polarbearRightEar, nosex + 80, nosey - 140, 100, 100);
     image(polarbearNose, nosex - 10, nosey, 70, 70);
   
   // Rudolf filter
   } else if (520 <= indexX && indexX <= 615 && 320 <= indexY && indexY <= 430) {
     image(rudolfNose, nosex, nosey, 30, 30);
     image(rLeftHorn, nosex - 80, nosey - 140, 100, 150);
     image(rRightHorn, nosex + 80, nosey - 140, 100, 150);
   
   // No filter
   } else {
     console.log("0");
   }

4. Tested and refined the outputs so filters lined up consistently with facial features.