Artwork that knows when you are taking a picture of it

This is my electronic artwork inspired by Peter Vogel’s and Walter Giers works. And I am pretty happy with how it came out. It looks cool and it has artificial intelligence (to a limited extent). It recognizes when you are taking a picture of it with a mobile phone. And it uses only a microcontroller to do it. All thank to TinyML.

It consists custom-made PCB (made by PCBway) where plugged MCU (Arduino Nano PLE sense), a camera (OV7670) module and some LEDs. The largest part is the old soviet doorbell module, made in 1987. And then the speaker is connected there. I also added some status LEDs, so I will know what the MCU is currently doing.

The camera takes a picture every few seconds and MCU analyzes it. It detects things in three categories: 1) humans, 2) when someone takes a picture of it with a cell phone 3) and all other scenes.

There are also some sensors that can sense the environment:

  • 9 axis inertial sensor LSM9DS1
  • humidity, and temperature sensor HTS22
  • barometric sensor LPS22HB
  • digital microphone MP34DT05-A
  • gesture, proximity, light colour and light intensity sensor APDS-9960

And there is a Bluetooth module. But I have not used it so far.

Most of my effort and time have been spent building a Machine Learning model to process images and recognise scenes. So far I have collected almost 700 images with this camera. And labelling all of them. I use Edge Impulse and it makes the process to build an ML model and library to embed it very easy.

This camera can take colour images in sizes 160×120 px, but the model uses size 96*96px. And it is quite small. You can imagine how few pixels take a mobile phone on these images. It’s just a black blob.

There is a big difference in model performance whether to use an unoptimised (float32) or the quantised (int8) model. The microcontroller’s small resources force us to use the quantised (int8) model. I have experimented with model parameters and found optimal results that work for me.

Details

Schematic:

Pin layout may be different on different camera modules!

The Soviet doorbell electronics board

Links

21 thoughts on “Artwork that knows when you are taking a picture of it

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.