Artwork that knows when you are taking a picture of it

This is my electronic artwork inspired by Peter Vogel’s and Walter Giers works. And I am pretty happy with how it came out. It looks cool and it has artificial intelligence (to a limited extent). It recognizes when you are taking a picture of it with a mobile phone. And it uses only a microcontroller to do it. All thank to TinyML.

It consists custom-made PCB (made by PCBway) where plugged MCU (Arduino Nano PLE sense), a camera (OV7670) module and some LEDs. The largest part is the old soviet doorbell module, made in 1987. And then the speaker is connected there. I also added some status LEDs, so I will know what the MCU is currently doing.

The camera takes a picture every few seconds and MCU analyzes it. It detects things in three categories: 1) humans, 2) when someone takes a picture of it with a cell phone 3) and all other scenes.

There are also some sensors that can sense the environment:

  • 9 axis inertial sensor LSM9DS1
  • humidity, and temperature sensor HTS22
  • barometric sensor LPS22HB
  • digital microphone MP34DT05-A
  • gesture, proximity, light colour and light intensity sensor APDS-9960

And there is a Bluetooth module. But I have not used it so far.

Most of my effort and time have been spent building a Machine Learning model to process images and recognise scenes. So far I have collected almost 700 images with this camera. And labelling all of them. I use Edge Impulse and it makes the process to build an ML model and library to embed it very easy.

This camera can take colour images in sizes 160×120 px, but the model uses size 96*96px. And it is quite small. You can imagine how few pixels take a mobile phone on these images. It’s just a black blob.

There is a big difference in model performance whether to use an unoptimised (float32) or the quantised (int8) model. The microcontroller’s small resources force us to use the quantised (int8) model. I have experimented with model parameters and found optimal results that work for me.



Pin layout may be different on different camera modules!

The Soviet doorbell electronics board


21 thoughts on “Artwork that knows when you are taking a picture of it

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.