Thursday, 28 March 2013

Measuring your dimensions with your webcam

Swami Gulagulaananda said:
"When you try to write a program that does what a human can do, you will appreciate the greatness of nature"

Microsoft's XBox Kinect is a wonderful device and an optional add-on to the XBox gaming console. The device allows you to play games without using any hand-held controllers. For example, if you were playing Fruit Ninja, you can slice fruits just by making a slicing action with your hand - Woop! The fruit on the screen gets sliced. There are many beautiful games that you can play with the device.

But from a technical standpoint, it is interesting how the Kinect actually determines your position. The Kinect has 3 sensors - One is a camera, one is an Infra Red sensor and one more I think is used to get wireframe images. In fact, there are special chips within the device that allows for motion and gesture recognition, 3D rendering etc. Read more on Wikipedia here.

Now let's have a quick look at the problem statement - It's a tool being developed for an online apparel store. A common problem that people face is - I am not sure what size I have to buy. So imagine if there was an online tool that tries to determine your size using your webcam?

Now let's have a look at the technical challenges. If you are standing in front of a webcam, the image that is obtained has you with some background that includes a window probably, table, chair, door etc. The first challenge is to separate the person from the background.

The next challenge is to get perspective. If a relatively wider person stands farther from the camera, his width will appear to be the same as that of a thinner person who's closer. As far as the image is concerned, both of them will appear to have the same width. This is because an image is 2-dimensional, and the depth is lost.

Of course, if you were doing this with Kinect, neither of this is particularly difficult. Primarily because the device can easily judge how far from the camera you are using IR beams. It can beam, get reflection and determine distance. And identifying where you are is also not an issue because they have specialised chips that do that work. In fact, it can even determine where your head, hands, legs, hips etc. are.

Then why not use a Kinect? For obvious reasons - A Kinect is not a device that everybody has, a webcam is something that almost everyone has because it comes with laptops. Besides, I don't think many would be driven to purchase a Kinect just to try out some app - it costs about Rs. 9000.

Ok, we have established that we have to use a webcam and we have a couple of challenges that a Kinect, an expensive device with special hardware can resolve.

To solve the first problem of extracting a person from the background, I first ask the subject to step out of the screen so that I can capture just the plain background. I then ask the subject to step in. This way, I will have two photographs - and subtracting one from the other will give the difference - which means, whatever is common should get erased. Whatever is different, the difference of their pixels will remain. What this means is that, the window was there in the photo without the subject and the photo with the subject. So the window gets erased, since the pixel values of the window will be the same (more or less) in both. I use a threshold to eliminate the problems that arise because of the slight differences in pixels that arise because of varied lighting. At the end of removing speckles and noise and some post processing, I will have an image that has the subject standing with a white background (the common background is made white)

Great! Now we have been able to extract the person. However, the problem of perspective still remains. A fat person at a distance and a thin person who is closer appear to be of the same width in a 2D image. The Kinect, as discussed previously, uses IR to solve this problem. A webcam can't do that. So instead, the idea was to ask the subject to hold an object of standard size and dimensions, something that everyone will have. A Rs. 100 note seemed ideal.

If the person is holding the note in his hand, we can determine the size of the note in the image. The size of the note in real life is already known. We get a ratio of sizes. We know the size of the person in the image. The ratios are going to be the same, thus we know the person's size in real life.

Great! Now we have resolved the second issue. However, there is a challenge. You need to be able to measure the size of the note in the image. For that you should know where the note is. A human can do that. A program cannot easily do that - it probably needs a note detection program. I tried to train one using OpenCV but unfortunately couldn't get it up.

So the next best solution was to draw a rectangle on the screen. What you have to do is hold the note against your body and align yourself to the rectangle. By doing this, the distance between the subject and the camera is always the same, no matter who is holding it, fat or thin. The ratio is also fixed.

The note has to be held against the body - this is the only way I know that the ratio of the note in the image to real life is the same as that of your image and you in real life. If the note is held a foot away from you, it defeats the entire purpose.

The next objective is to measure your chest. A human being knows where the chest begins and ends, and where the arms lie. The program is dumb. So I ask you to spread your arms apart. This way, your arms will not appear in the measurement of the chest. But I have to determine where your chest is. What I do is, I first determine where your head is using a standard face detection algorithm. I then go a little to the right, and then start going down till I get your arms - you had spread them wide, remember? After that, I continue moving further till I reach the end of your arms. I have almost reached the measurement region. The point of intersection between the horizontal line drawn from this point and the vertical line drawn from the centre of your face marks the centre of your chest. We have now reached the point of measurement.

I now measure this width of your chest and get some  value. If I use the ratio, will I get the right measurement? No, I won't. The human body is not a cuboid. Though we see a flat image, it is a curved body in reality. So, in the next step I ask you to turn to the right by 90 degrees and capture your side profile and measure that width as well.

The cross section of the human body when looked from the top, around the chest area is very close to an ellipse. What I have obtained here is the major axis and the minor axis of the ellipse.

The objective was to find the circumference. There is a formula for that. We can use the ratio to get the real width and depth and get the approximate circumference of the person. The formula is below

Are we done? There are other challenges too. Like, if you were wearing a loose shirt, the extra cloth stretches and makes it appear like you are wider than you really are.

When you are done aligning, how will the program know? If you try to click a button, you will have to move closer to the computer which means the position is lost. I am using a motion detection library there. So you just have to wave your hand at the top left to indicate that you are done.

And what if you were standing right in front of a wall? Then a shadow might get cast on the wall. The shadow was not there when the background was captured but is present when the person's photo is captured, making it seem to the program that the shadow is part of the person. So this also makes it feel like the person is broader than he actually is...

And what if you are wearing a white shirt? I told you that I was making the background white. Since your shirt is also white, I will consider that as part of the background, ergo I won't know where your body is.

These are a couple of problems that have to be solved, but for now, the only way to avoid it is to avoid the situation. Wear a darker tighter shirt, don't stand close to a wall.

I found that most people cannot understand the challenges faced in this problem. When they try the program out, they expect things to be way simpler and even the slightest discomfort made them say

"WTF is this shit?" and I find it hard to explain to them the various challenges.

Like I said before, if you use a specialised hardware like the Kinect, you don't have to do the dance of the sugar plum fairy. This project is purely an academic one and shows how to measure it with image processing and a simple webcam.

If you have a different or better way to solve this or these problems, I would definitely love to hear about it. I hope you liked what you read. Do leave me a comment with your opinions and suggestions.

A lot of the things that I have attempted to do programmatically are extremely trivial for a human being to achieve.

Comments from Facebook

Adarsh Basavalingappa: 
"necessity" is the mother of invention. Now let us know about the mother :P

Chiranth Ashok:
Hey this is a very interesting problem you are trying to solve. :)

You can check out the SIFT algorithm to find your hundred rupee note in the frame, its pretty good at such a job. You won't even need the whole damn note, any two distinct points on the note will do because you can get your ratio even that much by appropriate mapping.

I feel you could use the same algorithm to find other features (eyes, chest, arms, etc.) by comparing the input image against a template of a human body..

Let me know your thoughts on this approach and if you get anywhere this way. I worked on the SIFT algo a while back and it interests me what it can achieve. :)

Nikhil Baliga: 
Hey Chiranth Ashok - I vaguely remember trying that with OpenCV - for note detection. I couldn't get it up and running. Maybe I was doing something wrong. Another problem was that the program had to be done with Flash. So translating something that works in Python to something in Flash was another challenge

Adarsh Basavalingappa - Mother of what man? :P I posted it because I was tired of listening to "WTF is this shit" :P Thought it was better to show why I was doing what I was doing

Chiranth Ashok: 
Flash? I did that using cpp.. czech out this blog
I was able to get it running in a few hours I guess, you can do better :)

Nikhil Baliga: 
:) One of the first limitation is that it has to be done with actionscript because it should run within a browser... But I'll definitely have a look at it. Seems interesting

Saturday, 16 March 2013

The Patient

Swami Gulagulaananda said:
"It is the mind that makes you feel what it wants to make you feel"

You can now read the story from the book - Swami G's Short Stories

My head was feeling heavier and lighter alternately. I could feel myself letting go as I relaxed in my easy-chair. I imagined that this was the kind of headiness that was experienced by those who consumed liquor. I tried to open my eyes but could barely get my eyelids to separate. When I did manage to squeeze some sight through a narrow slit between my eyelids, all that I could see was some white light. I let out a soft groan.

"Are you in a condition to speak?" said a dignified voice. "Yes, I am just finding it a little hard to stay conscious" I replied. "That's good. A semi-conscious state seldom exercises restraint. When the inhibition centre shuts down, the invisible iron curtain crumbles and like water escaping through the breach in a dam, all your thoughts will come gushing out. I believe we can have a meaningful session today. Indeed, I do" came the reply.

I tilted my head gently to my left and slowly tried to open my eyes once more. Through the curtain behind the chair in front of me, a sliver of light crept in and illuminated the back of the man who sat in the chair, casting a silhouette. With quite some effort, I managed to focus my eyes on the stolid countenance of the psychiatrist. The elderly gentleman's bespectacled face betrayed no emotion, his thin lips set in a horizontal line. His eyes though, were alert, focused and strangely not blinking, as if he didn't want to miss out anything. It was hard for me to believe that a man like this could have such a baritone voice, deep and yet cultivated and pleasing to hear. I turned my head to my front once more, grateful that he had agreed to come for a house visit.

"So, when did you realise it?" asked the doctor persuasively, "Or would you like to start from the very beginning?" he added. I closed my eyes to focus better. I didn't want the strain of processing visual information to clutter my thoughts. My brain clearly didn't have adequate energy to process sight AND recall memories. "There's not much to say about the beginning. I was waiting at the bus-stop in front of my college when I saw her. She looked very beautiful and I was attracted to her. I knew she was from my college because she wore our ID card. I wanted to talk to her and so I approached her and not knowing what to speak, asked her if she had any idea when the bus would come." The doctor seemed to be jotting down something in a little notepad that he held in his hand. "Was there anyone else there besides the two of you?" he asked, to which I replied "I don't remember seeing anyone. Maybe I was so attracted to her that she became the only thing that filled my eyes. Or maybe there was nobody there, I really can't say." "Love, huh?" said the doctor and smiled. I said nothing.

"Proceed" he said. "She seemed sweet. I asked her about her class, she asked me about mine. We made fun of some teachers, I was pretty good at mimicry and made her laugh with some imitations. Her laugh had a very nice ring to it and I wanted to hear it all the time. I never do it, and she said neither did she, but we decided to exchange our phone numbers. We had let our respective buses go because we felt we were enjoying the conversation. We both didn't want it to end. I knew I was falling for her, and hard." I kept silent for some time. The doctor waited patiently. I think he could understand the pain I was going through. "We spoke over the phone. A lot. For hours together. We had a similar wavelength, it was like I was completing her sentences and she, mine. And we both knew what the other was thinking. I thought, there can be no way that we will have disagreements ever. This was what resonance meant."

I continued "I finally confessed that I was falling in love with her. She said that she too was falling in love with me, but for us to be together, she had a strange proposal. I should have guessed when she suggested that" I said.
"Uh huh? What was it that she proposed?" asked the doctor.
"She told that neither of us would tell anybody else about each other. That it was to be a complete secret"
"And you agreed to this?"
"Why? Didn't you think it was strange?" asked the doctor as he continued scribbling.
"I did. But then she said that if we both agreed and understood, that was more important. That it was pointless trying to convince others as to why we did what we did. In the end, we were to be with each other, and it was important that we understood each other"
"Interesting" said the doctor. "And did you ever feel that you were being manipulated?"
"No, never. I guess I have a quality of trusting people. Mistrust and distrust, I believe that the human society cannot live with it. We eventually have to trust somebody"
"I see. And none of your friends knew? What happened after that? When you got married, I mean."

I slowly opened my eyes. I was a little less drowsy. I recollected the conversation. "I told her that I wanted to get married in the conventional way, in front of friends and family. She said she wanted to live with me forever, and that marriage was a waste of time. However for legal reasons, she was willing to sign and get married in front of a judge. But the fanfare was something she wanted to avoid."

"And you agreed to this too?" asked the doctor. I could sense a certain element of bewilderment. A man I believed to be completely impassive possessed emotions afterall.  "I did. I loved her and I didn't want to lose her. I agreed to her silly conditions for that singular reason. And she added that she didn't want my friends or family to know. It was strange. So when I got an opportunity to move to a different country, I took it. My boss was pleased, my family was pleased because of the dollar rupee conversion, and she was pleased because nobody would bother us. And I was pleased, because it seemed I got it all"
"Indeed. But for how long? Sooner or later, your family would want you to come back and get married."
"I thought of that. And I convinced myself that I would cross that bridge when it came. The immediate plan seemed to work"

"So when did you realise it?" asked the doctor again
"I know introverts. But this was a different kind of introvert. She met nobody, she spoke to nobody else. She never went out of our house. Not even to purchase provisions. She bought them online and had them delivered. They would be lying outside the house, she wouldn't even collect it from the delivery boy. I used to bring them inside when I came home. At other  times, she would call me at work and ask me to get provisions on my way back. I began thinking then... What could be the reason for that"

"And that made you realise? That she was not real? That she was just a figment of your imagination? That that was the reason she was completing your sentences, they were the same sentence after all. That you were the only one who spoke to her? That she met none of your friends and you met none of hers? That that was the reason you had to move away from your friends and family because sooner or later you would realise that it was all not real? That it was you who was placing orders online? If there was nobody to collect it from the delivery boy, it will lie outside your house, am I not right? You subconsciously knew she was not real. You didn't want to consciously realise it. That was the reason you left home to an unknown land, didn't you? You had an idea that there was something wrong with you... mentally"

I turned to look at the doctor. His alert eyes were focused on me. His thin lips, expressionless. His eyes, expressionless. "Yes" I said. "It was then that I realised. I think I am schiezophrenic, and I need help" I said.

Suddenly the door opened and my wife walked into the room and said - "I thought I heard your voice. Who are you talking to?" I turned to look at the doctor, and his thin lips had a smile. "Indeed... You have a problem. You need help" he said.

You may also like to read
Other short stories, by Nikhil Baliga

Comments from Facebook

Prajwal M Sudarshan: 
Nicely written..

Aditi Srinivasan:

Suraj Srinivas:
Nice one! :-)

Aditya Kiran:
Awesome :-)

Vishwanath Korisetru:
Nice twist !!

Vishwanathan Shankar:
sooper maga :)

Madhura Basavaraju:
awesome!! i know that's an understatement.. u should think of writing a novel!

Ram G Kulkarni:
nice one !! :)

Roshni Ramesh:
I liked the ending a lot :)

Susheela Sadashivaiah:
Really nice! :)

Abhishek Kodankiry:
Mindblowing man. Very thrilling..

Comments from Twitter

Niranjan Lakshmanan ‏@niranjanl
awesome :-)

Lokesh Acharya ‏@acharya2
nice one sir.. :) most of your love stories make me get scared :-P :-$

Mithun Shroff ‏@MithunShroff
Dude.. Dafaq was that..!? Mind blowing man.. Bloody good.. You should write more! Reminded me of Shutter Island.

Sunday, 10 March 2013

Project Terrarium

Baba Gyani Triviani said:
"As long as man doesn't interfere, nature works perfectly."

The other day, I read a very interesting article that I would like to share with all of you. A man planted four seedlings in a glass bottle in 1960, and watered it once. He then sealed it and hasn't watered it for the past forty years. The plant is alive and wonderful.

 I felt this idea was truly brilliant because it is like a demonstration of quite a few things that we have studied in school. For example, plants consume oxygen and give out carbon di oxide during respiration, and consume carbon di oxide during photosynthesis to produce food and releases oxygen. The cycle repeats, and the plant is completely self sustained.

The plant absorbs water from the soil and consumes it. Excess water is given out into the atmosphere via transpiration. Water vapour form clouds and it rains to give the water back to the soil which is then absorbed by the plants and another cycle repeats.

If a plant dies, it gets decomposed and goes back to the soil as nutrients and another plant absorbs the nutrients to grow. Another cycle.

So, basically, plants are completely self sufficient, and the so called "Circle of Life" and other cycles are demonstrated, albeit subtly without anyone realising it.

I thought that this idea was very interesting. And just for the heck of it have decided to try it out myself. So I took an old Priya Pickles jar, took some good soil from my garden, some dried leaves sand crushed it, one semi dried leaf off the ground and tore it and mixed it with the soil, just to make it seem like some real soil.

And I chose Fenugreek (methi / menthya) as the plant of choice, mainly because I have grown it before and I know it grows very easily and quickly, not too tall and the seeds are available in the kitchen :-) Then I put it in and shook it like James Bond's Vodka Martini. Added some water, and left it yesterday. Today, I observed the bottle and saw the seeds have begun germinating. I am already excited.

Now what will happen in case of disease, I don't know. I know that there are some bacteria in the soil as well, though no insects. I think if this is successful, it would be great to add some small insects too, like an aquarium, a terrarium. As in, a small earth, an ecosystem of sorts. Of course, all this is mere wishful thinking. I have done no research on this, though I know that there are plenty of sites that explain a lot of things. Like adding charcoal to keep the soil fresh, etc. But as an idea, I liked it a lot. In terms of maintenance, it's nothing. And if this works, probably next time, I shall use a beautiful glass jar instead of a pickle bottle. And grow a Bonsai instead of Fenugreek.