plyrenderer

To display a PLY, there already exists a few solutions however I was trying to display them in the browser and there is no such solution. So, I wrote a first prototype in java with GWT.

As I forgot how to display 3d objects since school, I looked on the web and found a very interesting post from Markus Dwyer who wrote this tool: http://mdkey.org/?p=406. I took this code and converted it in java. You will ask me why I rewrote it in java, well there is no good reason except that I wanted to :-).

It will download the point cloud from Google App Engine and display it on a canvas. I will update the prototype very soon to download a ply using JSON to avoid google app engine.

Please note that the prototype only displays point clouds for the moment.

You can use:

• arrow keys to move
• shift + arrow keys to rotate
• Select the mode and use the mouse to rotate, move or zoom
• Touch screen works but only with one finger (I will update multi touch soon)

 

The source code can be found here.

Here are the photo I used to reconstruct the model:

How to download images from Bing?

Here is a small python script I used to download images from bing search engine. An extra test is performed to check if the image has the focal length in the exif information. This information is mandatory for bundler so the download system will skip the images which are not matching.

To download images from bing, you must first create an app_id for your application here:  http://www.bing.com/developers/createapp.aspx

There already exists libraries to get search result from bing: pybing and bingapi. They both work pretty well. I decided to use pybing. So let's install it:

sudo easy_install pybing

The next step is to install the python imaging library in order to capture the EXIF information:

sudo easy_install PIL

Script

1. from pybing.query.query import BingQuery
2. from pybing.resultset import BingResultSet
3. from pybing.query import BingQuery
4. from pybing.query import Pagable
6. import Image
7. import ExifTags
9. import os
10. import sys
12. __author__ = 'jacques fontignie'
14. from pybing import *
16. APP_ID = <ENTER_YOUR_APP_ID>
18. exifAttrs = dict(Model=True,Make=True,ExifImageWidth=True,ExifImageHeight=True,FocalLength=True)
19.                  
20. class ImageQuery(BingQuery, Pagable):
21.     SOURCE_TYPE = constants.IMAGE_SOURCE_TYPE
22.    
23.     def get_request_parameters(self):
24.         params = super(ImageQuery, self).get_request_parameters()
25.         params.update({
26.             'Image.Filters': 'Style:Photo'        
27.         })
28.         return params
30. def download(dir, filename, url):
31.     """Copy the contents of a file from a given URL
32.    to a local file.
33.    """
34.     import urllib2
35.    
36.     try:
37.         webFile = urllib2.urlopen(url)
38.     except:
39.         print "Unexpected error:", sys.exc_info()[0]
40.         return False
41.     extension = url.split('.')[-1].lower()
42.     if not extension == "jpg":
43.         print "not a jpg extension: " + url
44.         return False
45.     localPath = dir + "/" + filename + "." + extension
46.     localFile = open(localPath, 'w')
47.     localFile.write(webFile.read())
48.     webFile.close()
49.     localFile.close()  
50.    
51.     print "File successfully downloaded"
52.     try:
53.         photoHandle = Image.open(localPath)
54.        
55.         found = False
56.         exif = {}
57.         info = photoHandle._getexif()
58.         if info:
59.             for attr, value in info.items():
60.                 decodedAttr = ExifTags.TAGS.get(attr, attr)
61.                 if decodedAttr in exifAttrs: exif[decodedAttr] = value
62.                 if 'FocalLength' in exif:
63.                     found = True    
64.            
65.         if not found:
66.             print "Dropping the file as it does not contain any useful exif information"
67.             os.remove(localPath)
68.             return False
69.    
70.     except:
71.         print "Unexpected error:", sys.exc_info()[0]
72.         os.remove(localPath)
73.         return False
74.         # get EXIF information as a dictionary
75.    
76.    
77.     return True
79. def main():
80.    
81.     dir = "images"
82.     if not os.path.exists(dir):
83.         os.makedirs(dir)
84.    
85.    
86.     query = ImageQuery(APP_ID,query="rushmore")      
87.          
88.     results = query.execute()
89.     default_name = "img_"
90.     index = 0
91.     count = 0
92.    
93.     for result in results:
94.         count = count +1
95.        
96.         url = result.mediaurl
97.         print (str(index) + ") " + str(count) + "- " + url)
98.         filename = default_name + '%05d' % index
99.                  
100.         if download(dir,filename,url):
101.             index = index + 1
102.        
103.        
104.         if index == 500:
105.             break
107.        
108.     print "Images succesfully fetched ({})".format(index)
109.        
110.    
111. if __name__ == '__main__':
112.   main()

Compile pixelstruct

In order to compile pixelstruct. Here is how I did:

1. Download pixelstruct using SVN:  svn checkout http://pixelstruct.googlecode.com/svn/trunk/ pixelstruct-read-only
2. Install the library needed:  sudo apt-get install libqt4-dev libcgal-dev cmake
3. Go in src/glwidget.h and add the line: #include "GL/glut.h"
4. Remove the folder cmake as it contains an old script to find CGAL
5. Go in src/CMakeLists.txt and add the line: INCLUDE( ${CGAL_USE_FILE} )
6. Compile using make command

osm-bundler

Now let's look a bit on bundler and more particularly on the osm-bundler which is an open source software based on python.

Object to reconstruct

The goal of this example is to show how to reconstruct a simple object, For the demo, it will be a toy car. You can see here the photos, I used:

What a wonderful car, isn't? The photographs were taken with an iPad 2 main camera: you will see later that it is important to know the model.

Create the point cloud

So, I put all my photos in a folder and can now create the point cloud: to do it, I use: osm-bundler and run the following command:

python /XXX/osm-bundler/RunBundler.py --photos=.

The whole operation took: 1 min 43 seconds and if everything goes well, the output should be something like this:

Working directory created: /tmp/osm-bundler-ip82t8
BundlerMatching executable path: /XXX/osm-bundler/software/bundler/bin/KeyMatchFull
Sift executable path: /XXX/osm-bundler/software/vlfeat/bin/glx/sift

Processing photo 'IMG_0018.JPG':
        Extracting features with the SIFT method from VLFeat library...
        Found 8609 features

Processing photo 'IMG_0019.JPG':
        Extracting features with the SIFT method from VLFeat library...
        Found 8505 features

...

Performing feature matching...
[KeyMatchFull] Reading keys took 3.460s
[KeyMatchFull] Matching to image 0
[KeyMatchFull] Matching took 0.010s
[KeyMatchFull] Matching to image 1
[KeyMatchFull] Matching took 0.570s
[KeyMatchFull] Matching to image 2
[KeyMatchFull] Matching took 1.160s
[KeyMatchFull] Matching to image 3
[KeyMatchFull] Matching took 1.890s
[KeyMatchFull] Matching to image 4
[KeyMatchFull] Matching took 2.480s
[KeyMatchFull] Matching to image 5
...
[KeyMatchFull] Matching to image 9
[KeyMatchFull] Matching took 5.760s

Performing bundle adjustment...
Finished!
See the results in the '/tmp/osm-bundler-ip82t8' directory

A few explanations on the output:

1. The first step to perform is to analyze the different pictures and understand their features of key points. A feature is an area in the picture which could be "easily" recognized on other pictures. Calculating the features is relatively slow as it can take between 1 and 2 seconds per picture (and my pictures have a very poor quality). For more information about the algorithm used, you can go here. We will see later we could use other algorithms for the feature detection
2. When all the features have been determined, the algorithm will match them together, that means it will compare the features of two different images and understand if there are any relationship. This algorithm is the longest one as it O(n^2) where n is the number of images.
3. When all the relations have been determined, the bundle adjustment is performed to understand where are the cameras compared to the scene and add the different points in one unique scene.

Point cloud obtaining

If we go in the temporary folder, here is what we can see:

We have many files, but in particular, a folder called bundle, this one will contain the point cloud:

The ply files correspond to the point cloud found after each iteration, the last one is thus the most signification.

By opening it with Meshlab:

meshlab points010.ply

we get the following result:

It does not seem very relevant with one screenshot, but you can try by yourself and you will see that we see quite well the car in the middle of the cloud and the carpet around.

PixelStruct

You can then use pixelstruct to make an animation similar to photosynth:

pixelstruct .

OpenPhotoVR

A software I tried is called OpenPhotoVR, it is not manual at all but at least, it let me create quite simply a virtual tour.

How does it work?

OpenPhotoVR is a web tool that will store your images and allow you to display the thumbnail in a  pseudo 3D view. 

The idea behind the tool is to create links behind photographs and allow to transition between them with a 3D transition. You can find many examples here: http://www.openphotovr.org.

A simple example

To see the example, you can go here: http://openphotovr.org/view.html#FKuk4SNR

Let me describe how I made the example:

First, I made a test with the tool by taking some photographs of my house: 

The photographs are bad but it was raining the day I made the test, so you will see my hall :-).

1. Now, that I have made the photographs, I can go on www.openphotovr.org and I will create the album. 
2. Click on "Create new album"
3. Click on "Upload new photo" and select the image you want to upload
   
   
4. Now that you have uploaded all your photos, you can start to add the links between them. You could as well perform the step between each upload. Basically, a link specifies where the "child" image is located compared to the "parent" image.
5. 4 points should appear, they allow you to place the child image compared to its parent. You can move as well the child image. When you are happy with the position, click on "Save".
   
   
   
6. Reiterate the steps 4-5 until you have added all the links

The tool is very easy to use. It took less than 15 minutes to learn it and use it.