Note the many ways that 3D data is gradually emerging from research in many disciplines. I feel that our current visualization tools are not providing a smooth transition to 3D data analysis from the traditional 2D visualization approaches. Perhaps cross-disciplinary exchanges will provide the necessary catalyst.

The blog highlighted the team from MIT and Harvard who authored a research article in Science. An informative video (4:34) is a must-see! Note the short discussion on patterns detected around 2:00. Try analyzing those patterns with the typical statistical method!

The problem is scanning large amounts of data to find significant associations among the variables. They proposed a new correlation statistic - maximal information coefficient or MIC – that can find significant associations between two variables or attributes. Lots of statistical methods can find associations among variables and determine their strength, but all the methods make limiting assumption and tend to have narrow applicability. The MIC statistic appears to have fewer assumptions and broader applicability.

For example, the World Health Organization has collected 357 health related variables from over 200 countries for many years. On their website http://www.exploredata.net/ a download of the MIC program (in Java) is available, along with their sample data sets. I downloaded the MIC program and WHO data and run the pairwise analysis. It worked…in about 10 minutes of crunching…and generated a large CSV file showing over 64,000 associations, sorted on descending MIC strength. Impressive!

The potential is to quick map the variable association net for a large unexplored data set. An example from the article is shown in the sample at the right. A researcher can start the discovery process where the patterns are the richest.

UPDATE 12/30/2011: KD Nuggets caught extra commentaries to the Science article on MIC. First, the TechNewsWorld article by John Mello (12/21/2011) noted the ability of MIC to cope with noise. He also the MIC is part of a suite of data analysis tools called MINE for Maximal Information-based Nonparametric Exploration. Second, Science Daily reprinted an article from the Broad Institute of MIT and Harvard (“Tool detects patterns hidden in vast data sets.” ScienceDaily, 15 Dec. 2011. Web. 30 Dec. 2011). The article highlights that, if researchers print each potential relationship among bacteria in the human gut, the paper would reach to a height of 1.4 miles, as show in the figure on the left. Finally, here is a link to a YouTube video (4:46)  that is quite good in explaining how MIC can be used for exploratory analysis of large data sets.

Here is a list of possible choices:

• Plain terrain: A vast featureless plain with flat Euclidean geometry and downward gravity. Avatars can walk freely on the ground in the X and Z dimensions. There is a definite Y dimension (for the vertical “UP”). The ground level (Y=0) displays the base (or atomic) data, while the upward vertical contains various analytics derived from the base data, such as descriptive statistics, groupings, and clustering. [What would negative Y would imply? Some kind of “drill-down”.]
• Free space: A vast featureless space in all three dimensions, with flat geometry but no gravity. Avatars float with rotation, so that there is no definite vertical. Useful for complex 3D structures (like molecules, star clusters) where the camera perspective is critical for viewing specific behavior.
• Flowing Time: The analogy is like a river of time. Avatars flow along with time, which is attached to one of the three dimensions in either Plain Terrain or Free Space.
• Stretchy Dimension: Similar to Open Terrain, except one or more of XYZ dimensions are not linear scales. For instance, the X dimension could be logarithmic, so that you could compare the very small with the very large. [How could ordinal and interval scales (as opposed to the usual ratio scale) be rendered differently?]
• Closed Elliptical: This canvas is like creating info-objects on a sphere, although it could be any enclosed volume whose slices form ellipses.  The sum of angles of any triangle drawn on its surface is greater than 180°. An example of this canvas is the primary perspective used in the TeraViz project by FragileEarthStudios.
• Open Hyperbolic: In contrast to Closed Elliptical canvas, the Open Hyperbolic canvas is like a saddle between two mountain peaks, where the sum of angles of any triangle drawn on its surface is less than 180°. Avatars walking in any direction would continue forever, not returning to a previous place.
• Spacetime: This canvas is an imitation of Einstein’s special relativity that combines space with time. One approach is to render info-objects that automatically flow with time (like the Flowing Time), like an animated weather maps but in 3D. Another approach is to momentarily assigned one XYZ dimension to time so that info-objects are positioned according to time, but with the ability to quickly interchange time with another XYZ dimension. And yet another approach would be to concurrently display the four possible configurations (XYZ, XYT, XTZ, TYZ) in close proximity for easy comparison.
• Higher Dimension: This canvas is a challenge to design! The situation is a data set whose entities have many (thousands!) of attributes. Through dimension-reducing analytics (like cluster analysis), many attributes would be collapsed into a single scene dimension. The usefulness may not be the final result, but the intermediate processing showing the behavior of clustering similar entities.

To provide orientation experience for users of the virtual world, an initial scene should have a Viewing Platform where avatars initially “rez” and can easily return to as their home. For example, the NASA Victoria Crater rendition in SecondLife had a platform structure with information boards. The Viewing Platform could also contain Viewing Vehicles where one or more avatars could tour various highlights of the world.

This blog is obvious a discussion piece. So, please comment and share your ideas.

Check out this video by Bret Victor. Simple and thoughtful! Note how the multi-touch user interface (of the iPad) enhances the interaction with a complex set of differential equations.I like the intuitive way that he manipulates the equations to learn the behavior of the equations.

Also check out the other work that Bret has done!

• The Jibe discussions on Google Groups has some great links. Check out these examples of virtual worlds created by students to invoke specific emotions. The multi-user worlds were done using Jibe from the Reaction Grid on top of Unity3D, while the single-user examples were plain Unity3D.
• Check out Autodesk 123D. Totally unexpected offering from Autodesk — well designed and free! Focus is DIY 3D solid modeling whose output creates the physical object. The odd duck is 123D Scrulpt, which is an iPad app but has no way to get the results into a normal 3D environment (please prove me wrong on this point!).
• Julien’s class in Gaming
• eBooks: pro and con, good eReading devices.
• Through Ander’s Grondsedt Group, Andrew Boyarsky of Virtual World Workshop is offering a six-hour workshop on the “Strategic Design of VW Learning Programs”.
• Federal Virtual World Challenge is due December 8
• Ander’s diversity training for Microsoft using augmented reality. They were told “more Disney and less Harvard” when directed to revise their training program.
• Microsoft World Wide Telescope, which gave rise to SciDB
• Ander: tradeoffs in doing virtual worlds with Web.Alive versus Unity3D
• Whatever happened to those amazing claims for point-cloud rendering of high-resolution graphics? Oh, that was Bruce Robert, CEO of Euclideon. Remember that YouTube video that got 2,838,827 views? Has anything come from this? Their blog states that they received $2 million in funding from the Australian Government in May. Hmmmmm

1) Anders is using Avaya web.alive (http://www.avaya.com/usa/product/avaya-web-alive) for some of his weekly Train For Success and for private meetings.

2) Anders has a great line-up for the weekly Train For Success talks at 10:00am on Thursday. See http://www.facebook.com/TrainForSuccess for details

3) The serious game Fold-It (http://fold.it/) is making waves! We are going to schedule a Fold-It geeks-only evening at Dick’s house…

4) See the Rock Creek Park (created for the Dept of Energy) from the Portfolio tab of http://fragileearthstudios.com/

5) Eric just got back from a special gather-together of federal employees on “Games for America“. Lots of discussion… P.S. see http://www.gamesforamerica.org/

6) Eric will be spending a week in Seattle for the SuperComputing 2011 (http://sc11.supercomputing.org/).
See the list of 349 exhibitors! (http://iebms.heiexpo.com/iebms/oep/oep_p1_exhibitors.aspx?oc=34&ct=OEP&eventid=5011) Who is in booth 903?

Fascinating! Zoom and pan! Enjoy! What insights did you discover?

Note the contrast of GigaPan with Munroe’s giga-money chart. For instance, you could add a time elapse feature to visualize changes over the decades. Did you know that 60% of the total wealth generated by mankind over the centuries was created since 1980? Now find that fact on giga-money!

The idea behind GigaPan is simple. Take a lot of  high-resolution images over time and post to a website. However, the engineering and interface is amazing! Think 2D version of Google Earth, zooming across levels of detail, with a flexible time scale, plus the ability to create “tours” across any segment of space and time. …YES! You gotta play with this!

The main page at GigaPan shows more than a dozen amazing examples. Note you need to use Chrome or Safari browsers because of advanced HTML5 features. Firefox will be enabled in December 2011.

• Solar Dynamics Observatory: shows filament eruptions and coronal loops of recent solar flares. Images were taken over a 24-hour period in December 2010, at 24 seconds apart, which is a 600x speedup. Makes you appreciate just how unearthly the surface of the sun really is!
• Plant Growth: 32-day time elapse images at 15-minute intervals. Normally this would be as exciting as “watching grass grow”; however, the plants take on this weird dancing behavior with which we are totally unaware. Note the devastation wroth by the caterpillars!
• Early Universe: a massive cosmology simulation showing the formation of galaxies and black holes by the clumping of filaments of dark matter. Since anyone can see a galaxy forming, instead watch the time warp of intergalactic voids.
• Visible Human Female: We all have seen images of the sectioning of human cadavers, but take one of tours that travels through the human body! Note that the time dimension is NOT time, but the vertical dimension of the human body. So, this is a 2D representation of a 3D object. Think of other possibilities…
• NOAA Weather Radar: Remember when the southeast got hammered by tornadoes over a one-week period in April of 2011? Well, relive that tragic weather pattern by playing with GigaPan.

So what? This is a good example of tools for exploring a huge amount of visual data, simply and easily. Further, it is done with a social capability to share insights into this data. The About page also contains a longer video that lasts about an hour. Toward the end (at 45:00) is a discussion of future directions for the GigaPan research. Very insightful into the potential of this approach for science discovery and education. Further, there is a potential for 3D enhancement using the TeraViz approach from Fragile Earth Studios.

One of the Strata talks that caught my attention was Big Data and Big Analytics: SciDB is not Hadoop by Paul Brown, Chief Architect for Paradigm4 and SciDB. Brown’s slides are here.

What is SciDB? It is database management based on an array data model. Array operations are fundamental to much of mathematics, providing the basic for linear algebra and statistical computations. Considerable effort was focused on efficient storage and data manipulation for petabyte data sets. By building upon MapReduce’s open-source architecture, SciDB also has a massively parallel scalable architecture.

Array Storage from Brown's Strata Talk

SciDB evolved from discussions in 2007-2009, resulting in a document Requirements for Science Data Bases and SciDB, which was authored by:

• Michael Stonebraker, MIT
• Jacek Becla, SLAC
• David Dewitt, Microsoft
• Kian-Tat Lim, SLAC
• David Maier, Portland State University
• Oliver Ratzesberger, eBay, Inc.
• Stan Zdonik, Brown University

The objective was to merge the requirements for large scientific database management from: particle physics (the LHC project at CERN, the BaBar project at SLAC and Fermilab), biology and remote sensing applications (Pacific Northwest National Laboratory), remote sensing (University of California at Santa Barbara), astronomy (Large Synoptic Survey Telescope), oceanography (Oregon Health & Science University and the Monterey Bay Aquarium Research Institute), and eBay.

Paradigm4 is one of Stonebraker’s companies. Their focus is: “sponsoring & distributing enterprise releases of the open-source SciDB analytics software platform, along with Paradigm4 add-on components”. The SciDB extensions from Paradigm4 include data versioning, etc, as shown below. Click for high res.

Paradigm4 Extensions to SciDB

SciDB is yet another new database approach to watch over the coming years! I wonder when we will see the array data model as an extension to an SQL engine?

Gartner estimates that 70% of large corporations will use workplace gamification by 2014, while M2 Research predicts revenues of $938 million for gamification software and services by 2014.

Gamification vendors, such as Bunchball, Badgeville, and Seriousity offer services to gamify business processes. The article notes IBM, Deloitte Touche Tohmatsu, SAP AG, and LiveOps as examples of corporations using gamification.

Comment on ways that you have seen gamification used in corporations…or in large science organizations.