I am attending (when schedule permits) because I believe that Machine Learning (ML) will (has) become a basic analysis technique of any complex system. However, I was surprised by a recent poll in KDnuggets that asked: “Can Machine Learning on Big Data replace Domain Expertise?” The majority (55%) felt that “there are many domains where machine learning cannot beat domain expertise”.  However, Gregory Piatetsky-Shapiro (newsletter editor) argued that there are growing number examples where ML of Big Data outperform domain expertise. Many of the Knowledge Discovery (KD) competitions over the past ten years confirmed this.

I believe that successful applications of ML will involve a synthesis of ML with human domain expertise. The ML component will provide hints and basis instrumentation. However, humans will provide judgment and insights based on their domain expertise. Could a naive domain expert use ML functionality to perform useful analyses? Of course. However, a savvy domain expert could leverage ML much more.

In 1972 using “32 buttons and a lightpen” on “an Information Display’s IDIIOM refresh CRT driven by a Varian 620/i minicomputer linked to an IBM 360/91″, Tukey developed the PRIM-9 program to do multivariate analysis. It handled up to 9 dimensional data with the functions of “picturing, rotation, isolation and masking”. A paper in May of 1974 describes the operation of his program. Particularly insightful is the Discussion section at the end, in which Tukey gives his best practices for discovering meaningful relationships hidden within the 9 dimensions of the data. Nathan suggests that the GGobi visualization software by Hadley Wickham et al owes a bit of its heritage to Tukey’s PRIM-9.

The real treat is a 25-minute video from 1973. Take the time for view this! Despite the awkwardness of ancient computer equipment, Tukey teases out the patterns in what initially appear as random dots.

Toward the end of this video, I was stuck with the implications of Tukey’s data visualization work to Immersive Intelligence. Here is one of pioneers of modern data analysis showing us the value of 3-D visualizations…as an early approach to immersing oneself in the data. And this was forty years ago!

It seems that the criteria for network health is loosely defined. Any anomaly to the normal pattern could be an area of concern.

Normal operation is fairly easy to identify according to the data definitions. A policy status of “1″ is “machine is functioning normally and is healthy”. Likewise, a activity flag of “1″ is “normal with only normal activity detected on the equipment”.

A new table ‘health’ was created by joining ‘meta’ with ‘windowOneSingle’ on ‘ipaddr’. This generated health records for 809,216 devices in 51 business units and 206 facilities. For policy = ’1′ and activity = ’1′, there were 646,127 (80%) devices that were in normal operation.

We probably should add normal business hours (7am to 6pm Monday-Friday) to the criteria for ‘normal operation’ …at least for workstation (and not for servers and ATMs). Workstations that should be turned off outside of business hours.

The ‘health’ table had records for eight timezones – BMT-4 to BMT-11. This data was a snapshot at BMT = 14:00 (2:00pm) on February 2, which is a Thursday. Hence, normal business hours is only for timezones BMT-4 to BMT-7, implying the BMT-8 to BMT-11 are before normal business hours. There were records for 84,696 (10%) workstations that were in operation before normal business hours. NOTE: criteria for whether a workstation is turned off is uncertain.

Another criteria for ‘normal operation’ could be whether a device is under maintenance and whether is was ‘planned on a regular schedule’.

Out of 809, 216 devices, there were only 743 devices under maintenance (activity = ’2′) across all timezones. This seems odd! Not much maintenance being performed on this network. Further, there seems to be no indication as to whether this maintenance was planned or not.

More analysis on this ‘health’ table will performed using several data viz tools, like Quikview and Tableau, and reported in future blogs.

It is the same size of Earth, but consist of a single large land mass, about the size of Europe and Asia, but situated over North American, the north part of South American and the Pacific Ocean out to the Hawaii Islands.

The best way to visual this geography is Google Earth, especially since the challenge designers gave a set of KML layers. So, bring up your copy of Google Earth, position it over Mexico City so that the whole globe is visible, and turn off all the layers in the Primary Database (lower left box).

Ready? Find the folder “Mini-Challenge 1 Image and Google Earth Files” that you downloaded. I found that it was best to drag-drop each KML file individually onto the Earth …transforming it into BankWorld! Do it in this order: BankWorld, BankCenters, Large Regional Offices, Region Boundaries, Small Region Offices, Large Branch Offices, Small Branch Offices, or something like that…

Play with the check boxes in the upper left. You should see something like the image on the right. Click for hi res version.Get familiar with the geography and the icons. The yellow mailbox is global headquarters for the bank. The yellow ‘hairs’ (actually pushpins if you zoom in) are each of the branch offices, each of which have lots of workstations, servers, ATMs, etc.

The important aspect is to note the timezones (not shown) that starts in London (does not exist in BankWorld) and proceeds west on 15 degree intervals of longitude. Hence, a place with longitude of -70.5 would be -4 hours from the BankWorld Mean Time (BMT). IOW Timezone =  integer (Longitude / 15) – 1. Now you can determine the local time at a branch office, thus monitoring whether transactions are being performed during normal business hours.

This is certainly a creative transformation of Earth’s geography for this challenge and a fun use of Google Earth (and other KML tools).

The challenge deals with “Big Data” although the total amount of data is less than 10 GB. The situation is cyber-security for a large bank with hundreds of branch offices spread across a fictitious world, completed with lat/long geographic coordinates and KML annotations.

There are two mini-challenges, only the first of which has been released. Mini-challenge #1 is to provide “situation awareness of the cyber-health of the bank’s network. In their words, “how do you visualize data out of a network containing nearly a million computers in a way that you can perceive its state and identify problems?” Actually, the bank network consists of 895,025 IP addresses, as shown in the table at the right. 

Mini-challenge #1 requires two responses:

• A – Create a visualization of the health and policy status of the entire bank enterprise as of 2 pm on February 2. What areas of concern do you observe?
• B – Use your visualization tools to look at how the network’s status changes over time. Highlight up to five potential anomalies in the network and provide a visualization of each. When did each anomaly begin and end? What might be an explanation of each anomaly?

I download the data sets, consisting to three tables:

• Meta data about the organization and location of network nodes (workstations, routers, servers) – 1.1 M rows as 63 MB CSV file
• Health status data about each nodes through time – 80 M rows as 7.8 GB CSV file
• Health status data for a time window – 0.9 M rows as 40 MB CSV file

Here is an Excel file (2 MB) containing the first 20K rows for each of these three tables, along with two documentation files containing an overview of the banking world and a table explanation.

Using WAMP, the data was loaded into MySQL for profiling. The status data took about an hour to load, without any performance tuning. Still loading…

Yesterday the load of the large health status table aborted with the error “multi-statement transaction required more than ‘max_binlog_cache_size’ bytes of storage; increase this mysql variable and try again”. After consulting with my MySQL technical wizard Roland Bouman, I disable the binary logging for replication in ‘my.ini’ configuration file and rerun the load. It took all most two hours, but complete after loading 133 M rows. The stats on the tables are shown at the right…

David Burden of Daden Limited suggested that we form a team to conduct a workshop at the Federal Consortium for Virtual Worlds in May. As part of that workshop, we have been searching for a problem upon which to focus. So, we are discussing whether the VAST Challenge would be an appropriate context for the workshop. Please join us by commenting below, or by attending FCVW.

• Social and Information Network Analysis – how to analyze the structure and dynamics of large networks, how to model links, and how design algorithms that work with such large networks
• Machine Learning – Design and development of algorithms and techniques that allow computers to “learn” by extracting information from data automatically
• Mining Massive Data Sets – data mining of distributed file systems: Hadoop, map-reduce; PageRank, topic-sensitive PageRank, spam detection, hubs-and-authorities; similarity search; etc etc
• Information Retrieval and Web Search- efficient text indexing; Boolean and vector space retrieval models; evaluation and interface issues; Web search including crawling, etc etc

Combining this material with Immersive Intelligence would be awesome! Please contact me if you are enrolled in this program.

Note the many ways that 3D data is gradually emerging from research in many disciplines. I feel that our current visualization tools are not providing a smooth transition to 3D data analysis from the traditional 2D visualization approaches. Perhaps cross-disciplinary exchanges will provide the necessary catalyst.

Here is a list of possible choices:

• Plain terrain: A vast featureless plain with flat Euclidean geometry and downward gravity. Avatars can walk freely on the ground in the X and Z dimensions. There is a definite Y dimension (for the vertical “UP”). The ground level (Y=0) displays the base (or atomic) data, while the upward vertical contains various analytics derived from the base data, such as descriptive statistics, groupings, and clustering. [What would negative Y would imply? Some kind of “drill-down”.]
• Free space: A vast featureless space in all three dimensions, with flat geometry but no gravity. Avatars float with rotation, so that there is no definite vertical. Useful for complex 3D structures (like molecules, star clusters) where the camera perspective is critical for viewing specific behavior.
• Flowing Time: The analogy is like a river of time. Avatars flow along with time, which is attached to one of the three dimensions in either Plain Terrain or Free Space.
• Stretchy Dimension: Similar to Open Terrain, except one or more of XYZ dimensions are not linear scales. For instance, the X dimension could be logarithmic, so that you could compare the very small with the very large. [How could ordinal and interval scales (as opposed to the usual ratio scale) be rendered differently?]
• Closed Elliptical: This canvas is like creating info-objects on a sphere, although it could be any enclosed volume whose slices form ellipses.  The sum of angles of any triangle drawn on its surface is greater than 180°. An example of this canvas is the primary perspective used in the TeraViz project by FragileEarthStudios.
• Open Hyperbolic: In contrast to Closed Elliptical canvas, the Open Hyperbolic canvas is like a saddle between two mountain peaks, where the sum of angles of any triangle drawn on its surface is less than 180°. Avatars walking in any direction would continue forever, not returning to a previous place.
• Spacetime: This canvas is an imitation of Einstein’s special relativity that combines space with time. One approach is to render info-objects that automatically flow with time (like the Flowing Time), like an animated weather maps but in 3D. Another approach is to momentarily assigned one XYZ dimension to time so that info-objects are positioned according to time, but with the ability to quickly interchange time with another XYZ dimension. And yet another approach would be to concurrently display the four possible configurations (XYZ, XYT, XTZ, TYZ) in close proximity for easy comparison.
• Higher Dimension: This canvas is a challenge to design! The situation is a data set whose entities have many (thousands!) of attributes. Through dimension-reducing analytics (like cluster analysis), many attributes would be collapsed into a single scene dimension. The usefulness may not be the final result, but the intermediate processing showing the behavior of clustering similar entities.

To provide orientation experience for users of the virtual world, an initial scene should have a Viewing Platform where avatars initially “rez” and can easily return to as their home. For example, the NASA Victoria Crater rendition in SecondLife had a platform structure with information boards. The Viewing Platform could also contain Viewing Vehicles where one or more avatars could tour various highlights of the world.

This blog is obvious a discussion piece. So, please comment and share your ideas.

Check out this video by Bret Victor. Simple and thoughtful! Note how the multi-touch user interface (of the iPad) enhances the interaction with a complex set of differential equations.I like the intuitive way that he manipulates the equations to learn the behavior of the equations.

Also check out the other work that Bret has done!