I am attending (when schedule permits) because I believe that Machine Learning (ML) will (has) become a basic analysis technique of any complex system. However, I was surprised by a recent poll in KDnuggets that asked: “Can Machine Learning on Big Data replace Domain Expertise?” The majority (55%) felt that “there are many domains where machine learning cannot beat domain expertise”.  However, Gregory Piatetsky-Shapiro (newsletter editor) argued that there are growing number examples where ML of Big Data outperform domain expertise. Many of the Knowledge Discovery (KD) competitions over the past ten years confirmed this.

I believe that successful applications of ML will involve a synthesis of ML with human domain expertise. The ML component will provide hints and basis instrumentation. However, humans will provide judgment and insights based on their domain expertise. Could a naive domain expert use ML functionality to perform useful analyses? Of course. However, a savvy domain expert could leverage ML much more.

In 1972 using “32 buttons and a lightpen” on “an Information Display’s IDIIOM refresh CRT driven by a Varian 620/i minicomputer linked to an IBM 360/91″, Tukey developed the PRIM-9 program to do multivariate analysis. It handled up to 9 dimensional data with the functions of “picturing, rotation, isolation and masking”. A paper in May of 1974 describes the operation of his program. Particularly insightful is the Discussion section at the end, in which Tukey gives his best practices for discovering meaningful relationships hidden within the 9 dimensions of the data. Nathan suggests that the GGobi visualization software by Hadley Wickham et al owes a bit of its heritage to Tukey’s PRIM-9.

The real treat is a 25-minute video from 1973. Take the time for view this! Despite the awkwardness of ancient computer equipment, Tukey teases out the patterns in what initially appear as random dots.

Toward the end of this video, I was stuck with the implications of Tukey’s data visualization work to Immersive Intelligence. Here is one of pioneers of modern data analysis showing us the value of 3-D visualizations…as an early approach to immersing oneself in the data. And this was forty years ago!

It seems that the criteria for network health is loosely defined. Any anomaly to the normal pattern could be an area of concern.

Normal operation is fairly easy to identify according to the data definitions. A policy status of “1″ is “machine is functioning normally and is healthy”. Likewise, a activity flag of “1″ is “normal with only normal activity detected on the equipment”.

A new table ‘health’ was created by joining ‘meta’ with ‘windowOneSingle’ on ‘ipaddr’. This generated health records for 809,216 devices in 51 business units and 206 facilities. For policy = ’1′ and activity = ’1′, there were 646,127 (80%) devices that were in normal operation.

We probably should add normal business hours (7am to 6pm Monday-Friday) to the criteria for ‘normal operation’ …at least for workstation (and not for servers and ATMs). Workstations that should be turned off outside of business hours.

The ‘health’ table had records for eight timezones – BMT-4 to BMT-11. This data was a snapshot at BMT = 14:00 (2:00pm) on February 2, which is a Thursday. Hence, normal business hours is only for timezones BMT-4 to BMT-7, implying the BMT-8 to BMT-11 are before normal business hours. There were records for 84,696 (10%) workstations that were in operation before normal business hours. NOTE: criteria for whether a workstation is turned off is uncertain.

Another criteria for ‘normal operation’ could be whether a device is under maintenance and whether is was ‘planned on a regular schedule’.

Out of 809, 216 devices, there were only 743 devices under maintenance (activity = ’2′) across all timezones. This seems odd! Not much maintenance being performed on this network. Further, there seems to be no indication as to whether this maintenance was planned or not.

More analysis on this ‘health’ table will performed using several data viz tools, like Quikview and Tableau, and reported in future blogs.

It is the same size of Earth, but consist of a single large land mass, about the size of Europe and Asia, but situated over North American, the north part of South American and the Pacific Ocean out to the Hawaii Islands.

The best way to visual this geography is Google Earth, especially since the challenge designers gave a set of KML layers. So, bring up your copy of Google Earth, position it over Mexico City so that the whole globe is visible, and turn off all the layers in the Primary Database (lower left box).

Ready? Find the folder “Mini-Challenge 1 Image and Google Earth Files” that you downloaded. I found that it was best to drag-drop each KML file individually onto the Earth …transforming it into BankWorld! Do it in this order: BankWorld, BankCenters, Large Regional Offices, Region Boundaries, Small Region Offices, Large Branch Offices, Small Branch Offices, or something like that…

Play with the check boxes in the upper left. You should see something like the image on the right. Click for hi res version.Get familiar with the geography and the icons. The yellow mailbox is global headquarters for the bank. The yellow ‘hairs’ (actually pushpins if you zoom in) are each of the branch offices, each of which have lots of workstations, servers, ATMs, etc.

The important aspect is to note the timezones (not shown) that starts in London (does not exist in BankWorld) and proceeds west on 15 degree intervals of longitude. Hence, a place with longitude of -70.5 would be -4 hours from the BankWorld Mean Time (BMT). IOW Timezone =  integer (Longitude / 15) – 1. Now you can determine the local time at a branch office, thus monitoring whether transactions are being performed during normal business hours.

This is certainly a creative transformation of Earth’s geography for this challenge and a fun use of Google Earth (and other KML tools).

The challenge deals with “Big Data” although the total amount of data is less than 10 GB. The situation is cyber-security for a large bank with hundreds of branch offices spread across a fictitious world, completed with lat/long geographic coordinates and KML annotations.

There are two mini-challenges, only the first of which has been released. Mini-challenge #1 is to provide “situation awareness of the cyber-health of the bank’s network. In their words, “how do you visualize data out of a network containing nearly a million computers in a way that you can perceive its state and identify problems?” Actually, the bank network consists of 895,025 IP addresses, as shown in the table at the right. 

Mini-challenge #1 requires two responses:

• A – Create a visualization of the health and policy status of the entire bank enterprise as of 2 pm on February 2. What areas of concern do you observe?
• B – Use your visualization tools to look at how the network’s status changes over time. Highlight up to five potential anomalies in the network and provide a visualization of each. When did each anomaly begin and end? What might be an explanation of each anomaly?

I download the data sets, consisting to three tables:

• Meta data about the organization and location of network nodes (workstations, routers, servers) – 1.1 M rows as 63 MB CSV file
• Health status data about each nodes through time – 80 M rows as 7.8 GB CSV file
• Health status data for a time window – 0.9 M rows as 40 MB CSV file

Here is an Excel file (2 MB) containing the first 20K rows for each of these three tables, along with two documentation files containing an overview of the banking world and a table explanation.

Using WAMP, the data was loaded into MySQL for profiling. The status data took about an hour to load, without any performance tuning. Still loading…

Yesterday the load of the large health status table aborted with the error “multi-statement transaction required more than ‘max_binlog_cache_size’ bytes of storage; increase this mysql variable and try again”. After consulting with my MySQL technical wizard Roland Bouman, I disable the binary logging for replication in ‘my.ini’ configuration file and rerun the load. It took all most two hours, but complete after loading 133 M rows. The stats on the tables are shown at the right…

David Burden of Daden Limited suggested that we form a team to conduct a workshop at the Federal Consortium for Virtual Worlds in May. As part of that workshop, we have been searching for a problem upon which to focus. So, we are discussing whether the VAST Challenge would be an appropriate context for the workshop. Please join us by commenting below, or by attending FCVW.

As Nathan summarizes: People often miss this point about data — that it’s a representation of the physical world — and because of that, things like uncertainty and complexity come attached to the numbers. There are also actual human beings associated with a lot of data. So while optimization, maximization, and efficiency are well and good, stories, ethics, and lessons are pretty good takeaways, too.

Jer tells us a critical aspect about the future directions for Data Science…as the intersection of science, design, and art. Further, the extent that we connect humans – their stories, dreams, experiences – to the data will be the extend that we can understand the complexity of our world. What an ideal for Immersive Intelligence!

• Social and Information Network Analysis – how to analyze the structure and dynamics of large networks, how to model links, and how design algorithms that work with such large networks
• Machine Learning – Design and development of algorithms and techniques that allow computers to “learn” by extracting information from data automatically
• Mining Massive Data Sets – data mining of distributed file systems: Hadoop, map-reduce; PageRank, topic-sensitive PageRank, spam detection, hubs-and-authorities; similarity search; etc etc
• Information Retrieval and Web Search- efficient text indexing; Boolean and vector space retrieval models; evaluation and interface issues; Web search including crawling, etc etc

Combining this material with Immersive Intelligence would be awesome! Please contact me if you are enrolled in this program.

Note the many ways that 3D data is gradually emerging from research in many disciplines. I feel that our current visualization tools are not providing a smooth transition to 3D data analysis from the traditional 2D visualization approaches. Perhaps cross-disciplinary exchanges will provide the necessary catalyst.

The blog highlighted the team from MIT and Harvard who authored a research article in Science. An informative video (4:34) is a must-see! Note the short discussion on patterns detected around 2:00. Try analyzing those patterns with the typical statistical method!

The problem is scanning large amounts of data to find significant associations among the variables. They proposed a new correlation statistic - maximal information coefficient or MIC – that can find significant associations between two variables or attributes. Lots of statistical methods can find associations among variables and determine their strength, but all the methods make limiting assumption and tend to have narrow applicability. The MIC statistic appears to have fewer assumptions and broader applicability.

For example, the World Health Organization has collected 357 health related variables from over 200 countries for many years. On their website http://www.exploredata.net/ a download of the MIC program (in Java) is available, along with their sample data sets. I downloaded the MIC program and WHO data and run the pairwise analysis. It worked…in about 10 minutes of crunching…and generated a large CSV file showing over 64,000 associations, sorted on descending MIC strength. Impressive!

The potential is to quick map the variable association net for a large unexplored data set. An example from the article is shown in the sample at the right. A researcher can start the discovery process where the patterns are the richest.

UPDATE 12/30/2011: KD Nuggets caught extra commentaries to the Science article on MIC. First, the TechNewsWorld article by John Mello (12/21/2011) noted the ability of MIC to cope with noise. He also the MIC is part of a suite of data analysis tools called MINE for Maximal Information-based Nonparametric Exploration. Second, Science Daily reprinted an article from the Broad Institute of MIT and Harvard (“Tool detects patterns hidden in vast data sets.” ScienceDaily, 15 Dec. 2011. Web. 30 Dec. 2011). The article highlights that, if researchers print each potential relationship among bacteria in the human gut, the paper would reach to a height of 1.4 miles, as show in the figure on the left. Finally, here is a link to a YouTube video (4:46)  that is quite good in explaining how MIC can be used for exploratory analysis of large data sets.