Data Matters

In this post , the test results show that Tokyo Cabinet hashtable has a remarkable performance. It is pretty convenient to use it as a Key/Value store for data analysis. pytc is a python interface of Tokyo Cabinet. pytc doesn't provide a document, but we find some examples here .

Aardvark, which powers vark.com, published an interesting paper entitled " Anatomy of a Large-Scale Social Search Engine " at WWW 2010. The paper talks about the social search engine applied in vark.com. The search engine is based on social graphs and topics instead of keyword. The paper addresses search engine like Google as library paradigm and Aardvark as village paradigm. The search engine of village paradigm gets answers by asking the one who are expert in the underlying topic in social graphs. In library paradigm, the search engine needs to figure out what a user what based on keywords, search history and user profile, which considers to be a very difficult task. The village paradigm leaves the difficult part to human being, so, only problem is to find the right person. The model of Aardvark considers that a user u1 asks a question q, and the search engine should find the right user u2 to provide the answer. Aardvark associates both users and questions to topics. Aa...

Social Network Mining. Jennifer Neville, Purdue University, Foster Provost, New York University, Stern School of Business. AAAI 2008 Tutorial, Chicago, IL, USA webpage    slides    additional resources Modeling Social and Information Networks: Opportunities for ML. Jure Leskovec. ICML 2009.  webpage    slides    video

I found a very intering poster in WWW 2009 with a very cool title "The Web of Nations". The previous study found that the web at page level is looked like a bow-tie with regards to SCC(Strongly connected compoent), IN(link to SCC) and OUT(linked from SCC). But are every nations just like the world, are nations like USA affecting the structure of the web of the world? The authors of this 2 pages paper crawl billions of web pages of 200 nations to see what exactly the web of every single nations look like. Some interesting results comes out that nations like India and Vietnam have higher inlinks/outlinks rate than other countries, the inlinks and outlinks in those nations are more than other nations, but those nations have the lowest number of other nations related with their inlinks/outlinks.   I refer the paper as below: Chung, Sukwon and Shiowattana, Dungjit and Dmitriev, Pavel and Chan, Su (2009) The Web of Nations. In: 18th International World Wide Web Confere...

The prefixspan is a key algorithm for mining sequential patterns. I have implemented the algorithm in Python. The algorithm is based on the following paper: Jian Pei, Jiawei Han, Senior Member, Behzad Mortazavi-asl, Jianyong Wang, Helen Pinto, Qiming Chen, Umeshwar Dayal. Mining Sequential Patterns by Pattern-Growth: The PrefixSpan Approach. IEEE Transactions on Knowledge and Data Engineering, 2004. or their conference paper You may download the source code at the following addresses: Link1

R-mat is a method to generate graphs which follow power law distribution. A simple python implementation of R-mat to generate directed graphs could be downloaded here  .

Infer.NET is a .NET framework for machine learning. It provides state-of-the-art message-passing algorithms and statistical routines for performing Bayesian inference. It has applications in a wide variety of domains including information retrieval, bioinformatics, epidemiology, vision, and many others. Please refer to  http://research.microsoft.com/en-us/um/cambridge/projects/infernet/