I was talking to a Doctor today and we got to talking about what big data could mean for medicine. The answer is… it will change everything. Talking to him made me realize how under appreciated the long term impact of Hadoop and other big data technologies is in most disciplines. Right now we look at the patients through a key hole. You get a few measurements per patient per year when they go to the doctor and you enroll a few patients in a inquiry specific study and then watch them, often waiting years to test a single hypothesis. Then we lock this data up. Often literally, in filing cabinets where no one else can learn from it.
Let’s think about the kind of technology that is widely deployed today: Cell phones, networked fitness monitors, low cost DNA sequencing, digital health records, social data, search engine & machine learning. What could you build with this stuff? What about a continuous longitudinal study of everyone, all the time?
Today we could build a system that collected location, heart rate, activity, social and professional activity, diet (via phone camera), all medical records, etc from most everyone in a country, cradle to grave. Via medical records and social data and other sources we know a huge amount about everyone and we can or will soon be able to DNA sequence everyone. Put all of this in a searchable repository, keep if forever and then use machine learning techniques to infer many other facts about users (activity, life style, consumption patterns…).
If you had such a system, learning and hypothesis testing in medicine would accelerate tremendously! Some examples: You could do quality of care of outcome metrics on all procedures over time and ID best practices; You could quickly look for interesting correlations between infected patents to ID causes of medical conditions or at least correlations; You could datamine all reported illnesses against all unusual gene sequences; You could correlate activity, demographics, diet with outcomes and ID best practices; Etc, Etc, Etc. You would see an amazing acceleration of learning, quality of care, patent specific therapies, …
Unfortunately, it will probably take us 10-20 years for us to sort out policy & privacy issues, set standards, learn and deploy. Then we need to collect data… and change everything.
So yes, I think this “big data stuff” is going to have an impact on healthcare.