Tuesday, 24 March 2015

The problem with big data and machine learning


The advent of ‘big data’, coupled with fancy statistical machine learning techniques, is increasingly seducing people to believe that new insights and better predictions can be achieved in a wide range of important applications, without relying on the input of domain experts. The applications range from learning how to retain customers through to learning what makes people susceptible to particular diseases. I have written before about the dangers of this kind of 'learning' from data alone (no matter how 'big' the data is).

Contrary to the narrative being sold by the big data community, if you want accurate predictions and improved, decision-making then, invariably, you need to incorporate human knowledge and judgment. This enables you to build rational causal models based on 'smart' data. The main objections to using human knowledge - that it is subjective and difficult to acquire - are, of course,  key drivers of the big data movement. But this movement underestimates the typically very high costs of collecting, managing and analysing big data. So, the sub-optimal outputs you get from pure machine learning do not even come cheap.

To clarify the dangers of relying on big data and machine learning, and to show how smart data and causal modelling (using Bayesian networks) gives you better results, I have collected together the following short stories and examples:
The whole subject of 'smart data' rather than 'big data' is also the focus of the research project BAYES-KNOWLEDGE.

44 comments:

  1. The advent of big data coupled with fancy statistic machine learning techniques is increasingly seducing people to believe that new insights very high costs of collecting managing and analysing big data the sub-optimal outputs you get from pure machine learning.Its nice article in related giving more information post!.

    Regards
    safety training in Chennai
    safety training institute in Chennai
    safety institute Chennai
    fire and safety courses in chennai
    fire and safety course in chennai

    ReplyDelete
  2. The advent of big data coupled with fancy statistic machine learning techniques is increasingly seducing people to believe that new insights very high costs of collecting managing and analysing big data the sub-optimal outputs you get from pure machine learning.Its nice article in related giving more information post!.

    Regards
    safety training in Chennai
    safety training institute in Chennai
    safety institute Chennai
    fire and safety courses in chennai
    fire and safety course in chennai

    ReplyDelete
  3. Wonderful blog & good post.Its really helpful for me, awaiting for more new post. Keep Blogging!

    fire safety course

    ReplyDelete
  4. Hello admin, your technical information related to big data is very informative. I have gathered some new information about big data through your blog. Its really useful for me. Keep blogging. Thanks.
    Big Data Training in Chennai

    ReplyDelete
  5. Introduction to Nebosh International General Certificate
    The Nebosh International General Certificate in Occupational Health and Safety is best suited for managers, supervisors and workers in an organisation for making day – to- day decisions at workplace that need a broad understanding of health and safety issues and be able to manage risks effectively. Over 45,000 people having achieved this qualification since it was introduced in 2004.
    The NEBOSH International General Certificate is also suitable for those embarking on a career in health and safety, providing a valuable foundation for further professional study (such as the NEBOSH International Diploma in Occupational Health and Safety).

    nebosh igc course in chennai
    nebosh igc exam techniques

    nebosh course in Chennai
    nebosh courses in Chennai
    nebosh
    nebosh in Chennai
    nebosh hsw course in Chennai
    IOSH course in Chennai
    HABC course in Chennai
    IASP (OSHA Standard)course in Chennai
    OHSAS 18001:2007 Lead Auditor course in Chennai
    safety course in Chennai
    fire safety training in Chennai

    ReplyDelete
  6. Its great post this is machine level of the safety training in enmeshment on do working on this construction safety is first!!
    best safety training institute in Chennai
    nebosh course in Chennai
    Nebosh in Chennai
    Nebosh safety course in Chennai

    ReplyDelete
  7. Contrary to the narrative being sold by the big data community, if you want accurate predictions and improved, decision-making then, invariably, you need to incorporate human knowledge and judgment. This enables you to build rational causal models based on 'smart' data.

    health and safety courses

    ReplyDelete
  8. Very Nice Blog I like the way you explained these things.
    Indias Fastest Local Search Engine
    CALL360
    Indias Leading Local Business Directory

    ReplyDelete
  9. GREEN WOMEN HOSTELGreen Women hostel is one of the leading Ladies hostel in Adyar and we serving an excellent service to Staying people, We create a home atmosphere, it is the best place for Working WomenOur hostel Surrounded around bus depot, hospital, atm, bank, medical Shop & 24 hours Security Facility



    ReplyDelete
  10. Just found your post by searching on the Google, I am Impressed and Learned Lot of new thing from your post. I am new to blogging and always try to learn new skill as I believe that blogging is the full time job for learning new things day by day.
    "Emergers Technologies"

    ReplyDelete
  11. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much.
    failure to yield right of way course

    ReplyDelete
  12. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharng this information,this is useful to me...
    Android training in chennai
    Ios training in chennai

    ReplyDelete


  13. Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...Really it was an awesome article...very interesting to read..please sharing like this information......
    Web Design Development Company
    Mobile App Development Company

    ReplyDelete
  14. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Web Design Company
    Web Development Company

    ReplyDelete