"Many of these funds are putting time and resources into securing data sets" - this is key.
back
4 comments
Whenever I see announcements like this, it's very unclear to me what is meant by "AI." Are they talking about basically getting the most out of the current ML/deep learning type systems? If so then I guess building data sets makes sense but it seems more like an uninteresting business strategy than what I think of as pushing AI forward.
If, on the other hand, they are talking about making progress on the more traditional dream of AI, the focus on building these data sets seems to be a sad way to lock us into a local maximum for a long time.
Massive curated data sets are crutches that lead us into narrow-minded hyper-specialized systems. Are any of these funds investing in people working on systems that try to make sense of raw sensory data streams? I don't think we're ever going to move away from data sets by creating more and better data sets.
I am of the opinion that `true AI` is the science/engineering of understanding and replicating human intelligence. Why are we able to come up with abstract concepts from the surrounding physical environments? Why do we look at the stars and wonder what they are (and why)? How are we able to communicate with one another through pictures, words, writings, snapchat. Is that something special about our brains, our collective society, or something else, that enables such remarkable different behaviors from other any animal on earth? I don't know which direction we can start to go down to answer these questions, but collecting good data sets is probably as good as anything. Maybe we'll get the `quantity` of smarter specialized systems first, and once we get the `quantity`, maybe the `quality` will follow?
I agree. I think the fields of "computational cognitive science" and developmental psychology are the ones to look into to make progress towards the "hard fundamental problems". Some of the leading labs working on this are MIT CBMM (https://cbmm.mit.edu/, they have a nice youtube channel) and Berkeley Cocosci (https://cocosci.berkeley.edu/index.php).
Google Brain/DeepMind are also pushing some of those ideas. They must be, since they aggressively poach all the top researchers from those labs...
Ng approach is different: he wants a world powered by Deep Learning, so his goal is to make applied deep learning thrive. His strategy to do that: give those data-hungry models even more data, which is completely reasonable.
Those two approaches - fundamental research and applied deep learning - are often referred to as AI, causing much confusion.
Don't have anything relevant to say except to make a fun note that we can change your second sentence, keeping it correct, while increasing its ambiguity. If we restate it as:
Are they talking about basically getting the most out of the current ML type systems?
Since Deep learning is a subset of machine learning, the sentence retains correctness but there are now two equally valid interpretations.
Okay, I do have something relevant to say. I don't see what advantage there is in raw sensory data streams, models can be trained off-line to operate on sensor streams just fine. What we actually want are systems that learn adaptively and on-line. To do that well, they'd need to also be data efficient.
models can be trained off-line to operate on sensor streams just fine.
I mentioned this in another comment, but I don't know that we can train these models just fine, a human can get a lot more information out of an audiovisual stream than a rudimentary transcription of recognized speech, objects and/or text.
If we make these models more sophisticated to capture more information (e.g. body language, tone, context), we have to decide how that information is structured and communicated to the "higher level meaning interpretation" stage. No matter what, their output is going to be more rigidly structured and will contain less information than the raw stream. The extent to which the sensory processing model captures the human-recognizable information in the stream is the extent to which you have created an intelligent system.
There is some form of this structuring and reducing happening inside our brains, but we will never get machines to do that if we continue curating structured data sets. We, the researchers, are using our human intelligence to process raw sensory data and put it into a nice format for the AI. They need to be able to do that themselves.
> we have to decide how that information is structured and communicated to the "higher level meaning interpretation" stage.
Do we? I'm not convinced. I've joined different sensory inputs before and just glommed together the top of two nets, I didn't have to create any of my own representations.
> but we will never get machines to do that if we continue curating structured data sets
Curated datasets are important though. Want your machine to understand more about the world around it? Then you need high quality inputs in formats you can load in. These data also need to be licensed correctly.
> They need to be able to do that themselves.
We don't chuck kids out into the wilderness and expect them to come back as a useful member of society. We have a huge range of inputs specifically curated to help (from toys and shows to school curricula). Later on with specialisation we pay for extremely carefully selected data, presented in a specific order!
Creating high quality datasets is vital for really anything from niche specialist systems to large general ones.
What is the difference between a data set and a raw sensory data stream? More specifically, isn't a raw sensory data stream just a data set? I think you are getting hung up on semantics.
Or is it just the time correlation that interests you? Because some of these data sets are very likely to indeed be time correlated. Like a video/audio data set for example.
Yes in the most general sense, a "data set" could be anything, but I'm talking about the highly curated and labeled data sets that are used to train contemporary ML systems.
You can feed a million images that are labeled "cat" or "no cat" to one of these systems and it can achieve a human-level proficiency at identifying cats in images. But, it won't be able to do anything other than identify cats, it's far too narrow to be considered intelligent in any way.
If you can feed a series of timestamped photos to a system, basically an unlabeled arbitrary video stream, and you can demonstrate that it formed some notion of what a cat is, that would be very interesting indeed.
Have you ever seen a child learn how to speak? He also needs loads of "labelled" data to initially learn about concepts.
The data is not labelled in the same way. The child has still acquired their knowledge solely from sensory experience. How do they know to apply the spoken label "mom" to the recurring pattern in their visual data stream?
With modern data sets and labelling, so much of the problem domain is deeply hard-coded into the system. It doesn't have to learn what letters and words mean and how to identify them in a totally arbitrary* visual or audio stream. It just gets a relatively minute amount of structured data that it has an embedded understanding of what to do with.
Sure there are things like OCR and speech-to-text, but I don't think you could just run your streams through those, there's just so much subtle information loss. In order to make AI that really has a chance of reaching what humans would call intelligence, I firmly believe it has to make meaning out of some kind of raw sensory experience analogous to ours.
*Ok, not totally arbitrary, a human would not learn language from a video of a forest, and that's where the "labelling" comes in, from observing other people using language, but a child can still learn any language from just that, and the labels are often vague, inaccurate, contradictory, complex, abstract, etc. There's no master training set with the right answers, you have to decide for yourself. And humans were also capable of bootstrapping language from nothing. I just don't see modern supervised learning systems ever doing things like that.
I would disagree on this point, humans unlike current AI systems can learn from one or two data points, especially at easier tasks like identifying cats. Current AI algorithms need huge labeled data sets for solving narrow problems so one needs to build more generalization ability to our current AI systems.
Are you comparing a child learning to speak with a set of conditional statements?
I'm personally most interested in approaches to AI not based as heavily on dataset-collecting. There's a now almost standard method of: 1) first curate a nice, large dataset (often explicitly labeled by humans), then 2) carefully engineer a model architecture that has good performance on the problem represented by that dataset. Example: dataset is ImageNet, problem is tagging images. Obviously there's a lot of practical value in that, and it's a perfectly reasonable thing to research and use, but that's only one specific kind of inference.
this all seems quite naive of the literature in the field. what other "kind of inference" do you wish he would pursue?
Quite possible. I do have a recent PhD in AI, but it's a big field and I can't claim to know more than a small percentage of it.
The problem with the dataset-first approach is that humans are still providing a lage part of the intelligence: defining the domain, defining what good performance on it looks like, carefully designing model architectures, collecting and labeling large datasets, etc. This is fine for narrow task-specific problems, but is not really the end-all of AI, and does not even seem to work well on all well-defined tasks. As an example of another kind of inference, how about mathematical reasoning? I purposely pick one here that is seemingly very formal; should be possible for a computer to do it. Mathematicians are somehow able to invent conjectures and prove theorems without first being exposed to terabytes of labeled mathematical facts. The scientific method is kind of an even messier version of this. Or to take something laypeople do, people can usually learn games to at least a passable level from just a handful of playthroughs, not AlphaGo-style millions of plays (imagine if you had to play even 1,000 games of MT:G before you got the basic hang of it...).
All this kind of stuff is quite well-represented in the literature though, if you mean the scientific literature. Pop-press AI writing tends to cover a pretty specific subset of what's going on at AI conferences.
I don't think you're up to speed with the latest in ML approaches. You have to be narrow before you can be broad, drawing cross correlations between great narrow inference towards more generalized problem sets.
I would be grateful if you could point me toward some relevant resources.
This is one of the better examples I've seen recently that I really love:
> Are any of these funds investing in people working on systems that try to make sense of raw sensory data streams?
Yes. That's also my total focus right now. Happy to discuss, email on profile.
Have you considered writing a blog post? I'm sure it'd be very well received on HN.
The building blocks of AI products are not stable enough for the industry to be respected outside of specialized cases. Too many crap models, not enough simple tools ensuring that models are not crap. I'm guessing these funds would go toward producing those tools. Data quality is a huge issue, so curated data sets are vital, at least until the standard toolchain is strong enough to reliably work on noisy information.
Looks like Data sets are going to be the moat of AI companies.
I've been involved in this a couple of times and want to share a thought.
If you are just aggregating data, that's not a moat.
A lot of folks are buying data from a bunch of sources to give complete coverage - e.g. a dataset of all plane ticket prices, where before you could only get separate datasets from each of amadeus and rivals.
Someone, usually someone big, just goes round you to build their own infrastructure for some internal function, e.g. Google flights api, and then realises they can replace you as a secondary revenue stream.
Instead, I think you gotta somehow add value to your data, which is best done as a side effect of another buisness.
Reuters have a huge news dataset with amazing annotation because they got thier editors to curate it as it was produced. That's an unassailable free text training set that no one else is gonna match.
So build a dating app that causes users to create a curated dataset. Or a game. Or a buisness tool. Or an api. Or a really good AI secret sauce built in an expensive privately curated training set that ate most of your funds.
What kind of useful curated datasets do you imagine could be created by a game? Like driving data from a GTA-like game?
Some of the more interesting AI research is being done in the area of developing very accurate models of the real world. This way the many varied iterations needed to develop the AI model can be done without the same physical limitations of aggregating human data. Games are definitely in this same realm, although it's not always practical to make a human playable simulation-game and an AI playable simulation game given development constraints.
What I find fascinating is what AR might do for training AI models. To execute on AR we'll need to digitize a model of our physical surroundings so the software can interact. At that point we'll have a compelling pipeline of actionable data in regards to machine learning - especially for robotics.
Why do you think companies like Google release so many things open source related with AI (papers, models, frameworks)? Because they have the data to train them. In my company 90% of my time is dealing with the dataset, having a good and big dataset is first step to train any algorithm.
I'd never thought of this angle before, but maybe to establish prior art? They have several high level competitors all working on the same stuff. The algorithms aren't that useful without the dataset anyway, so they release the algorithms in order to block any future attempts to patent the tech?
They release the algorithms so other researches can improve them and they can use the new ones with their dataset.
They release the frameworks so people learn them and then it is easier and cheaper to find employees.
That is what I think.
Licensing quality, trusted data is very expensive right now, but as IoT, device tracking, and other data-producing technologies come online there will be a race to the bottom in cost of quality. A lot of older data providers who haven't kept up with the times will be put out of business. I doubt it will take that long; I've had a few ideas in the space and they're not all that complicated...
open/free data sources are likely to become very important. AI hasn't yet been super-important in the open data world, but I'd expect it to gain a lot of prominence as time goes by.
Starting a data set company would probably be a good idea. Necessarily has some humans labeling them, but you could probably build a lot of tools around it to make it as smooth as possible. Also, task rabbit and Amazon turk workers could be used.
Yep, open data and models with state-of-the-art performance are popping up more and more. I expect companies to appear which will sell data and models as a service, too.
Two additional points are (1) dataset collection is low variance relative to fundamental algorithmic advances, and (2) dataset collection relies less on having tip-top research talent (than fundamental algorithmic advances).
Definitely an echo of what I just commented (didn't see your comment until after I posted mine); data sets and the collection of them already are seen as very valuable, and I would imagine this would only become more true as time passes, if we don't hit another winter, that is...
It was the reason Blekko had value to IBM's Watson effort, the crawler was state of the art. With that and the 'web' you can create data sets that others can't. It is the not-so-secret advantage that both Google and Microsoft leverage (their search engine crawlers).
Could you describe at a high level(or whatever level you'd like,) what goes into, or more specifically, what are the components of a crawler that makes it superior for these purposes?
Sure;
Doesn't die -- HTML being a specification in name only, there are a lot of really crazy web pages out there that render on browsers but are pathological edge cases.
Does a good job of distinguishing 'good' links from 'bad' links on a page. -- Lots of pages have links that should not be followed, some are easy they are rendered in the same color as the background (SEO black hat link juice) and others refer to crawler traps.
Crawler traps come in many forms -- Rich Skrenta created a great example one where the page generated a random number and said "%d is an interesting number" here are two more interesting numbers "%d and %d" the each link went to a new URL that ended in the number. So if you tried to crawl that site exhaustively you would fill your entire crawler cache with random number pages.
Dynamic importance scaling -- you want to crawl the 'best' pages for a topic so you need to figure out a way to measure which pages are important and which aren't. This was the secret sauce of the PageRank patent Google had but it's been gamed to death by SEO types. So now you need better heuristics to understand which are the more important links to follow.
Effective crawl frontier management - for every billion pages you decide to crawl there are probably 20 to 50 billion pages you "know about". These URIs that are known but not yet crawled are referred to as the 'crawl frontier'. Picking where to go looking in the crawl frontier to find useful new pages is half art and half good machine learning.
Good algorithmic de-packing -- many many pages today are generated algorithmicly from a set of rules, whether it is the product pages on Amazon or posts in a PHP forum, if you can recognize the algorithm early, you can effectively avoid crawling pages that are duplicates or not useful.
Good page de-duping -- There is a lot of repetition on the web. Whether it is the 'how to sign up' page of every PHPBBB site ever or the same product with 10 different keywords in the URI.
Selective JS interpretation -- sometimes the page exists in the JS code, not in the HTML code, so unless you want to store 'this page needs Javascript enabled to run' into your crawler cache you need to recognize this situation and get the page out of the Javascript.
That's just off the top of my head.
Until one shot and transfer learning work.
All the one-shot approaches I've seen are IMO transfer learning in disguise, which raises the question of "what are you transferring from?". So while these should hopefully reduce the need for truly gigantic datasets about everything, there is still a limit of how much info you can extract from a fixed amount of data.
Large datasets will still be important for all but strictly perceptual tasks.
And the primary advantage of China's AI push.
Definitely - without the proper data and large amounts of it, things are pretty dead in the water for research, let alone building actual products.
More and more varied datasets are needed for this (but understandably they can be seen as valuable on their own, so reluctance to share is understandable - at least from a business perspective).
People say that but I don't see many ML companies being acquired or funded because of their data sets.
Big companies have valuable datasets, small companies have talent. It makes sense for big companies to acquire talent, not datasets.
Increasingly, small companies have really novel and valuable data sets.
like?
Like Comma.ai, Pair (my company), Travelflan handful of others
plus acquiring data is trivial compared to devising a good model
I think (hope) you missed a sarcasm tag there?
I spend 90% of my time manipulating data to try to build bigger, better datasets, and only 5% modelling.