Hmm, given that a few hundred line python script fed with large stream of characters can start producing sentences with correct grammar (however mostly nonsensical), I agree that we, given the complexity of our neural networks, don't need any pre-existing machinery to learn languages.
back
1 comments
But in order to do that, the "model" (i.e. the python program itself) needs to be carefully selected. Most python programs - or other programs - won't do this. Indeed, you need to construct the program to pay special attention to particular kinds of statistics, and the choice of those structures would correspond to the hypothetical "universal grammar" - at least, as far as I understand without being overly encumbered by any expertise on the matter ;-).
I wrote the parent comment on the go so here's more details. I'm specifically referring to the python script [1] mentioned in the article titled "The Unreasonable Effectiveness of Recurrent Neural Networks" [2]. I don't know much about ANNs but after a quick glance at the code, it seems to be setting up a generic three layer RNN. Based on this, I would argue that ability to learn patterns in sequence of data is all that's needed to learn a language. Calling it a "grammar" ties it too much to concept of language since such learning ability extends beyond human languages.
[1]: https://gist.github.com/karpathy/d4dee566867f8291f086
[2]: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Chomsky based his original argument that language is an innate ability of the human species on a famous mathematical result by Mark E. Gold, from a paper titled "Language identification in the limit" published in 1967 [1]
Gold's result basically says that it's impossible for a learner to learn a non-finite language (which means anything from regular languages and above) without access to an infinite number of examples, both positive and negative, of the language, or failing that, access to an oracle that knows the language perfectly and can confirm or reject the learner's evolving model of the language. In some versions, the oracle must respond to errors with a negative counter-example, teaching the learner why its model is wrong ("if the sentence you just formed was correct then you could form this other sentence, which is obviously wrong").
As far as I know, Gold's result still stands. So I think you may be overestimating the power of the python script you link to.
I should say that I'm aware of Karpathy's project, and it's really cool (or awesome if you prefer), but reproducing text is not the same as learning language.
In other words, if you expect any program to learn language, you better have infinite data to train it with and also an infinite amount of time. I stress that this doesn't only apply to human languages. You can try teaching a network to learn the syntax of Java, for example. You'll still need infinite resources, or an Oracle.
++ Edited to remove unfortunate scare quotes and also to add: if you could machine-learn a language, even "just" a CFG, we wouldn't need to write parsers for programming language compilers anymore. It would suffice to present a machine learning algorithm with some examples (a lot of them probably) and the algorithm would form a model of the language, that we could then use as a parser.
This is far from the state of the art today, however.
__________________
[1] http://web.mit.edu/~6.863/www/spring2009/readings/gold67limi...