Structured forgetfulness has been proposed by Facebook researchers as a way for AI to clear the decks a little bit, thereby improving performance and getting that much closer to how the human mind works. The researchers describe the issue by illustrating how humans and artificial intelligence agents might approach a similar situation. By assigning an expiration date to data, Facebook researchers hope to aid future AIs in paying attention. It’s called Expire-Span and it’s intended to assist neural networks in sorting and storing the information most relevant to their assigned tasks more efficiently.
Expire-Span works by predicting which data will be most useful to the network in a given context and then assigning that data an expiration date. The more critical an item of information is to Expire-Span, the further out the expiration date is assigned, Angela Fan and Sainbayar Sukhbaatar, research scientists at FAIR, explained in a Friday blog post. Thus, neural networks will be able to retain critical information for a longer period of time while continuously clearing memory space through forgetting irrelevant data points. Each time a new piece of data is added, the system evaluates not only its relative importance, but also the significance of previously added data points.
This will also aid AI in learning to make better use of available memory, resulting in increased scalability. Expire-Span calculates the expiration value for each hidden state each time a new piece of information is presented and determines how long that information is retained as a memory, they explained. This gradual deterioration of some information is critical for retaining critical information without obscuring it. Expire-Span makes a prediction based on context learned from data and influenced by its immediate environment’s memories.
Though the research is still in its infancy, the research team stated that as a next step in our research toward more human like AI systems, we are investigating how to incorporate different types of memories into neural networks. The team hopes to develop an even more accurate approximation of human memory in the future that is capable of learning new information much more quickly than current technology allows. As an example, suppose there are ten doors of varying colours. That information was almost immediately discarded by your brain.
An animation depicting the vanishing of an AI’s memories. Each individual memory has a predicted expiration date, and the size of the memory is task-dependent explained Angela Fan, a Facebook AI researcher who contributed to the Expire-Span paper. Thus, in the case of the doors, the colours of the non-yellow doors are critical until the yellow one is discovered. At that point, it is safe to disregard the remainder, though depending on how many additional doors need to be checked, the memory may be retained for varying lengths of time.
However, an AI may have retained the colours and locations of the remaining nine doors in its memory. This is not an issue when dealing with relatively small amounts of data, but machine learning algorithms now routinely handle millions of data points and ingest terabytes of imagery or language during training. And, because they are designed to constantly compare new data to their accumulated knowledge, they become bogged down by constant references to irrelevant or outdated data points. Previous attempts to overcome this obstacle involved compressing less useful data to conserve memory space, but those efforts failed because compression results in blurry versions of the data, according to Fan and Sukhbaatar. you can read more details in this research paper.
Because the system has no way of knowing whether the other doors are important, that information is kept on hand, increasing the model’s size and decreasing its speed. Fan stated that the models trained with Expire-Span outperformed and were more efficient, requiring less memory and compute time. Get code of Expire-Span is hosted on github .