2023's Biggest Breakthroughs in Computer Science

From ChatGPT to Dall-E, it might seem like the pace of progress in artificial intelligence is unstoppable. But the artificial neural networks that underpin these programs face some important limitations. On the one hand, it is difficult for them to reason. Human brains are capable of reasoning by analogy. When we see something new, we don't have to develop new neurons. We can generalize new concepts from existing knowledge, but artificial neural networks have difficulty reasoning by analogy. In this way, when faced with new information, they often need more artificial nodes to expand their statistical capabilities, allowing them to learn new concepts.

This approach is called statistical AI. Statistically, AI is at the heart of deep neural networks, but there is another competing approach known as symbolic AI, which uses logic-based programming and symbols to represent concepts and rules. Therefore, by their very nature, they are incompatible. And the big challenge is how we can combine them to get the best of both worlds. An emerging approach called hyperdimensional computing aims to do just that. It harnesses the power of statistical AI while attempting to emulate symbolic computing. To do this, it uses vectors, ordered lists of numbers that can also represent arrows in space.

Depending on where they point, these vectors can represent information in a multidimensional and highly complex way. They can incorporate nuances, traits or concepts with only small changes in their orientation. These vectors can be combined to represent new concepts and then separated again to discern how they were formed. More importantly, these hyperdimensional vectors can encode information without having to add more nodes to the network. The moment we bring in hyperdimensional computing or better symbolic architectures and combine them with statistical methods like deep neural networks, we can have certain advantages. In March

2023

,

computer

scientists at IBM Research in Zurich made a dramatic breakthrough by combining statistical and symbolic methods to solve Ravens' progressive matrix, a puzzle that asks an AI to predict an image to fill the final square of a grid. three by three.

This is a classic problem. The first aspect is the perception part, where we look at different visual objects and then we have to recognize certain aspects of the objects. For example, what is the shape of the object? What is the color, what is the size? All this type of property. And the second aspect is to do some abstract reasoning about these types of attributes of objects in order to solve a puzzle. Thus, for the first time we combine hyperdimensional computing, or better yet symbolic architecture machinery, with deep neural networks to be able to solve abstract reasoning problems at scale.

Essentially, we can build within the neural network machinery that is capable of emulating any symbolic computing task that is difficult for the neural network itself to perform. The researchers also broke new ground by solving the problem at record speed. We were able to significantly speed up inference times for some abstract reasoning tasks by 250 times. By having a method that is able to search this type of space quickly, it is crucial to be able to make them quite practical for larger scales. and real world problems. Although more work is needed, researchers like Rahimi hope that hyperdimensional computing can be faster, more transparent and more energy efficient than current AI platforms.

We can substantially reduce energy consumption. We are talking about energy consumption ten times, 100 times lower that can be directly transferred to the reduction of the carbon footprint once we carry out the inference of this type of models. The solution for driving progressive mattresses was just a prototype. So far it's been amazing, but the fun is just beginning, that's the beautiful thing. In the 1990s, mathematician Peter Shor developed an algorithm that threatened to destroy the Internet. He was trying to understand the power of quantum

computer

s and realized that they could quickly decompose large numbers into their prime factors.

That would undermine one of the central methods of modern cryptography that has safeguarded online privacy for decades. So when I discovered the algorithm, I just called it factorization algorithm, quantum factorization algorithm, and published it. And soon people started calling it Shor's algorithm. At the heart of Shor's algorithm is the act of repeatedly multiplying a number with itself until the output of a given function begins to repeat. The required number of multiplications called the period of functions can be used to find the prime factors of any large number. For the past 30 years, this has been the fastest known algorithm for factoring integers.

Then, in August

2023

, mathematician Oded Regev published a groundbreaking new paper. Shor's algorithm is great in the sense that it allows quantum computers to factorize integers and do it quickly. I was wondering if it can be done even faster. Regev suspected that he could speed up Shor's algorithm by transforming the periodic function from one dimension to multiple dimensions. That meant including more numbers, multiplying each by itself many times to find the repetition period in the results. In doing so, Regev established an improved method for factoring integers more quickly and efficiently. Shor's algorithm is to find how long you have to walk on the line to get back to the starting point.

In the new algorithm, we no longer walk the line. We walk in the plane or even in three-dimensional space or more. The idea is that you can walk both to the right and to the left and also up and down the plane. So it could be, for example, that if you walk four steps to the right and five steps up, you will return to the starting point. Oded's improvement on my algorithm came out of nowhere. There was no work prior to this. He just had a really good idea. And he sat down, figured it out, and published a paper.

Like Shor's algorithm. Regev's new algorithm remains theoretical in nature. Talking about a potential future when quantum computing becomes more practical. I would be happy if it turns out that this new algorithm is more practical, it has some of the components that could make it more practical. But it's still unclear if it's really there. Is my algorithm now 30 years old? And Oded found an improvement that no one had thought of before. I wouldn't be surprised if people come up with more innovative quantum algorithms. And it is quite possible that someone will improve the factorization algorithm even further.

We already knew how to factor integers using Shor's algorithm. So this new algorithm is simply a speedup. What I would really like to know is whether quantum computers can do things that we simply didn't know how to do. What movie do these emojis represent? When the researchers asked three different models of large languages, they got very different answers. The simplest AI had a hard time coming up with a coherent answer. A somewhat more complex model spits out something reasonable but still wrong. But the most complex system achieved it in a single attempt. This impressive result can be explained by a number of interesting new abilities.

AI researchers call these emergent behaviors. It can be said that something has emerged that is not present in a small language model and that it becomes present in a larger model. We don't think it's going to be there and then we scale it up and suddenly this phenomenon is already there. Therefore, it is not necessarily a predictable phenomenon. Emergence occurs when a system behaves in a way that its individual units do not do on their own. This phenomenon can be seen in nature, as when lifeless atoms give rise to living cells, water molecules create waves, and starlings swarm and dart as one.

Researchers are seeing that if enough digital nodes are combined into large language models, surprising new behaviors can emerge. These behaviors are a clear departure from older neural networks that process painstakingly word by word. In 2017, a key piece of code known as a transformer was introduced to LLMs, allowing neural networks to analyze words in parallel. Now, LLMs can process text strings in an instant. The models we were training before seemed to struggle with a lot of more abstract tasks, more generalizations. And what we've seen since Transformers is that if we train much larger models, they are better at those tasks.

The magic of transformers is that they are extremely good at solving sequence prediction problems. So, given a bunch of elements in a sequence, predict the following. And this is particularly good for building generative models of, for example, language and also images. This year, researchers reported on a variety of emerging behaviors that allow LLMs to solve problems they have never seen before. This is known as zero-shot or few-shot learning and has long been a goal in artificial intelligence. But as exciting as the results are, researchers still have questions about what exactly underpins these emerging abilities. People are trying to figure out when a tipping point occurs.

One theory suggests that emergence occurs when large linguistic models break down complex tasks into smaller steps. This process is called chain of thought reasoning. But the true source of the tipping point toward emergence remains a mystery. Models get better as they get bigger and people release larger models at a pretty fast pace. Meanwhile, the

science

of understanding what's going on under the hood is advancing more slowly than that. So one of the challenges of emergence is that it is unpredictable. Because, on the one hand, bigger models usually mean better models. But, on the other hand, larger models can also demonstrate novel behaviors that developers did not anticipate a priori.

These can be beneficial behaviors, but they can also be harmful behaviors. Therefore, the ways in which they will or are used are still under development. And therefore, it is equally important to measure the possible types of damages that could be included in these models and how they do or do not emerge at scale. And that's why things seem pretty unpredictable right now. But I don't think they are always unpredictable.