What is vector space when it comes to LLMs?
I've been doing a deep dive with my spaced repetition memorization process where I go through various important terms related to machine learning like CUDA, transformers, embeddings, and attention.
One such term, vector space, has taken a lot of imagination power to understand.
Machine learning basically breaks down really high-density objects in our universe like an apple and then makes representations of them that are much lower density and able to be computed, essentially a low-density linguistic model of an apple. This process keeps the relevant information about an apple like semantic similarity with other objects in our universe (like an orange) while discarding the giant amount of information that is not relevant.
The brilliance of this is this is what humans do as well except our models of objects in our universe are much higher density, something about the brain allows us to process a whole lot more than these machines in real-time with a fraction of the power (I learned from Subutai Ahmad that the brain does it all on 20 watts of power, please correct me if I misheard)
The way that the machine learning algorithms do this is they create a simulation of our universe called a vector space and then attach objects that are similar to each other and places them together in that vector space. The model of an apple is closer to the model of an orange than it is to a model of a cat.
Did this make sense? If so, what have you recently learned about machine learning that you can share with us fellow nerds?