Vector databases have been around for several years, yet they gained significant attention with the emergence of Generative AI, boosted by the announcement of ChatGPT in late 2022. But what exactly sets vector databases apart from traditional databases, and why are they well-suited for AI applications?
What are vector databases and why are they well-suited for AI applications?
The answer lies in the fact that vector databases are designed explicitly to store and manage high-dimensional vector data, which represents the semantics of unstructured data such as text, video, and audio. This makes it possible to search for similar files given a specific input based on advanced indexing and search techniques such as Hierarchical Navigable Small Worlds (HNSW).
In contrast, traditional databases like relational and NoSQL databases are designed to provide exact answers to precise queries. Vector similarity search, on the other hand, enables users to find semantically similar texts or images, also known as k-Nearest-Neighbor search.
CrateDB, for example, has been designed to combine structured data (tables, time series, geospatial) with semi-structured data (JSON/documents) and unstructured data (text) in a single record of the database. This allows users to store a vast amount of data and apply various query techniques, ranging from simple queries to complex aggregations and full-text search in milliseconds across billions of data points.
According to Gartner, around 80% of enterprise data is unstructured and stored in various formats such as documents, emails, support information, videos, audio files, etc. Therefore, it makes sense to extend databases with vector store capabilities to make unstructured data even more usable.
The advantages of vector stores are not limited to advanced search techniques such as hybrid search, which combines exact text matches and full-text search with similarity search based on vectors. They also have several applications in machine learning, such as serving as a vector store for embeddings created by machine learning models. These embeddings can be used for Retrieval Augmented Generation (RAG), which provides additional context for queries to a large language model, thereby customizing chatbots with company-specific data.
Advantages of vector stores over fine-tuning
Vector stores are crucial for AI in enterprises because public large language models (LLMs) like OpenAI's GPT are trained on publicly available data. However, for enterprises, storing information in vectors/embeddings is essential when creating company-specific Generative AI applications and making them available for RAG. This is beneficial over fine-tuning LLMs for several reasons.
Firstly, vector stores offer enhanced security and privacy. Unlike AI models, databases have built-in roles and security that restrict who sees what by standard access controls, allowing fine-grained control over which context is provided to the LLM. On the other hand, fine-tuning provides all data to the LLM, making it accessible to anyone who queries the LLM. This is a major concern, especially in scenarios where customer data or other PII is involved, where not having this control is a no-go.
Secondly, RAG is more scalable than fine-tuning, as it does not require updating all the parameters of an LLM, which requires extensive computing power. Additionally, it does not require labeling and crafting training data, which is a labor-intensive process that can take weeks or months.
Finally, RAG increases trust in the results, as it works better with dynamic data, generating more deterministic results from curated, up-to-date data. Fine-tuning acts more like a black box and makes it difficult to explain why an LLM provided a specific answer. Hallucinations can be reduced with RAG instead of relying on the model's weights that encode business information.
Wrapping-up
In conclusion, vector databases are an essential tool for enterprise AI applications. With their advanced indexing and search techniques, vector stores can store and manage high-dimensional vector data efficiently, making it possible to search for similar files based on advanced search techniques. Additionally, they offer several advantages over traditional databases, such as enhanced security and privacy, scalability, and improved trust in the results.