Qdrant + FastEmbed with Rust #2
aka “part 2” 🙂
So in part 1 we got the basics working, we set up a connection to Qdrant, made a new collection, and then checked the collection exists.
We also wired up the code for FastEmbed to produce embeddings, so what next? How do we actually make it usable for a real project? We want to pass in text files, chunk them, and then upsert the chunks with their matching embeddings and payload as Qdrant “Points”.
We will eventually build a server with API endpoints, but before that, let’s take care of “chunking” our text, this is one of the fundamentals of vector embeddings for Qdrant, or *another brand of vector database (*not recommended).
Chunking (1 block of text)
So in part 1 we already had a bag of diced carrot ‘chunks’. (This is an analogy). But we ideally we want to make our own “chunks”. We need to progress to a more realistic scenario where we have a bunch of files (or carrots 🥕).
For the purposes of developing the code I made a “/data” directory with a load of “.txt” files inside. You will want to ad more error checking, eg file not found etc.
Chunking, why?
We have a maxiumum context window*
more vectors == better retrieval
*Context window is a limitation of an LLM — Qdrant will return only the most relevant chunks (once you have made them and stored them inside a Qdrant “collection”)
Chunking a group of files
Putting Qdrant and FastEmbed to one side, for just a moment, chunking in Rust is much like using the built in “window” function, but window could leave out an incomplete final chunk, so we will do it from scratch. We have 3 variables, chunk size, overlap, and step. This is not the only way to do chunking but to save us getting distracted we will use this for starters:
| Term | Meaning |
|---|---|
| chunk size | how many words per chunk |
| overlap | how many words repeat between chunks |
| step | chunk size − overlap |
Rust Playground:
Experiment, delete, change the code and finally see if you can write it from scratch?
The code is fairly logical, you may or may not have come across usize::min before – here is a summary of why we use it:
Why usize::min is used
When you get near the end of the text, there may be fewer than 20 words left. (where 20 is the chunk size for our toy example). A more realistic chunk size would be 200 with an overlap of 20. There is no right answer, and there is much more to it than I have covered here. Check out the Qdrant docs
usize::min(i + chunk_size, vs.len()) ensures the last chunk still includes the tail of the text without crashing/panic’ing
And now, here is the full Rust code, including the iteration over all text files in the data directory:
Thanks for making it to the end!
