[1/3 - anomaly detection] [1/2 - KNN classification] Batch upload dataset to Qdrant (crops dataset) n8n Workflow Template

Setting up variables

1. Cloud URL - to connect to Qdrant Cloud (your personal cluster URL)
2. Collection name in Qdrant
3. Size of Voyage embeddings (needed for collection creation in Qdrant)
4. Batch size for batch embedding/batch uploading to Qdrant

Google Storage node returns mediaLink, which can be used directly for downloading images; however, we just need a public image URL so that Voyage API can process it; so here we construct this public link and extract a crop name from the folder in which image was stored (for example, "cucumber")

Since we build anomaly detection based on the crops dataset, to test it properly, I didn't upload to Qdrant pictures of tomatoes at all; I filter them out here

Since Voyage API requires a specific json structure for batch embeddings, as does Qdrant's API for uploading points in batches, I am adapting the structure of jsons

[NB] - [payload = meta data in Qdrant]

Embedding images with Voyage model (mind input_type)

Batch Uploading Dataset to Qdrant

This template imports dataset images from storage, creates embeddings for them in batches, and uploads them to Qdrant in batches. In this particular template, we work with crops dataset. However, it's analogous to lands dataset, and in general, it's adaptable to any dataset consisting of image URLs (as the following pipelines are).

• First, check for an existing Qdrant collection to use; otherwise, create it here. Additionally, when creating the collection, we'll create a payload index, which is required for a particular type of Qdrant requests we will use later.
• Next, import all (dataset) images from Google Storage but keep only non-tomato-related ones (for anomaly detection testing).
• Create (per batch) embeddings for all imported images using the Voyage AI multimodal embeddings API.
• Finally, upload the resulting embeddings and image descriptors to Qdrant via batch uploading.

For anomaly detection

1. This is the first pipeline to upload (crops) dataset to Qdrant's collection.

1. The second pipeline is to set up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.
2. The third is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.

For KNN (k nearest neighbours) classification

1. This is the first pipeline to upload (lands) dataset to Qdrant's collection.

1. The second is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.

To recreate both

You'll have to upload crops and lands datasets from Kaggle to your own Google Storage bucket, and re-create APIs/connections to Qdrant Cloud (you can use Free Tier cluster), Voyage AI API & Google Cloud Storage

In general, pipelines are adaptable to any dataset of images

In Qdrant, you can create a collection once; if you try to create it two times with the same name, you'll get an error, so I am adding here a check if a collection with this name exists already

If a collection with the name set up in variables doesn't exist yet, I create an empty one;

Collection will contain named vectors, with a name "voyage" For these named vectors, I define two parameters

1. Vectors size (in our case, Voyage embeddings size)
2. Similarity metric to compare embeddings: in our case, "Cosine".

In the next workflow, we're going to use Qdrant to get the number of images belonging to each crop type defined by crop_name (for example, "cucumber"). To get this information about counts in payload fields, we need to create an index on that field to optimise the resources (it needs to be done once). That's what is happening here

Now it's time to embed & upload to Qdrant our image datasets; Both of them, crops and lands were uploaded to our Google Cloud Storage bucket, and in this workflow we're fetching the crops dataset (for lands it will be a nearly identical workflow, up to variable names) (you should replace it with your image datasets)

Datasets consist of image URLs; images are grouped by folders based on their class. For example, we have a system of subfolders like "tomato" and "cucumber" for the crops dataset with image URLs of the respective class.

I regroup images into batches of batchSize size and, to make batch upload to Qdrant possible, generate UUIDs to use them as batch point IDs (Qdrant doesn't set up id's for the user; users have to choose them themselves)