Typesense app benchmark
Build product search, filters, and catalog tools using the Typesense search engine. Tasks and sources
| Setup | Passed | Attempts | Pass rate | Interval | Tiers | Cost per pass | Cost basis | Median run |
|---|---|---|---|---|---|---|---|---|
| opencode · gpt-5.6-luna (no-web) | 1 | 26 | 4% | 0%–12% (95% task-clustered bootstrap, 26 attempts) | 1B 1C 24F | $0.593 | receipts, 432 receipts | 4.6 min |
| opencode · gpt-5.6-terra (no-web) | 5 | 26 | 19% | 0%–38% (95% task-clustered bootstrap, 26 attempts) | 5B 21F | $1.09 | receipts, 370 receipts | 4.3 min |
| Setup | Pass rate | Passed | Interval | Cost per pass | Cost basis | On frontier |
|---|---|---|---|---|---|---|
| opencode · gpt-5.6-terra (no-web) | 19% | 5/26 | 0%–38% (95% task-clustered bootstrap, 26 attempts) | $1.09 | receipts | not computed |
| opencode · gpt-5.6-luna (no-web) | 4% | 1/26 | 0%–12% (95% task-clustered bootstrap, 26 attempts) | $0.593 | receipts | not computed |
| Task | opencode · gpt-5.6-luna (no-web) | opencode · gpt-5.6-terra (no-web) |
|---|---|---|
| T01 | 0/2 · 2F · $0.042 | 0/2 · 2F · $0.352 |
| T02 | 0/2 · 2F · $0.046 | 0/2 · 2F · $0.484 |
| T03 | 0/2 · 2F · $0.048 | 0/2 · 2F · $0.381 |
| T04 | 0/2 · 2F · $0.041 | 2/2 · 2B · $0.352 |
| T05 | 0/2 · 2F · $0.040 | 0/2 · 2F · $0.527 |
| T06 | 0/2 · 2F · $0.050 | 0/2 · 2F · $0.473 |
| T07 | 0/2 · 2F · $0.052 | 0/2 · 2F · $0.399 |
| T08 | 0/2 · 2F · $0.047 | 0/2 · 2F · $0.353 |
| T09 | 0/2 · 2F · $0.045 | 0/2 · 2F · $0.416 |
| T10 | 0/2 · 2F · $0.052 | 0/2 · 2F · $0.470 |
| T11 | 0/2 · 2F · $0.037 | 2/2 · 2B · $0.388 |
| T12 | 1/2 · 1B 1C · $0.052 | 1/2 · 1B 1F · $0.494 |
| T13 | 0/2 · 2F · $0.042 | 0/2 · 2F · $0.350 |
Task prompts
Tangle’s VerticalBench generated these app-building tasks from the search engine’s documentation and release notes. The prompts and their sources are below.
T01 · Product search with shared synonyms
Build a small internal search console on top of a Typesense product index called 'products' (fields 'title' and 'description', both text). The tool must let a store editor define a reusable, multi-way synonym group named 'product-synonyms' by entering two or more words that should be treated as equivalent, apply that group to the 'products' index so it takes effect for search, and run a product search that benefits from the group. A user should be able to type one word from the group into the search box and see products whose text only contains a different word from that same group.
T02 · Pinned product results
Build a Next.js app on a Typesense-powered product catalog named 'products', where each document has a title and a price. Add Typesense's search curation feature (pinned results): let an operator define a named curation rule that fixes one specific product to the very first search result whenever a shopper's search text exactly matches a phrase the rule specifies, even when that product's title shares no words with the search text. Shoppers use a search box to look up products by title, and the pinned product must appear first whenever their search matches an active rule.
T03 · Catalog duplication
Build a Next.js app for a product catalog stored in a Typesense collection named `products` (fields: `title`, `description`). Add a 'Duplicate with data' action that creates a full copy of the collection, including every existing document, under the name `products_copy`. After duplicating, show the new collection's documents (their titles) so the user can confirm everything copied over, not just the schema.
T04 · Loyalty-tier facets via a referenced customers collection
Build a Next.js product catalog search page on Typesense. The 'products' and 'customers' collections already exist and are seeded; the app only searches them. A 'products' document holds id, name and customer_id, which references the customer who bought the product most, and the 'customers' collection holds id and loyalty_tier (gold, silver or bronze). The two collections are linked with Typesense JOINs (collection references). Users type a term into a search box and run the search to find matching products. Alongside the results, show a sidebar that groups the matches by the referenced customer's loyalty_tier with a count for each tier, and let the user click a tier to narrow the product list to just that tier. The tier comes only from the customers collection; product documents do not carry it.
T05 · Author directory with book counts
Build "Author Shelf", a small bookstore author directory on Typesense 30.2. The search product already holds two collections. The authors collection has the fields name and country, and its document ids look like a1, a2. The books collection has the fields title and author_id, where author_id is a reference to the id of the author who wrote the book. A visitor opens the home page, types a name into an author search box and runs the search. The page then lists every matching author that has books, showing the author's name, their country and how many books the store holds by that author, for example "3 books". Authors that do not match the search must not appear. The book count must come from Typesense itself, using its JOINs support in the same search that finds the authors, rather than from fetching book records and counting them in your own code. All calls to Typesense are made from server-side code.
T06 · Diversified recipe search with Typesense MMR
Build a recipe discovery web app on Typesense's result diversification (Maximum Marginal Relevance), so a search does not return ten near-identical recipes in a row. The product already holds a collection named recipes with the fields title (string), tags (string array, faceted) and popularity (integer, the collection's default ranking, most popular first). It also already holds an empty curation set named recipe_diversity that is linked to the recipes collection. Every recipe title contains the word people search for, and the most popular recipes all share the same tags, so a plain search shows five desserts in a row. The home page has a recipe search box and a button that runs the search. After a search the page shows two lists side by side, each with exactly five recipes shown as title and tags: a Standard list, the plain search results in popularity order with no curation applied, and a Diversified list, the same search after Typesense's result diversification has spread apart recipes that share tags. Use tag overlap as the similarity measure and nothing else. The app itself must create the diversification rule inside the recipe_diversity curation set, with the rule id spread and the rule tag spread (your key is allowed to manage curation sets and to search the recipes collection, nothing more), and must apply that rule to the Diversified search. Both lists must be exactly what Typesense returns: do not reorder, group, filter or re-rank results in your own code. All calls to Typesense are made from server-side code.
T07 · Post search across topic and audience filters
Build a search page over a Typesense 'posts' collection (fields: title, topic, audience). Let a visitor filter posts by a topic term and separately by an audience option, then run a Typesense union search that merges the matches from both filters into a single ranked list, removing any post that matches both filters so it appears once by default. Also give the visitor a way to reveal every duplicate match instead, so a post matching both filters can appear twice.
T08 · Publisher Catalog Search With Capped Recent Books
Build a Next.js app backed by Typesense with two collections: `authors` (field: `name`) and `books` (fields: `title`, `seq`, and `author_id`, a reference back to the author). Users type an author name into a search box and run a search. For each matching author, use Typesense's join feature to show that author's related books, but cap each author's shown books to their 2 highest-`seq` entries only, with the highest `seq` first. All Typesense calls must run server-side using the scoped TYPESENSE_KEY; never expose that key to the browser.
T09 · Library admin that keeps books when an author is retired
Build a small library-admin web app on top of an empty Typesense. On first load the app itself must create the two collections and seed them, so a brand-new Typesense works with no manual step. Data model: - Collection `authors`: id and name. Seed author id a1 named "Ursula Le Guin" and author id a2 named "Octavia Butler". - Collection `books`: title and author_id, where author_id holds the id of the book's one author. Seed four books: "The Dispossessed" and "A Wizard of Earthsea" by a1, "Kindred" and "Parable of the Sower" by a2. Use Typesense JOINs so every book is returned together with its author's name. The home page shows: - An authors list. Each author row has a button labelled "Retire author" whose accessible name also says which author it retires. One click deletes only that author's document from `authors`, with no confirmation dialog, and the page then shows the updated data. - A form to add a book: a title field, an author choice listing the current authors, and an "Add book" button. The new book is saved in `books` under the chosen author and appears in the table. - A books table with one row per book showing its title and its author's name. A book whose author no longer exists stays in the table with an empty author cell (a dash counts as empty). Business rule: retiring an author must NEVER remove that author's books. After the operator clicks "Retire author" for Ursula Le Guin, `authors` no longer holds a1, but every book that was listed before is still stored in `books` and still shown in the table, now with an empty author. All Typesense calls happen on the server. The page must read live data on every request.
T10 · Asset registry exact key lookup on Typesense
Build a small asset-registry lookup web app on Typesense 30.2. The search product starts empty, so the app owns its data: the first time it runs it creates a collection named `assets` with two string fields, `label` and `asset_key`, and seeds four assets. Seeding must be safe to repeat: reloading the page or looking up a key must never create duplicate assets. The four assets (label, then asset_key) are: - Cooling pump alpha: k7q2m9x4wdr3t8v1z5bnh6j0c2y7fps4g9l3u8eam1x6d0q5wkz2n7b4t9vrc8h3f5j1ypu6a0e4s2glq9w3k7m5xdt1v8z2r6bnj4c0h9y3fpl5g7u1s8ead2x6m4q0wkn8p2r6t0v4b3d7f1h5j9x2z6c0e4g8 - Cooling pump beta: k7q2m9x4wdr3t8v1z5bnh6j0c2y7fps4g9l3u8eam1x6d0q5wkz2n7b4t9vrc8h3f5j1ypu6a0e4s2glq9w3k7m5xdt1v8z2r6bnj4c0h9y3fpl5g7u1s8ead2x6m4q0wky9w5u1s7q3m4k8i2g6e0a7c3f9h1j5 - Boiler valve gamma: p3o8i2u7y1t6r0e5w9q4l2k7j1h6g0f5d9s3a8z2x7c1v6b0n4m9q3w8e2r7t5y0u4i9o1p6a2s7d3f8g1h5j0k4l9z6x2c7v3b8n0m5q1w6e4r9t3y8u2i7o1p6a0s5d4f8g3h7j2k6l9z1x5c0v4b7n2m6q3w8 - Boiler valve delta: p3o8i2u7y1t6r0e5w9q4l2k7j1h6g0f5d9s3a8z2x7c1v6b0n4m9q3w8e2r7t5y0u4i9o1p6a2s7d3f8g1h5j0k4l9z6x2c7v3b8n0m5q1w6e4e5r0t4y9u1i6o2p7a3s8d1f5g0h4j9k3l7z2x6c1v8b4n9m5q0 Every asset_key is an unbroken 160-character string of lowercase letters and digits, with no spaces or separators. The keys of Cooling pump alpha and Cooling pump beta share their first 130 characters and differ only in their last 30. The keys of Boiler valve gamma and Boiler valve delta share their first 110 characters and differ in their last 50. Each asset has a distinct label, and the label is the only thing besides the key that identifies it. The page is a single lookup screen with a box labelled 'Look up by key' and a button that runs the lookup. A user pastes a full asset key and the app asks Typesense for the asset whose asset_key is exactly that key (an exact-match filter, not a fuzzy text search). The result area shows exactly the one asset with that key, its label and its key, and never a second look-alike asset. The page does not list the other assets. A key that matches nothing shows a clear 'No asset found' message. All Typesense calls happen on the server, and the page reads live data from Typesense on every request.
T11 · Menu explorer on Typesense nested ingredients
Build a menu explorer on Typesense. A collection named menu already exists with nested fields enabled. Each dish has a name (string) and an ingredients field (object[]) whose entries each carry a name (string) and a concentration (integer percent); the collection declares the fields ingredients (object[]), ingredients.name (string[]) and ingredients.concentration (int32[]). Three dishes are stored, each with three ingredient entries. Dish names and ingredient names are ordinary words, so use what the user types and never hard-code dishes. The page has an ingredient-name text box, a comparison drop-down select with two options labelled exactly Less than and At least, and a concentration number box, plus a button that lists dishes. Listing shows every dish that has one ingredient entry whose name equals the typed name AND whose concentration satisfies the chosen comparison (At least means greater than or equal) in that same ingredient entry. A dish must not appear when the name matches in one ingredient entry and the concentration condition is met only by a different ingredient entry of the same dish. Show the matching dish names as a list and an explicit empty message when nothing matches. The app is read-only: it searches the menu collection through Typesense from server-side code with the scoped key it is given, and never creates or changes collections or documents.
T12 · Pet-shop search with an irregular-plurals stemming dictionary
Build a pet-shop catalogue search app on Typesense that uses Typesense's custom stemming dictionaries, so that a search for a singular word also finds products titled with its irregular plural. The dictionary is named irregular-plurals and must teach the search that geese means goose, mice means mouse and children means child. The search index is a collection named products. Every document has one string field, title. The app has a one-click 'Set up catalogue' action that (1) uploads the irregular-plurals dictionary to the server, (2) creates the products collection so that title searches use that dictionary, and (3) loads four starter titles: 'Decorative geese set', 'Goose feather duster', 'Field mice plush' and 'Children building blocks'. Running the set-up action more than once must be safe and leave the catalogue holding those four starter titles again. The home page is the whole app. Before set-up it says the catalogue is not set up yet and does not crash. After set-up it shows: every product title in the catalogue; a list of the stemming dictionaries stored on the server, which must include irregular-plurals; a small form where staff type a new product title and add it to the catalogue; and a search box with a search button. After a search, the page lists only the matching product titles. Searching the singular word goose must return both 'Decorative geese set' and 'Goose feather duster', mouse must return 'Field mice plush', and child must return 'Children building blocks'. All calls to Typesense come from server-side code using the base URL and scoped key in the environment. Never expose the key to the browser.
T13 · Delivery-zone checker on Typesense
Build a delivery-zone checker for a small courier company on Typesense 30.2. The app keeps its delivery zones in a Typesense collection named `zones`. Each zone document has a `name` (text) and an `area` that holds the zone's boundary as a polygon. The collection does not exist when the app first runs: the app must create it, and load these three rectangular zones, described by their corners as (latitude, longitude) in order around the boundary. North: (10, 0), (10, 10), (20, 10), (20, 0). Centre: (5, 0), (5, 10), (15, 10), (15, 0). Far: (50, 50), (50, 60), (60, 60), (60, 50). Creating and loading must be safe to repeat, so a second visit never duplicates zones. The home page has a latitude input, a longitude input and a button that checks the point. Submitting shows the name of every zone whose boundary contains the entered point, one per line, and nothing else about other zones; when no zone contains the point it shows the text No delivery. Zones can overlap, so a point can belong to more than one zone. Typesense itself must decide which zones contain the point, using its native support for polygon boundaries on the stored `area`; do not implement point-in-polygon geometry in your own code.
| Comparison | Pairs | Minimum effect | Interval | Exact interval |
|---|---|---|---|---|
| opencode · gpt-5.6-terra (no-web) − opencode · gpt-5.6-luna (no-web) | 26 | +40 pts | 0 pts to +38 pts (95% task-clustered bootstrap, n=26) | −7 pts to +23 pts (95% exact McNemar, n=26) |