--- title: "Getting Started with pairwiseLLM" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting Started with pairwiseLLM} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", eval = TRUE ) library(pairwiseLLM) library(dplyr) ``` # 1. Introduction `pairwiseLLM` provides a unified workflow for generating and analyzing **pairwise comparisons of writing quality** using live LLM APIs (OpenAI, Anthropic, Gemini Developer API, Vertex AI Gemini API, Together.ai), local models via Ollama, and batch APIs for OpenAI, Anthropic, and Gemini Developer API. A typical workflow: 1. Select writing samples 2. Construct pairwise comparison sets 3. Submit comparisons to an LLM (live or batch API) 4. Parse model outputs 5. Fit Bradley–Terry or Elo models to obtain latent writing-quality scores For prompt evaluation and positional-bias diagnostics, see: * [Prompt Template Positional Bias Testing](https://shmercer.github.io/pairwiseLLM/articles/prompt-template-bias.html) For advanced batch processing workflows, see: * [Advanced: Submitting and Polling Multiple Batches](https://shmercer.github.io/pairwiseLLM/articles/advanced-batch-workflows.html) --- # 2. Setting API Keys `pairwiseLLM` reads provider keys **only from environment variables**, never from R options or global variables. | Provider Surface | Environment Variable | |------------------|----------------------| | [OpenAI](https://openai.com/api/) | OPENAI_API_KEY | | [Anthropic](https://platform.claude.com/) | ANTHROPIC_API_KEY | | Gemini Developer API | GEMINI_API_KEY | | Vertex AI Gemini API | VERTEX_API_KEY | | [Together](https://www.together.ai/) | TOGETHER_API_KEY | `backend = "gemini"` uses the Gemini Developer API and `backend = "vertex"` uses the Vertex AI Gemini API. Those backends use separate API-key surfaces. Configure only the key for the cloud backend you plan to use; you do not need keys for every provider. You can put the relevant key in your `~/.Renviron`: ``` OPENAI_API_KEY="sk-..." ANTHROPIC_API_KEY="..." GEMINI_API_KEY="..." VERTEX_API_KEY="..." TOGETHER_API_KEY="..." ``` Check which keys are available: ``` library(pairwiseLLM) check_llm_api_keys() #> Some LLM API keys are not set. This is expected when you use only selected backends. #> # A tibble: 5 × 4 #> backend service env_var has_key #> #> 1 openai OpenAI OPENAI_API_KEY TRUE #> 2 anthropic Anthropic ANTHROPIC_API_KEY TRUE #> 3 gemini Google Gemini GEMINI_API_KEY TRUE #> 4 vertex Vertex AI Gemini API VERTEX_API_KEY TRUE #> 5 together Together.ai TOGETHER_API_KEY TRUE ``` [Ollama](https://ollama.com/) runs locally and does not require an API key, just that the Ollama server is running. --- # 3. Example Writing Data The package ships with 20 simulated student writing samples with clear differences in quality: ```{r} data("example_writing_samples", package = "pairwiseLLM") dplyr::slice_head(example_writing_samples, n = 3) ``` Each sample has: - `ID` - `text` --- # 4. Constructing Pairwise Comparisons Create all unordered pairs: ```{r} pairs <- example_writing_samples |> make_pairs() dplyr::slice_head(pairs, n = 5) ``` Sample a subset of pairs: ```{r} pairs_small <- sample_pairs(pairs, n_pairs = 10, seed = 123) ``` Randomize SAMPLE_1 / SAMPLE_2 order: ```{r} pairs_small <- randomize_pair_order(pairs_small, seed = 99) ``` --- # 5. Traits and Prompt Templates ## 5.1 Using a built-in trait ```{r} td <- trait_description("overall_quality") td ``` Or define your own: ```{r} td_custom <- trait_description( custom_name = "Clarity", custom_description = "How clearly and effectively ideas are expressed." ) ``` ## 5.2 Using or customizing prompt templates Load default prompt: ```{r} tmpl <- set_prompt_template() cat(substr(tmpl, 1, 300)) ``` Placeholders required in custom prompt templates: - `{TRAIT_NAME}` - `{TRAIT_DESCRIPTION}` - `{SAMPLE_1}` - `{SAMPLE_2}` Load a template from file: ```{r, eval=FALSE} set_prompt_template(file = "my_template.txt") ``` For named registration, replacement, removal, and the schemas used in later steps, see [Data Schemas and Prompt Management](https://shmercer.github.io/pairwiseLLM/articles/data-and-prompts.html). --- # 6. Live Pairwise Comparisons The unified wrapper works for **OpenAI, Anthropic, Gemini Developer API, Vertex AI Gemini API, Together.ai, and Ollama**. It supports **parallel processing** and **incremental output file saving** (resume capability) for **all** supported backends. The function returns a list containing: - `$results`: observed outcomes only (canonical schema) - `$failed_pairs`: scheduled pairs with no observed outcome - `$failed_attempts`: attempt-level failures (retries, timeouts, parse errors, invalid winners) ```{r, eval=FALSE} # Example using parallel processing and incremental saving res_list <- submit_llm_pairs( pairs = pairs_small, backend = "openai", # also "anthropic", "gemini", "vertex", "together", "ollama" model = "gpt-4o", trait_name = td$name, trait_description = td$description, prompt_template = tmpl, # New features: parallel = TRUE, workers = 2, save_path = "live_results.csv" ) ``` `service_tier` is provider-specific rather than portable across backends. Gemini Developer API and Vertex AI Gemini API currently accept the public values `"standard"`, `"flex"`, and `"priority"`, but Vertex is live-only and uses a different wire format internally. ```{r, eval=FALSE} # Gemini Developer API live request res_gemini <- submit_llm_pairs( pairs = pairs_small, backend = "gemini", model = "gemini-3.5-flash-lite", trait_name = td$name, trait_description = td$description, prompt_template = tmpl, service_tier = "priority" ) # Vertex AI Gemini API live request res_vertex <- submit_llm_pairs( pairs = pairs_small, backend = "vertex", model = "gemini-3.8-flash", trait_name = td$name, trait_description = td$description, prompt_template = tmpl, service_tier = "flex" ) ``` For OpenAI, `service_tier = "flex"` requests lower-cost, slower Flex processing when the selected model supports it. Capacity may be unavailable; Flex is not priority routing. Service-tier meanings are provider-specific. Preview results: ```{r, eval=FALSE} # Successes are in the $results tibble dplyr::slice_head(res_list$results, 5) # Failures (if any) are in $failed_pairs if (nrow(res_list$failed_pairs) > 0) { print(res_list$failed_pairs) } # Attempt-level failures (if any) are in $failed_attempts if (nrow(res_list$failed_attempts) > 0) { print(res_list$failed_attempts) } ``` Each row in `$results` includes: - `custom_id` (uses `pair_uid` if supplied; otherwise defaults to `LIVE__vs_`) - `ID1`, `ID2` - parsed `` tag → `better_sample` and `better_id` - canonical aliases/keys: `A_id`, `B_id`, `winner_pos`, `ordered_key`, `unordered_key`, `pair_uid`, `received_at`, `backend`, `model` - thoughts (reasoning text, if available) and content (final answer) --- # 7. Preparing Data for BT or Elo Modeling Convert the LLM output (specifically the `$results` tibble for `submit_llm_pairs()` output) to a 3-column BT dataset: ```{r, eval=FALSE} # res_list: output list from submit_llm_pairs() # We extract the $results tibble for modeling bt_data <- build_bt_data(res_list$results) dplyr::slice_head(bt_data, 5) ``` and/or a dataset for Elo modeling: ```{r, eval=FALSE} # res_list: output from submit_llm_pairs() elo_data <- build_elo_data(res_list$results) ``` --- # 8. Bradley–Terry Modeling Fit model: ```{r, eval=FALSE} bt_fit <- fit_bt_model(bt_data) ``` Summarize results: ```{r, eval=FALSE} summarize_bt_fit(bt_fit) ``` The output includes: - latent θ ability scores - SEs - reliability (when using `sirt` engine) --- # 9. Elo Modeling ```{r, eval=FALSE} elo_fit <- fit_elo_model(elo_data, runs = 5) elo_fit ``` Outputs: - Elo ratings for each sample - unweighted and weighted reliability - trial counts --- # 10. Batch APIs (Large Jobs) ## 10.1 Submit a batch Batch helpers support `"openai"`, `"anthropic"`, and `"gemini"`. `backend = "vertex"` is rejected explicitly because Vertex batch mode is not implemented in this series. ```{r, eval=FALSE} batch <- llm_submit_pairs_batch( backend = "gemini", model = "gemini-3.8-flash", pairs = pairs_small, trait_name = td$name, trait_description = td$description, prompt_template = tmpl, service_tier = "priority" ) ``` ## 10.2 Download results ```{r, eval=FALSE} res_batch <- llm_download_batch_results(batch) head(res_batch) ``` ## 10.3 Multi‑Batch Jobs In addition to the standard batch helpers, you can split a large job into multiple segments using `llm_submit_pairs_multi_batch()` and then poll all of them with `llm_resume_multi_batches()`. This is particularly useful when you have many pairs or want to ensure that you can resume if the session ends. ``` {r, eval=FALSE} # Generate a small set of pairs pairs_small <- example_writing_samples |> make_pairs() |> sample_pairs(n_pairs = 10, seed = 4321) |> randomize_pair_order(seed = 8765) td <- trait_description("overall_quality") tmpl <- set_prompt_template() # Split into two batches and include reasoning/chain-of-thought multi_job <- llm_submit_pairs_multi_batch( pairs = pairs_small, backend = "openai", model = "gpt-5.1", trait_name = td$name, trait_description = td$description, prompt_template = tmpl, n_segments = 2, output_dir = "myjob", write_registry = TRUE, include_thoughts = TRUE ) # Poll and merge results. Combined results are written to # "myjob/combined_results.csv" or the directory you specify. res <- llm_resume_multi_batches( jobs = multi_job$jobs, interval_seconds = 30, write_combined_csv = TRUE ) head(res$combined) ``` ## 10.4 Estimating cost before you run For large jobs, it is often useful to estimate token usage and cost before launching a live run or submitting a batch. `pairwiseLLM` includes `estimate_llm_pairs_cost()`, which runs a small **pilot** (paid live calls) and then estimates the rest of the job by calibrating input tokens from prompt byte length. The output includes both: - **Expected cost** (using mean output tokens from usable pilot calls) - **Budget cost** (using a high quantile of pilot output tokens, controlled by `budget_quantile`) If you are running a discounted batch workflow, set `mode = "batch"` and supply a `batch_discount` multiplier. ```{r, eval=FALSE} # Create a moderate set of pairs pairs_big <- example_writing_samples |> make_pairs() |> sample_pairs(n_pairs = 200, seed = 123) |> randomize_pair_order(seed = 456) td <- trait_description("overall_quality") tmpl <- set_prompt_template() est <- estimate_llm_pairs_cost( pairs = pairs_big, backend = "anthropic", # "openai", "anthropic", "gemini", "together" model = "claude-sonnet-4-5", trait_name = td$name, trait_description = td$description, prompt_template = tmpl, mode = "batch", batch_discount = 0.5, # set to 1 for no discount n_test = 10, # paid pilot calls (live) budget_quantile = 0.9, # p90 output tokens cost_per_million_input = 3.0, # fill in your provider pricing cost_per_million_output = 15.0 ) est$summary ``` ### Deterministic calculation example Suppose a two-pair live pilot records input-token counts of 20 and 40 and completion-token counts of 10 and 30. The prompt-byte calibration predicts 60 and 80 input tokens for the two remaining pairs. With `budget_quantile = 0.9`, R's type-7 sample quantile of `c(10, 30)` is 28. At input and output prices of 1 currency unit per million tokens and `batch_discount = 0.5`, the estimator calculates: - observed live pilot tokens: 60 input and 40 output; - remaining expected tokens: 140 input and `mean(c(10, 30)) * 2 = 40` output; - remaining budget output tokens: `28 * 2 = 56`; - expected cost: `(60 + 40 + 0.5 * (140 + 40)) / 1e6 = 0.000190`; and - budget cost: `(60 + 40 + 0.5 * (140 + 56)) / 1e6 = 0.000198`. The pilot observations are included once at live prices. They are not averaged into the estimated remaining count and are not discounted. Avoid paying twice: reuse pilot results The estimator returns the original pilot output object and the pairs not included in the pilot (`remaining_pairs`). Use `remaining_pairs` to submit only the remaining work after you are satisfied with the estimate. Pilot judgments are not automatically merged into the later submission result: ```{r, eval=FALSE} remaining_pairs <- est$remaining_pairs # Example: submit only the remaining pairs as a batch batch <- llm_submit_pairs_batch( backend = "anthropic", model = "claude-sonnet-4-5", pairs = remaining_pairs, trait_name = td$name, trait_description = td$description, prompt_template = tmpl) results <- llm_download_batch_results(batch) ``` Notes: - The estimator does not require a provider tokenizer; it uses prompt byte length calibrated on the pilot. - Ollama is not supported in the estimator (local models do not incur token costs). - Reasoning/thinking tokens are treated as output tokens for pricing. - Supply current prices for the selected provider, model, endpoint, and mode; the package does not maintain or validate a pricing catalog. --- # 11. Backend-Specific Tools Most users use the unified interface, but backend helpers are available. ### 11.1 OpenAI - `submit_openai_pairs_live()` - `build_openai_batch_requests()` - `run_openai_batch_pipeline()` - `parse_openai_batch_output()` ### 11.2 Anthropic - `submit_anthropic_pairs_live()` - `build_anthropic_batch_requests()` - `run_anthropic_batch_pipeline()` - `parse_anthropic_batch_output()` ### 11.3 Google Gemini - `submit_gemini_pairs_live()` - `build_gemini_batch_requests()` - `run_gemini_batch_pipeline()` - `parse_gemini_batch_output()` ### 11.4 Vertex AI Gemini API (live only) - `submit_vertex_pairs_live()` - `vertex_compare_pair_live()` ### 11.5 Together.ai (live only) - `together_compare_pair_live()` - `submit_together_pairs_live()` ### 11.6 Ollama (local, live only) - `ollama_compare_pair_live()` - `submit_ollama_pairs_live()` - `ensure_only_ollama_model_loaded()` Reasoning/thinking constraints, partial-result handling, retries, and local-resource guidance are covered in [Provider Controls and Recovery](https://shmercer.github.io/pairwiseLLM/articles/provider-controls-and-recovery.html). --- # 12. Troubleshooting ### Missing API keys ```{r} check_llm_api_keys() ``` ### Timeouts Use batch APIs for >40 pairs. Split a large job into multiple segments using `llm_submit_pairs_multi_batch()` and then poll/download all of them with `llm_resume_multi_batches()` ### Positional bias Use `compute_reverse_consistency()` + `check_positional_bias()`; see [Prompt Template Positional Bias Testing](https://shmercer.github.io/pairwiseLLM/articles/prompt-template-bias.html) for a full example. --- # 13. Citation > Mercer, S. H. (2026). *Getting started with pairwiseLLM* [R package vignette]. Comprehensive R > Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM .