> ## Content Index > Fetch the complete content index at: https://expected-parrot.ghost.io/llms.txt > Use this file to discover other available public pages before exploring further. # How to launch a survey with LLMs and humans - URL: https://expected-parrot.ghost.io/how-to-launch-a-survey-with-llms/ - Published: 2025-07-22T20:55:04.000Z - Updated: 2026-10-17T15:55:08.000Z - Description: Our tools let you design and run the same survey with AI agents and human respondents, and seamlessly compare results. - Author: John Horton - Tags: Newsletter, #Migrated-1792252444613, #Import 2026-10-17 15:54 There may be a number of reasons to incorporate human feedback into your AI experiment. When using AI agents and LLMs to simulate human responses—whether for surveys, [data labeling](http://performing a data labeling task), or user research—you often need real human validation. This might involve cognitive testing of your questions and instructions, or verifying that AI responses align with realistic human behavior. We've built features that streamline this human-AI comparison process in both our open-source [EDSL](https://github.com/expectedparrot/edsl?ref=expected-parrot.ghost.io) package and our [no-code builder](https://www.expectedparrot.com/getting-started/build?ref=expected-parrot.ghost.io). The workflow consists of: creating a survey, running it with AI agents, then sending it to human participants. Since AI and human results use identical formatting, you can analyze both datasets with the [same methods](https://docs.expectedparrot.com/en/latest/results.html?ref=expected-parrot.ghost.io). Below we demonstrate this process with a simple survey about some unforgettable Jane Austen quotes, gathering responses from both AI agents and human participants. *Note*: Every step shown in code can also be accomplished through our no-code builder—reach out if you need help getting started! # Create a survey We start by creating a [survey](https://docs.expectedparrot.com/en/latest/surveys.html?ref=expected-parrot.ghost.io). EDSL comes with [many common question types](https://docs.expectedparrot.com/en/latest/questions.html?ref=expected-parrot.ghost.io) we can choose from based on the form of the response that we want to get back from models (and humans): ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/db742f75-e924-4537-aac7-4a3e66e93beb_1752x1026.png) Note the use of **{{ scenario.placeholders }}** in the questions. [Scenarios](https://docs.expectedparrot.com/en/latest/scenarios.html?ref=expected-parrot.ghost.io) allow you to create versions of your questions efficiently with different texts, PDFs, CSVs, images, videos, tables, lists, dicts, etc. Here we post some images to Coop and inspect them: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/589e460e-56de-4ed1-88ad-b8128e679d5c_1952x894.png) ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/d74accfb-fa3b-4ae2-bbb3-3058bf71dc29_3102x1598.png) ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/ac4cc4f9-57d5-4292-8f04-534d4b7de687_3102x1598.png) When we create scenarios we can add any extra fields for metadata that we want to include in the survey results, without having to add it to the questions directly—e.g., we did not include placeholders in the questions for “novel” and “speaker” but we can make them columns of the results that are generated: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/3072d389-e8a5-4b3c-851a-45713337bcb0_1950x828.png) # Run with AI Next we can [design AI agents](https://docs.expectedparrot.com/en/latest/agents.html?ref=expected-parrot.ghost.io) to answer the questions and [select models](https://docs.expectedparrot.com/en/latest/language%5Fmodels.html?ref=expected-parrot.ghost.io) to generate those responses. EDSL works with many popular models; you can check details on current available models [here](https://www.expectedparrot.com/models?ref=expected-parrot.ghost.io) and see which ones are being used the most at our [cache stats](https://www.expectedparrot.com/cache-stats?ref=expected-parrot.ghost.io) page. Here we create some simple personas and select a couple vision models to use with the survey. Note that agent traits and personas can be \*much\* more detailed than this—*we’re working on lots of features for helping you generate them, please get in touch to test*—and you can specify the [model parameters](https://docs.expectedparrot.com/en/latest/language%5Fmodels.html?ref=expected-parrot.ghost.io) (here we use defaults). We add the scenarios, agents and models to the survey when we run it: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/47a33535-6b74-4fcf-a7bc-57bc04acb3e4_1950x950.png) This generates a dataset of results that we can immediately begin analyzing (see all results in the [notebook](https://www.expectedparrot.com/content/RobinHorton/pride-and-prejudice-politeness-notebook?ref=expected-parrot.ghost.io) and at [Coop](https://www.expectedparrot.com/content/RobinHorton/pride-prejudice-quotes-survey?ref=expected-parrot.ghost.io)). # Run with humans To run the same survey with humans, we start by calling the [humanize](https://docs.expectedparrot.com/en/latest/humanize.html?ref=expected-parrot.ghost.io) method to generate a web version of the survey and a **Project** for collecting responses: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/288c998e-93e6-4ed5-ad62-b6b29e929a03_1952x280.png) We can inspect and share the respondent URL: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/c00bf829-b377-4951-b78b-c54495ba966f_2234x1155.png) We can create a [study](https://docs.expectedparrot.com/en/latest/prolific.html?ref=expected-parrot.ghost.io) to launch the same survey at Prolific: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/3812b1d4-67de-4d67-a705-d2d2c9eb808c_1958x664.png) ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/675f2bed-3083-4b9a-b6fa-2cd04540e39f_3102x1598.png) Responses are automatically added to your Project page and can be pulled into your notebook for analysis: ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/ae7c02cd-ba3c-4886-a927-013fe404ad37_3102x1425.png) ![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/41bb4986-e8e9-421a-8f41-9df4193d47d1_2148x1028.png) # References Details on methods, tutorials and example code for a variety of use cases are available at our [**documentation page**](https://docs.expectedparrot.com/en/latest?ref=expected-parrot.ghost.io). A complete notebook of example code shown above is available [**here**](https://www.expectedparrot.com/content/RobinHorton/pride-and-prejudice-politeness-notebook?ref=expected-parrot.ghost.io). A slide deck version of the code is available [**here**](https://docs.google.com/presentation/d/1oenBc49ElXmXvtCnGiEE%5FqZXBYyaoTlzIg17cHUPCh0/edit?usp=sharing&ref=expected-parrot.ghost.io): [![](https://storage.ghost.io/c/bc/8c/bc8c40e2-e3d8-4888-8bb9-3276553ff8a4/content/images/2026/10/1282ece0-9650-4e49-8d1c-90888fc405d3_2926x1310.png)](https://docs.google.com/presentation/d/1oenBc49ElXmXvtCnGiEE%5FqZXBYyaoTlzIg17cHUPCh0/edit?usp=sharing&ref=expected-parrot.ghost.io) [Subscribe](#/portal/signup)