{"version":"2026-10-19.boundary.1","asOf":"2026-10-19","month":"2026-10","domain":{"id":"computing","title":"计算机 · 软件工程","summary":"从写小程序、修缺陷到维护和整仓迁移，最明显的变化是交接范围扩大；真实收益因任务和验收标准而异。","frontierId":"F08","sourceIds":["BC03","BC06","BC08","BC10"],"terms":[{"term":"仓库 / issue / PR","definition":"仓库是项目代码集合；issue是待解决的问题单；PR是送给人审查的代码变更，draft PR是其草稿。"},{"term":"HTTP server","definition":"接收网络请求并返回响应的小程序；此处实验任务规定了规格和测试。"},{"term":"BM25 / oracle","definition":"BM25按关键词找文件；oracle用参考修复指定相关文件，代表理想检索条件。"},{"term":"Agent框架 / token","definition":"框架组织模型、工具与运行步骤；token是模型处理的文字片段，13k约为1.3万片段。"},{"term":"技术栈 / 反例","definition":"技术栈指语言、框架等组成；反例是可执行输入，显示新旧程序行为不一致。"}]},"focusTask":null,"caseStudy":null,"caseOptions":[{"id":"C03","title":"熟悉项目维护：先看完整交付，再看生成速度","lead":"补全新代码和改熟悉仓库，是两种任务。AI生成得快，不代表你连同核对和返工也更快。"}],"tasks":[{"id":"C01","domainId":"computing","title":"从规格写出小型程序","input":"简短接口规格、语言与测试环境","output":"实现及可执行测试","human":"核对需求、边界输入、安全与发布","acceptance":["约定输入输出一致","独立边界测试通过","记录生成与返工时间"],"relation":"受控HTTP server实验直接覆盖一种小程序；代码补全产品只支持部分动作","workTaskId":null,"actualCoverage":null},{"id":"C02","domainId":"computing","title":"定位并修复仓库缺陷","input":"固定仓库版本、问题描述与复现条件","output":"补丁及回归测试","human":"审查设计约束、遗漏行为与合并","acceptance":["复现测试由失败转通过","无关行为保持","真实PR审查通过"],"relation":"SWE-bench是相近公开缺陷修复；本站私有任务的完成性未知","workTaskId":"DEV-21670","actualCoverage":null},{"id":"C03","domainId":"computing","title":"完成熟悉项目中的维护任务","input":"长期维护者选定的缺陷、功能或重构","output":"符合团队标准的可审阅PR","human":"记录准备、提示、核对、返工与接管的活动人时；机器运行、等待与历时另记，重叠活动不重复累计","acceptance":["质量相当再比较耗时","全部尝试/未完成记录","区分并行等待与人时"],"relation":"真实工作RCT，但仅代表参加研究的仓库、人群和当时工具","workTaskId":null,"actualCoverage":null},{"id":"C04","domainId":"computing","title":"迁移整个仓库的技术栈","input":"可运行旧仓库、迁移目标与行为约束。技术栈是使用的语言、框架、平台和构建方式，例如C库改写为Rust、Maven构建改为Gradle。","output":"新技术栈实现及行为保持证据","human":"架构取舍、反例审查与上线","acceptance":["旧栈确实被替换","固定行为检查全过","独立差分反例检查"],"relation":"20项迁移是限定公开任务集；不把混合成绩映射为企业项目通过率","workTaskId":null,"actualCoverage":null},{"id":"C05","domainId":"computing","title":"根据黑盒行为重建程序","input":"可执行程序、行为说明与允许的查询","output":"替代源码与构建脚本","human":"补足隐藏行为、授权与部署验收","acceptance":["接口兼容","完整隐藏测试通过","记录查询/重试与时间预算"],"relation":"ProgramBench黑盒重建；与日常已有源码的开发不同","workTaskId":null,"actualCoverage":null}],"events":[{"id":"C01-2021","taskId":"C01","availableAt":"2021-06-29","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"编辑器开始提供代码建议","kind":"product","mode":"assist","result":"Copilot技术预览能在编辑器中建议代码和函数。","aiRole":"产品提供代码行或函数候选；开发者选择、整合并检查。","config":"技术预览、编辑器上下文；开发者选择和修订建议","metric":"功能存在；无通用完成率","meaning":"从逐字编写转为接受/修改候选，但完整程序仍需验收。","limit":"发布方功能说明，没有真实项目交付成功率。","sourceIds":["BC01"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C01-2022","taskId":"C01","availableAt":"2022-09-07","eventDate":"2022年研究；具体运行日期未披露","dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"受控小程序实验显示耗时减少","kind":"field","mode":"assist","result":"95名开发者随机分组，编写JavaScript HTTP server（接收请求并返回响应的小程序）。公告报告完成率78%对70%，平均完成耗时71分钟对161分钟。","aiRole":"开发者使用代码建议完成指定HTTP server；该实验报告更短的任务耗时。","config":"相同单一任务及自动测试；开发者在AI辅助下完成","metric":"公告报告的任务完成耗时约减少55%，另列完成率；不是工作覆盖率或全体尝试人时","meaning":"这项规格明确的小程序实验提示辅助潜力，但有未完成者；真实试点需同时记录完成比例与全部人时。","limit":"发布方实验，单一任务。公告未明示均值的完成者处理；不能视为95人全完成或所有尝试的净人时收益。","sourceIds":["BC02"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C01-2023","taskId":"C01","availableAt":"2023-02-13","eventDate":"2022年同一受控实验的后续论文说明","dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"论文说明：耗时只比较完成者","kind":"caution","mode":"assist","result":"同一研究的论文明确：只有完成任务者进入71.17分钟对160.89分钟的均值比较，约减少55.8%耗时。","aiRole":"AI辅助下的已完成任务耗时更短；这个均值不等于所有尝试者的净人时收益。","config":"同一HTTP server实验；95人随机分组；论文Results说明以完成任务为条件","metric":"已完成任务者的平均完成时间；不包括所有未完成尝试的人时","meaning":"把完成率与完成者耗时分开记录；未完成、失败和返工不能从实际工作成本中消失。","limit":"此澄清来自2023公开论文，与2022公告同一实验，不能累计为独立复验或回填为2022已读全文。","sourceIds":["BC11"],"comparison":"后出的同实验方法说明；不是又一次能力进步实验","supersedes":"C01-2022","actualCoverage":null,"measurementWindow":null},{"id":"C02-2023","taskId":"C02","availableAt":"2023-10-10","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"已有真实问题集，但检索仍是瓶颈","kind":"benchmark","mode":"bounded","result":"2294题、12个Python仓库；Claude2+BM25检索13k上下文解决1.96%，oracle文件条件4.8%。","aiRole":"根据问题与检索到的仓库内容生成补丁；BM25条件仅1.96%测试全过。","config":"固定首稿；BM25在项目代码集合（仓库）中按关键词找文件，13k为约1.3万token输入（模型处理的文字片段）；oracle为参考修复指定相关文件的理想检索条件。","metric":"解决指定题目的比例；检索条件分开","meaning":"让AI先写局部补丁已可测量，完整修复当时仍很有限。","limit":"oracle使用参考补丁指示文件，1.96%不应被称为最高可达到能力。","sourceIds":["BC03"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C02-2024","taskId":"C02","availableAt":"2024-08-13","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"更可信的缺陷修复子集出现","kind":"benchmark","mode":"bounded","result":"500道人工筛选Verified题；GPT-4o+Agentless报告33.2%解决率。","aiRole":"指定Agent生成修复补丁；Verified子集33.2%通过要求的测试。","config":"500题Verified子集；gpt-4o-2024-05-13 + Agentless（组织查找和改代码的评测框架）；每框架单一种子，使用最接近文档或默认参数。","metric":"该子集测试通过比例","meaning":"开始有更扎实的范围明确缺陷修复证据，但仍需团队自己的验收。","limit":"与2023全2294题和BM25不同；不是从1.96到33.2的同条件曲线。","sourceIds":["BC04"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C02-2025","taskId":"C02","availableAt":"2025-05-19","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"产品把issue推进到待审PR","kind":"product","mode":"bounded","result":"Copilot coding agent接收问题单（issue），在环境中修改和测试，提交待审代码变更草稿（draft PR）并等待人审阅。","aiRole":"产品可后台探索仓库、修改和测试，交付待人工审阅的PR。","config":"当时GitHub Actions环境、分支规则、日志与人工审查","metric":"可用工作流；未提供统一通过率","meaning":"交接单位从代码片段扩为一个待审变更；人负责合并与发布判断。","limit":"产品发布不等于每个issue完成；当前权限配置可能不同于发布时。","sourceIds":["BC05"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C03-2025","taskId":"C03","availableAt":"2025-07-10","eventDate":"2025-02至2025-06","dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"熟悉大仓库的开发者反而更慢","kind":"caution","mode":"assist","result":"16名开发者完成246个自己项目的真实任务，允许AI时耗时增加19%（区间+2%至+39%）。","aiRole":"辅助编码、查找与迭代；在这批真实维护任务中，净耗时增加19%。","config":"2025年2–6月，主要Cursor Pro+Claude3.5/3.7；熟悉仓库、真实PR质量要求","metric":"相近质量下的任务耗时变化","meaning":"评审与返工可能吃掉编码收益；是否采用需记录全程人时。","limit":"不是所有软件工作的普遍结论；不能与2022 HTTP server实验相减。","sourceIds":["BC06"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":{"start":"2025-02","end":"2025-06","precision":"month","label":"2025年2–6月；仅月粒度，不表示每天都在观测"}},{"id":"C02-2026","taskId":"C02","availableAt":"2026-02-23","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"旧基准的成绩解释需要收紧","kind":"caution","mode":"unknown","result":"OpenAI审计138道困难Verified题，59.4%存在重大测试/描述问题，并发现部分题训练污染迹象。","aiRole":"旧基准仍保留局部补丁结果；新的审计降低用其分数推断真实修复能力的把握。","config":"针对o3多次未一致解决的困难子集；发布方审计","metric":"测量有效性；不是模型能力下降","meaning":"保留历史结果，换用新鲜或私有题验收；不要据饱和榜单外推整个工程。","limit":"138题并非随机代表全部500题；反例不否定所有历史补丁。","sourceIds":["BC08"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":"C02-2024","actualCoverage":null,"measurementWindow":null},{"id":"C03-2026","taskId":"C03","availableAt":"2026-02-24","eventDate":"2025-08起的后续研究","dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"后续看似加速，研究方认为测量仍不可靠","kind":"caution","mode":"assist","result":"后续57名开发者/800多任务；旧参与者点估计耗时−18%，新参与者−4%，两个区间均跨过零。","aiRole":"后续工具继续辅助维护；因样本选择和并行计时，当前净收益大小未可靠确定。","config":"2025年8月起；参与选择、报酬变化与并行计时影响结果","metric":"后续耗时点估计；仅弱证据","meaning":"工具和工作方式有变化，真实净人时趋势仍需新的研究设计。","limit":"研究方明确认为该数据不能可靠衡量当前增益大小；不画收益增长线。","sourceIds":["BC07"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":"C03-2025","actualCoverage":null,"measurementWindow":null},{"id":"C03-field-comparison","taskId":"C03","availableAt":"2026-02-27","eventDate":null,"dateBasis":"按所引用固定版本的公开日进入；不是首次发现或运行日","title":"不同开发情境出现任务数收益","kind":"field","mode":"assist","result":"三项企业现场实验合计4,867名开发者；随机提供代码补全工具的合并估计为完成任务数增加26.08%（标准误10.3%，表示估计的不确定性）。","aiRole":"代码补全辅助日常开发；这批研究记录任务数，未提供本任务完整净人时。","config":"Microsoft、Accenture及匿名大型企业；工具访问随机分配，三项结果有噪声和差异","metric":"完成任务数变化，不能改写为耗时下降","meaning":"与熟悉仓库的耗时实验并列，提示选择本团队任务并测审查返工。","limit":"相近开发情境，不是C03同任务重测；质量、利润与全流程人时不能由摘要推出。","sourceIds":["FS071"],"comparison":"只解释这一来源/任务/配置；不与其他题集连成增长曲线","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C05-2026","taskId":"C05","availableAt":"2026-05-05","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"黑盒重建显示另一种边界","kind":"benchmark","mode":"bounded","result":"固定首稿评测200个程序重建任务，九个系统没有完整解决任何一题。","aiRole":"系统探查黑盒并编写替代程序；固定首稿中没有完整通过的任务。","config":"9模型：Claude Opus4.7、Opus4.6、Sonnet4.6、Haiku4.5，Gemini3.1 Pro、3 Flash，GPT5.4、5.4 mini、5 mini。mini-SWE-agent框架、发布方默认参数、每模型每题一次、最多6小时/1000步；不给项目源码、不联网。","metric":"完整隐藏测试全过；不是部分测试率","meaning":"即使会写代码，接口隐藏行为重建仍需大量独立核对。","limit":"这是固定五月快照；不宣称此后所有模型仍为零。","sourceIds":["BC09"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null},{"id":"C04-2026","taskId":"C04","availableAt":"2026-08-24","eventDate":null,"dateBasis":"按该条支持材料的公开日期进入时间轴；不是实验统一发生日","title":"全仓迁移进入严格多关验收","kind":"benchmark","mode":"bounded","result":"20任务、8模型/26配置共520运行；340通过迁移审查、118全过固定行为，二者交集88，最终28全关通过。","aiRole":"系统尝试替换技术栈并保持行为；520次混合运行只有28次全关通过。","config":"模型：claude-opus-5、claude-sonnet-5、gpt-5.6-luna、gpt-5.6-sol、kimi-k3、qwen3.8-max、dsv4-flash、glm-5.2；GPT两模型用Codex，其余用Claude Code框架。各配置各任务一次，预算6–30小时，无网络（模型端点除外），隐藏验收；末关6个编码Agent各用1小时寻找原程序与新程序的可复现行为差异。","metric":"28/520混合运行全关通过；三个集合不能画成逐层340→118漏斗","meaning":"明确迁移与保持行为是两件事，有限测试全过后仍需查反例。","limit":"28/520是混合配置运行结果，不是企业项目或单模型通用通过率；全关通过表示这组验证器未找到反例，不是数学证明所有行为等价。","sourceIds":["BC10"],"comparison":"任务/配置或指标不同，不计算统一增长率","supersedes":null,"actualCoverage":null,"measurementWindow":null}],"sources":[{"id":"BC01","title":"GitHub Copilot技术预览","url":"https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/","publishedAt":"2021-06-29","kind":"产品发布","locator":"标题、技术预览与编辑器建议介绍","limit":"现页面后续更新；只取初始代码建议流程，不反推后续功能","family":"copilot-product","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC02","title":"Copilot受控开发实验","url":"https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/","publishedAt":"2022-09-07","kind":"发布方实验","locator":"Results of the experiment：95名开发者、JavaScript HTTP server任务","limit":"发布方研究，单一任务；不是整个软件生命周期的收益","family":"copilot-rct","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC03","title":"SWE-bench首稿","url":"https://arxiv.org/html/2310.06770v1","publishedAt":"2023-10-10","kind":"研究基准","locator":"§2任务；§4/表4–5 BM25与oracle条件","limit":"1.96%是Claude2+BM25结果，oracle条件4.8%；不选择性把前者说成最高能力","family":"BC03","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC04","title":"SWE-bench Verified发布","url":"https://openai.com/index/introducing-swe-bench-verified/","publishedAt":"2024-08-13","kind":"发布方基准","locator":"Our Approach；Performance on SWE-bench Verified；脚注3/4：模型版本、单种子及默认/文档参数。","limit":"500题筛选子集与原2294题不同；当前页面有2025更新","family":"swe-verified","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC05","title":"Copilot coding agent发布","url":"https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/","publishedAt":"2025-05-19","kind":"产品发布","locator":"发布标题；draft pull request；Integrated, configurable, and secure","limit":"功能说明不提供整体成功率；引用当时流程，不作为现产品权限配置说明","family":"copilot-product","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC06","title":"METR早期2025开发效率RCT","url":"https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/","publishedAt":"2025-07-10","kind":"独立工作实验","locator":"Methodology；Core Result；FAQ","limit":"16位熟悉所维护仓库的开发者/246任务；不代表所有程序员","family":"metr-productivity","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC07","title":"METR后续研究设计修正","url":"https://metr.org/blog/2026-02-24-uplift-update/","publishedAt":"2026-02-24","kind":"方法修正","locator":"首段及Wider adoption/Details of the productivity study","limit":"选择、支付和并行计时问题导致结果仅弱证据；不能据此计算可靠趋势","family":"metr-productivity","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC08","title":"SWE-bench Verified测量审计","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/","publishedAt":"2026-02-23","kind":"方法修正","locator":"Background；Too narrow and too wide tests；Contamination","limit":"59.4%针对138道困难审计题，不是全部500题；发布方审计","family":"swe-verified","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC09","title":"ProgramBench固定首稿","url":"https://arxiv.org/html/2605.03546v1","publishedAt":"2026-05-05","kind":"研究基准","locator":"§2.1任务；§3配置；§4/表2","limit":"200项黑盒重建，禁止源码/联网；固定首稿不等于动态最新榜","family":"BC09","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC10","title":"SWE Refactor Bench首稿","url":"https://arxiv.org/html/2608.23564v1","publishedAt":"2026-08-24","kind":"研究基准","locator":"§2.2任务预算；§3.1指标；§3.2/表2","limit":"28/520为8模型26配置混合运行结果，不是单一模型通用通过率","family":"BC10","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面标示发布日期或固定预印本v1页首日期","status":"active"},{"id":"BC11","title":"Copilot受控实验：论文方法澄清","url":"https://arxiv.org/html/2302.06590v1","publishedAt":"2023-02-13","kind":"发布方实验论文","locator":"§2 Experiment Design；§3 Results，conditioning on completing the task","limit":"与2022公告同一实验家族；均值仅完成者，不是全体尝试净收益","family":"copilot-rct","readAt":"2026-10-14","readingScope":"本轮直接阅读所列原文位置；未复跑实验","dateBasis":"页面/论文首稿标示公开日期","status":"active"},{"id":"FS071","title":"开发者三项现场实验（正式期刊版本）","url":"https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535","publishedAt":"2026-02-27","kind":"现场随机实验论文","locator":"Abstract；Article Information / Published Online；Microsoft Research对应记录","limit":"复用现站FS071同一研究；4,867开发者，完成任务数指标，不是耗时/质量；正式版本日不是首次披露日","family":"developer-three-field-rcts","readAt":"2026-10-15","readingScope":"本轮直接核对所列原文位置/版本；未复跑外部实验","dateBasis":"本次引用期刊Published Online；机构记录已有2025年6月公开摘要","status":"active"}],"scope":"16个编辑选择的代表任务切片；不是完整职业清单或任务覆盖比例。","actualCoverage":null,"knowledgeRule":"主轴按原始材料公开日期；实验/上线时点另列。历史空白仅表示本次未收录证据；后来的质疑不偷偷改写早期结果。","method":{"coverage":"覆盖的含义：在指定输入、环境和验收下，材料支持AI参与哪些动作；未知比例保持null。","speed":"速度的含义：证据节点之间的日历间隔、流程交接变化；指标/样本/预算不一致时，不计算能力增长率或外推一年三年。","sourcePolicy":"综述用于发现任务和原始文献；每条公开主张引用原始论文/官方记录。转载及同一文献家族不累计成独立验证。","selection":"目的性选择三个领域的代表任务，不是穷尽检索或无偏职业采样；基础时间轴于2026-10-14核对，本次精选补充与证据截止2026-10-15，固定论文优先，不把动态榜排名倒填历史。"}}