[prompt-analysis] Copilot PR Prompt Analysis - May 20, 2026

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - May 20, 2026

Summary

Analysis Period: Last 30 days (Apr 20 - May 20, 2026)
Total PRs: 1,000 | Merged: 806 (80.6%) | Closed: 187 (18.7%) | Open: 7 (0.7%)

Key Finding: Success rate has improved to 80.6%, up from 79.3% on May 12 and marking the highest rate in the past month. Bug fix prompts show the strongest success rate at 82.1%.

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Bug Fix	857	704	82.1% ⭐
Test	792	646	81.6%
Feature Addition	913	740	81.1%
Refactoring	263	211	80.2%
Documentation	603	479	79.4%
Security	114	90	78.9%
Other	5	3	60.0%

Note: Categories are not mutually exclusive - a single PR may be tagged with multiple categories based on title and body content.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 988 words (detailed but focused)
• File references: 78% include specific file paths
• Code blocks: 81% include code examples or snippets
• Top action verbs: add, update, test, fix, create

Key success factors:

1. Specificity matters: Prompts with file references have 78% inclusion in merged PRs
2. Context-rich: Including code blocks correlates with 81% of merged PRs
3. Clear objectives: Using concrete action verbs (add, fix, update) rather than vague terms
4. Comprehensive detail: Nearly 1000 words on average suggests thorough problem/solution description

Example successful prompts:

1. PR Add OTLP data quality validator workflow for end-to-end telemetry integrity checks #33623: Add OTLP data quality validator workflow for end-to-end telemetry integrity checks

   • Prompt preview: "This adds a new gh-aw workflow that validates OTLP telemetry quality across traces, metrics, and logs from app emission through Collector and backend visibility..."
   • Why it worked: Specific feature addition, clear scope, detailed technical context
   • → MERGED

2. PR Normalize report formatting guidelines across 7 agentic workflows #33596: Normalize report formatting guidelines across 7 agentic workflows

   • Prompt preview: "Eight active workflows generating user-facing reports lacked consistent formatting guidelines for header hierarchy and progressive disclosure..."
   • Why it worked: Identified specific problem across multiple workflows, concrete solution
   • → MERGED

3. PR fix: guard create_discussion against PLACEHOLDER-only bodies #33595: fix: guard create_discussion against PLACEHOLDER-only bodies

   • Prompt preview: "The safe-output-health workflow submitted a discussion with a body of literally "PLACEHOLDER" (11 chars), consuming 30.4M tokens..."
   • Why it worked: Bug fix with clear reproduction, quantified impact, specific function target
   • → MERGED

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 1,179 words (20% longer than merged PRs!)
• File references: 79% include file paths (similar to merged)
• Code blocks: 76% include code snippets (slightly lower)
• Top action verbs: add, update, test, fix, resolve

Issues identified:

1. Over-specification: Closed PRs average 20% more words but lower success - suggests excessive detail may obscure core intent
2. Alignment issues: Several closed PRs addressed schema/specification misalignments that were already resolved or not needed
3. Verification gaps: Some PRs confirmed non-issues (e.g., "no code delta" PRs)

Example unsuccessful prompts:

1. PR Align main workflow schema with parser allowlist by adding top-level applyTo and inputs #33594: Align main workflow schema with parser allowlist by adding top-level applyTo and inputs

   • Prompt preview: "The parser accepts applyTo and inputs in shared/main workflow frontmatter, but main_workflow_schema.json did not define them..."
   • Why it failed: Schema alignment issue that may have been superseded or unnecessary
   • → CLOSED

2. PR Confirm CLI short-description punctuation consistency (no code delta) #33575: Confirm CLI short-description punctuation consistency (no code delta)

   • Prompt preview: "The reported consistency issue (trailing periods in cobra.Command.Short text) is no longer reproducible..."
   • Why it failed: Verification-only PR with no actual changes needed
   • → CLOSED

3. PR Serialize create_issue handler execution to eliminate shared-state race conditions #33574: Serialize create_issue handler execution to eliminate shared-state race conditions

   • Prompt preview: "create_issue kept mutable Maps (temporaryIdMap, dedup caches, parentIssueCache) in handler scope while allowing concurrent async invocations..."
   • Why it failed: Complex concurrency fix that may have had implementation issues or was addressed differently
   • → CLOSED

Key Insights

1. Conciseness paradox: Merged PRs average 988 words while closed PRs average 1,179 words - suggesting that focused detail beats exhaustive specification. Too much information may dilute the core intent.

2. Bug fixes win: The highest success rate (82.1%) belongs to bug fix prompts, likely because they have clear success criteria (does it fix the bug?) versus features which may face scope/design debates.

3. Verification work is risky: PRs that confirm "no issue exists" or "already compliant" tend to get closed - consider checking current state before creating a PR for verification-only work.

Recommendations

Based on today's analysis, here are actionable guidelines for writing successful Copilot prompts:

DO:

1. Be specific about files and functions: 78% of merged PRs reference specific code locations
2. Include code context: 81% of merged PRs include code blocks showing current/desired state
3. Keep it focused: Target ~1,000 words - enough detail but not overwhelming
4. Frame as fixes when possible: Bug fix prompts have 82.1% success vs 60% for "other"
5. Use concrete action verbs: add, fix, update, implement rather than vague terms

AVOID:

1. Overly long specifications: Beyond 1,000 words, additional detail may hurt rather than help
2. Verification-only PRs: If confirming "no changes needed", consider investigating first
3. Schema/spec alignment without confirmation: Check if the issue still exists before creating PR
4. Vague or exploratory scopes: "Improve performance" is less successful than "Reduce API latency by caching X"

Historical Trends

7-Day Comparison (completed PRs only):

Date	Total	Merged	Success Rate	Trend
May 04	1000	771	78.0%	↔️
May 06	1000	780	78.0%	↔️
May 07	1000	777	77.7%	↓
May 11	1000	786	79.1%	↑
May 12	1000	793	79.3%	↑
May 20	1000	806	80.6%	↑↑

30-Day Range: 77.7% - 80.6% (2.9 percentage point range)

Trend Analysis: Success rate has steadily improved over the past week, from 77.7% on May 7 to 80.6% today - a +2.9 percentage point improvement. This suggests prompt quality or CI robustness is improving.

Category Performance Over Time

Bug Fix consistently performs best:

• May 11: 80.0%
• May 12: 80.0%
• May 20: 82.1% ⬆️ (new high)

Feature Addition shows steady improvement:

• May 12: 75.0%
• May 20: 81.1% ⬆️ (+6.1 pp)

Test category strong and stable:

• Recent range: 78.6% - 100%
• May 20: 81.6% (solid)

Methodology Notes

• Data source: GitHub search API for is:pr repo:github/gh-aw head:copilot/* created:>=2026-04-20
• Outcome classification:
  • Merged = mergedAt is not null
  • Closed = state is "CLOSED" and mergedAt is null
  • Open = state is "OPEN"
• Category assignment: Keyword-based from title and body (non-exclusive)
• Prompt extraction: Full PR body text used as prompt content
• Sample size: 1,000 PRs analyzed (806 merged, 187 closed, 7 open)

---

Next Steps: Consider automating prompt quality checks pre-PR using these patterns, or adding a "prompt template" system based on the successful patterns identified here.

Generated by 🔍 Copilot PR Prompt Pattern Analysis · ● 4.5M · ◷

• expires on May 21, 2026, 9:22 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-21T21:22:50.843Z.

Closed by Workflow