Can AI Detect AI Generated Code in GitHub? The Reality

How AI Code Detection Actually Works

1. Token Perplexity and Log-Probability Analysis

• Perplexity scoring: The detector runs the source code through a reference LLM and calculates how "surprised" the model is by each variable name, function call, and control flow.
  
  
• Entropy thresholds: Human developers frequently introduce idiosyncratic naming, quirky loop logic, and non-standard spacing that produce sudden spikes in entropy.
  
  
• The structural flaw: Well-written, idiomatic code (like standard React hooks or Go boilerplate) naturally has low perplexity, causing massive false positives on experienced human coders.
  
  

2. AST Fingerprinting and Structural Regularity

• Tree depth and branching: LLMs trained on standard libraries favor textbook structural patterns with balanced branching and predictable nesting levels.
  
  
• Canonical construct preference: AI models consistently generate orthodox control structures (e.g., opting for explicit for...of iteration over obscure pointer arithmetic or clever bitwise hacks).
  
  
• Dead-code absence: Raw LLM outputs rarely contain dead variables, abandoned draft routines, or orphaned imports that routinely clutter human work branches.
  
  

3. Stylistic and Semantic Artifacts

• Over-documenting standard patterns: AI models frequently insert textbook docstrings explaining self-explanatory standard library calls.
  
  
• Defensive boilerplate: Models over-index on generic try/catch wrappers and redundant null/undefined validation checks.
  
  
• Variable naming verbosity: Instead of terse variables like i, buf, or ctx, synthetic code leans heavily toward ultra-descriptive identifiers like processedUserPayloadList.
  
  

4. Client-Side Telemetry and Provenance Tracking

• IDE-level attestation: Tracking whether code was pasted from an external window, generated via inline tab-completion, or typed keystroke-by-keystroke.
  
  
• Git Notes integration: Storing cryptographic attestations directly inside Git metadata to declare authorship origin at generation time.
  
  
• Keystroke dynamics: Measuring typing rhythm and burst rates to mathematically separate 80-WPM human typing from instantaneous 500-token paste events.
  
  

Pro Tip: If you want to spot synthetic pull requests without automated tooling, inspect the comments and commit messages. LLMs notoriously write immaculate, grammatically flawless docstrings for trivial 4-line helper functions, yet omit context on complex architectural race conditions.

The Core Technical Dilemma: Code vs. Natural Language

       Natural Language (High Freedom)             Source Code (Strict Freedom)
       ┌─────────────────────────────┐             ┌─────────────────────────────┐
       │ - Billions of valid phrasing│             │ - Rigid grammar & AST rules │
       │ - Subjective rhythm/style   │    VS       │ - Compiler rejects variance │
       │ - High statistical entropy  │             │ - Idiomatic code looks AI   │
       └─────────────────────────────┘             └─────────────────────────────┘

# Is this human code following standard conventions, or GPT-4o output?
def calculate_moving_average(data: list[float], window_size: int) -> list[float]:
    if window_size <= 0 or window_size > len(data):
        return []
    
    averages = []
    current_sum = sum(data[:window_size])
    averages.append(current_sum / window_size)
    
    for i in range(len(data) - window_size):
        current_sum += data[i + window_size] - data[i]
        averages.append(current_sum / window_size)
        
    return averages

Comparing AI Code Detection Approaches

Real-World Testing: What Breaks AI Code Detectors?

Test 1: The Raw Generation Baseline

• The Code: An asynchronous rate-limiter built using Redis and TypeScript, generated entirely via Claude Code in a single prompt.
  
  
• The Result: Heuristic scanners flagged the file with an 88% probability score, triggered primarily by exhaustive JSDoc annotations and defensive parameter validations.
  
  
• The Reality: The tool successfully identified the code, but only because the raw prompt output was committed without any human cleanup.
  
  

Test 2: Stripping Comments and Minifying Identifiers

• The Code: The exact same Redis rate-limiter, with docstrings removed and variable names shortened to standard engineering conventions (e.g., rateLimiterBucket renamed to bucket).
  
  
• The Result: The AI detection score collapsed from 88% down to 24%.
  
  
• The Reality: A basic 30-second manual cleanup completely blinded the token-probability classifier.
  
  

Test 3: The Hybrid Refactor

• The Code: Core business logic generated by an AI assistant, then wired into an existing codebase by a senior engineer who altered the method signatures and injected custom error handling.
  
  
• The Result: Detection tools classified the file as 100% human-authored.
  
  
• The Reality: Real software development is collaborative. Once an engineer edits 15% of a synthetic snippet, AST boundaries blur beyond statistical recovery.
  
  

Pro Tip: If your engineering team must enforce AI auditing, stop spending budget on standalone post-commit AI scanners. Focus your CI pipeline on strict static analysis (like Semgrep or SonarQube) paired with branch commit velocity checks. Bad code is dangerous regardless of whether an LLM or an exhausted developer wrote it.

Hands-On Workflow: Auditing GitHub Repositories for AI Code

                     Pull Request Opened
                              │
                              ▼
                ┌───────────────────────────┐
                │ Analyze Commit Velocity   │
                │ & File Change Spikes      │
                └─────────────┬─────────────┘
                              │
                              ▼
                ┌───────────────────────────┐
                │ Run Static Analysis &     │
                │ AST Anomaly Scanner       │
                └─────────────┬─────────────┘
                              │
                              ▼
                ┌───────────────────────────┐
                │ Check Dependency Registry │
                │ (Hallucination Detection) │
                └─────────────┬─────────────┘
                              │
                              ▼
                  Triaged Review Status

Step 1: Flagging Unnatural Commit Velocities

# Calculate average lines changed per commit in the current pull request
COMMITS_COUNT=$(git rev-list --count origin/main..HEAD)
LINES_CHANGED=$(git diff --shortstat origin/main..HEAD | awk '{print $4+$6}')
AVG_LINES_PER_COMMIT=$((LINES_CHANGED / COMMITS_COUNT))

if [ "$AVG_LINES_PER_COMMIT" -gt 450 ]; then
  echo "::warning title=High Velocity PR::Average commit density exceeds typical human baseline ($AVG_LINES_PER_COMMIT lines/commit)."
fi

Step 2: Scanning for Hallucinated Dependencies

name: Dependency Hallucination Check
on: [pull_request]

jobs:
  verify-deps:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Verify npm package existence
        run: |
          # Extract newly added packages from package.json diff
          git diff origin/main..HEAD package.json | grep '+   "' | awk -F'"' '{print $2}' | while read pkg; do
            if [ ! -z "$pkg" ]; then
              STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://registry.npmjs.org/$pkg)
              if [ "$STATUS" -eq 404 ]; then
                echo "CRITICAL: Package $pkg does not exist on npm registry! Potential AI hallucination."
                exit 1
              fi
            fi
          done

Step 3: Enforcing AST Linting and Complexity Gates

• Configure strict linter rules (such as ESLint or Flake8) to block pull requests containing unreferenced parameters, duplicate imports, or excessive nesting.
  
  
• Require unit test coverage thresholds on all new code paths to guarantee functional validity, regardless of the author.
  
  

Honest Limitations: Why Detectors Keep Falling Behind

Fast Evolution of Frontier Coding Models

The Paraphrasing and Refactoring Dilemma

The False Positive Threat to Junior Developers

Practical FAQs

Can GitHub natively tell if I used Copilot to write my pull request?

Can automated tools detect AI-generated code in compiled binaries?

Why do open-source maintainers care about detecting AI code?

Is watermarking code possible like it is for images?

The Verdict