protocol-common Git Command Guide

The Git protocol common elements define the shared functionality, data structures, and communication patterns used across all Git network protocols. Understanding these common elements is essential for implementing Git servers and debugging protocol-level issues.

Git Protocol Fundamentals:

Packet Format:

Git Protocol Packets:
├── Length prefix: 4 hex digits + LF (e.g., "001e\n")
├── Data payload: variable length
├── Packet termination: 0000 (flush packet)
└── Side-band channels: 1=data, 2=progress, 3=error

Example packet:
001ahello world\n0000
├── 001a = length (26 bytes)
├── hello world\n = data
└── 0000 = flush

Reference Format:

Reference Lines:
<SHA-1> <ref-name> [capabilities]
<SHA-1> <ref-name>
...

Example:
a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s refs/heads/main
b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s refs/heads/develop

Object ID Format:

SHA-1 Hash: 40 hexadecimal characters
SHA-256: 64 hexadecimal characters (future)

Examples:
SHA-1:  a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s
SHA-256: a665c0c5b6b8d7e7f7b9c9b9e9b9e9b9e9b9e9b9e9b9e9b9e9b9e9b9e9b9e9b9

Common Protocol Elements:

Capability Advertisement:

Server Capabilities:
agent=<agent>
ls-refs
fetch=<options>
server-option
object-info
bundle-uri
object-format=<format>
session-id=<id>
symref=<target>:<name>

Error Handling:

Error Packets:
ERR <error-message>

Examples:
ERR repository not found
ERR authentication required
ERR insufficient permissions

Progress Reporting:

Progress Messages:
side-band: 000eside-band data
progress:  Counting objects: 123
progress:  Compressing objects: 100% (50/50)

Reference Discovery:

Initial Reference Advertisement:

# Client request
git ls-remote <url>

# Server response
a1b2c3d4... refs/heads/main
b2c3d4e5... refs/heads/develop
c3d4e5f6... refs/tags/v1.0
0000

Symref Handling:

Symbolic References:
symref=HEAD:refs/heads/main
ref: refs/heads/main HEAD

Hidden References:

Hidden Ref Filtering:
unborn HEAD handling
filtered ref advertisement
capability-based ref visibility

Object Transfer Protocol:

Want/Have Negotiation:

Client → Server:
want <SHA-1>
want <SHA-1>
have <SHA-1>
have <SHA-1>
done

Server → Client:
ACK <SHA-1> [common]
NAK

Pack File Transfer:

Pack Stream:
PACK
<signature>
<version>
<object-count>
<objects...>
<trailer>

Side Band Protocol:

Multi-channel Transfer:
Channel 1: Pack data
Channel 2: Progress messages
Channel 3: Error messages

Packet format: \1<data> \2<progress> \3<error>

Authentication and Security:

Credential Handling:

Authentication Methods:
├── HTTP Basic Auth
├── SSH public key
├── Personal access tokens
├── OAuth tokens
└── Kerberos tickets

Transport Security:

Security Layers:
├── SSL/TLS encryption (HTTPS)
├── SSH host verification
├── Certificate validation
├── Man-in-the-middle protection
└── Credential encryption

Access Control:

Permission Models:
├── Repository-level access
├── Branch-level permissions
├── Tag creation rights
├── Force push restrictions
└── Administrative controls

Error Conditions and Recovery:

Common Error Scenarios:

Network Errors:
├── Connection timeout
├── DNS resolution failure
├── SSL certificate issues
├── Firewall blocking
└── Proxy configuration

Protocol Errors:
├── Malformed packets
├── Invalid capabilities
├── Reference conflicts
├── Object corruption
└── Capability mismatches

Recovery Strategies:

Error Recovery:
├── Automatic retry with backoff
├── Fallback to different transports
├── Partial transfer resumption
├── Reference validation
└── Repository integrity checking

Performance Optimization:

Connection Management:

Connection Optimization:
├── Connection pooling
├── Keep-alive handling
├── Parallel transfers
├── Compression negotiation
└── Bandwidth throttling

Caching Strategies:

Protocol Caching:
├── Reference advertisement caching
├── Object existence checking
├── Capability negotiation caching
├── DNS resolution caching
└── SSL session resumption

Bandwidth Optimization:

Transfer Optimization:
├── Delta compression
├── Thin pack usage
├── Shallow clone support
├── Partial clone filtering
└── Bundle URI support

Debugging Protocol Issues:

Packet-Level Tracing:

# Enable protocol tracing
GIT_TRACE_PACKET=1 git clone <url>

# Trace with timestamps
GIT_TRACE_PACKET=2 git fetch origin

# Debug specific operations
GIT_CURL_VERBOSE=1 git push origin main

Capability Analysis:

# Check server capabilities
git ls-remote --upload-pack="git upload-pack --advertise-capabilities" <url>

# Test capability negotiation
echo "0000" | git upload-pack --stateless-rpc <url> | head -10

# Debug version negotiation
GIT_TRACE=1 git clone --verbose <url> 2>&1 | grep -i "version\|capability"

Network Diagnostics:

# Test basic connectivity
telnet github.com 9418  # Git protocol
curl -I https://github.com  # HTTPS

# Check DNS and routing
traceroute github.com
dig github.com

# SSL certificate verification
openssl s_client -connect github.com:443 -servername github.com

Implementing Custom Git Services:

Basic Protocol Handler:

#!/bin/bash
# Basic Git protocol handler

handle_git_request() {
    local service="$1"
    local repo_path="$2"

    case "$service" in
        "git-upload-pack")
            # Handle clone/fetch
            git upload-pack "$repo_path"
            ;;
        "git-receive-pack")
            # Handle push
            git receive-pack "$repo_path"
            ;;
        *)
            echo "Unsupported service: $service"
            exit 1
            ;;
    esac
}

# Usage
handle_git_request "git-upload-pack" "/path/to/repo.git"

Advanced Protocol Implementation:

#!/bin/bash
# Advanced Git protocol server

git_protocol_server() {
    local repo_path="$1"

    # Read initial client request
    read -r line

    # Parse service request
    if [[ "$line" == "git-upload-pack "* ]]; then
        service="upload-pack"
        repo="${line#git-upload-pack }"
    elif [[ "$line" == "git-receive-pack "* ]]; then
        service="receive-pack"
        repo="${line#git-receive-pack }"
    else
        echo "ERR Unsupported service"
        exit 1
    fi

    # Validate repository access
    if ! validate_repo_access "$repo" "$service"; then
        echo "ERR Access denied"
        exit 1
    fi

    # Handle the request
    case "$service" in
        "upload-pack")
            handle_upload_pack "$repo"
            ;;
        "receive-pack")
            handle_receive_pack "$repo"
            ;;
    esac
}

validate_repo_access() {
    local repo="$1"
    local service="$2"
    # Implement access control logic
    return 0  # Placeholder
}

handle_upload_pack() {
    local repo="$1"
    # Implement upload-pack logic
    git upload-pack "$repo"
}

handle_receive_pack() {
    local repo="$1"
    # Implement receive-pack logic
    git receive-pack "$repo"
}

Custom Capability Server:

# Server with custom capabilities
custom_capability_server() {
    local repo="$1"

    # Advertise custom capabilities
    echo "version 2"
    echo "capability-advertisement"
    echo "fetch"
    echo "server-option"
    echo "custom-feature"
    echo "custom-backup"
    echo ""

    # Handle client commands
    while read -r line; do
        case "$line" in
            "command=fetch")
                handle_custom_fetch "$repo"
                ;;
            "command=custom-backup")
                handle_custom_backup "$repo"
                ;;
            "done")
                break
                ;;
        esac
    done
}

handle_custom_fetch() {
    local repo="$1"
    # Custom fetch implementation
    echo "Custom fetch operation"
}

handle_custom_backup() {
    local repo="$1"
    # Custom backup implementation
    echo "Custom backup operation"
}

Configuration and Tuning:

Protocol Configuration:

# Configure protocol behavior
git config protocol.version 2                    # Use protocol v2
git config protocol.allow always                 # Allow all protocols
git config protocol.allow never "file"           # Disable file protocol

# Transport-specific settings
git config http.postBuffer 524288000             # 500MB buffer
git config http.maxRequests 100                  # Connection pooling
git config ssh.variant ssh                        # SSH variant

Performance Tuning:

# Optimize transfer performance
git config pack.window 10                         # Delta search window
git config pack.depth 50                          # Delta chain depth
git config pack.threads 0                         # Auto-detect threads

# Network optimization
git config core.compression 9                     # Maximum compression
git config transfer.fsckObjects false             # Skip fsck for speed
git config receive.fsckObjects false              # Skip receive fsck

Security Configuration:

# Configure security settings
git config transfer.fsckObjects true              # Enable transfer fsck
git config receive.fsckObjects true               # Enable receive fsck
git config receive.denyDeletes true               # Deny delete operations
git config receive.denyDeleteCurrent true         # Deny current branch delete

Troubleshooting Common Issues:

Protocol Version Conflicts:

# Force specific protocol version
GIT_PROTOCOL=version=1 git clone <url>
GIT_PROTOCOL=version=2 git clone <url>

# Check version support
git ls-remote --upload-pack="git upload-pack --advertise-capabilities" <url> | grep "version"

Capability Negotiation Failures:

# Debug capability issues
GIT_TRACE=1 git clone <url> 2>&1 | grep -i "capability\|feature"

# Test specific capabilities
git ls-remote --upload-pack="git upload-pack --advertise-capabilities" <url>

# Check client capabilities
git config --list | grep protocol

Network and Transport Issues:

# Diagnose connection problems
timeout 10 git ls-remote <url>

# Test different transports
git clone git://github.com/user/repo.git
git clone https://github.com/user/repo.git
git clone ssh://git@github.com/user/repo.git

# Debug proxy issues
export GIT_CURL_VERBOSE=1
export GIT_TRACE_CURL=1

Object Transfer Problems:

# Debug pack transfer issues
GIT_TRACE_PACKET=1 git fetch origin

# Check pack integrity
git verify-pack .git/objects/pack/*.pack

# Test object transfer
git cat-file -e <object-id>  # Check object existence

Caution

Git protocol common elements are fundamental to Git’s network operation. Changes can break compatibility. Test thoroughly before implementing custom protocol handlers.

Real-World Usage Examples:

Custom Git Proxy Implementation:

#!/bin/bash
# Git protocol proxy for caching and filtering

git_protocol_proxy() {
    local upstream_url="$1"
    local cache_dir="$2"

    # Read client request
    read -r line

    # Parse service and repository
    if [[ "$line" == git-upload-pack* ]]; then
        service="upload-pack"
        repo="${line#git-upload-pack }"
    elif [[ "$line" == git-receive-pack* ]]; then
        service="receive-pack"
        repo="${line#git-receive-pack }"
    else
        echo "ERR Invalid request"
        exit 1
    fi

    # Check cache first
    cache_key="$(echo "$upstream_url$repo" | sha256sum | cut -d' ' -f1)"
    cache_file="$cache_dir/$cache_key"

    if [ -f "$cache_file" ] && [ -n "$GIT_PROXY_CACHE" ]; then
        echo "Serving from cache: $cache_file"
        cat "$cache_file"
        exit 0
    fi

    # Proxy to upstream
    echo "Proxying to upstream: $upstream_url"
    case "$service" in
        "upload-pack")
            git upload-pack "$upstream_url$repo" | tee "$cache_file"
            ;;
        "receive-pack")
            git receive-pack "$upstream_url$repo"
            ;;
    esac
}

# Usage
git_protocol_proxy "https://github.com/" "/tmp/git-cache"

Enterprise Protocol Gateway:

# Enterprise Git protocol gateway
enterprise_protocol_gateway() {
    local internal_url="$1"
    local external_url="$2"

    echo "Enterprise Git Protocol Gateway"
    echo "Internal: $internal_url"
    echo "External: $external_url"

    # Read and validate client request
    read -r line

    # Apply enterprise policies
    if ! apply_security_policies "$line"; then
        echo "ERR Security policy violation"
        log_security_event "policy_violation" "$line"
        exit 1
    fi

    # Route to appropriate backend
    if is_internal_client; then
        echo "Routing to internal server"
        proxy_to_backend "$internal_url" "$line"
    else
        echo "Routing to external server"
        proxy_to_backend "$external_url" "$line"
    fi
}

apply_security_policies() {
    local request="$1"

    # Implement security checks
    # - Rate limiting
    # - Access control
    # - Content filtering
    # - Audit logging

    return 0  # Placeholder
}

is_internal_client() {
    # Check client IP, certificates, etc.
    return 0  # Placeholder
}

proxy_to_backend() {
    local backend_url="$1"
    local request="$2"

    # Implement backend proxying
    echo "Proxying to $backend_url"
}

log_security_event() {
    local event="$1"
    local details="$2"

    echo "$(date): $event - $details" >> security-audit.log
}

# Usage
enterprise_protocol_gateway "https://internal.git.company.com/" "https://github.com/company/"

Protocol Analysis and Monitoring:

# Analyze Git protocol traffic
analyze_protocol_traffic() {
    local log_file="$1"

    echo "Git Protocol Traffic Analysis"
    echo "============================"

    # Count protocol versions
    echo "Protocol versions:"
    grep -o "version [0-9]" "$log_file" | sort | uniq -c | sort -nr

    # Analyze capabilities requested
    echo ""
    echo "Capabilities requested:"
    grep -o "cap-[a-zA-Z0-9_-]*" "$log_file" | sort | uniq -c | sort -nr | head -10

    # Count operations by type
    echo ""
    echo "Operations by type:"
    grep -o "command=[a-zA-Z0-9_-]*" "$log_file" | sed 's/command=//' | sort | uniq -c | sort -nr

    # Performance metrics
    echo ""
    echo "Performance metrics:"
    grep -o "duration=[0-9.]*" "$log_file" | sed 's/duration=//' | awk '
        {sum+=$1; count++} END {
            if(count>0) {
                print "Average duration:", sum/count, "seconds"
                print "Total operations:", count
                print "Total time:", sum, "seconds"
            }
        }
    '

    # Error analysis
    echo ""
    echo "Error analysis:"
    grep -i "err\|error" "$log_file" | head -10
}

# Usage
analyze_protocol_traffic "/var/log/git-server.log"

Tip

Understand Git protocol common elements for implementing custom servers and debugging network issues. Protocol fundamentals are shared across all Git transports.

What’s the difference between Git protocol v1 and v2?

V1 uses text-based negotiation with stateful connections; V2 uses binary command structure with stateless connections, better performance and cleaner architecture.

How do Git packets work?

Packets start with 4 hex digits indicating length, followed by data, terminated by 0000 flush packets. Side-band channels allow progress and error reporting.

What are Git capabilities?

Capabilities are features advertised by servers that clients can use. They control available operations like shallow clones, side-band progress, and atomic pushes.

How does reference discovery work?

Client requests initial reference advertisement, server responds with SHA-1/ref-name pairs and capabilities, client uses this for want/have negotiation.

What is the want/have negotiation?

Client tells server what objects it wants (want) and what it has (have), server finds common base and sends minimal pack with differences.

How do side-band channels work?

Git uses channel 1 for pack data, channel 2 for progress messages, channel 3 for errors. Allows progress reporting during long transfers.

What are symrefs in Git protocol?

Symbolic references like HEAD pointing to actual refs. Server advertises symref=HEAD:refs/heads/main to tell client where HEAD points.

How does Git handle authentication?

Different transports handle authentication differently: HTTP uses Basic Auth, SSH uses public keys, Git protocol typically anonymous.

What is a flush packet?

0000 packet that terminates a section of protocol communication. Used to separate reference advertisement from pack data.

How do Git servers advertise capabilities?

In initial response after client request, server sends capability list as space-separated values in first reference line.

What is the object-info capability?

Allows clients to query object information without full clone. Useful for checking object existence and types.

How does bundle-uri work?

Server can advertise bundle URIs for clients to download pack files from CDNs or caches, reducing server load.

What is session-id for?

Unique session identifier for tracking and debugging protocol interactions across multiple requests.

How do Git servers handle errors?

Send ERR packets with error messages, or use side-band channel 3 for error reporting during transfers.

What is the difference between stateless and stateful protocols?

Stateless protocols (v2) don’t require server to maintain state between requests; stateful protocols (v1) maintain connection state.

How does Git optimize pack transfers?

Uses delta compression, thin packs, and negotiates common objects to send only differences between repositories.

Applications of Git protocol common elements

1. Custom Server Implementation: Build specialized Git servers using shared protocol elements
2. Network Debugging: Diagnose and resolve Git network communication issues using protocol knowledge
3. Proxy and Gateway Development: Implement Git proxies and enterprise gateways
4. Protocol Analysis: Monitor and analyze Git network traffic for performance optimization
5. Security Implementation: Add authentication and access control to Git services
6. Transport Development: Create new Git transport methods using protocol fundamentals