Witness Troubleshooting

Receipt verification returns valid:false

Cause: One or more verification checks failed.

Fix:

  1. Check which specific check failed in the checks object:

    • signature_valid: false — Ed25519 signature verification failed
    • content_hash_matches: false — Log has been modified
    • chain_link_valid: false — Chain hash doesn’t match computed value

  2. Common causes:

    • Modified receipt — use the complete unmodified receipt from the notarize endpoint
    • Wrong public key — ensure you’re using the public key from the receipt itself
    • JSON serialization differences — content_hash is computed from JSON.stringify(receipt.log)

Quick test: Verify the receipt server-side first before debugging offline verification:

curl -X POST https://api.solenoid.systems/v1/witness/verify \
  -H "Authorization: Bearer sm_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d @receipt.json

Storage status is “buffered” instead of “delivered”

Cause: S3 storage is unavailable or not configured. Receipts are safely held in purgatory (R2).

Fix:

  1. Check if S3 is configured:

    # Try notarizing again - 202 means purgatory fallback
curl -X POST https://api.solenoid.systems/v1/witness/log \
  -H "Authorization: Bearer sm_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{"log": {"test": true}}'

  2. If you haven’t configured S3, call POST /v1/witness/config with credentials

  3. If S3 is configured but receipts are buffered:

    • Check S3 credentials are still valid
    • Verify S3 endpoint is accessible (no firewall/network issues)
    • Check S3 bucket permissions (PutObject required)

  4. Receipts in purgatory will be delivered automatically when S3 recovers

Note: Buffered receipts are NOT lost. They’re stored in R2 and delivered when S3 is available.

Chain not initialized error

Cause: Attempting to notarize logs before calling POST /v1/witness/sys/init.

Fix:

curl -X POST https://api.solenoid.systems/v1/witness/sys/init \
  -H "Authorization: Bearer sm_your_api_key_here"

Initialization is one-time per account. If you’ve already initialized, you’ll get success: false but can proceed to notarize.

Client-side verification fails but server works

Cause: Local verification implementation differs from server.

Diagnosis:

  1. Verify content_hash computation:

    const logJson = JSON.stringify(receipt.log);
const logBytes = new TextEncoder().encode(logJson);
const contentHash = await crypto.subtle.digest('SHA-256', logBytes);
const contentHashHex = Array.from(new Uint8Array(contentHash))
  .map(b => b.toString(16).padStart(2, '0')).join('');
console.log('Computed:', contentHashHex);
console.log('Receipt:', receipt.content_hash);

  2. Verify chain_hash computation:

    const prevHashBytes = hexToBuffer(receipt.prev_hash);
const sequenceIdBytes = new Uint8Array(8);
new DataView(sequenceIdBytes.buffer).setBigUint64(0, BigInt(receipt.sequence_id), false);
const contentHashBytes = hexToBuffer(receipt.content_hash);

const concatenated = new Uint8Array(72);
concatenated.set(prevHashBytes, 0);      // offset 0, 32 bytes
concatenated.set(sequenceIdBytes, 32);   // offset 32, 8 bytes
concatenated.set(contentHashBytes, 40);  // offset 40, 32 bytes

const chainHash = await crypto.subtle.digest('SHA-256', concatenated);

  3. Verify Ed25519 key import:

    const publicKey = await crypto.subtle.importKey(
  'jwk',
  receipt.public_key,
  { name: 'Ed25519' },
  false,
  ['verify']
);

  4. Use the client verifier as a reference implementation

S3 endpoint blocked

Cause: SSRF protection blocks private IPs and localhost.

Fix:

  • Use public HTTPS S3 endpoints only:

    • AWS: https://s3.{region}.amazonaws.com
    • R2: https://{account}.r2.cloudflarestorage.com
    • MinIO: https://{public-domain}

  • Do NOT use:

    • http:// (HTTPS required)
    • Private IPs (10.x, 172.16.x, 192.168.x)
    • Localhost (127.0.0.1, ::1)
    • Link-local addresses

202 Accepted response

Cause: This is normal behavior when S3 is unavailable. Not an error.

Fix: None needed. The receipt is valid and stored in purgatory. It will be delivered to S3 when available.

When to investigate:

  • If ALL notarizations return 202 for extended periods (hours)
  • Check S3 configuration and credentials
  • Verify S3 endpoint is accessible from Cloudflare Workers

Rate limit errors (429)

Cause: Exceeded your tier’s request limit. See Rate Limits.

Fix:

  • Implement exponential backoff on 429 responses
  • Cache public keys locally (don’t fetch on every verification)
  • Use server-side verification to avoid rate limits on public endpoints

Getting help

  1. Check status.solenoid.systems
  2. Review Witness Errors for error code details
  3. Contact support@solenoid.systems with your chain ID, sequence IDs, and error messages