gitlab-deployment/docs/TROUBLESHOOTING.md

GitLab Troubleshooting Guide

Common Issues and Solutions

1. GitLab Not Starting

Symptoms:

• Services won't start
• Services keep crashing
• 502 Bad Gateway error

Diagnosis:

# Check service status
sudo gitlab-ctl status

# Check logs for errors
sudo gitlab-ctl tail

# Check disk space
df -h

# Check memory
free -h

Solutions:

A. Out of Memory:

# Check memory usage
free -h

# If memory is full, restart services
sudo gitlab-ctl restart

# Add swap if not present
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Consider upgrading droplet size if issue persists

B. Disk Space Full:

# Check disk usage
df -h

# Find large files
sudo du -h /var | sort -rh | head -20

# Clean up old backups
sudo find /var/opt/gitlab/backups -type f -mtime +7 -delete

# Clean up logs
sudo gitlab-ctl cleanup-logs

# Consider adding more storage or upgrading droplet

C. Services Not Starting:

# Check specific service
sudo gitlab-ctl status servicename

# View service logs
sudo gitlab-ctl tail servicename

# Restart specific service
sudo gitlab-ctl restart servicename

# Full reconfigure
sudo gitlab-ctl reconfigure

2. SSL Certificate Issues

Symptoms:

• Certificate not issuing
• HTTPS not working
• Browser shows certificate error
• Let's Encrypt failing

Diagnosis:

# Check Let's Encrypt logs
sudo gitlab-ctl tail lets-encrypt

# Check certificate status
echo | openssl s_client -servername gitlab.yourdomain.com -connect gitlab.yourdomain.com:443 2>/dev/null | openssl x509 -noout -dates

# Verify DNS is correct
dig gitlab.yourdomain.com

Solutions:

A. DNS Not Pointing to Server:

# Verify A record
dig gitlab.yourdomain.com

# Should return your droplet IP
# If not, update DNS and wait for propagation (up to 48 hours, usually 10-60 minutes)

B. Ports Not Open:

# Check firewall
sudo ufw status

# Allow HTTP and HTTPS
sudo ufw allow http
sudo ufw allow https
sudo ufw reload

C. Manual Certificate Renewal:

# Force certificate renewal
sudo gitlab-ctl renew-le-certs

# If fails, try reconfigure
sudo gitlab-ctl reconfigure

3. 502 Bad Gateway

Symptoms:

• 502 error when accessing GitLab
• Page won't load

Diagnosis:

# Check if services are running
sudo gitlab-ctl status

# Check nginx logs
sudo gitlab-ctl tail nginx

Solutions:

A. Services Starting Up: GitLab can take 5-10 minutes to fully start. Wait and refresh.

B. Services Crashed:

# Restart all services
sudo gitlab-ctl restart

# Wait 5 minutes then check status
sudo gitlab-ctl status

# If still failing, check logs
sudo gitlab-ctl tail

C. Nginx Configuration Error:

# Test nginx configuration
sudo gitlab-ctl nginx -t

# Reconfigure
sudo gitlab-ctl reconfigure

4. Email Issues

Quick Checks:

# Test SMTP connection
gitlab-rake gitlab:smtp:check

# Send test email
gitlab-rails runner "Notify.test_email('your@email.com', 'Test', 'Body').deliver_now"

# Check email logs
tail -f /var/log/gitlab/gitlab-rails/production.log | grep -i mail

See docs/EMAIL_SETUP.md for comprehensive email troubleshooting.

5. Git Push/Pull Failures

Symptoms:

• Can't push or pull
• Authentication errors
• Connection refused

Diagnosis:

# Test HTTPS git access
git clone https://gitlab.yourdomain.com/root/test.git

# Test SSH git access
ssh -T git@gitlab.yourdomain.com

Solutions:

A. SSH Key Issues:

# Check SSH keys in GitLab UI: User Settings → SSH Keys

# Test SSH connection
ssh -vT git@gitlab.yourdomain.com

# Generate new SSH key if needed
ssh-keygen -t ed25519 -C "your_email@example.com"
cat ~/.ssh/id_ed25519.pub  # Add to GitLab

B. HTTPS Authentication:

# Use personal access token instead of password
# GitLab UI: User Settings → Access Tokens → Create token

# Clone with token
git clone https://oauth2:TOKEN@gitlab.yourdomain.com/user/repo.git

6. Backup Failures

Symptoms:

• Backup script failing
• Backups not completing
• Backup files missing

Diagnosis:

# Check disk space
df -h

# Check backup logs
tail -f /var/log/gitlab_backup.log

# Try manual backup
sudo gitlab-backup create

Solutions:

A. Out of Disk Space:

# Clean old backups
sudo find /var/opt/gitlab/backups -type f -mtime +7 -delete

# Move backups to object storage
# Configure s3cmd for DO Spaces or AWS S3

B. Permissions Issues:

# Fix backup directory permissions
sudo chown -R git:git /var/opt/gitlab/backups
sudo chmod 0700 /var/opt/gitlab/backups

7. Slow Performance

Symptoms:

• GitLab is slow to load
• Git operations timeout
• High CPU or memory usage

Diagnosis:

# Check resource usage
htop

# Check disk I/O
iostat -x 1

# Check GitLab performance
sudo gitlab-rake gitlab:check

Solutions:

A. Insufficient Resources: Upgrade your droplet:

• Minimum: 4GB RAM, 2 vCPUs
• Recommended: 8GB RAM, 4 vCPUs
• For >50 users: 16GB RAM, 8 vCPUs

B. Database Issues:

# Analyze and optimize database
sudo gitlab-rake db:migrate

# Vacuum database
sudo gitlab-psql -c "VACUUM ANALYZE;"

C. Performance Tuning: Edit /etc/gitlab/gitlab.rb:

# PostgreSQL tuning
postgresql['shared_buffers'] = "256MB"
postgresql['work_mem'] = "16MB"
postgresql['maintenance_work_mem'] = "64MB"

# Sidekiq tuning
sidekiq['max_concurrency'] = 10

# Puma tuning
puma['worker_processes'] = 2
puma['max_threads'] = 4

Then reconfigure:

sudo gitlab-ctl reconfigure

8. User Can't Login

Symptoms:

• "Invalid login or password" error
• Account locked
• 2FA issues

Solutions:

A. Reset Root Password:

# Access GitLab console
sudo gitlab-rails console

# Find and reset password
user = User.where(username: 'root').first
user.password = 'newpassword'
user.password_confirmation = 'newpassword'
user.save!
exit

B. Unlock Account:

sudo gitlab-rails console

user = User.find_by(username: 'username')
user.unlock_access!
exit

C. Disable 2FA:

sudo gitlab-rails console

user = User.find_by(username: 'username')
user.disable_two_factor!
exit

9. Database Connection Issues

Symptoms:

• "Could not connect to database" error
• Database timeout errors

Solutions:

A. Restart Database:

sudo gitlab-ctl restart postgresql

B. Check Database Status:

sudo gitlab-ctl status postgresql

# Check connections
sudo gitlab-psql -c "SELECT count(*) FROM pg_stat_activity;"

C. Reset Database Connections:

sudo gitlab-rake db:migrate:status
sudo gitlab-ctl restart

Getting More Help

Check System Status

# Comprehensive check
sudo gitlab-rake gitlab:check

# Environment info
sudo gitlab-rake gitlab:env:info

# Check configuration
sudo gitlab-rake gitlab:check_config

View All Logs

# Tail all logs
sudo gitlab-ctl tail

# Specific service
sudo gitlab-ctl tail nginx
sudo gitlab-ctl tail gitlab-rails
sudo gitlab-ctl tail sidekiq
sudo gitlab-ctl tail postgresql

Useful Commands Reference

# Service Management
sudo gitlab-ctl start
sudo gitlab-ctl stop
sudo gitlab-ctl restart
sudo gitlab-ctl status

# Configuration
sudo gitlab-ctl reconfigure
sudo gitlab-ctl show-config

# Logs
sudo gitlab-ctl tail
sudo gitlab-ctl tail SERVICE_NAME

# Maintenance
sudo gitlab-ctl cleanup-logs
sudo gitlab-rake gitlab:check

# Backups
sudo gitlab-backup create
sudo gitlab-backup restore BACKUP=timestamp

# Console Access
sudo gitlab-rails console
sudo gitlab-psql

# Updates
sudo apt update
sudo apt upgrade gitlab-ce

Prevention Best Practices

1. Monitor Resource Usage

   • Set up alerts for disk space (<20% free)
   • Monitor memory usage
   • Check CPU load regularly

2. Regular Backups

   • Automate daily backups
   • Test restore procedure monthly
   • Store backups off-server

3. Keep Updated

   • Update GitLab monthly
   • Subscribe to security announcements
   • Test updates in staging first

4. Monitor Logs

   • Check logs weekly for errors
   • Set up log aggregation
   • Configure error notifications

5. Document Everything

   • Keep change log
   • Document customizations
   • Maintain runbook

Emergency Contacts

• GitLab Community Forum: forum.gitlab.com
• GitLab Documentation: docs.gitlab.com
• Digital Ocean Support: cloud.digitalocean.com/support