Troubleshooting#

Checking Agent Status#

Use the built-in status command to get a comprehensive overview of the agent:

Windows (PowerShell as Administrator):

.\senhub-agent.exe status

Linux:

sudo /opt/senhub/bin/senhub-agent status

The status command displays:

• Service status (Running / Stopped)
• Agent version and build information
• Health status (healthy / degraded / unhealthy)
• Operating mode (online / offline)
• Memory and CPU usage
• Number of active probes and cached metrics
• Uptime

You can also check the service directly using system commands:

Windows:

Get-Service "SenHub Agent"

Linux:

sudo systemctl status senhub-agent

Verifying Agent Health#

Health Endpoint#

The health endpoint is the quickest way to verify the agent is running and responding:

curl http://localhost:8080/health

Expected response:

{
  "status": "ok",
  "version": "0.1.80",
  "uptime": "2h30m15s",
  "probes_active": 4,
  "metrics_cached": 156
}

If the agent does not respond, check that the service is running and the port is not blocked.

System Information#

For detailed system information (requires authentication key):

curl http://localhost:8080/api/{key}/info/system

Response:

{
  "status": "running",
  "version": "0.1.80",
  "os": "linux",
  "arch": "amd64",
  "port": 8080,
  "uptime": "2h30m15s",
  "health": {
    "status": "healthy",
    "services": {
      "http_server": "running",
      "cache": "running",
      "mode": "offline"
    }
  },
  "cache": {
    "total_metrics": 156,
    "ttl": "5m0s",
    "memory_usage": "2.45 MB"
  },
  "resources": {
    "memory_usage_mb": 45.67,
    "cpu_percent": 2.5,
    "goroutines": 42
  }
}

Probes Status#

To check which probes are active and how many metrics each is collecting:

curl http://localhost:8080/api/{key}/info/probes

Response:

{
  "probes": ["CPU", "Memory", "Citrix Production"],
  "probe_metrics": {
    "CPU": 4,
    "Memory": 3,
    "Citrix Production": 85
  },
  "total_metrics": 92
}

If a probe shows 0 metrics, it may be encountering errors. Check the logs for details.

Viewing Logs#

Agent logs are stored in dedicated directories:

Windows:

%ProgramData%\SenHub\logs\senhubagent.log

Typically: C:\ProgramData\SenHub\logs\senhubagent.log

Linux:

/var/log/senhub/senhubagent.log

If the Linux log directory is not writable, logs are written to the agent binary directory.

Log rotation: 10 MB maximum per file, 5 backup files, 30-day retention, compressed. Old log files are named senhubagent-YYYY-MM-DD.log.gz.

Viewing Recent Logs#

Windows (PowerShell):

Get-Content "C:\ProgramData\SenHub\logs\senhubagent.log" -Tail 100

To follow logs in real-time:

Get-Content "C:\ProgramData\SenHub\logs\senhubagent.log" -Tail 50 -Wait

Linux:

tail -100 /var/log/senhub/senhubagent.log

To follow logs in real-time:

tail -f /var/log/senhub/senhubagent.log

You can also view logs via systemd journal on Linux:

sudo journalctl -u senhub-agent -n 100

Understanding Log Format#

Log entries are structured in JSON format:

{"level":"info","module":"probe.citrix","time":"2025-03-03T10:30:00Z","message":"Collection completed: 85 metrics"}
{"level":"error","module":"probe.netscaler","time":"2025-03-03T10:30:05Z","message":"Connection refused","error":"dial tcp 192.168.1.100:443: connect: connection refused"}

Key fields:

Filtering Logs#

To find errors:

Linux:

grep '"level":"error"' /var/log/senhub/senhubagent.log | tail -20

Windows (PowerShell):

Select-String -Path "C:\ProgramData\SenHub\logs\senhubagent.log" -Pattern '"level":"error"' | Select-Object -Last 20

To find logs for a specific probe:

grep '"module":"probe.citrix"' /var/log/senhub/senhubagent.log | tail -20

Enabling Debug Logging#

Runtime Debug (No Restart Required)#

You can enable debug logging for specific modules at runtime via the API. This is the recommended approach as it does not require restarting the service:

curl -X POST http://localhost:8080/api/{key}/debug/logs \
  -H "Content-Type: application/json" \
  -d '{"module_levels": [{"module": "probe.citrix", "level": "debug"}]}'

You can enable debug for multiple modules at once:

curl -X POST http://localhost:8080/api/{key}/debug/logs \
  -H "Content-Type: application/json" \
  -d '{"module_levels": [
    {"module": "probe.citrix", "level": "debug"},
    {"module": "probe.netscaler", "level": "debug"},
    {"module": "strategy.http", "level": "debug"}
  ]}'

To check current log levels for all modules:

curl http://localhost:8080/api/{key}/debug/logs

Response:

{
  "module_levels": [
    {"module": "strategy.http", "level": "info"},
    {"module": "probe.citrix", "level": "debug"},
    {"module": "probe.netscaler", "level": "debug"},
    {"module": "cache", "level": "info"}
  ]
}

To revert a module to normal logging:

curl -X POST http://localhost:8080/api/{key}/debug/logs \
  -H "Content-Type: application/json" \
  -d '{"module_levels": [{"module": "probe.citrix", "level": "info"}]}'

Available Debug Modules#

Console Debug Mode#

For troubleshooting startup issues or when the API is not available, run the agent interactively in verbose mode:

All modules (verbose):

senhub-agent run --authentication-key KEY --verbose

Specific modules only:

senhub-agent run --authentication-key KEY --verbose --debug-modules probe.citrix,strategy.http

This runs the agent in the foreground (not as a service) and outputs detailed logs to the console. Press Ctrl+C to stop. This is useful for:

• Diagnosing startup failures
• Verifying credentials and connectivity to target systems
• Debugging probe collection in detail
• Identifying configuration issues

Common Problems#

Agent Does Not Start#

Symptom: The service fails to start or exits immediately after starting.

Possible causes and solutions:

Configuration Changes Not Applied#

Symptom: Changes to agent-config.yaml are not reflected in agent behavior.

Possible causes and solutions:

• YAML syntax error in the modified section: The agent keeps the previous valid configuration when it encounters a syntax error. Check logs for configuration reload errors.
• Config file path mismatch: Verify the agent is monitoring the correct file. Check with senhub-agent status or logs.
• File not saved: Ensure the editor saved the file (some editors use temporary files).

The agent detects file changes automatically within a few seconds. No restart is required.

Probe Not Collecting Metrics#

Symptom: A probe is configured but returns 0 metrics or does not appear in the probes list.

Possible causes and solutions:

Enable debug logging for the specific probe module to get detailed error information:

curl -X POST http://localhost:8080/api/{key}/debug/logs \
  -H "Content-Type: application/json" \
  -d '{"module_levels": [{"module": "probe.citrix", "level": "debug"}]}'

API Not Accessible#

Symptom: curl http://localhost:8080/health returns “Connection refused” or times out.

Possible causes and solutions:

• Service not running: Check with senhub-agent status and start if needed
• Firewall blocking the port: See Firewall Configuration in the HTTP/HTTPS section
• Agent bound to a different interface: Check bind_address in the storage config. If set to 127.0.0.1, the API is only accessible from localhost
• Different port configured: Check the port in the storage config section of agent-config.yaml
• HTTPS enabled: If HTTPS is enabled, use https:// instead of http://

TLS / HTTPS Errors#

Symptom: HTTPS connections fail with certificate errors or TLS handshake failures.

Possible causes and solutions:

License Errors#

Symptom: A probe type is rejected with a license error, or premium probes are not available.

Possible causes and solutions:

• No license activated: Check with senhub-agent license show. Free tier only allows cpu, memory, logicaldisk, network.
• License expired: Check the expiration date. There is a 7-day grace period after expiration. Contact support for renewal.
• Probe not in license tier: Verify the probe type is included in your tier. See the License Tiers table in the Configuration section.

Check the full license status via the API:

curl http://localhost:8080/api/{key}/license/status

Response for an expired license:

{
  "status": "grace_period",
  "tier": "pro",
  "expires_at": "2026-01-01T00:00:00Z",
  "days_remaining": 5,
  "message": "License expired but in grace period (5 days remaining)"
}

PRTG Sensor Shows No Data#

Symptom: PRTG sensor is configured but shows “No data” or errors.

Possible causes and solutions:

• Wrong sensor type: Use HTTP Data Advanced (not HTTP XML/REST Value or other types)
• Wrong URL: Verify the URL format: /api/{key}/prtg/metrics/{probe-name}
• Probe name mismatch: Check available probe names with curl http://localhost:8080/api/{key}/prtg/probes
• Authentication key invalid: Verify the key in the URL matches the agent’s key
• Agent not reachable from PRTG server: Test with curl from the PRTG server itself
• Missing lookups: Install PRTG Lookups for status metrics (see Web Interface section)

High Memory Usage#

Symptom: The agent uses more memory than expected.

Possible causes and solutions:

• Too many probes: Each probe maintains its own cache. Reduce the number of probes or increase the collection interval.
• Cache retention too long: Reduce cache.retention_minutes in the configuration.
• Debug logging enabled: Debug logging generates more data. Disable debug logging when not needed.

Check current memory usage:

curl http://localhost:8080/api/{key}/info/system

Clear the cache if needed:

curl -X POST http://localhost:8080/api/{key}/admin/cache/clear

Diagnostic Checklist#

When troubleshooting, follow these steps in order:

1. Service running? senhub-agent status
2. Health OK? curl http://localhost:8080/health
3. Probes active? curl http://localhost:8080/api/{key}/info/probes
4. Errors in logs? Check the last 50 lines of the log file for errors
5. Configuration valid? Validate YAML syntax
6. Network reachable? Test connectivity from the agent host to target systems
7. License active? senhub-agent license show
8. Enable debug for the relevant module and check detailed logs

Getting Support#

If you cannot resolve the issue:

• Email: support@senhub.io
• Include in your support request:
  • Agent version (senhub-agent version)
  • Operating system and version
  • Relevant log entries (last 50 lines with errors)
  • Your agent-config.yaml (with passwords removed)
  • The output of senhub-agent status