--- title: "TACACS+ Integration on NVIDIA Cumulus Linux with Cisco ISE" slug: "tacacs-integration-on-nvidia-cumulus-linux-with-cisco-ise" description: "Learn about TACACS+ on Cumulus Linux for centralized AAA, SSH logins, and user management without local accounts. Maximize stability with version 5.11." updated: 2026-10-30T22:10:29Z published: 2026-10-30T22:10:29Z canonical: "docs.vastdata.com/tacacs-integration-on-nvidia-cumulus-linux-with-cisco-ise" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # TACACS+ Integration on NVIDIA Cumulus Linux with Cisco ISE ## 1. What is TACACS+ on Cumulus? Cumulus Linux implements TACACS+ as a client for centralized AAA (Authentication, Authorization, and Accounting) of administrative access to the switch. It uses PAM and NSS to integrate TACACS+ transparently into the Linux login system no local accounts need to be created for TACACS+ users. **What TACACS+ controls:** - SSH logins - Console access - `sudo` command execution - NVUE (`nv set`, `nv show`, `nv apply`) commands **What TACACS+ does NOT control:** - Data plane traffic forwarding through the switch - Routing protocol operation - OpenTelemetry (OTEL) telemetry export see [Section 9](/develop/docs/tacacs-integration-on-nvidia-cumulus-linux-with-cisco-ise#9-otel-telemetry-is-not-affected-by-tacacs) > **Note:** The TACACS+ client is only supported through the management interface (`eth0`) or management VRF. It is not supported through bond, SVI, or `swp` interfaces. --- ## 2. Version Support Matrix | Version | Status | NVUE TACACS+ | Notable Changes | | --- | --- | --- | --- | | 5.5 | ✅ GA | ✅ GA | **Practical minimum** for NVUE-based TACACS+ | | 5.11 | ✅ ESR | ✅ GA | **ESR/LTS** recommended for stable production; minimum for SN5600 (800G) | | 5.12 | ✅ GA | ✅ GA | Added explicit `nv set system aaa tacacs enable` command | | 5.13 | ✅ GA | ✅ GA | Minimum version for SN5610 switches (5.13.1 specifically) | | 5.14 | ✅ GA | ✅ GA | Default VRF support for per-command authorization | | 5.15 | ✅ GA | ✅ GA | ⚠️ **Breaking syntax change** server config and auth order commands changed | | 5.16 | ✅ Latest | ✅ GA | NVUE tab completion for per-command auth; max 8 servers | > **Recommendation:** Use **5.11 (ESR)** for maximum stability or **5.16 (latest)** for the most features. > **Warning:** Upgrading from 5.14 to 5.15+ requires updating TACACS+ NVUE command syntax. `exclude-users` changed to `exclude-user` (singular), and server config syntax changed. Update any automation scripts before upgrading. ### 5.15 Syntax Change Reference | Setting | 5.14 and earlier | 5.15+ | | --- | --- | --- | | Add server | `nv set system aaa tacacs server 5 host <IP>` | `nv set system aaa tacacs server <IP> priority 5` | | Set secret | `nv set system aaa tacacs server 5 secret 'key'` | `nv set system aaa tacacs server <IP> secret 'key'` | | Auth order | `nv set system aaa authentication-order 5 tacacs` | `nv set system aaa authentication order tacacs local` | | Enable | `nv set system aaa tacacs enable on` | *(no longer required separately)* | | Exclude user | `nv set system aaa tacacs exclude-users <user>` | `nv set system aaa tacacs exclude-user <user>` | --- ## 3. How Authentication Works ``` User SSH login → Cumulus PAM → TACACS+ server (port 49) → ISE/AAA server │ Identity lookup (AD/LDAP) │ priv_lvl attribute returned │ Privilege level enforced on switch ``` **Key behaviors:** - Authentication uses PAM and includes login, SSH, sudo, and su - NSS maps TACACS+ users to local privilege-level accounts (`tacacs0` through `tacacs15`) - No need to create local accounts for TACACS+ users - Accounting records are sent to all configured TACACS+ servers by default - Up to **8 TACACS+ servers** supported (5.16+); 7 in earlier versions --- ## 4. Privilege Levels The TACACS+ server returns a `priv_lvl` attribute (0–15) after authentication. Cumulus enforces access based on this value. | Privilege Level | Access on Cumulus Linux | | --- | --- | | **15** | Full sudo access + all NVUE commands (`nv set`, `nv unset`, `nv apply`, `nv show`) | | **1 – 14** | `nv show` commands only no configuration changes, no sudo | | **0** | Shell built-ins only used with per-command authorization (restricted shell) | > **Note:** The privilege level is configured on the TACACS+ server (e.g., ISE Shell Profile), not on the Cumulus switch. The switch receives and enforces the level returned by the server. --- ## 5. Cumulus Linux Configuration ### Required Settings ``` # TACACS+ server syntax for Cumulus 5.15+ nv set system aaa tacacs server priority 5 nv set system aaa tacacs server secret '' # VRF use mgmt for out-of-band management interface nv set system aaa tacacs vrf mgmt # Authentication order TACACS+ first, local fallback nv set system aaa authentication order tacacs local # Exclude local accounts from TACACS+ (always authenticate locally) nv set system aaa tacacs exclude-user cumulus # Enable accounting nv set system aaa tacacs accounting state enabled nv config apply ``` > **Note on priority:** The `priority` value controls which server is tried first when multiple servers are configured. It is **not** the TACACS+ privilege level. Lower number = higher priority. ### Multiple Servers ``` nv set system aaa tacacs server priority 5 nv set system aaa tacacs server priority 10 nv config apply ``` ### Verify Configuration ``` nv show system aaa tacacs nv show system aaa tacacs server ``` ### Optional Parameters ``` # Change timeout (default: 5 seconds) nv set system aaa tacacs timeout 10 # Change port (default: 49) nv set system aaa tacacs server port 49 # Set source IP for TACACS+ packets (useful for ACLs on ISE) nv set system aaa tacacs source-ip # Change authentication mode (default: pap) nv set system aaa tacacs authentication mode chap # Send accounting to first responding server only (default: all) nv set system aaa tacacs accounting send-records first-response ``` ### Test Connectivity ``` # Verify reachability ping -I eth0 # Verify port 49 is open nc -zv 49 # Expected: Connection to 49 port [tcp/tacacs] succeeded! ``` > **Warning:** `Connection refused` on port 49 means the Device Admin Service is not enabled on ISE. This is the most common initial issue. --- ## 6. Cisco ISE Configuration ### Prerequisites on ISE 1. **Enable Device Admin Service** on the PSN node: `Administration → System → Deployment → [node] → Enable Device Admin Service` *(Without this, ISE will not listen on port 49)* 2. **Join Active Directory** (if using AD authentication): `Administration → Identity Management → External Identity Sources → Active Directory` 3. **Import AD Groups** to use as authorization conditions: AD join point → Groups tab → Add → Select Groups from Directory ### Add the Cumulus Switch as a Network Device `Work Centers → Device Administration → Network Resources → Network Devices → Add` | Field | Value | | --- | --- | | Name | descriptive name for the switch | | IP Address | management IP of the Cumulus switch | | Device Type | your device type (e.g. CUMULUS_SWITCH) | | TACACS Authentication Settings | ✅ Checked | | Shared Secret | must match the secret configured on Cumulus | | Enable Single Connect Mode | ❌ Unchecked (not supported on Cumulus) | ### Create a TACACS Shell Profile `Work Centers → Device Administration → Policy Elements → Results → TACACS Profiles → Add` | Field | Value | | --- | --- | | Name | e.g. `Cumulus_Admin` | | Default Privilege | `15` (for full admin) | | Maximum Privilege | `15` | ### Create a Device Admin Policy Set `Work Centers → Device Administration → Device Admin Policy Sets` 1. Add a new policy set with a condition matching your network devices (e.g. by Device Type) 2. Set Allowed Protocols to `Default Device Admin` 3. Open the policy set and configure: **Authentication Policy:** - Default rule → Identity Source: your AD join point **Authorization Policy (add above the default deny rule):** | Field | Value | | --- | --- | | Condition | AD ExternalGroups EQUALS `<your-domain>/<your-admin-group>` | | Shell Profile | `Cumulus_Admin` (priv 15) | > **Note:** ISE 3.x does not require an Essentials/Advantage/Premier tier license for TACACS+ Device Administration. Only the Device Admin license (`L-ISE-TACACS-ND=`) one per PSN is required. --- ## 7. Local User Behavior with TACACS+ Enabled When TACACS+ is enabled, local users and TACACS+ users coexist on the switch. ### Authentication Order ``` nv set system aaa authentication order tacacs local ``` | Scenario | Result | | --- | --- | | TACACS+ server reachable | All logins (except excluded users) go to ISE | | TACACS+ server unreachable | Automatically falls back to local accounts | | User in `exclude-user` list | Always authenticates locally | | User not found on TACACS+ server | Falls back to local accounts | ### Local vs TACACS+ Users Compared | Aspect | TACACS+ Users | Local Users | | --- | --- | --- | | Authentication | Validated by ISE / AD | Local `/etc/passwd` | | Password management | Managed in AD or ISE | Managed on switch | | Privilege level | Set by ISE (0–15) | Set by local role | | Accounting | Full command logging to TACACS+ server | Local syslog only | | Home directory | Mapped to `tacacs0`–`tacacs15` | Own home directory | ### Excluding Users from TACACS+ Always exclude accounts that should never go to TACACS+: ``` # Exclude the default admin account nv set system aaa tacacs exclude-user cumulus # Exclude a monitoring service account nv set system aaa tacacs exclude-user nv config apply ``` > **Critical:** Always keep at least one local break-glass account excluded from TACACS+. If ISE becomes unreachable and all accounts go through TACACS+, no one can log in to the switches. ### Known Limitation: Simultaneous Logins If two TACACS+ users log in simultaneously with the **same privilege level**, they share the same mapped local UID (e.g., both map to `tacacs15`). Files and processes are attributed to both users. This is an inherent Linux TACACS+ limitation. TACACS+ accounting on the server side tracks the real username regardless. --- ## 8. Monitoring User Best Practice Automated monitoring scripts that SSH into Cumulus switches should use a **dedicated local account** with minimum required privileges not a TACACS+ user and not the `cumulus` admin account. ### Why a Local Account - Monitoring must continue even if ISE is unreachable - Avoids generating large volumes of TACACS+ accounting records from automated polling - Easier to audit and scope to exactly the commands needed ### Minimum Required Access for Common Monitoring Tasks | Command | Purpose | Access Required | | --- | --- | --- | | `nv show system` | Firmware version, hostname, system info | NVUE read-only (`nv show`) | | `/usr/cumulus/bin/decode-syseeprom` | Serial number, part number | Local execution, no sudo | | `nv show interface` | Interface state and counters | NVUE read-only | | `nv show system aaa` | AAA configuration | NVUE read-only | ### Create the Monitoring User ``` # Create local user with read-only NVUE role nv set system aaa user role nvue-monitor nv set system aaa user password # Exclude from TACACS+ always authenticate locally nv set system aaa tacacs exclude-user nv config apply ``` The built-in `nvue-monitor` role grants: - ✅ All `nv show` commands - ✅ Read-only system commands including `decode-syseeprom` - ❌ No `nv set`, `nv unset`, or `nv apply` - ❌ No sudo access ### User Account Summary | Account | Type | Auth | Purpose | | --- | --- | --- | --- | | `cumulus` | Local | Always local | Break-glass emergency access | | `<monitoring-user>` | Local | Always local | Automated monitoring scripts | | AD admin users | TACACS+ | Via ISE → AD | Network engineers full admin | | AD read-only users | TACACS+ | Via ISE → AD | NOC / read-only access | --- ## 9. OTEL Telemetry is Not Affected by TACACS+ TACACS+ and OpenTelemetry (OTEL) operate on completely separate planes of the switch. They are independent, use different protocols, and enabling or configuring TACACS+ has zero effect on OTEL operation. ### Comparison | | TACACS+ | OTEL | | --- | --- | --- | | **What it does** | Controls who can administer the switch | Exports switch metrics for observability | | **Plane** | Management plane | Data/network plane | | **Protocol** | TCP port 49 → TACACS+ server | gRPC/OTLP → OpenTelemetry collector | | **Triggered by** | SSH login, CLI commands | Continuous scheduled collection | | **Auth required** | Yes user identity | No system service | | **Consumers** | Security / audit teams | Prometheus, Grafana, observability stacks | ### What OTEL Collects (Always Independent of TACACS+) - Interface statistics counters, errors, drops, utilization - Buffer statistics ingress/egress occupancy and watermarks - Histogram data queue depth and latency - Platform statistics CPU, memory, temperature, fans, PSUs - Routing metrics BGP peer state, route table counts - Control plane statistics control plane packet counters - ACL statistics packet/byte counts per rule (5.15+) - AI Ethernet statistics adaptive routing, SRv6, packet trimming (5.14+) - Systemd service health statistics ### OTEL Hardware Support | ASIC | OTEL Support | Minimum Cumulus Version | | --- | --- | --- | | Spectrum-1 | ✅ | 5.11 | | Spectrum-2 | ✅ | 5.11 | | Spectrum-3 | ✅ | 5.11 | | Spectrum-4 (SN5600, SN5610) | ✅ | 5.10 | ### Basic OTEL Setup (Unchanged by TACACS+) ``` # Enable OTEL completely independent of TACACS+ state nv set system telemetry enable on nv set system telemetry export otlp state enabled nv set system telemetry export otlp grpc destination port 4317 nv set system telemetry export otlp grpc insecure enabled nv set system telemetry export vrf mgmt # Enable specific metric categories nv set system telemetry interface-stats egress-buffer traffic-class 3 nv set system telemetry interface-stats export state enabled nv set system telemetry interface-stats ingress-buffer priority-group 3 nv set system telemetry interface-stats sample-interval 30 nv set system telemetry interface-stats switch-priority 3 nv set system telemetry label cluster description nv set system telemetry label switch_hostname description nv set system telemetry label switch_ip description nv config apply ``` > **Warning:** Do **not** enable OTEL and gNMI streaming simultaneously, they conflict with each other. This restriction is unrelated to TACACS+. --- ## 10. Accounting Accounting is enabled by default when TACACS+ is configured. All commands run by TACACS+ users generate records sent to the configured server(s). ### What Gets Logged - All Linux commands - All NVUE commands - Login and logout events - Sub-processes spawned by commands - Sudo commands (logged against the original TACACS+ login name) ### What Does NOT Get Logged - Inline editor actions (vi, nano file edits) - vtysh internal actions - Commands longer than 240 bytes are truncated (TACACS+ protocol limit) ### Accounting Commands ``` # Enable accounting nv set system aaa tacacs accounting state enabled # Disable accounting nv set system aaa tacacs accounting state disabled # Send to first responding server only (default: all servers) nv set system aaa tacacs accounting send-records first-response # Send to all servers (default) nv set system aaa tacacs accounting send-records all nv config apply ``` --- ## 11. Per-Command Authorization Per-command authorization allows granular control over which commands specific privilege levels can run. It operates at two layers: ### Layer 1 Cumulus Switch (Coarse) On Cumulus, authorization is enforced at the **initial word** of the command only: ``` # Allow privilege level 0 to run ip and nv commands nv set system aaa tacacs authorization 0 command ip nv set system aaa tacacs authorization 0 command nv nv config apply ``` `nv` permits **all** NVUE commands beginning with `nv`. For more granular control of specific sub-commands, use ISE. ### Layer 2 ISE (Granular) Configure TACACS Command Sets in ISE for fine-grained per-command control: `Work Centers → Device Administration → Policy Elements → Results → TACACS Command Sets` Example command sets: - **Permit show only:** permit `nv show`, deny everything else - **Permit interface config:** permit `nv set interface`, deny `nv set system` Attach command sets to authorization rules in your Device Admin Policy Set. > **Note:** As of Cumulus 5.16, per-command authorization supports NVUE tab completion and `?` option listing. --- ## 12. Troubleshooting ### Common Issues | Symptom | Likely Cause | Fix | | --- | --- | --- | | `Connection refused` on port 49 | Device Admin Service not enabled on ISE | `Administration → System → Deployment → Enable Device Admin Service` | | `tac_connect_single: getpeername failed` | Network cannot reach ISE on port 49 | Check routing/firewall; test with `nc -zv <ISE-IP> 49` | | Password prompt appears but login fails | Wrong password, user not found, or auth policy not matching | Check ISE Live Logs for exact reason | | `incorrect shared secret` in syslog | Shared secret mismatch between switch and ISE | Verify secret is identical on both sides case-sensitive | | Login succeeds but all commands denied | Authorization policy returning deny or wrong profile | Check ISE Live Logs verify AD group membership and rule match | | Local fallback not working | `cumulus` not in `exclude-user` list | Add `nv set system aaa tacacs exclude-user cumulus` | ### Debug on Cumulus ``` # Watch TACACS+ events in real time tail -f /var/log/syslog | grep -i # Check audit records cat /var/log/audit/audit.log # Debug per-command authorization (run as TACACS+ user) export TACACSAUTHDEBUG=1 # Restart audit daemon if accounting records are missing sudo systemctl restart auditd.service # Notify audisp plugin after config changes sudo killall -HUP audisp-tacplus # Add debug logging to accounting config # Edit /etc/audisp/audisp-tac_plus.conf → add: debug=1 ``` ### Debug on ISE **Operations → TACACS → Live Logs** shows every authentication and authorization attempt in real time with the exact pass/fail reason, matched policy, and applied shell profile. --- ## 13. Reference ### Key Files on Cumulus | File | Purpose | | --- | --- | | `/etc/tacplus_servers` | TACACS+ server config (Linux/file method) | | `/etc/tacplus_nss.conf` | NSS lookup configuration and timeouts | | `/etc/audisp/audisp-tac_plus.conf` | Accounting plugin configuration | | `/var/log/syslog` | TACACS+ authentication and error logs | | `/var/log/audit/audit.log` | Command accounting audit records | | `/etc/pam.d/` | PAM configuration files | | `/etc/nsswitch.conf` | Name Service Switch lookup order | ### NVUE Quick Reference ``` # Show TACACS+ config nv show system aaa tacacs # Show auth order nv show system aaa # Show server status nv show system aaa tacacs server # Show monitoring user nv show system aaa user ``` ### Documentation Links - **Cumulus Linux TACACS+ (5.16):** [https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-516/System-Configuration/Authentication-Authorization-and-Accounting/TACACS/](https://www.cisco.com/c/en/us/td/docs/security/ise/3-5/admin_guide/b_ise_admin_3_5/b_ISE_admin_device_admin.html) - **Cisco ISE Device Administration:** [https://www.cisco.com/c/en/us/td/docs/security/ise/3-4/admin_guide/b_ise_admin_3_4/b_ISE_admin_Device_Administration.html](https://www.cisco.com/c/en/us/td/docs/security/ise/3-4/admin_guide/b_ise_admin_3_4/b_ISE_admin_Device_Administration.html) - **Cisco ISE Licensing Guide:** [https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/ise-licensing-guide-og.html](https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/ise-licensing-guide-og.html) - **Cumulus Linux Support Policy:** [https://docs.nvidia.com/networking-ethernet-software/knowledge-base/Support/Support-Offerings/Cumulus-Linux-Release-Versioning-and-Support-Policy/](https://docs.nvidia.com/networking-ethernet-software/knowledge-base/Support/Support-Offerings/Cumulus-Linux-Release-Versioning-and-Support-Policy/) --- ### Author by: Dima Kaputkin