Quick troubleshooting procedure
An agent may not be able to connect to the NetBeez server for various reasons, such as:
- The agent is not powered on:
- Check the status lights on hardware agents
- The agent can't resolve the server's address (FQDN)
- Ping the server
- A network firewall is blocking the agent
- Telnet to the server port 20018
- The server doesn't have a valid license or is maxed out
- Check the license in the NetBeez settings
- The agent is faulty
- Open an RMA case with NetBeez support
Detailed troubleshooting procedure
Step 1 - Agent is powered up and shows activity (Hardware agents only*)
* Skip this step if you are troubleshooting a software/virtual agent.
This step applies to hardware sensors. If you are troubleshooting a software or virtual agent, verify that the host or virtual appliance is powered up.
If you are troubleshooting a Fast Ethernet and WiFi beez
These sensors are based on the Raspberry Pi platform. The Raspberry Pi has two LEDs:
Power Led - Solid, it signals that the unit is receiving power via power supply or PoE.
Activity Led - Blinking based on SD card activity.
A unit where the red led is solid, but the green one doesn’t show any activity for around 10 seconds may be faulty. In this case, open an RMA to support@netbeez.net.
If you are troubleshooting a GigE beez
Gigabit Ethernet sensors are based on the Intel Up-Board platform. The Up-Board has one blue LED that is visible from the top of the case. You can verify that the blue light is one, and eventually that there is network activity on the Ethernet interface:
A unit where the blue light is not visible, or the Ethernet doesn't show any activity, may be faulty. In this case, open an RMA to support@netbeez.net.
Step 2 - Get troubleshooting information by initiating an ssh session
If there is routing between your machine and the agent and port 22 is not blocked and you know the agent's IP, you can get useful troubleshooting information if you try to initiate an ssh session with the agent. By doing that you will get an output similar to this one:
$> ssh 172.31.0.85
nb_troubleshoot.sh: Tue May 19 13:00:18 EDT 2020
nb_troubleshoot.sh: Agent version: 1.0.0
nb_troubleshoot.sh: Linux Kernel: Linux nbagent 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux$
nb_troubleshoot.sh: OS: Debian GNU/Linux 7.11 (wheezy)
nb_troubleshoot.sh: eth0: MAC: b8:27:eb:31:42:4a, IP: 172.31.0.14, status: up, configuration: static, type: wired
nb_troubleshoot.sh: iface eth0 inet static
nb_troubleshoot.sh: address 172.31.0.14
nb_troubleshoot.sh: netmask 255.255.255.0
nb_troubleshoot.sh: gateway 172.31.0.1
nb_troubleshoot.sh: dns-nameservers 8.8.8.8
nb_troubleshoot.sh: Host: wifi.netbeez.net
nb_troubleshoot.sh: Unencrypted port: 20019 (default route:heartbeat detected, eth0:open)
nb_troubleshoot.sh: Encrypted port: 20018 (heartbeat:detected on default route, eth0:open)
nb_troubleshoot.sh: Dashboard port: 443 (eth0:open)
nb_troubleshoot.sh: Websocket port: 443 (eth0:open)
You don't need to have any credentials or login username and password or key to get this output. You should be able to get information by using any kind of ssh utility (e.g. putty) or console.
Step 3 - Is the network configured to enable agent-server communication?
For an agent to successfully connect to the dashboard, the network must be configured to resolve the server's Fully Qualified Domain Name (FQDN) and allow the agent to establish a TCP connection to port 20018. If you are able to ssh to a NetBeez agent you can perform the following two tests. If you are not, you'd be able to get the most reliable information if you performed these steps on a host in the same subnet with the agent in question.
1- Verify that the agent is able to resolve the server's fully qualified domain name (FQDN):
ping <server_FQDN>
2- Verify with telnet that no network firewalls are blocking the agent's connection to the NetBeez server by using any of the following two commands:
telnet <server_FQDN> 20018
nc -zv <server_FQDN> 20018
Step 4 - Is the agent receiving a DHCP lease?
Check the logs of your DHCP server and verify that the server offers an IP address to the agent’s MAC address. The agent's MAC address can be located on the case.
Step 5 - Check the NetBeez license file
Check your license file by clicking the NetBeez Settings gear icon in the upper right corner and then the License tab.
Check if you are licensed for the agent you trying to connect to your dashboard. For example, if you are licensed for three virtual agents and you already have three virtual agents connected, then any additional virtual agents trying to connect to your dashboard will be rejected.
If you would like to replace a virtual agent with a new one, you must delete an existing virtual agent and then spin up the new virtual agent. For instructions for deleting an agent, please see the following article:
Agent Configuration - Delete Agent
Other checks
- If you are able to ssh or console to the agent, please send to support@netbeez.net the last 100 lines of the agent log file with the following command:
tail -n 100 /var/log/netbeez/netbeez-agent.log
Troubleshoot via SSH
The agent is set up to return its troubleshooting information when you try to ssh to it. It's not necessary to have the ssh credentials to actually authenticate and connect. By only trying to initiate an ssh session, you should see on your screen the troubleshooting information. Here is an example:
$> ssh netbeez@192.168.86.29
Oct 8 16:14:02 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Agent version: 6.0.5
Oct 8 16:14:02 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Linux Kernel: Linux localhost 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux$
Oct 8 16:14:02 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: OS: Raspbian GNU/Linux 9.11 (stretch)
Oct 8 16:14:02 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: eth0: MAC: b8:27:eb:47:2f:c2, IP: 172.31.0.243, status: up, configuration: dhcp, type: wired
Oct 8 16:14:02 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: wlan0: MAC: 2c:4d:54:03:0d:42, IP: 192.168.86.29, status: up, configuration: dhcp, type: wireless (driver version: 88XXau v5.3.4_28226.20180614_COEX20171103-6c6d)
Oct 8 16:14:03 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Host: customer.netbeezcloud.net (3.8.11.3)
Oct 8 16:14:24 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Unencrypted port: 20019 (eth0:heartbeat detected, wlan0:heartbeat detected, eth0:open, wlan0:open)
Oct 8 16:14:46 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Encrypted port: 20018 (eth0:heartbeat detected, wlan0:heartbeat detected, eth0:open, wlan0:open)
Oct 8 16:14:48 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Dashboard port: 443 (eth0:open/established, wlan0:open)
Oct 8 16:14:50 localhost nb_troubleshoot.sh: nb_troubleshoot.sh: Websocket port: 443 (eth0:open/established, wlan0:open)
netbeez@192.168.86.29: Permission denied (publickey).
As you can see, the authentication failed, but I can see the following information:
- The agent version number
- The kernel version
- The OS version
- Information about the wired eth0 interface (MAC, IP, status, dhcp/static configuration, type)
- Information about the wireless wlan0 interface (MAC, IP, status, dhcp/static configuration, type, driver)
- The URL and IP of its dashboard
- For the different ports (20019, 20018, 443) whether they are open and which one is used for the agent-server communication (in this case port 443)
This information is updated every two minutes.
If you ssh to the agent, you can get this information by running the troubleshooting script as follows:
nb_troubleshoot
You can get the full usage information about this script by typing:
nb_troubleshoot --help
Procedure to upload log files to the NetBeez server
If you have an issue with the agent (e.g. doesn't connect to the server), the NetBeez support team may ask you to upload the agent's logs to the NetBeez log server to help with the troubleshooting.
The command to upload the logs looks like this:
$> nb_troubleshoot --upload-logs
nb_troubleshoot: Thu 8 Oct 16:29:30 EDT 2020
nb_troubleshoot: Uploading: "/var/log/netbeez/netbeez-agent.log" -> "wifi.netbeez.net_b8.27.eb.47.2f.c2_2020-10-08_16.29.30EDT_netbeez-agent.log.gz"
nb_troubleshoot: Uploading: "/var/log/netbeez/banner-netbeez-agent.log" -> "wifi.netbeez.net_b8.27.eb.47.2f.c2_2020-10-08_16.29.30EDT_banner-netbeez-agent.log.gz"
nb_troubleshoot: Uploading: "/tmp/wpa_supplicant.log" -> "wifi.netbeez.net_b8.27.eb.47.2f.c2_2020-10-08_16.29.30EDT_banner-netbeez-agent.log.gz"
As you can see the log files "/var/log/netbeez/netbeez-agent.log", "/var/log/netbeez/banner-netbeez-agent.log", "/tmp/wpa_supplicant.log" are uploaded.
If NetBeez support asks you to upload a different file from the agent, they will give you specific instructions and the related commands to do that.
0 Comments