After configuring the vCenter Syslog Collector using this excellent article by Jason Boche, my vCenter Syslog Collector was not receiving messages from the configured ESXi servers. After a little research I found out that I forget to open the outbound firewall ports on my ESXi Server. After reconfiguring the firewall, the vCenter Syslog Collector was still not receiving any log message.What was the problem here?
Well, if the ESXi syslog service cannot find the syslog server (in this case because of a firewall misconfiguration), the syslog service will stop functioning. VMware is talking about this issue in KB article 2003127. The solution? Restart the syslog service on the ESXi server(s), using the esxcli command:
esxcli system syslog reload
Although this solution works, it’s not really a good option if you have to restart the service on 10+, 100+ or even more ESXi servers.
Note: After a reboot of the vCenter Server, e.g. because of applying a Windows Update, an ESXi server will also loose its connection with vCenter Syslog Collector (in case you installed it on your vCenter!).
So can we think of an alternative? The answer is yes…PowerCLI is the solution here:
foreach ($esxhost in (Get-Cluster -Name $cluster | Get-VMHost )){
Write-Host "ESX: $esxhost"
$esxcli = Get-EsxCli -VMhost $esxhost
$esxcli.system.syslog.reload()
}
The Get-EsxCli commandlet let you use the esxcli commandline tool on the ESXi server. With $esxcli.system.syslog.reload() you’re actually executing esxcli system syslog reload. Don’t forget the brackets ‘()’ at the end, because otherwise the command will not work. And don´t forget to set the clustername in the $cluster variable.
This script will restart/refresh the syslog deamon on all ESXi hosts in the selected cluster. After running this script, syslogging should be up and running again!
14 Comments
manish
Can I use the powercli script to update the article 2003127 for the syslog service function?
viktorious
Of course, no problem! Any reference would be appreciated (if possible) 🙂
Peter
I’ve just changed to using UDP hoping that will fix the problem. By default it is TCP.
viktorious
Interesting thought, what are your results Peter?
Peter
I unfortunately found out the same day that it still stops. This is the only syslog client I know of that acts like this. Most just ship out the logs and if the server is down they are lost. I think VMware needs to fix this.
Smock
I’ve hit this problem too. I’m setting it set up so I ship logs to a CentOS box, but every time I reconfigure the rsyslog config and restart the rsyslog service or reboot the log server, the connection is lost, and I have to reload the syslog config on the vm hosts. This script will save quite a bit of time, but ideally the vm host should poll the syslog server every X seconds or minutes (could be a custom setting in the Syslog.global section). Once setup it should be less of a problem, but it’s a major annoyance at the moment!
viktorious
Hi, also check this post on this subject:
http://www.viktorious.nl/2012/07/30/esxi-syslogger-problem-with-interrupted-syslog-service-partially-solved/
This issue is solved for syslog based on UDP.
Important: You have to explicitly patch you ESXi servers with path ESXi 201207401!
Ed
ESXi 5.0 wants to use tcp for syslog. When you kill and restart your rsyslog service, that connection is lost. Try ‘service rsyslog reload’ instead.
viktorious
You can also use UDP for syslog. On top of this VMware solved this issue for UDP, see this article:
http://www.viktorious.nl/2012/07/30/esxi-syslogger-problem-with-interrupted-syslog-service-partially-solved/
I have successfully configured ESXi 5.0 syslog with UDP several times and it works like charm.
JustDave
I am new to scripting, but I am familiar with PowerCLI/vMA. How do I create/run the script you listed above? thanks!
viktorious
You just run the script from your favourite scripting editor!
Pingback: Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm | virtuallyGhetto
jawed
is this really working for you guys? I can’t see how it works for you and not for me.
here is what i am getting this error which I understand is happening because we are not even storing the variable in $cluster and it becomes a null string.
PowerCLI C:\> foreach ($esxhost in (Get-Cluster -Name $cluster | Get-VMHost )){Write-Host “ESX: $esxhost”$esxcli = Get-EsxCli -VMhost $esxhost $esxcli.system.syslog.reload()}
Get-Cluster : Cannot validate argument on parameter ‘Name’. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
At line:1 char:41
+ foreach ($esxhost in (Get-Cluster -Name $cluster | Get-VMHost )){Write-Host “ESX …
+ ~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Get-Cluster], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetCluster
viktorious
Hi, did you put the clustername in $cluster variable?