I’ve been working with Fortigate firewalls since the early 2000s. They have been affordable, providing high-end features and good support. I’ve deployed hundreds of various units and have never once had any issues with functionality, security or updates. Not one time have I applied a firmware update that has rendered the unit non-functional or has crippled the unit in any way. I have always looked forward to new releases as they would provide performance and feature enhancements. All of this has been the norm, until the recent release of version 6.2.0.
I recently upgraded to version 6.2.0 to a Fortigate 100E, which went very smooth as has always been the case. I check the performance indicators and all looked normal. Several days later I get a call that the client’s Internet access is down. I am on the road at the moment, but the client says he will call their Internet provider to see what is going on. In past this has been the cause when Internet access is down. The provider suggests restarting the firewall as part of their process, which I didn’t think much about. Internet access back online and all is well until it happened again and again. After checking the logs, they were filled with duplicate messages that were repeating every couple of minutes. Something was not right, so I called Fortinet support. After looking at the logs, the technician states that this is a known bug and proceeded to disable IPv6 to clear up the logging situation and also stated that version 6.2.1 would be out within days and to upgrade at that point. At that point, I accepted this short term solution.
Well, days later, down again. I called again and now it was stated, by the same tech as before, that the IPS engine was the problem and she had a script that would take care of the issues until the new firmware was available. The solution again seemed reasonable, but now the new firmware was a couple of weeks out.
I thought that now, we have a good short term solution, but that was short lived as the firewall was reset again 2 days later. The customer has now grown accustomed to fixing this issue by unplugging the firewall. After restart, IPv6 is re-enabled which invokes the first issue of filling the logs and overwriting itself so I don’t see what is going on. I decide to schedule a daily restart at 3:00 a.m., which does the trick for about 9 days and then once again, no Internet. By this time, my customer is beginning to question the choice of purchasing this firewall. I can’t believe in Fortinet support or in any workaround. I have to resolve this now.
So this time, I drop everything, disable IPv6 and monitor the firewall. As I watch, I see memory usage start at 69% and slowly make it’s way upward until conserve mode kicks into gear killing Internet access. Solution, in my opinion, downgrade. So I read the warnings and prepared for the worst case scenario. I had the previous firmware and good backups and a console cable, TFTP server setup and anything else I might need. I chose the firmware and hit the downgrade option and got an error that the firmware couldn’t be downloaded. Not a good start. I chose the browse option as I had the firmware downloaded. The process pulled it in and proceeded to reboot. About 2 minutes later, the firewall was back up and all looked good. Memory was down to 43% usage from 69% and stayed down as I continued to monitor.
Long story short is that I broke my usual cautionary rule of waiting before rolling out new software versions on any product unless there is an urgent need. I put too much trust in support technicians that I know only roll through a short list of quick fixes. Mostly, I let my customer down by not ensuring this issue was resolved and/or a temporary solution was actually working. Lessons learned!
