NZFF.org outage (15-16/09-2020) - UPDATE

[aerofoto]

Thank you for your HU Dan

Mark C
AKL/NZ

[deaneb]

Dan - big thanks to you for your work in getting the site back up and running. Much appreciated. Cheers Deane

[Adamski]

A huge thank-you from me also. I know this "behind-the-scenes" stuff isn't trivial, so ... much appreciated!!!

Adam.

[emfrat]

Well done, Dan
MikeW

[Charl]

It was like missing an old friend!
Thanks for mending it, Dan

[Fozzer]

Many thanks, Chums!
It was getting very lonely in the World of: "Denial Of Service"!.... ...!
Well done all!

Paul... ....with a brand new password!... .... .. ...!

[optrex]

I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.

[Zsolt]

I second optrex's opinion. Too many users received an email regarding a PM and the server flagged it as suspicious traffic.
If the PM in question had only one recipient and lots of users received it, it could be a phpbb bug.

[emfrat]

I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.

So, having figured that out, why did you PM me? Or did you also use 'Reply All?'

[Adamski]

I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.

I *also* got a PM from you [optrex] just now? It says:

"why does this system allow you to message the entire board?"

Was this a test? Are you saying that the blue line in the "To" field that says "Registered Users" should not be there?

It's not something I'd like to test - for fear of another bout of outgoing messages.

Adam.

UPDATE: OK ... now I get it!!! I just tried composing a NEW PM ... and there's a GROUPS selection box (to the right of "Find a member") that appears to allow you to select entire groups. NOT GOOD!

So ... I think optrex is indicating that this may be a BB misconfiguration <??>.

[optrex]

I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.

[optrex]

... or are you saying that the blue line in the "To" field that says "Registered Users" should not be there?

It's not something I'd like to test - for fear of another bout of outgoing messages.

Adam.

Hi Adam,
The ability to message registered users should not be there, but also the number of recipients should be limited (for multiple reasons).
Ideally the largest "group chat" you'd want is say 4 or 5 recipients, otherwise what's the purpose of having a forum discussion?
A better system would also rate limit the notification emails by cron job so you're not pushing out 1500 or so in one go, which I guarantee is the reason the site got "suspended".

[Adamski]

I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.

Here's what we shouldn't be seeing, I suspect!



Adam.

[Adamski]

Ideally the largest "group chat" you'd want is say 4 or 5 recipients, otherwise what's the purpose of having a forum discussion?
A better system would also rate limit the notification emails by cron job so you're not pushing out 1500 or so in one go, which I guarantee is the reason the site got "suspended".

Not sure how a CRON job could be set up for PMs, but I use CRON scripts quite often on some of my sites for newsletter mailouts etc. My main hosting company sets limits for the number of outgoing emails per hour, so I edit the CRON script accordingly. Many PHP mailout scripts usually allow you to set this up quite easily.

I don't think regular users should have a "Groups" option showing for them at all and - I agree - that in the event of an admin (or mod) needing to send out to large sections of the membership then the outgoing emails should be controlled/choked in some way.

[optrex]

If the admin needed to email the membership they would send a newsletter. The cron is for the email notifications associated with the PM, not the PM itself.

There are a few site issues that need addressing for security too

phpinfo.php

this should not be visible to anyone as it details the server configuration and makes it easy for anyone who does want to do malicious things to determine how to attack the weaknesses. Most of the time this will be scripted by a bot searching for these files.

However it does show the fundamentals of the server config are very old and insecure.
Php 5.4 support ended in September 2015 for instance. Thats 5 years of no security patches.
Post size is 512Mb - it doesn't need to be this massive, especially when the memory limit is half that.

There's quite a big list to be fair.

Also when you post a reply to a topic, the site SSL certificate becomes insecure > I'm not 100% with phpbb as my sites are IPB or XF, but probably down to the text editor or related.
I'll stop there, dont want to be too depressing

[Radar88]

I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.

I was the intended recipient of a PM sent on Monday evening From another Member, that was accidentally sent to the Registered Users Group which is All 1142 members of the NZFF

The member has since publicly apologize in the forum post concerned and other members myself included have accepted the apology as a genuine mistaken error in the process of sending a PM

Subsequently as a result, this incident has brought other members "out of the wood work" and woken them up to log back into the Forum to check out and see what all the fuss is about.

In this incident concerning a New private message has arrived "NZFF" with the following subject:

Re: Thanks for the help

PLEASE DO NOT click on Reply to message or Reply to sender and all recipients.



This makes the matter worse as you are effectively Spamming all 1142 members of the NZFF Community all over and over again.

I should clarify that I've had 10 years + working experience, with an internationally recognized large Computing Organization in the area of Information Technology Security.

It's highly likely that the Outage that occurred Tuesday morning was potentially caused after another member whom unwittingly Reply to message early on Tuesday morning after receiving a message on Monday evening, that again spammed everyone's mail boxes later on Tuesday morning, this appears to have overloaded the forum PM and external email address notification system.

So far I have noted a couple of members have unwittingly done a Reply to message or Reply to sender and all recipients.

The Outage is likely a result of unwittingly Spamming a Reply to message that includes all registered members external personal email address's, this likely can result in a process known as Email Spoofing that may or may not have potentially occurred.

Hence the precaution action taken by Dan.

Please simply ignore the PM Forum Message with subject Re: Thanks for the help.

Thanking everyone for their patience and understanding in this rather unfortunate incident.

[Radar88]

I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.

Thanking you for all your input regarding the incident.

I respect your input, only would please ask, that you refrain from jumping in and doing that in the future.

In doing that, as I see it only compounds the issue further and potentially contributes to making it worse for everyone.

It would be better for the Site Administrators to do a thorough in-depth investigation of the issue over the next few days.

[Adamski]

It would be better for the Site Administrators to do a thorough in-depth investigation of the issue over the next few days.

It's also possible that the admin[s] got the site up and running quickly for us and haven't had a chance to finish re-configuring things.

[Aharon]

Thanks all for your hard work to restore this magnificent forum back to exciting action.

Regards,

Aharon

[optrex]

Hi guys,

Thank you for the constructive feedback - I agree with most of it. I have already made changes to the PM system so that only admins can PM groups. The forum settings are fairly limited so we can either enable PMs to groups and multiple users or disable them altogether - meaning you can only PM a single user. To avoid further Reply All "incidents" I disabled it for now.

I also agree that PhPBB might not be the most modern/secure platform but there is a reason why we chose it. Here's a brief background to those of you that joined us more recently. Until about 4 years ago the forum was run on a forum software from stone-age (defunct now), using a very outdated server and database. We looked at different options for safely migrating everything (and keep all posts and attachments) to a more modern and customizable system that is at least somewhat compatible with the old database and forum software. At the time only PhPBB had migration scripts that were proven working fine so we went with that - none of us had the time on our hands to do a full manual migration. However, to address those security holes we're going to schedule in a software update from v3.0 to the current v3.3.

Another factor is that this forum (as most other forums) doesn't have a full-time admin. We're trying to fit things into our busy lives, work, kids, jobs, etc, therefore a "fully blown" IT support with continuous maintenance and updates is not achievable at this point. I'm open to discussions about options or volunteers to do certain tasks, obviously someone with some IT background and a bit of a change management (ITIL) experience.

All in all you're right, it's far from perfect, but I think a single 24-hours outage in 4 years is pretty good in terms of availability and stability. I'm not trying to use this as an excuse or brag - it's merely an objective observation.

Kind Regards
Dan

Hey Dan,

Just looking at your comments around IT support and running the site and looking for volunteers. We mentioned previously the out of date nature of the hosting/server platforms, but I also now see you run 3.0.14 and note that that was end of life in 2015. All in all, this still makes things very old, and gives the feeling of things being neglected (and I say that from a tech point, I know the demands of running communities and real life don't always align and that behind the scenes can be quite demanding at times), so quite a bit of work just to bring the existing up to at least a supported and more secure environment. And I have to say it was the number one factor for me not contributing to the site before now, despite joining quite a while ago.

That said, I'm a sucker for volunteering and I'm happy to help with LAMP server config/snapshots/backups/database management/SSL/patching and upgrades, SEO and hosting config and even bringing the forum upto spec. It's something I do already for a number of forum based sites. However, I would like to ask why stop at there?
There's clearly room for some feature improvements:

A gallery for the screen captures - ongoing photo comps and an interest point to bring in new content
A document storage area for manuals, instructions, modifications, heck you could even monetise it to allow developers to charge for a download.
A decent review system, allow people to recommend equipment/suppliers/developers
An event calendar - get the community together for a flying event.
Bring in community features - reactions, the ability to recommend posts, promote topics, up voting and down voting ideas etc, the list is endless.

Q&A forums, blogs, articles from members willing to act as regular content providers.
With a bit of time, you could bring in tactful donation, sponsorship, advertising opportunities to support the running costs, and if cost is the stumbling block, I would be happy to support a software purchase/license. If time is the issue, I maybe could possibly be talked into potentially doing it for you.

Obviously we'd start off with a ticketing support system and change log and a test environment.
Let me know what you think.

Grant