NZFF.org outage (15-16/09-2020) - UPDATE

This is a private developers-only forum where any issues which may not be appropriate on the open forum can be discussed.

NZFF.org outage (15-16/09-2020) - UPDATE

Postby gojozoom » Wed Sep 16, 2020 11:48 am

Hi guys,

Thank you for the constructive feedback - I agree with most of it. I have already made changes to the PM system so that only admins can PM groups. The forum settings are fairly limited so we can either enable PMs to groups and multiple users or disable them altogether - meaning you can only PM a single user. To avoid further Reply All "incidents" I disabled it for now.

I also agree that PhPBB might not be the most modern/secure platform but there is a reason why we chose it. Here's a brief background to those of you that joined us more recently. Until about 4 years ago the forum was run on a forum software from stone-age (defunct now), using a very outdated server and database. We looked at different options for safely migrating everything (and keep all posts and attachments) to a more modern and customizable system that is at least somewhat compatible with the old database and forum software. At the time only PhPBB had migration scripts that were proven working fine so we went with that - none of us had the time on our hands to do a full manual migration. However, to address those security holes we're going to schedule in a software update from v3.0 to the current v3.3.

Another factor is that this forum (as most other forums) doesn't have a full-time admin. We're trying to fit things into our busy lives, work, kids, jobs, etc, therefore a "fully blown" IT support with continuous maintenance and updates is not achievable at this point. I'm open to discussions about options or volunteers to do certain tasks, obviously someone with some IT background and a bit of a change management (ITIL) experience.

All in all you're right, it's far from perfect, but I think a single 24-hours outage in 4 years is pretty good in terms of availability and stability. I'm not trying to use this as an excuse or brag - it's merely an objective observation.

Kind Regards
Dan
User avatar
gojozoom
Sim-holic
 
Topic author
Joined: Fri Dec 21, 2007 10:37 am
Posts: 943
Location: Wellington

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby aerofoto » Wed Sep 16, 2020 2:12 pm

Thank you for your HU Dan ;)

Mark C
AKL/NZ
aerofoto
Member
 
Joined: Thu Mar 23, 2017 5:37 pm
Posts: 113

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby deaneb » Wed Sep 16, 2020 2:32 pm

Dan - big thanks to you for your work in getting the site back up and running. Much appreciated. Cheers Deane
User avatar
deaneb
Senior Member
 
Joined: Sat Aug 12, 2006 4:40 pm
Posts: 1356
Location: Blenheim

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Adamski » Wed Sep 16, 2020 4:22 pm

A huge thank-you from me also. I know this "behind-the-scenes" stuff isn't trivial, so ... much appreciated!!! :cheers:

Adam.
Image
User avatar
Adamski
NZFF Pro
 
Joined: Thu Nov 01, 2007 2:22 am
Posts: 4564
Location: Birkenhead, Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby emfrat » Wed Sep 16, 2020 4:28 pm

:hesaid:
Well done, Dan :thumbup:
MikeW
MikeW
'Propliner' is actually short for 'Proper airliner, with big rumbly radials'

Image
OnlineUser avatar
emfrat
NZFF Pro
 
Joined: Sat May 07, 2011 7:41 pm
Posts: 2996
Location: 50 DME YBBN

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Charl » Wed Sep 16, 2020 4:56 pm

:hesaid: :hesaid:
It was like missing an old friend!
Thanks for mending it, Dan
User avatar
Charl
NZFF Pro
 
Joined: Mon May 01, 2006 8:28 am
Posts: 7532
Location: Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Fozzer » Wed Sep 16, 2020 7:21 pm

Many thanks, Chums!
It was getting very lonely in the World of: "Denial Of Service"!.... :blink: ...!
Well done all!

Paul... :uk: ....with a brand new password!... :rockon: .... :lol: .. :lol: ...!
User avatar
Fozzer
NZFF Pro
 
Joined: Tue Aug 17, 2010 10:29 pm
Posts: 2177
Location: Hereford, Herefordshire, England

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby optrex » Wed Sep 16, 2020 7:35 pm

I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.
optrex
Member
 
Joined: Fri Feb 24, 2017 9:19 pm
Posts: 41

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Zsolt » Wed Sep 16, 2020 8:03 pm

I second optrex's opinion. Too many users received an email regarding a PM and the server flagged it as suspicious traffic.
If the PM in question had only one recipient and lots of users received it, it could be a phpbb bug.
Zsolt
Newbie
 
Joined: Sat Feb 20, 2016 7:18 pm
Posts: 6

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby emfrat » Wed Sep 16, 2020 8:07 pm

optrex wrote:I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.


So, having figured that out, why did you PM me? Or did you also use 'Reply All?'
MikeW
'Propliner' is actually short for 'Proper airliner, with big rumbly radials'

Image
OnlineUser avatar
emfrat
NZFF Pro
 
Joined: Sat May 07, 2011 7:41 pm
Posts: 2996
Location: 50 DME YBBN

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Adamski » Wed Sep 16, 2020 8:43 pm

optrex wrote:I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.

I *also* got a PM from you [optrex] just now? It says:

"why does this system allow you to message the entire board?"

Was this a test? Are you saying that the blue line in the "To" field that says "Registered Users" should not be there?

It's not something I'd like to test - for fear of another bout of outgoing messages.

Adam.

UPDATE: OK ... now I get it!!! I just tried composing a NEW PM ... and there's a GROUPS selection box (to the right of "Find a member") that appears to allow you to select entire groups. NOT GOOD!

So ... I think optrex is indicating that this may be a BB misconfiguration <??>.
Image
User avatar
Adamski
NZFF Pro
 
Joined: Thu Nov 01, 2007 2:22 am
Posts: 4564
Location: Birkenhead, Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby optrex » Wed Sep 16, 2020 8:44 pm

I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.
optrex
Member
 
Joined: Fri Feb 24, 2017 9:19 pm
Posts: 41

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby optrex » Wed Sep 16, 2020 8:52 pm

Adamski wrote:
... or are you saying that the blue line in the "To" field that says "Registered Users" should not be there?

It's not something I'd like to test - for fear of another bout of outgoing messages.

Adam.


Hi Adam,
The ability to message registered users should not be there, but also the number of recipients should be limited (for multiple reasons).
Ideally the largest "group chat" you'd want is say 4 or 5 recipients, otherwise what's the purpose of having a forum discussion?
A better system would also rate limit the notification emails by cron job so you're not pushing out 1500 or so in one go, which I guarantee is the reason the site got "suspended".
optrex
Member
 
Joined: Fri Feb 24, 2017 9:19 pm
Posts: 41

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Adamski » Wed Sep 16, 2020 8:56 pm

optrex wrote:I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.


Here's what we shouldn't be seeing, I suspect!

Image

Adam.
Image
User avatar
Adamski
NZFF Pro
 
Joined: Thu Nov 01, 2007 2:22 am
Posts: 4564
Location: Birkenhead, Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Adamski » Wed Sep 16, 2020 9:06 pm

optrex wrote:Ideally the largest "group chat" you'd want is say 4 or 5 recipients, otherwise what's the purpose of having a forum discussion?
A better system would also rate limit the notification emails by cron job so you're not pushing out 1500 or so in one go, which I guarantee is the reason the site got "suspended".

Not sure how a CRON job could be set up for PMs, but I use CRON scripts quite often on some of my sites for newsletter mailouts etc. My main hosting company sets limits for the number of outgoing emails per hour, so I edit the CRON script accordingly. Many PHP mailout scripts usually allow you to set this up quite easily.

I don't think regular users should have a "Groups" option showing for them at all and - I agree - that in the event of an admin (or mod) needing to send out to large sections of the membership then the outgoing emails should be controlled/choked in some way.
Image
User avatar
Adamski
NZFF Pro
 
Joined: Thu Nov 01, 2007 2:22 am
Posts: 4564
Location: Birkenhead, Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby optrex » Wed Sep 16, 2020 9:22 pm

If the admin needed to email the membership they would send a newsletter. The cron is for the email notifications associated with the PM, not the PM itself.

There are a few site issues that need addressing for security too

phpinfo.php

this should not be visible to anyone as it details the server configuration and makes it easy for anyone who does want to do malicious things to determine how to attack the weaknesses. Most of the time this will be scripted by a bot searching for these files.

However it does show the fundamentals of the server config are very old and insecure.
Php 5.4 support ended in September 2015 for instance. Thats 5 years of no security patches.
Post size is 512Mb - it doesn't need to be this massive, especially when the memory limit is half that.

There's quite a big list to be fair.

Also when you post a reply to a topic, the site SSL certificate becomes insecure > I'm not 100% with phpbb as my sites are IPB or XF, but probably down to the text editor or related.
I'll stop there, dont want to be too depressing :huh:
optrex
Member
 
Joined: Fri Feb 24, 2017 9:19 pm
Posts: 41

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Radar88 » Wed Sep 16, 2020 9:34 pm

optrex wrote:I doubt it was spam emails. It actually co-incided wit a genuine user pm'ing the whole membership and its subsequent reply. Based on your membership that would generate a huge amount of sudden email traffic in the form of notifications and look like spam, depending on your setup.

You should never have a configuration that would allow that to happen.


I was the intended recipient of a PM sent on Monday evening From another Member, that was accidentally sent to the Registered Users Group which is All 1142 members of the NZFF

The member has since publicly apologize in the forum post concerned and other members myself included have accepted the apology as a genuine mistaken error in the process of sending a PM

Subsequently as a result, this incident has brought other members "out of the wood work" and woken them up to log back into the Forum to check out and see what all the fuss is about.

In this incident concerning a New private message has arrived "NZFF" with the following subject:

Re: Thanks for the help

PLEASE DO NOT click on Reply to message or Reply to sender and all recipients.

Image

This makes the matter worse as you are effectively Spamming all 1142 members of the NZFF Community all over and over again.

I should clarify that I've had 10 years + working experience, with an internationally recognized large Computing Organization in the area of Information Technology Security.

It's highly likely that the Outage that occurred Tuesday morning was potentially caused after another member whom unwittingly Reply to message early on Tuesday morning after receiving a message on Monday evening, that again spammed everyone's mail boxes later on Tuesday morning, this appears to have overloaded the forum PM and external email address notification system.

So far I have noted a couple of members have unwittingly done a Reply to message or Reply to sender and all recipients.

The Outage is likely a result of unwittingly Spamming a Reply to message that includes all registered members external personal email address's, this likely can result in a process known as Email Spoofing that may or may not have potentially occurred.

Hence the precaution action taken by Dan.

Please simply ignore the PM Forum Message with subject Re: Thanks for the help.

Thanking everyone for their patience and understanding in this rather unfortunate incident.
Last edited by Radar88 on Thu Sep 17, 2020 10:26 am, edited 1 time in total.
Radar88
Forum Addict
 
Joined: Sun Feb 15, 2015 2:32 pm
Posts: 202

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Radar88 » Wed Sep 16, 2020 10:14 pm

optrex wrote:I used reply to all to see if

A) that was the issue
B) the configuration error had been closed by the site owner

It's a bit of a rookie flaw to be fair, but also phpbb isn't the best or safest form platform out there either, so I can understand the hosting companies concern.


Thanking you for all your input regarding the incident.

I respect your input, only would please ask, that you refrain from jumping in and doing that in the future.

In doing that, as I see it only compounds the issue further and potentially contributes to making it worse for everyone.

It would be better for the Site Administrators to do a thorough in-depth investigation of the issue over the next few days.
Radar88
Forum Addict
 
Joined: Sun Feb 15, 2015 2:32 pm
Posts: 202

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Adamski » Thu Sep 17, 2020 12:27 am

Radar88 wrote:It would be better for the Site Administrators to do a thorough in-depth investigation of the issue over the next few days.

It's also possible that the admin[s] got the site up and running quickly for us and haven't had a chance to finish re-configuring things.
Image
User avatar
Adamski
NZFF Pro
 
Joined: Thu Nov 01, 2007 2:22 am
Posts: 4564
Location: Birkenhead, Auckland

Re: NZFF.org outage (15-16/09-2020) - Change your password please!

Postby Aharon » Thu Sep 17, 2020 2:38 am

Thanks all for your hard work to restore this magnificent forum back to exciting action.

Regards,

Aharon
Aharon
Sim-holic
 
Joined: Sun Jul 23, 2017 11:39 am
Posts: 953

Next

Return to Developer's discussion

Who is online

Users browsing this forum: No registered users and 1 guest