Webmaster's Blog

[GDII]

Thanks Michael! It did have a different error this time vs the previous ones so it does appear to be something else.

[mknz]

Forum went down, but fix was easy, just needed to clear the cache

[mknz]

Hmm started getting weird logging in behaviour, so I disabled a bunch of CDN features to see if that resolves the issue

[mknz]

I've enabled uploading *.heic and *.heif images, but none of the browsers support viewing them so you have to download them to see them.

You can convert them to JPEG pictures on your phones:
https://www.howtogeek.com/666363/how-to ... -and-ipad/

Samsung devices still need to use a third-party converter in the meantime, but you can disable using HEIF/C in the meantime.

[mknz]

Sorry, had to get home to fix the DB log file index. It got corrupted as it probably auto-rebooted on a crash. If this happens again I'll update the filesystem mounting to be stricter, and potentially automate fixing the index.

[Benckj]

Was starting to wonder why it was down for so long. Thnx for fixing issue.

[mknz]

We paid 44.23 USD more than usual for hosting last month because of far higher bandwidth usage. The CloudFlare dashboards showed Singapore was 90% of requests and the AWS billing showing about ~450GB transferred. We only have about ~60GB of files. Looking at the access logging, it's almost entirely Bytedance's (TikTok) crawler. They are known for using Singapore data centres including AWS to get around blocks, a good chunk of traffic was going to AWS Singapore. CloudFlare also presented many anti-bot challenges last month, entirely in Singapore. This robot doesn't respect robots.txt and I don't think we have many MR2 owners in the PRC nor Singapore so I've blocked Bytedance/Bytespider and the two countries.

Source: https://wordpress.org/support/topic/psa ... -blocking/

Now that it's blocked we should see a reduction in our bill back to normal this month, but I'll keep an eye on it to make sure it's working. I can see the blocking rule is already doing something. I can escalate by rate-limiting them instead to a very low value if they try crawling from somewhere else and by throwing the anti-bot page from CloudFlare more often.

I've also lowered the billing alert so I get notified sooner if we're going to exceed our normal spend.

Thanks Gary for noticing it and messaging me.

EDIT: For reference, in April, we only transferred ~1.5GB in and ~24GB out which is within the global free tier of AWS.

[Benckj]

Good job Michael. Was wonderful what was going on with the below average posts.

[mknz]

Been getting hammered by the bots recently. I had to rollback to a couple of days ago to right before the DB got corrupted

[mknz]

I moved the SSDs to the newer set while it was down anyway. Both cheaper and faster, should theoretically increase capacity, but realisitically the constraint is probably the limited number of hardware threads.

[Benckj]

Thought there was something going on in the backgraound. Thnaks for sorting out.

[mknz]

Bots at it again, I've been blocking them proactively, but they keep finding new ways.

In the meantime I'd tuned the database to only use as much RAM as is available so the operating system wouldn't kill it and thus cause data corruption. This means that the website has been smart enough to just refuse serving traffic when there's been too much load.

This is still better than before where it would occasionally require my manual intervention to go fix the DB corruption and then restart the server. The database is by far the bottleneck and it'll just say "too many connections".

The search performance did suffer a little though, so I have a little bit of further tweaking to do.

[Trotbags]

Appreciate you keeping this going. It is one of the reliable go-to places for MR2 knowledge.

[mknz]

Oh man, LLMs have figured out the Q&A question of user signups, which means I've had a flood of user signups. The question was already a bit esoteric for new owners. Switching back to using reCaptcha which isn't perfect, but hopefully harder.

I approve new accounts manually, so none of them got through

[MIKEP]

So any one called Ima Bot wont get on. Good one thanks Mike