chipmunk.land services

Tags
chipmunk.land servers services technology 
Labels
Sticky
Members allowed to view this conversation
Everyone

admin of eso.chipmunk.land please ban the mysticgiggle bot on code.chipmunk.land and add auth using ww.scr.chipmunk.land using a fork of scratch auth that you should make and do a forgejo instance for bots called botforgejo.chipmunk.land and the mystic* bots should be on botforgejo.chipmunk.land please


max

1 month ago (edited by max 1 month ago) Administrator

mysticgiggle -

admin of eso.chipmunk.land please ban the mysticgiggle bot on code.chipmunk.land and add auth using ww.scr.chipmunk.land using a fork of scratch auth that you should make and do a forgejo instance for bots called botforgejo.chipmunk.land and the mystic* bots should be on botforgejo.chipmunk.land please

What the fuck am I reading? lmfao
Scratch is like, probably the weirdest way one could ever provide authentication for a website. Maybe it's too niche to get spammed, but I already mitigated the current spambot attack anyway. You personally are blocked from registering an account for a very good reason.
Edit: nevermind your request makes more sense than I thought… I think those are joke accounts though, and nah they make more sense to have on the main site.


max -

What the fuck am I reading? lmfao
Scratch is like, probably the weirdest way one could ever provide authentication for a website. Maybe it's too niche to get spammed, but I already mitigated the current spambot attack anyway. You personally are blocked from registering an account for a very good reason.
Edit: nevermind your request makes more sense than I thought… I think those are joke accounts though, and nah they make more sense to have on the main site.

MysticBrewer


opt - go to this post

MysticBrewer

stfu


opt - go to this post

MysticBrewer

stfu


max

1 month ago Administrator

Alright so I've been thinking about something lately. The R/W bandwidth of the current HDD in server0.local has seemingly been exceeded before by two particular services, Forgejo and Synapse, and so I'm interested in not letting this happen.

Regarding Forgejo, as you may know there have been downtime, slowness, breakage, etc instances due to a bunch of crawlers making tons of requests to the site. I have noticed, however, some related slow-down incidents affected not only the Forgejo daemon, but disk operations across the entire server0.local machine.
Now I've blocked some of the most easily fingerprintable crawlers, which has reduced though not eradicated the load on the service. Still, I fear requests causing such disk operations may still get in. Now you may be like "oh just act like every request is malicious and send them all JS like Anubis!" but I believe that would degrade the site too much, and I believe this is a deeper issue than a few crawlers getting in anyway.
Once (even with all the filtering) one git process to spawn was enough to bring the disk to its knees. I say this because I only had to kill one process in htop before everything went back to normal. I don't remember the command, just that it might have had the word graph somewhere. It could have been some batch operation, but I still see this as a bad sign.

It doesn't end there, of course. Synapse, a Matrix homeserver implementation, runs on server0.local and is hooked up to a PostgreSQL database on the same machine. Many times, I have seen postgres processes relating to synapse at the top of htop, when set to sort by I/O.
I'm not as certain this is an issue, but I do vaguely remember an incident. Once disk I/O was slow on server0.local, so I tried to stop forgejo, just for nothing to happen. I could be wrong, but I think synapse was, instead, what caused the most I/O operations to be happening at that time.
Either way there have been incidents when this service has frozen, and while it may not be for disk-related reasons I still think it's a possibility.

For the sake of reliability I think I should adopt a storage setup that can take more I/O operations at a time, to avoid incidents like the ones from the two aforementioned services from happening again. Looking at the mobo (archive) of the machine, we can see that there are three SATA ports we can make use of (one is likely for the DVD drive, but that's basically useless on a server).
Perhaps we could make use of solid state drives?

  • server1.local has a fast NVME SSD, though also a worse CPU than server0.local. I could move the database daemons to it, however this could add latency and wouldn't fix the Forgejo issue. Network storage is also possible but could also add latency. Since I have a backdoored ISP router, I also might want encryption (more latency).
  • I have a 1TB M.2 SSD from another laptop, perhaps we could get it into server0.local?
  • This brings up the question of what should go on the SSD and what shouldn't, though (OS files too for boot speed? what about static stuff?).
  • While it's likely these drives can handle more operations I'm honestly unsure.

Using a RAID array is another possibility. Aside from the 1TB SSD, I also have a 1.5TB external HDD (can be disassembled prolly) and a smaller (laptop-sized?) 1TB external HDD (ditto). Still, the amount of SATA ports is limited, and while I also have USB 3.0 ports they are limited too. USB would imply putting disks *outside* of the chassis which could quickly get messy and/or ugly too.

These solutions have other reliability concerns of course. I've had issues with external HDDs failing; this could simply be from physical damage (like from dropping) that would never happen in a server setting but I'm unsure. SSDs are known for failing after long use, though they can last a while beforehand. Also using some RAID types will cause all data to be lost if a single drive fails.

Well… am I a lunatic at this point?


max - go to this post

Alright so I've been thinking about something lately. The R/W bandwidth of the current HDD in server0.local has seemingly been exceeded before by two particular services, Forgejo and Synapse, and so I'm interested in not letting this happen.

Regarding Forgejo, as you may know there have been downtime, slowness, breakage, etc instances due to a bunch of crawlers making tons of requests to the site. I have noticed, however, some related slow-down incidents affected not only the Forgejo daemon, but disk operations across the entire server0.local machine.
Now I've blocked some of the most easily fingerprintable crawlers, which has reduced though not eradicated the load on the service. Still, I fear requests causing such disk operations may still get in. Now you may be like "oh just act like every request is malicious and send them all JS like Anubis!" but I believe that would degrade the site too much, and I believe this is a deeper issue than a few crawlers getting in anyway.
Once (even with all the filtering) one git process to spawn was enough to bring the disk to its knees. I say this because I only had to kill one process in htop before everything went back to normal. I don't remember the command, just that it might have had the word graph somewhere. It could have been some batch operation, but I still see this as a bad sign.

It doesn't end there, of course. Synapse, a Matrix homeserver implementation, runs on server0.local and is hooked up to a PostgreSQL database on the same machine. Many times, I have seen postgres processes relating to synapse at the top of htop, when set to sort by I/O.
I'm not as certain this is an issue, but I do vaguely remember an incident. Once disk I/O was slow on server0.local, so I tried to stop forgejo, just for nothing to happen. I could be wrong, but I think synapse was, instead, what caused the most I/O operations to be happening at that time.
Either way there have been incidents when this service has frozen, and while it may not be for disk-related reasons I still think it's a possibility.

For the sake of reliability I think I should adopt a storage setup that can take more I/O operations at a time, to avoid incidents like the ones from the two aforementioned services from happening again. Looking at the mobo (archive) of the machine, we can see that there are three SATA ports we can make use of (one is likely for the DVD drive, but that's basically useless on a server).
Perhaps we could make use of solid state drives?

  • server1.local has a fast NVME SSD, though also a worse CPU than server0.local. I could move the database daemons to it, however this could add latency and wouldn't fix the Forgejo issue. Network storage is also possible but could also add latency. Since I have a backdoored ISP router, I also might want encryption (more latency).
  • I have a 1TB M.2 SSD from another laptop, perhaps we could get it into server0.local?
  • This brings up the question of what should go on the SSD and what shouldn't, though (OS files too for boot speed? what about static stuff?).
  • While it's likely these drives can handle more operations I'm honestly unsure.

Using a RAID array is another possibility. Aside from the 1TB SSD, I also have a 1.5TB external HDD (can be disassembled prolly) and a smaller (laptop-sized?) 1TB external HDD (ditto). Still, the amount of SATA ports is limited, and while I also have USB 3.0 ports they are limited too. USB would imply putting disks *outside* of the chassis which could quickly get messy and/or ugly too.

These solutions have other reliability concerns of course. I've had issues with external HDDs failing; this could simply be from physical damage (like from dropping) that would never happen in a server setting but I'm unsure. SSDs are known for failing after long use, though they can last a while beforehand. Also using some RAID types will cause all data to be lost if a single drive fails.

Well… am I a lunatic at this point?

What are the specifications of the current drive in server0.local? Sounds to me like the HDD in server0.local is on its way out…

A single instance of git probably shouldn't be enough to bring your drive to its knees!


max

1 month ago Administrator

opt - go to this post

What are the specifications of the current drive in server0.local? Sounds to me like the HDD in server0.local is on its way out…

server0:~# smartctl -i /dev/sda
smartctl 7.5 2025-04-30 r5714 [x86_64-linux-6.18.35-0-lts] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA100
Serial Number:    7846XA8MS
LU WWN Device Id: 5 000039 fd8c32494
Add. Product Id:  DELL(tm)
Firmware Version: MS2OA810
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.5/5706
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jul  1 13:09:03 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Perhaps I should be more careful with backups, though I'm not sure if the drive dying is the concern.
Then again I do have an almost certainly dying drive (in my laptop, has been clicking since December) and usually the laptop does slow down during the clicks. I haven't really associated the clicks with any process on the laptop, though, unlike here. But maybe…
(and don't worry I have a recent backup of the laptop's files)

opt - go to this post

A single instance of git probably shouldn't be enough to bring your drive to its knees!

To be fair I don't remember the exact command, nor the repo involved. Most git commands execute just fine, but the idea that such a destructive command which the invocation of can be caused by a web request is what I mainly find concerning.


how to use this?


max

1 month ago Administrator

DankPods - go to this post

how to use this?

Uhh see some of the stickied threads I guess. esoBB is basically a forum with less common conventions for finding threads, i.e. categories are missing in favor of tags.


Hey! You need to log in or create an account to do anything on this forum.

507 posts - 56 conversations - 1 member online

  • Display avatars