• too many open files - user.dat

    From DaiTengu@VERT/ENSEMBLE to All on Sat May 28 14:24:51 2022
    I'm running into an issue where my system will lock up due to too many open files. It's been happening for a couple months. I upped the open file limit, but that didn't seem to help.

    [root@bbs ctrl]# ulimit -a
    core file size (blocks, -c) 0
    data seg size (kbytes, -d) unlimited
    scheduling priority (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) 15605
    max locked memory (kbytes, -l) 8192
    max memory size (kbytes, -m) unlimited
    open files (-n) 950000
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) 819200
    real-time priority (-r) 0
    stack size (kbytes, -s) 8192
    cpu time (seconds, -t) unlimited
    max user processes (-u) 950000
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited

    I'm still getting lots of this, and it will refuse to let anyone log in.

    May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:45:33 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4671 (daily_maint) opening "/sbbs/data/mail" access=4294967096 info=smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd
    May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: Running system event
    May 28 13:45:33 bbs sbbs: evnt BBS Events !JavaScript : can't open /sbbs/exec/logonlist.js: No such file or directory
    May 28 13:45:33 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in exec.cpp line 662 (js_execfile) compiling "/sbbs/exec/logonlist.js" access=0
    May 28 13:45:33 bbs sbbs: evnt BBS Events Daily event: '?logonlist -m' returned -1
    May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
    May 28 13:45:37 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
    May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in logon.cpp line 603 (logonstats) opening "/sbbs/data/logon.lst" access=577
    May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
    May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
    May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
    May 28 13:45:37 bbs sbbs: evnt BBS Events ERROR -200 (smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd) opening mail base
    May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
    May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 1" access=4294967295
    May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 2" access=4294967295
    May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 3" access=4294967295
    <continues for every user>

    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-

    and then it tries to re-run the daily event.. because it thinks it's Epoch:

    May 28 13:37:57 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
    May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
    May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
    May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
    May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
    May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:37:58 bbs sbbs: evnt BBS Events DAILY: Running system event
    May 28 13:37:58 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
    May 28 13:38:03 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Running system event
    May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
    May 28 13:38:07 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Running system event
    May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
    May 28 13:38:12 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
    May 28 13:38:12 bbs journal: Suppressed 4062 messages from /system.slice/sbbs.service
    May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in logon.cpp line 603 (logonstats) opening "/sbbs/data/logon.lst" access=577
    May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
    May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
    May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
    May 28 13:38:12 bbs sbbs: evnt BBS Events ERROR -200 (smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd) opening mail base
    May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
    May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 1" access=4294967295
    May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 2" access=4294967295
    May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 3" access=4294967295
    May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 4" access=4294967295


    -=-=-=-=-==-=-=-=-=-=-=-




    Now, when I checked, lsof | wc -l only showed about 72K open files. the majority were by sbbs/term, and sbbs/http to /sbbs/data/user/user.dat

    after killing sbbs and restarting, my open files were down to 12k.


    The kicker is, 72k is far less than the 950K my system allows. is there an open file limit built into SBBS?

    DaiTengu

    ... I'd give my right arm to be ambidextrous.

    ---
    Synchronet War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Digital Man@VERT to DaiTengu on Sat May 28 12:42:38 2022
    Re: too many open files - user.dat
    By: DaiTengu to All on Sat May 28 2022 02:24 pm

    I'm running into an issue where my system will lock up due to too many open files. It's been happening for a couple months. I upped the open file limit, but that didn't seem to help.

    [root@bbs ctrl]# ulimit -a
    open files (-n) 950000

    Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as:
    http://wiki.synchro.net/faq:nix#maximum_open_files

    Now, when I checked, lsof | wc -l only showed about 72K open files.

    Unfortunately, lsof is not accurate with multi-threaded programs since it'll list/count all open files for the process, repeated for each thread of the process. So lsof will report a much larger number than the actual number of open files.

    the
    majority were by sbbs/term, and sbbs/http to /sbbs/data/user/user.dat

    after killing sbbs and restarting, my open files were down to 12k.


    The kicker is, 72k is far less than the 950K my system allows. is there an open file limit built into SBBS?

    No. I suspect you just haven't correctly adjusted your open file limit for the sbbs user/process yet.
    --
    digital man (rob)

    Sling Blade quote #17:
    Charles Bushman: A shovel just makes too goddamned much racket.
    Norco, CA WX: 63.0F, 77.0% humidity, 0 mph ENE wind, 0.00 inches rain/24hrs ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From DaiTengu@VERT/ENSEMBLE to Digital Man on Sun May 29 11:04:36 2022
    Re: too many open files - user.dat
    By: Digital Man to DaiTengu on Sat May 28 2022 12:42 pm

    Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as: http://wiki.synchro.net/faq:nix#maximum_open_files

    Found the problem, which the wiki link helped me with. I had to up the open file limit in the systemd start config. I had it set to 950k for both users and root, but didn't update the startup config.

    LimitNOFILE=150000

    Have I ever mentioned how much I hate systemd? :)

    Thanks!

    DaiTengu

    ... DOS never says EXCELLENT command or filename...

    ---
    Synchronet War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Gamgee@VERT/PALANT to DaiTengu on Sun May 29 12:18:00 2022
    DaiTengu wrote to Digital Man <=-

    Re: too many open files - user.dat
    By: Digital Man to DaiTengu on Sat May 28 2022 12:42 pm

    Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as: http://wiki.synchro.net/faq:nix#maximum_open_files

    Found the problem, which the wiki link helped me with. I had to
    up the open file limit in the systemd start config. I had it set
    to 950k for both users and root, but didn't update the startup
    config.

    LimitNOFILE=150000

    Have I ever mentioned how much I hate systemd? :)

    Slackware.
    MX Linux.
    Devuan.
    <a few more>

    ;-)



    ... So easy, a child could do it. Child sold separately.
    --- MultiMail/Linux v0.52
    Synchronet Palantir BBS * palantirbbs.ddns.net * Pensacola, FL
  • From DaiTengu@VERT/ENSEMBLE to Gamgee on Mon May 30 00:51:31 2022
    Re: Re: too many open files - user.dat
    By: Gamgee to DaiTengu on Sun May 29 2022 12:18 pm

    Slackware.
    MX Linux.
    Devuan.
    <a few more>

    Gentoo can run on either. My BBS runs under CentOS though, as it was just easier to deal with for the most part.

    at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

    DaiTengu

    ... Humour is emotional chaos remembered in tranquillity.

    ---
    Synchronet War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Gamgee@VERT to DaiTengu on Mon May 30 08:33:00 2022
    DaiTengu wrote to Gamgee <=-

    Slackware.
    MX Linux.
    Devuan.
    <a few more>

    Gentoo can run on either. My BBS runs under CentOS though, as
    it was just easier to deal with for the most part.

    at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

    Okay..... but <confused look> that isn't going to solve anything
    regarding using 'systemd', which is what the whole point of my comment
    was...

    Are you locked in to running it only on RedHat variants, or something?



    ... Want to meet new people? Pick up the wrong golf ball.
    === MultiMail/Linux v0.52
    --- SBBSecho 3.15-Linux
    * Origin: Palantir * palantirbbs.ddns.net * Pensacola, FL * (1:123/115)
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From Belly@VERT/BRAZINET to Gamgee on Mon May 30 12:17:53 2022
    Re: Re: too many open files - user.dat
    By: Gamgee to DaiTengu on Mon May 30 2022 08:33 am

    Are you locked in to running it only on RedHat variants, or something?

    Maybe he prefers it because of familiarity? I certainly prefer Debian, although I started on Slackware in the early 90's, and use SuSE at work currently (because we're licensed, and run software packaged for it) and Amazon Linux, because that's the best-supported distro to run on an EC2 instance.

    o
    (O)
    BeLLy

    ---
    Synchronet bbs.brazi.net www.brazi.net WARNING: May contain nuts
  • From DaiTengu@VERT/ENSEMBLE to Gamgee on Tue May 31 00:28:30 2022
    Re: Re: too many open files - user.dat
    By: Gamgee to DaiTengu on Mon May 30 2022 08:33 am

    Gentoo can run on either. My BBS runs under CentOS though, as
    it was just easier to deal with for the most part.

    at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

    Okay..... but <confused look> that isn't going to solve anything regarding using 'systemd', which is what the whole point of my comment was...

    I can hate systemd and still use it.

    RedHat variants are my OS of choice for systems I want to "just work" because I'm familiar with them.

    DaiTengu

    ... A oscillator will oscillate at the wrong frequency ...if it oscillates.

    ---
    Synchronet War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Gamgee@VERT/PALANT to DaiTengu on Tue May 31 10:28:00 2022
    DaiTengu wrote to Gamgee <=-

    Gentoo can run on either. My BBS runs under CentOS though, as
    it was just easier to deal with for the most part.

    at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

    Okay..... but <confused look> that isn't going to solve anything
    regarding using 'systemd', which is what the whole point of my comment was...

    I can hate systemd and still use it.

    Sure. Just seems a little odd. I also hate systemd and won't use a
    distro for that very reason.

    RedHat variants are my OS of choice for systems I want to "just
    work" because I'm familiar with them.

    Okey dokey.



    ... Gone crazy, be back later, please leave message.
    --- MultiMail/Linux v0.52
    Synchronet Palantir BBS * palantirbbs.ddns.net * Pensacola, FL
  • From DaiTengu@VERT/ENSEMBLE to Gamgee on Tue May 31 17:43:40 2022
    Re: Re: too many open files - user.dat
    By: Gamgee to DaiTengu on Tue May 31 2022 10:28 am

    Okay..... but <confused look> that isn't going to solve anything
    regarding using 'systemd', which is what the whole point of my
    comment was...

    I can hate systemd and still use it.

    Sure. Just seems a little odd. I also hate systemd and won't use a
    distro for that very reason.

    For the most part, I don't have that luxury. managing thousands of CentOS boxes is my day job. At least until everything moves over to Kubernetes.

    DaiTengu

    ... Heisenberg may have slept here

    ---
    Synchronet War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Nelgin@VERT/EOTLBBS to DaiTengu on Wed Jun 1 00:04:04 2022
    On Sat, 28 May 2022 14:24:51 -0500
    "DaiTengu" <daitengu@VERT/ENSEMBLE> wrote:

    I'm running into an issue where my system will lock up due to too
    many open files. It's been happening for a couple months. I upped
    the open file limit, but that didn't seem to help.

    Digital Man submitted a fix this afternoon which may address the open
    files issue. Mine seem to be at a steady level for now so you may want
    to update.
    --
    End Of The Line BBS - Plano, TX
    telnet endofthelinebbs.com 23
    ---
    Synchronet End Of The Line BBS - endofthelinebbs.com