On 23/02/2021 18:15, steve-ALUG@hst.me.uk wrote:

Hi folks,

My *buntu server runs a mail server.  At 00:00 on some days, it complains that it can't reach the spam analyser or the virus analyser, despite the fact that they're still running and always have been, as far as I can tell.

Other than stopping up to see what's running at midnight, is there a way of monitoring what's running at a particular time.  I know you can see what's slowing things down at boot time using "system-analyze blame", or using pybootchartgui.  Is there a way to do something like this around midnight?

Also, of the top of your collective heads, can you think of any services update or hog processor at 00:00.

How are scheduled tasks started, so I can have a root around and see what's happening?

I can think of various ways.

1. A program (daemon) can constantly run, and may update at a specific

time. 2) crontab 3) "at"  perhaps. Any others?

I've also found/remembered

/etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/cron.monthly /etc/cron.weekly

I've also thought that logrotate may be running...

Any thoughts?

On Tue, 23 Feb 2021 18:15:26 +0000 steve-ALUG@hst.me.uk allegedly wrote:

My *buntu server runs a mail server.  At 00:00 on some days, it complains that it can't reach the spam analyser or the virus analyser, despite the fact that they're still running and always have been, as far as I can tell.

Other than stopping up to see what's running at midnight, is there a way of monitoring what's running at a particular time.  I know you can see what's slowing things down at boot time using "system-analyze blame", or using pybootchartgui.  Is there a way to do something like this around midnight?

Also, of the top of your collective heads, can you think of any services update or hog processor at 00:00.

Steve

What (if anything) does syslog say? Have you checked your (r)syslog.conf to see where cron (or any odd daemon) is logging?

Mick

--------------------------------------------------------------------- Mick Morgan gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312 https://baldric.net/about-trivia ---------------------------------------------------------------------

On 24/02/2021 14:17, mick wrote:

On Tue, 23 Feb 2021 18:15:26 +0000 steve-ALUG@hst.me.uk allegedly wrote:

...

My *buntu server runs a mail server.  At 00:00 on some days, it complains that it can't reach the spam analyser or the virus analyser, despite the fact that they're still running and always have been, as far as I can tell.

Other than stopping up to see what's running at midnight, is there a way of monitoring what's running at a particular time.  I know you can see what's slowing things down at boot time using "system-analyze blame", or using pybootchartgui.  Is there a way to do something like this around midnight?

Also, of the top of your collective heads, can you think of any services update or hog processor at 00:00.

Steve

What (if anything) does syslog say? Have you checked your (r)syslog.conf to see where cron (or any odd daemon) is logging?

Mick

Thanks Mick,

That's a start.  rsyslog.conf has no uncommented lines for cron, so cron logs to syslog.

Syslog says, the last time the error happened,

00:35 systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab... 00:36 exim4: ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail system possibly broken 00:36 systemd[1]: Starting exim4-base housekeeping... 00:36 systemd[1]: Starting Daily man-db regeneration... 00:37 systemd[1]: fstrim.service: Succeeded. 00:37 systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab. 00:37 exim[339807]: 2021-02-22 00:37 MAIL_HEADER_ID failed to write to main log: length=91 result=-1 errno=9 (Bad file descriptor) 00:37 exim[339807]: write failed on panic log: length=116 result=-1 errno=9 (Bad file descriptor) 00:37 systemd[1]: exim4-base.service: Succeeded. 00:37 systemd[1]: Finished exim4-base housekeeping. 00:37 systemd[1]: Starting Rotate log files...

So a shed-load of things were getting started.  These seem to have been started by systemd using config files in /etc/systemd/system/timers.target.wants

Some of the systemd timers specify a time to run, the rest don't and say something like daily.  The daily entries are started at 00:00, so I guess systemd is trying to run multiple jobs a the same time.

I shall try changing the OnCalendar=daily to OnCalendar=*-*-* 01:09:00 to see if it works better at a different time.

Thanks for starting me in the right direction.  I shall see where it leads.

I didn't know systemd ran scheduled tasks too!

Steve

On Wed, 24 Feb 2021 21:09:35 +0000 steve-ALUG@hst.me.uk allegedly wrote:

I didn't know systemd ran scheduled tasks too!

In my view systemd does a shit load of stuff it has no rational business being involved in. (DNS FFS?)

Feature creep on a gargantuan scale. (But then I'm a beardy old skool unix fan.)

Mick

--------------------------------------------------------------------- Mick Morgan gpg fingerprint: FC23 3338 F664 5E66 876B 72C0 0A1F E60B 5BAD D312 https://baldric.net/about-trivia ---------------------------------------------------------------------

On 25/02/2021 19:24, steve-ALUG@hst.me.uk wrote:

On 24/02/2021 21:09, steve-ALUG@hst.me.uk wrote: [SNIP]

My change has successfully changed the restart time.  Time will tell if this fixes my problem.

<chagrin mode>

OK, so having successfully managed to change the, the next time I got an error it occurred at the newly scheduled time.  Gosh what's happening at that time thinks me.

Then I think a bit more.  The email's being generated at that time, but the error is happening at sometime else.  I would have helped if I had looked a the contents of the error email, and not the timestamp of the error email.

Numpty!

</chagrin mode>

Steve