Service Restart Errors?

[Xuru]

I think I am having issues with the service monitors thinking services are down and restarting them.

from the logs


Feb 8 23:40:13 sv013 named[1291]: starting (/etc/namedb/named.conf). named 8.3.4-REL-p1 Thu Jan 8 14:11:45 CST 2004 root@sv013.xuru.com:/usr/obj/usr/src/usr.sbin/named
Feb 8 23:40:13 sv013 named[1291]: limit files set to fdlimit (1024)
Feb 8 23:40:13 sv013 named[1291]: ctl_server: bind: /var/run/ndc: Address already in use
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [69.56.219.154].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [69.56.219.154].53
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [69.56.219.155].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [69.56.219.155].53
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [69.56.219.156].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [69.56.219.156].53
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [69.56.219.157].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [69.56.219.157].53
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [69.56.219.158].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [69.56.219.158].53
Feb 8 23:40:13 sv013 named[1291]: bind(dfd=20, [127.0.0.1].53): Address already in use
Feb 8 23:40:13 sv013 named[1291]: deleting interface [127.0.0.1].53
Feb 8 23:40:13 sv013 named[1291]: not listening on any interfaces
Feb 8 23:40:13 sv013 named[1293]: Ready to answer queries.
Feb 8 23:40:16 sv013 proftpd[1298]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:40:16 sv013 proftpd[1298]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 8 23:40:19 sv013 sshd[1303]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 8 23:40:19 sv013 sshd[1303]: fatal: Cannot bind any address.
Feb 8 23:41:09 sv013 proftpd[1530]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:41:09 sv013 proftpd[1530]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 8 23:42:07 sv013 proftpd[1750]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:42:07 sv013 proftpd[1750]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 8 23:43:09 sv013 proftpd[1977]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:43:09 sv013 proftpd[1977]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 8 23:44:13 sv013 named[2236]: starting (/etc/namedb/named.conf). named 8.3.4-REL-p1 Thu Jan 8 14:11:45 CST 2004 root@sv013.xuru.com:/usr/obj/usr/src/usr.sbin/named
Feb 8 23:44:13 sv013 named[2236]: limit files set to fdlimit (1024)
Feb 8 23:44:13 sv013 named[2236]: ctl_server: bind: /var/run/ndc: Address already in use
Feb 8 23:44:13 sv013 named[264]: ctl_writedone: /var/run/ndc: Broken pipe
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [69.56.219.154].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [69.56.219.154].53
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [69.56.219.155].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [69.56.219.155].53
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [69.56.219.156].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [69.56.219.156].53
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [69.56.219.157].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [69.56.219.157].53
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [69.56.219.158].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [69.56.219.158].53
Feb 8 23:44:13 sv013 named[2236]: bind(dfd=20, [127.0.0.1].53): Address already in use
Feb 8 23:44:13 sv013 named[2236]: deleting interface [127.0.0.1].53
Feb 8 23:44:13 sv013 named[2236]: not listening on any interfaces
Feb 8 23:44:13 sv013 named[2238]: Ready to answer queries.
Feb 8 23:44:16 sv013 proftpd[2243]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:44:16 sv013 proftpd[2243]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 8 23:44:19 sv013 sshd[2248]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 8 23:44:19 sv013 sshd[2248]: fatal: Cannot bind any address.


As you can see it keeps spawning new named processes over and over.

I am not sure if the same issue is causing the ftp and sshd issues over and over. I can find nowhere, where they would be trying to bind thierselves to 0.0.0.0

[DirectAdmin Support]

Hello,

Have a look at the boot scripts for named:

/usr/local/etc/rc.d/named

... have a look at the "restart" portion. Right now it will probably show:
restart() {
echo -n "Restarting Named: "
/usr/sbin/ndc restart > /dev/null
RETVAL=$?
if [ "$RETVAL" = "0" ];
then
echo -e "\t\t[ OK ]";
else
echo -e "\t\t[ FAILED ]";
fi
}.. but ndc might not be working properly, so you can just change it to:
restart() {
stop
start
}
John

[Xuru]

I thried that and no change.

But this is happening over and over every few minutes. So it's like it thinks Named is down, and tries to start a new version without shutting down the old. So I don't think it has to do with that script, I think it has to do with the service checking scripts.

I did look and there is no named..pid in /var/run/. Shouldn' t there be?

[Xuru]

I just checked the errortaskq.log and it was line after line for days of this


2004:02:09-13:33:03: service directadmin wasn't running, starting it
2004:02:09-13:33:06: service exim wasn't running, starting it
2004:02:09-13:33:09: service mysqld wasn't running, starting it
2004:02:09-13:33:12: service named wasn't running, starting it
2004:02:09-13:33:15: service proftpd wasn't running, starting it
2004:02:09-13:33:18: service sshd wasn't running, starting it
2004:02:09-13:33:21: service vm-pop3d wasn't running, starting it
2004:02:09-13:34:00: service mysqld wasn't running, starting it
2004:02:09-13:35:01: service httpd wasn't running, starting it
2004:02:09-13:35:04: service directadmin wasn't running, starting it
2004:02:09-13:35:07: service mysqld wasn't running, starting it
2004:02:09-13:35:10: service proftpd wasn't running, starting it
2004:02:09-13:35:13: service sshd wasn't running, starting it
2004:02:09-13:35:16: service vm-pop3d wasn't running, starting it
2004:02:09-13:36:00: service mysqld wasn't running, starting it
2004:02:09-13:37:00: service mysqld wasn't running, starting it
2004:02:09-13:38:01: service exim wasn't running, starting it
2004:02:09-13:38:04: service mysqld wasn't running, starting it
2004:02:09-13:39:00: service mysqld wasn't running, starting it
2004:02:09-13:40:00: service httpd wasn't running, starting it
2004:02:09-13:40:03: service directadmin wasn't running, starting it
2004:02:09-13:40:06: service mysqld wasn't running, starting it
2004:02:09-13:40:09: service proftpd wasn't running, starting it
2004:02:09-13:40:12: service sshd wasn't running, starting it
2004:02:09-13:40:15: service vm-pop3d wasn't running, starting it
2004:02:09-13:41:00: service mysqld wasn't running, starting it


They are definitally running becasue I can ftp, ssh, and so on with the syetm.

[UltimeWWW]

Feb 8 23:44:19 sv013 sshd[2248]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 8 23:44:19 sv013 sshd[2248]: fatal: Cannot bind any address.


Check if there's not some process running.

[DirectAdmin Support]

Hello,

Are you ssh'ed in at the time, and is it kicking you out?

John

[Xuru]

Yes I am ssh'ed in and no I am not kicked out. Also the same goes if I'm using ftp.

It's not shutting down the service and restarting them. It just keeps trying to spawn new services for each over and over again. I assume it thinks they are down.

[Xuru]

I am still lost on why it is doing this. I needed to do a few security updates anyway so I went ahead and did a full CVSup on bith src and ports and did a buildworld on th whole server.

Issue is still there :(

[Xuru]

Where are the serverice checker scriipts and thier names so I can have a look at them?

[DirectAdmin Support]

Hello,

They are in:

/usr/local/etc/rc.d/*

John

[Xuru]

Ok I did some testing.

I ran "/usr/local/etc/rc.d/named" with each of it's vars at the end "{start|stop|restart|reload|status}". The script and every one of it's commands works fine. Even the status returns the correct info. This confirms that ndc is working fine.

In the messages log file, the lines about bind not being able to bind to the addresses was happening every minute to every few minutes. So I ran a test. When I run this command

/usr/local/etc/rc.d/named start

without every stopping named in the firt place it returns that it was started ok but in the log files it has the same errors as you see above in the first post. I can do this every 5 seconds and the long list of named errors repeats itself each time. This confirms that something keeps running the "./named start" over and over even though the service is running.

This of course is happening to many different services, I am just concentrating on named as an example. My question is, what on the system is going to be calling the start scripts over and over?

[DirectAdmin Support]

Hello,

Not too sure. DirectAdmin only runs a "start" if it can't find any instances of the program running. You can check /var/log/directadmin/system.log to see DA's service running history, and see if any of those match up.

John

[Xuru]

Well I checked that log and the errortaskq log and both say that it is DA trying to start the services over and over. It seems to think that they are not up when they definitally are. I checked and there are pid files in the /var/run for each. Also I can ftp and ssh into the server and stay in so I know they never go down. But for some reason DA seems to thing they are down, doesn't attempt a restart, and instead does a start without shutting down the old. Below are logs from DA.

errortaskq.log over the past few minutes

2004:02:11-13:07:00: service httpd wasn't running, starting it
2004:02:11-13:07:04: service directadmin wasn't running, starting it
2004:02:11-13:07:07: service mysqld wasn't running, starting it
2004:02:11-13:07:10: service proftpd wasn't running, starting it
2004:02:11-13:07:13: service sshd wasn't running, starting it
2004:02:11-13:07:16: service vm-pop3d wasn't running, starting it
2004:02:11-13:08:00: service httpd wasn't running, starting it
2004:02:11-13:08:03: service directadmin wasn't running, starting it
2004:02:11-13:08:06: service mysqld wasn't running, starting it
2004:02:11-13:08:09: service proftpd wasn't running, starting it
2004:02:11-13:08:12: service sshd wasn't running, starting it
2004:02:11-13:08:15: service vm-pop3d wasn't running, starting it
2004:02:11-13:09:00: service httpd wasn't running, starting it
2004:02:11-13:09:03: service directadmin wasn't running, starting it
2004:02:11-13:09:06: service exim wasn't running, starting it
2004:02:11-13:09:10: service mysqld wasn't running, starting it
2004:02:11-13:09:13: service named wasn't running, starting it
2004:02:11-13:09:16: service proftpd wasn't running, starting it
2004:02:11-13:09:19: service sshd wasn't running, starting it
2004:02:11-13:09:22: service vm-pop3d wasn't running, starting it
2004:02:11-13:10:00: service proftpd wasn't running, starting it
2004:02:11-13:10:03: service sshd wasn't running, starting it
2004:02:11-13:10:06: service vm-pop3d wasn't running, starting it
2004:02:11-13:11:00: service httpd wasn't running, starting it
2004:02:11-13:11:03: service directadmin wasn't running, starting it
2004:02:11-13:11:06: service exim wasn't running, starting it
2004:02:11-13:11:09: service mysqld wasn't running, starting it
2004:02:11-13:11:12: service named wasn't running, starting it
2004:02:11-13:11:16: service proftpd wasn't running, starting it
2004:02:11-13:11:19: service sshd wasn't running, starting it
2004:02:11-13:11:22: service vm-pop3d wasn't running, starting it
2004:02:11-13:12:00: service httpd wasn't running, starting it
2004:02:11-13:12:03: service directadmin wasn't running, starting it
2004:02:11-13:12:06: service exim wasn't running, starting it
2004:02:11-13:12:09: service mysqld wasn't running, starting it
2004:02:11-13:12:12: service named wasn't running, starting it
2004:02:11-13:12:15: service proftpd wasn't running, starting it
2004:02:11-13:12:18: service sshd wasn't running, starting it
2004:02:11-13:12:21: service vm-pop3d wasn't running, starting it
2004:02:11-13:13:00: service httpd wasn't running, starting it
2004:02:11-13:13:03: service directadmin wasn't running, starting it
2004:02:11-13:13:06: service mysqld wasn't running, starting it
2004:02:11-13:13:09: service proftpd wasn't running, starting it



system.log over the last few minutes

2004:02:11-13:11:03: directadmin started
2004:02:11-13:11:06: exim started
2004:02:11-13:11:09: mysqld started
2004:02:11-13:11:12: named started
2004:02:11-13:11:16: proftpd started
2004:02:11-13:11:19: sshd started
2004:02:11-13:11:22: vm-pop3d started
2004:02:11-13:12:00: httpd started
2004:02:11-13:12:03: directadmin started
2004:02:11-13:12:06: exim started
2004:02:11-13:12:09: mysqld started
2004:02:11-13:12:12: named started
2004:02:11-13:12:15: proftpd started
2004:02:11-13:12:18: sshd started
2004:02:11-13:12:21: vm-pop3d started
2004:02:11-13:13:00: httpd started
2004:02:11-13:13:03: directadmin started
2004:02:11-13:13:06: mysqld started
2004:02:11-13:13:09: proftpd started
2004:02:11-13:13:13: sshd started
2004:02:11-13:13:16: vm-pop3d started
2004:02:11-13:14:00: httpd started
2004:02:11-13:14:03: directadmin started
2004:02:11-13:14:06: mysqld started
2004:02:11-13:14:09: proftpd started
2004:02:11-13:14:12: vm-pop3d started
2004:02:11-13:15:00: httpd started
2004:02:11-13:15:03: directadmin started
2004:02:11-13:15:06: exim started
2004:02:11-13:15:09: mysqld started
2004:02:11-13:15:12: named started
2004:02:11-13:15:15: proftpd started
2004:02:11-13:15:18: sshd started
2004:02:11-13:15:22: vm-pop3d started
2004:02:11-13:16:00: httpd started
2004:02:11-13:16:03: directadmin started
2004:02:11-13:16:06: mysqld started
2004:02:11-13:16:09: proftpd started
2004:02:11-13:16:12: vm-pop3d started

[DirectAdmin Support]

Oh... you probably don't have a /proc partition??

Check to see if it exists.

If not, run:


#to add it for future reboots
echo -e "proc\t\t\t/proc\t\tprocfs\trw\t\t0\t0" >> $FSTAB;

#to mount it now
/sbin/mount_procfs procfs /proc
John

[Xuru]

No there is a "/proc" and it gets mounted at boot

sv013# cat /var/log/mount.today
/dev/ad0s1a / ufs rw 1 1
devfs /dev devfs rw 0 0
procfs /proc procfs rw 0 0

I did your mount command to test anyway but even after that the repeated startups are still happening.

[Xuru]

I did find something strange. Take a look at the dir permissions


sv013# ll /proc
total 0
dr-xr-xr-x 1 ericmose ericmose 0 Feb 12 16:59 1000
d--------- 1 nobody nobody 0 Feb 12 16:59 10064
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 135
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 19
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 20
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 21
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 22
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 23
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 24
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 25
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 257
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 26
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 27
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 28
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 29
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 30
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 31
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 32
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 33
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 34
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 35
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 36
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 37
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 38
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 39
d--------- 1 root smmsp 0 Feb 12 16:59 399
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 40
d--------- 1 smmsp smmsp 0 Feb 12 16:59 402
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 41
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 42
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 420
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 43
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 44
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 45
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 46
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 47
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 48
d--------- 1 ftp ftp 0 Feb 12 16:59 48802
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 5
d--------- 1 mail mail 0 Feb 12 16:59 507
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 520
d--------- 1 root wheel 0 Feb 12 16:59 58762
d--------- 1 sv013 sv013 0 Feb 12 16:59 58764
dr-xr-xr-x 1 sv013 sv013 0 Feb 12 16:59 58765
d--------- 1 root wheel 0 Feb 12 16:59 58959
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 58966
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 592
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 593
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 594
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 595
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 596
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 59656
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 59694
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 597
d--------- 1 apache apache 0 Feb 12 16:59 59700
d--------- 1 apache apache 0 Feb 12 16:59 59701
d--------- 1 apache apache 0 Feb 12 16:59 59702
d--------- 1 apache apache 0 Feb 12 16:59 59703
d--------- 1 apache apache 0 Feb 12 16:59 59704
d--------- 1 apache apache 0 Feb 12 16:59 59705
d--------- 1 apache apache 0 Feb 12 16:59 59706
d--------- 1 apache apache 0 Feb 12 16:59 59707
d--------- 1 root mail 0 Feb 12 16:59 59726
d--------- 1 root mail 0 Feb 12 16:59 59727
d--------- 1 root mail 0 Feb 12 16:59 59728
d--------- 1 root mail 0 Feb 12 16:59 59729
d--------- 1 root mail 0 Feb 12 16:59 59730
d--------- 1 root mail 0 Feb 12 16:59 59731
d--------- 1 root mail 0 Feb 12 16:59 59732
d--------- 1 root mail 0 Feb 12 16:59 59733
d--------- 1 root mail 0 Feb 12 16:59 59734
d--------- 1 root mail 0 Feb 12 16:59 59735
d--------- 1 root mail 0 Feb 12 16:59 59736
d--------- 1 nobody nobody 0 Feb 12 16:59 59740
d--------- 1 nobody nobody 0 Feb 12 16:59 59741
d--------- 1 nobody nobody 0 Feb 12 16:59 59742
d--------- 1 nobody nobody 0 Feb 12 16:59 59743
d--------- 1 nobody nobody 0 Feb 12 16:59 59744
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 598
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 599
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 59969
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 6
dr-xr-xr-x 1 randjack randjack 0 Feb 12 16:59 60313
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 60415
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 60663
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 610
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 61131
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 61373
d--------- 1 mysql mysql 0 Feb 12 16:59 616
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 61632
d--------- 1 root mail 0 Feb 12 16:59 64237
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 64271
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 64921
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 7
d--------- 1 root wheel 0 Feb 12 16:59 70976
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 8
dr-xr-xr-x 1 root wheel 0 Feb 12 16:59 9
dr-xr-xr-x 1 jimbrown jimbrown 0 Feb 12 16:59 963
dr-xr-xr-x 1 kylehall kylehall 0 Feb 12 16:59 964
dr-xr-xr-x 1 stephorn stephorn 0 Feb 12 16:59 965
dr-xr-xr-x 1 wwwsecur wwwsecur 0 Feb 12 16:59 966
dr-xr-xr-x 1 richcoll richcoll 0 Feb 12 16:59 967
dr-xr-xr-x 1 chmccray chmccray 0 Feb 12 16:59 968
dr-xr-xr-x 1 danielco danielco 0 Feb 12 16:59 969
dr-xr-xr-x 1 robutler robutler 0 Feb 12 16:59 970
dr-xr-xr-x 1 tommbata tommbata 0 Feb 12 16:59 971
dr-xr-xr-x 1 jasonmeek jasonmeek 0 Feb 12 16:59 977
dr-xr-xr-x 1 philgarm philgarm 0 Feb 12 16:59 978
dr-xr-xr-x 1 garymccl garymccl 0 Feb 12 16:59 979
dr-xr-xr-x 1 ryangart ryangart 0 Feb 12 16:59 980
dr-xr-xr-x 1 biospons biospons 0 Feb 12 16:59 981
dr-xr-xr-x 1 bryanaue bryanaue 0 Feb 12 16:59 982
dr-xr-xr-x 1 baotran baotran 0 Feb 12 16:59 983
dr-xr-xr-x 1 drabbani drabbani 0 Feb 12 16:59 984
dr-xr-xr-x 1 joshparr joshparr 0 Feb 12 16:59 986
dr-xr-xr-x 1 jasofree jasofree 0 Feb 12 16:59 987
dr-xr-xr-x 1 naimeest naimeest 0 Feb 12 16:59 988
dr-xr-xr-x 1 caseycoa caseycoa 0 Feb 12 16:59 989
dr-xr-xr-x 1 bretbear bretbear 0 Feb 12 16:59 990
dr-xr-xr-x 1 bradscot bradscot 0 Feb 12 16:59 991
dr-xr-xr-x 1 paulswea paulswea 0 Feb 12 16:59 992
dr-xr-xr-x 1 markeric markeric 0 Feb 12 16:59 993
dr-xr-xr-x 1 mikechri mikechri 0 Feb 12 16:59 994
dr-xr-xr-x 1 chrismol chrismol 0 Feb 12 16:59 995
dr-xr-xr-x 1 briankli briankli 0 Feb 12 16:59 996
dr-xr-xr-x 1 afulford afulford 0 Feb 12 16:59 997
dr-xr-xr-x 1 ccalahan ccalahan 0 Feb 12 16:59 998
dr-xr-xr-x 1 shanewol shanewol 0 Feb 12 16:59 999


Shouldn't all of those dirs be r-xr-xr-x? Some are no rwx by anything at all.

[DirectAdmin Support]

Hello,

The /proc partition isn't an actual "disk".. it's created by the kernel from memory, which is why you can't write to it. It's basically just information about the system shown in "file" form.

John

[Xuru]

But why doesn't DA recognize that these services are running is what I need to figure out.

[DirectAdmin Support]

Hello,

Try upgrading to the latest version of DA. The older versions didn't use root to check the services, so if the permission don't allow "nobody" to read the files in the proc directory, then DA won't know that the programs are runnning, thus giving the restart-fest :)

Hmm.. but actually looking at the apache pid files, the permissions are:
d--------- 1 apache apache 0 Feb 12 16:59 59701which isn't readable by *anyone*.. hopefully root will override that.. else the system will have to be changed (I'm not sure how) so that the pid directories are at least readable by someone.

John

[Xuru]

I updated to the latest version. Well it changed how often the named is started when it shouldn't be. Now instead of every few minutes it starts named when it shouldn't every hour, to the exact second every hour. And things like proftp and ssh are still doing it every few minutes.


Feb 13 07:49:07 sv013 proftpd[95200]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:49:10 sv013 sshd[95204]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:49:10 sv013 sshd[95204]: fatal: Cannot bind any address.
Feb 13 07:50:06 sv013 proftpd[95427]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:50:06 sv013 proftpd[95427]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:50:09 sv013 sshd[95431]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:50:09 sv013 sshd[95431]: fatal: Cannot bind any address.
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [69.56.219.154].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [69.56.219.154].53
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [69.56.219.155].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [69.56.219.155].53
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [69.56.219.156].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [69.56.219.156].53
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [69.56.219.157].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [69.56.219.157].53
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [69.56.219.158].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [69.56.219.158].53
Feb 13 07:50:37 sv013 named[528]: bind(dfd=20, [127.0.0.1].53): Address already in use
Feb 13 07:50:37 sv013 named[528]: deleting interface [127.0.0.1].53
Feb 13 07:50:37 sv013 named[528]: not listening on any interfaces
Feb 13 07:51:07 sv013 proftpd[95656]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:51:07 sv013 proftpd[95656]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:51:10 sv013 sshd[95660]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:51:10 sv013 sshd[95660]: fatal: Cannot bind any address.
Feb 13 07:52:09 sv013 proftpd[95886]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:52:09 sv013 proftpd[95886]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:52:12 sv013 sshd[95890]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:52:12 sv013 sshd[95890]: fatal: Cannot bind any address.
Feb 13 07:53:09 sv013 proftpd[96115]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:53:09 sv013 proftpd[96115]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:53:12 sv013 sshd[96119]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:53:12 sv013 sshd[96119]: fatal: Cannot bind any address.
Feb 13 07:54:06 sv013 proftpd[96339]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:54:06 sv013 proftpd[96339]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:54:09 sv013 sshd[96343]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:54:09 sv013 sshd[96343]: fatal: Cannot bind any address.
Feb 13 07:55:09 sv013 proftpd[96585]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:55:09 sv013 proftpd[96585]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:55:12 sv013 sshd[96589]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:55:12 sv013 sshd[96589]: fatal: Cannot bind any address.
Feb 13 07:56:10 sv013 proftpd[96815]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:56:10 sv013 proftpd[96815]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:56:13 sv013 sshd[96819]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:56:13 sv013 sshd[96819]: fatal: Cannot bind any address.
Feb 13 07:57:09 sv013 proftpd[97046]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:57:09 sv013 proftpd[97046]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:57:12 sv013 sshd[97050]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:57:12 sv013 sshd[97050]: fatal: Cannot bind any address.
Feb 13 07:58:10 sv013 proftpd[97275]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:58:10 sv013 proftpd[97275]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:58:13 sv013 sshd[97279]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:58:13 sv013 sshd[97279]: fatal: Cannot bind any address.
Feb 13 07:59:06 sv013 proftpd[97499]: sv013.xuru.com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 13 07:59:06 sv013 proftpd[97499]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.
Feb 13 07:59:09 sv013 sshd[97503]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 13 07:59:09 sv013 sshd[97503]: fatal: Cannot bind any address.


In the permissions post you'll notice that the users that were not created by da and are running apps have no permission problems but for some reason all the permission problems seem to be with apps started by DA.

Can soneone else post the contents with "ll /proc" so I can compare?

Thanks

[DirectAdmin Support]

Hmm.. actually, now that I look at mine, the proc permissions on mine are similar to yours. I can cd into the directoires that are d---------. What does your system informatino show? are the visible? how about the "services status" ... they all use the same method for reading the proc directory.

One option is to turn off the checking in the /usr/local/directadmin/data/admin/services.status.. just set them to OFF.

John

[Xuru]

I tried to get some help in my favorite BSD forum and one of the mods over there said that /proc is no longer supported in 5.x.

http://www.freebsdforums.org/forums/showthread.php?s=&threadid=18892

He said it should use ioctl / sysctl. But of course other people have it running ok from what I can see. So I am lost at this point. Maybe since it has been downgraded and no longer officially supported something has to be configured to get it to work correctly that I'm not doing.

Can you post your rc.conf info and pkg_info readout?

[DirectAdmin Support]

defaultrouter="192.168.0.1"
hostname="my.hostname.com"
ifconfig_xl0="inet 192.168.0.100 netmask 255.255.255.0"
inetd_enable="YES"
ipv6_enable="YES"
kern_securelevel_enable="NO"
nfs_client_enable="YES"
nfs_server_enable="YES"
rpcbind_enable="YES"
sendmail_enable="NO"
sshd_enable="YES"
usbd_enable="YES"
and pkg_info was posted in another thread...
http://www.directadmin.com/forum/showthread.php?s=&threadid=1726&highlight=pkginfo

I think I'll look into reading it from the ioctl.. as that would be far more efficient anyway... we'll see how it turns out.

John

[davef8]

We are receiving the following error message on our Red Hat 9 box. I just pasted from the previous post obviously the domain name is different but the error is still the same.

Feb 8 23:40:16 sv013 proftpd[1298]: .com - Failed binding to 0.0.0.0, port 21: Address already in use
Feb 8 23:40:16 sv013 proftpd[1298]: sv013.xuru.com - Check the ServerType directive to ensure you are configured correctly.

This is showing up exactly once every minute.

No other services are producing errors. I think this might be caused by the fact that directadmin is registered to our domain name and then we have that same domain name setup in the user panel? Would this cause two instances of proftpd to start? I'm not sure if that is even remotely a potential cause. Any other ideas why this might be happening?

[DirectAdmin Support]

Hello,

Make sure that you have the hostname IP setup correctly in the /etc/hosts files. The hostname needs to point to the server IP, while localhost can go to 127.0.0.1. You can just change the hostname, then change it back through the admin settings screen to setup the /etc/hosts file correctly.

OR.. there may exist another ftp program running... you can check to see if there is another ftp program running by typing:

telnet localhost 21

The ServerType setting should be:

ServerType standalone

John

[davef8]

I checked the etc/hosts file and everything looked correct.

However I noticed we set the hostname to use the same domain as our primary hosting account name. Instead of using sever.domain.com for the host name and then domain.com for our primary hosting account name. The installation instructions specifically say not to do this.

Would this cause the error we see above? I turned off ftp status in
/usr/local/directadmin/data/admin/services.status

Does this solve the problem or just stop notification of the problem? Next I think I will change the hostname and see if that resolves this issue.

Update:
Tried changing the hostname and this had no effect.

[jmstacey]

I am also getting these errors and I think it may be the cause of my ftp problem with it getting slower and slower...

Running Freebsd 4.9 so perhaps this is a bug with da and freebsd?

-Jon-

[Xuru]

Originally posted by jmstacey
I am also getting these errors and I think it may be the cause of my ftp problem with it getting slower and slower...

Running Freebsd 4.9 so perhaps this is a bug with da and freebsd?

-Jon-

Are you talking about my erros, with all services or davef8 single error?

[jmstacey]

Yours. The services keep trying to start even thought their already running and I get errors like /kernel is using the ip address and no modules for 'ftp' etc and can't bind to 0.0.0.0.0 port whatever..

-Jon-

[davef8]

As it turns out we had vsftpd running the whole time. This was creating conflicts with proftpd. Once we turned vsftpd off, these errors stopped happening.

[Xuru]

jmstacey, did you ever find a solution, we are still looking for one :(


By the way I did find an ndc.core file in the directadmin directory which suggests there was an issue with ndc that started all this. But I have cvsup'ed the complete source of the server since then and tested that ndc works at the cmd line so I am not sure why da would be having an issue with it. That also doesn't explian why all other programs are restarting to.

[jmstacey]

Yes, but I don't think the cause is the same as yours.

I found my problem was due to a faulty router which tried to take the server ip address even though I set it to be static. I replaced the router and the problem is solved so I guess it was checking by its ip address as as it was trying to be stolen by another system it tried to restart it..

[DirectAdmin Support]

Xuru:
If your services are being restarted all the time, check the /var/log/directadmin/system.log to see if DA is doing it. If it is, check the /proc directory to make sure it exists. If DA can't read inside that, then it won't be able to see if the program is running and will assume its not. Next release the FreeBSD 5.x's will use the direct sysctl method, so the /proc directory shouldn't be an issue.

John

[Xuru]

Yea

The "/var/log/directadmin/system.log" shows it is definitally DA restarting them over and over.

I also checked and /proc is there, and it contains all the proper files with showing services are up.

I finally did as you suggested and in "/usr/local/directadmin/data/admin/services.status" set everything to off. But then DA kept turning them back on. I finally had to turn them off in that file and do a "chflags schg services.status" so it would keep them off.

I guess I'll just keep it that way until the sysctl method is implemened.

[DirectAdmin Support]

Hello,

DA is being bad if the services.status file is being changed every minute. I just had a look in the code and it's not being written by the dataskq... The only place it's written is when you turn programs on and off through the Admin Panel -> Services Info. When you click "start" it's set to on. If you click "stop" it's set to "OFF". Also having them set to OFF doesn't prevent DA from starting them when they're supposed to be restarted (like when a domain is added)... it only prevents DA from checking if they're running every minute.

John

[Xuru]

Yes Da only seems to be trying to write the file when I mess with the start/stop/restart in DA.

In the error.log I have

2004:02:28-20:13:23: Unable to move ./data/admin/services.status.temp to ./data/admin/services.status : Operation not permitted

It only has that logged at times that match up to when I was messing with da in the admin section.

So I guess it's not being bad on that end. but I still had to lock the file because even if it's not checking the services every minute with them being off, I don't need them turned back on when I used DA to restart them manually.

[DirectAdmin Support]

Hmm.. I'm wondering if turning it off should set them to OFF, but starting them should leave it alone.. If you turn them off and not change the file, then they'll just be started up again in a minute.. so it's hard to say what the correct answer is :)

John

[eagle]

Originally posted by jmstacey
Yours. The services keep trying to start even thought their already running and I get errors like /kernel is using the ip address and no modules for 'ftp' etc and can't bind to 0.0.0.0.0 port whatever..

-Jon-
I have the same on an install today on FreeBSD 4.9. Can't seem to get proftpd started. Looks like no other ftpd running.
Keeps restarting the thing every minute. Manually stopping gives an error. proftpd.vhost etc are empty. Any idea?

[existenz]

Did you reboot the server after install. I know is sounds stupid but I had same problems getting services to bind to the server ip after installs so now I just reboot when I get the error. I suspect I know what the problem is but I would rather reset everything.

[DirectAdmin Support]

Are there any errors in the /var/log/messages? .. or /var/log/proftpd/* ??

Also make sure that your /etc/hosts file has the hostname set as the main IP, and NOT 127.0.0.1... eg:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost localhost.localdomain
1.2.3.4 your.hostname.comJohn

[eagle]

hmmm, a reboot did the thing.....

tx existenz. i allways try to reboot as less as possible, but this was unavoidable??

[aleborg]

I know this thread is old but we're having the exact same problem as Xuru on a fresh installation.
system.log
2006:09:02-23:04:11: vm-pop3d started
2006:09:02-23:04:21: httpd started
2006:09:02-23:04:41: named started
2006:09:02-23:04:51: proftpd started
2006:09:02-23:05:11: vm-pop3d started
2006:09:02-23:31:00: directadmin started
2006:09:02-23:31:11: exim started
2006:09:02-23:31:21: httpd started
2006:09:02-23:31:41: named started
2006:09:02-23:31:51: proftpd started
2006:09:02-23:32:11: vm-pop3d started
2006:09:02-23:44:00: directadmin started
2006:09:02-23:44:11: exim started
2006:09:02-23:44:21: httpd started
2006:09:02-23:44:41: named started
2006:09:02-23:44:51: proftpd started
2006:09:02-23:45:11: vm-pop3d started
errortaskq.log
2006:09:02-23:37:00: getProcessList returned 19 processes
2006:09:02-23:38:00: getProcessList returned 18 processes
2006:09:02-23:39:00: getProcessList returned 19 processes
2006:09:02-23:40:00: getProcessList returned 20 processes
2006:09:02-23:41:00: getProcessList returned 18 processes
2006:09:02-23:42:00: getProcessList returned 19 processes
2006:09:02-23:43:00: getProcessList returned 20 processes
2006:09:02-23:44:00: service directadmin wasn't running, starting it
2006:09:02-23:44:10: getProcessList returned 17 processes
2006:09:02-23:44:10: service exim wasn't running, starting it
2006:09:02-23:44:21: getProcessList returned 17 processes
2006:09:02-23:44:21: service httpd wasn't running, starting it
2006:09:02-23:44:31: getProcessList returned 18 processes
2006:09:02-23:44:31: service mysqld wasn't running, starting it
2006:09:02-23:44:31: Error starting service mysqld : uid 0 gid 0
: /usr/local/etc/rc.d/mysqld start >>/dev/null 2>>/dev/null
2006:09:02-23:44:41: getProcessList returned 18 processes
2006:09:02-23:44:41: service named wasn't running, starting it
2006:09:02-23:44:51: getProcessList returned 18 processes
2006:09:02-23:44:51: service proftpd wasn't running, starting it
2006:09:02-23:45:00: getProcessList returned 17 processes
2006:09:02-23:45:01: getProcessList returned 19 processes
2006:09:02-23:45:01: service sshd wasn't running, starting it
2006:09:02-23:45:01: Error starting service sshd : uid 0 gid 0
: /usr/local/etc/rc.d/sshd start >>/dev/null 2>>/dev/null
2006:09:02-23:45:11: getProcessList returned 19 processes
2006:09:02-23:45:11: service vm-pop3d wasn't running, starting it
2006:09:02-23:45:21: getProcessList returned 18 processes
2006:09:02-23:46:00: getProcessList returned 17 processes
2006:09:02-23:47:00: getProcessList returned 17 processes
2006:09:02-23:48:00: getProcessList returned 18 processes
2006:09:02-23:49:00: getProcessList returned 17 processes
Direct Admin thinks the processes aren't running and tries to restart them although they are already up and running. We're using FreeBSD 6.1 on this machine.

What is the solution to this?

PS. The /proc file system is present.

[aleborg]

This seems to affect MySQL which sometimes instead shuts down for a while until DA tries to start it again.