Icecast with Liquidsoap not working - ubuntu-server

All;
Ubuntu 18.04 - Icecast / LiquidSoap / opam / (liquidsoap-daemon)
Installed: taglib mad lame Vorbis cry ssl samplerate magic opus
Plugins: liquidsoap-plugin-all
System Specs:
-System - VirtualBox
-CPU = Intel(R) Core(TM) i7-2720QM CPU # 2.20GHz
-Memory = 1969MiB (2GB assigned to VM Client)
I have been messing with this for a while now and to no avail.
I have this thread that is now closed, that has near about everything that you would need to look at about my setup. icecast request parsing failed
The playlist I'm testing:
#!/usr/bin/liquidsoap
# Log dir
set("log.file.path","/var/log/liquidsoap/radioliq.log")
# Music
myplaylist = playlist("/home/UserName/music-store/")
# Some jingles
jingles = playlist("/home/UserName/music-store/jingles/jingles.m3u")
# If something goes wrong, we'll play this
security = single("/home/UserName/music-store/Sec.mp3")
# Start building the feed with music
radio = myplaylist
# Now add some jingles
radio = random(weights = [1, 4],[jingles, radio])
# And finally the security
radio = fallback(track_sensitive = false, [radio, security])
# Stream it out
output.icecast(%vorbis,
radio, host = "192.168.2.20", port = 8000,
password = "********", mount = "test.ogg")
When I run this
sudo /etc/init.d/liquidsoap restart
I get this
Stopping liquidsoap channels: no script found in /etc/liquidsoap
Starting liquidsoap channels: radio1.liq OK
It shows starting radio1.liq, however, it is STOPPING, because it cannot find anything in that directory, however. That is where it is picking up on the radio1.liq file.
When I run
sudo systemctl status liquidsoap
My output is this:
● liquidsoap.service - LSB: Starts the liquidsoap daemon
Loaded: loaded (/etc/init.d/liquidsoap; generated)
Active: active (exited) since Tue 2018-06-12 17:47:52 EDT; 3h 0min ago
Docs: man:systemd-sysv-generator(8)
Process: 1312 ExecStart=/etc/init.d/liquidsoap start (code=exited, status=0/SUCCESS)
Jun 12 17:47:47 liquid-ice systemd[1]: Starting LSB: Starts the liquidsoap daemon...
Jun 12 17:47:52 liquid-ice liquidsoap[1312]: Starting liquidsoap channels: radio1.liq OK
Jun 12 17:47:52 liquid-ice systemd[1]: Started LSB: Starts the liquidsoap daemon.
As you can tell, the Active is (exited) when other times, it is (running) like Icecast is. However, I usually have to do a reboot on the system, in order to get it to show (running)
I just did a reboot, and it is still showing (exited), this is the first time that it has not changed to (running). And usually it will give me an error that I have to fix something, but it is not doing it now.
● icecast2.service - LSB: Icecast2 streaming media server
Loaded: loaded (/etc/init.d/icecast2; generated)
Active: active (running) since Tue 2018-06-12 17:47:50 EDT; 3h 1min ago
Docs: man:systemd-sysv-generator(8)
Process: 1338 ExecStart=/etc/init.d/icecast2 start (code=exited, status=0/SUCCESS)
Tasks: 4 (limit: 2289)
CGroup: /system.slice/icecast2.service
└─1403 /usr/bin/icecast2 -b -c /etc/icecast2/icecast.xml
Jun 12 17:47:48 liquid-ice systemd[1]: Starting LSB: Icecast2 streaming media server...
Jun 12 17:47:48 liquid-ice icecast2[1338]: * Starting streaming media server icecast2
Jun 12 17:47:50 liquid-ice icecast2[1338]: ...done.
Jun 12 17:47:50 liquid-ice systemd[1]: Started LSB: Icecast2 streaming media server.
when I open the web interface, I only get the following tabs (pages)
[Admin] | [Version]
All the other pages are all blank:
[Server Status] => Nothing, completely blank.
[Mountpoint List] => Active Mountpoints (This is all it shows on the page)
I also changed the username in the /etc/init.d/liquidsoap
user=username
group=username
Username Is what I am logged in as. And then gave permission to my user over the locations.
/etc/liquidsoap/radio1.liq
/run/liquidsoap/
using the following scripts:
sudo chmod o+w /run/liquidsoap/
sudo chmod o+w /etc/liquidsoap/radio1.liq
Here is the latest error log from the icecast error.log
[2018-08-08 02:22:18] INFO sighandler/_sig_die Caught signal 15, shutting down...
[2018-08-08 02:22:18] INFO main/main Shutting down
[2018-08-08 02:22:18] INFO fserve/fserve_shutdown file serving stopped
[2018-08-08 02:22:18] INFO slave/_slave_thread shutting down current relays
[2018-08-08 02:22:18] INFO slave/_slave_thread Slave thread shutdown complete
[2018-08-08 02:22:18] INFO auth/auth_shutdown Auth shutdown
[2018-08-08 02:22:18] INFO yp/yp_shutdown YP thread down
[2018-08-08 02:22:19] INFO stats/stats_shutdown stats thread finished
[2018-08-08 02:22:19] INFO main/main Icecast 2.4.3 server started
[2018-08-08 02:22:19] INFO connection/get_ssl_certificate No SSL capability
[2018-08-08 02:22:19] INFO stats/_stats_thread stats thread started
[2018-08-08 02:22:19] INFO yp/yp_update_thread YP update thread started
Exerts from the radio.log file
2018/08/09 04:35:03 [dynamic.loader:3] Could not find dynamic module for fdkaac encoder.
2018/08/09 04:35:03 [dynamic.loader:3] Could not find dynamic module for aacplus encoder.
..........
2018/08/09 04:35:03 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz master.
2018/08/09 04:35:03 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2018/08/09 04:35:03 [frame:3] Targetting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2018/08/09 04:35:03 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2018/08/09 04:35:03 [decoder:3] Method "MAD" accepted "/home/darkice/music/Security/sec.mp3".
2018/08/09 04:35:03 [single:3] "~/music/Security/sec.mp3" is static, resolving once for all...
2018/08/09 04:35:03 [threads:3] Created thread "generic queue #1".
2018/08/09 04:35:03 [threads:3] Created thread "non-blocking queue #1".
2018/08/09 04:35:03 [Jingle(dot)mp3:3] Loading playlist...
2018/08/09 04:35:05 [Jingle(dot)mp3:3] No mime type specified, trying autodetection.
2018/08/09 04:35:05 [playlist.basic:3] Wrong mime type audio/mpeg for playlist!
2018/08/09 04:35:05 [playlist parser:3] No format found
2018/08/09 04:35:05 [Jingle(dot)mp3:3] Could not parse playlist: Not_found
2018/08/09 04:35:05 [Jingle(dot)mp3:3] Got an empty list: keeping the old one.
2018/08/09 04:35:05 [musicList:3] Loading playlist...
2018/08/09 04:35:05 [musicList:3] Playlist is a directory.
2018/08/09 04:35:05 [musicList:3] Successfully loaded a playlist of 1 tracks.
2018/08/09 04:35:05 [single_6363:3] Prepared "/home/darkice/music/Security/sec.mp3" (RID 0).
2018/08/09 04:35:05 [test(dot)ogg:3] Connecting mount test.ogg for source#192.168.2.4...
2018/08/09 04:35:05 [test(dot)ogg:2] Connection failed: 401, Authentication Required (HTTP/1.0)
2018/08/09 04:35:05 [test(dot)ogg:3] Will try again in 3.00 sec.
2018/08/09 04:35:05 [threads:3] Created thread "wallclock_main" (1 total).
2018/08/09 04:35:05 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.
2018/08/09 04:43:34 [decoder:3] Method "MAD" accepted "/home/darkice/music/List/01 Send Her To Me.mp3".
2018/08/09 04:43:34 [test(dot)ogg:2] Connection failed: 401, Authentication Required (HTTP/1.0)
2018/08/09 04:43:34 [test(dot)ogg:3] Will try again in 3.00 sec.
2018/08/09 04:43:34 [threads:3] Created thread "wallclock_main" (1 total).
2018/08/09 04:43:34 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.
2018/08/09 04:43:34 [fallback_6368:3] Switch to single_6363.
2018/08/09 04:43:34 [carrzkiss:3] Prepared "/home/darkice/music/List/01 Send Her To Me.mp3" (RID 3).
2018/08/09 04:43:34 [fallback_6368:3] Switch to random_6366 with transition.
2018/08/09 04:35:05 [random_6366:3] Switch to musicList.
2018/08/09 04:35:09 [test(dot)ogg:3] Connecting mount test.ogg for source#192.168.2.4...
2018/08/09 04:35:09 [test(dot)ogg:2] Connection failed: 401, Authentication Required (HTTP/1.0)
2018/08/09 04:35:09 [test(dot)ogg:3] Will try again in 3.00 sec.
I followed this page here, to get the logging to start.
Is there something I am missing here? is there something else that needs to be installed, in order to make liquidsoap function properly?
I followed this tutorial on the installation. Create an internet radio station with Icecast and liquidsoap
I really do not know what else to add to this. I have gone through everything and even did some extras while writing this up. Please let me know if there is anything else that you may need for me to check on. And please, have a look at my other thread, for more information on what I have set up here. icecast request parsing failed
running the following code.
curl -v 192.168.2.4:8000 >/dev/null
gives me this.
* Rebuilt URL to: 192.168.2.4:8000/
* Trying 192.168.2.4...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.2.4 (192.168.2.4) port 8000 (#0)
> GET / HTTP/1.1
> Host: 192.168.2.4:8000
> User-Agent: curl/7.58.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: Icecast 2.4.3
< Date: Sun, 12 Aug 2018 18:28:52 GMT
< Content-Type: text/xml; charset=UTF-8
< Cache-Control: no-cache
< Expires: Mon, 26 Jul 1997 05:00:00 GMT
< Pragma: no-cache
< Access-Control-Allow-Origin: *
< Content-Length: 1144
<
{ [1144 bytes data]
100 1144 100 1144 0 0 95333 0 --:--:-- --:--:-- --:--:-- 101k
* Closing connection 0
Thank You.
Wayne

OK, now that I think I have enough information/data, let's pick this apart:
2018/08/09 04:35:03 [single:3] "~/music/Security/sec.mp3" is static, resolving once for all...
2018/08/09 04:35:05 [single_6363:3] Prepared "/home/darkice/music/Security/sec.mp3" (RID 0).
Your liquidsoap internal 'fallback' configured by 'security' just became available.
2018/08/09 04:35:03 [Jingle(dot)mp3:3] Loading playlist...
2018/08/09 04:35:05 [Jingle(dot)mp3:3] No mime type specified, trying autodetection.
2018/08/09 04:35:05 [Jingle(dot)mp3:3] Could not parse playlist: Not_found
2018/08/09 04:35:05 [Jingle(dot)mp3:3] Got an empty list: keeping the old one.
There is no playlist for jingles. I'm guessing it will just ignore that.
2018/08/09 04:35:05 [musicList:3] Loading playlist...
2018/08/09 04:35:05 [musicList:3] Playlist is a directory.
2018/08/09 04:35:05 [musicList:3] Successfully loaded a playlist of 1 tracks.
"/home/darkice/music/List/01 Send Her To Me.mp3" is the only file in the specified directory.
2018/08/09 04:35:05 [test(dot)ogg:3] Connecting mount test.ogg for source#192.168.2.4...
2018/08/09 04:35:05 [test(dot)ogg:2] Connection failed: 401, Authentication Required (HTTP/1.0)
It's trying to connect to your Icecast server, but Icecast rejects the connection.
The most likely reason for this is that the password doesn't match the one configured as <source-password> in the Icecast config.
You will find details as to why the connection failed in the Icecast error.log at the timestamp 2018/08/09 04:35:05 - please note that there may be an offset if this is a different computer and the system time of either machine is not synchronized by e.g. NTP.

It has been a while since I messed with Icecast2 and Liquidsoap.
However, I did just that a few weeks ago.
After struggling for some time, I figured it out after getting help from another person and watching a video.
It was all a permissions issue.
Once you create your user, "Icedemo."
(logged in as root)
adduser Icedemo
Give user SUDO rights.
usermod -aG sudo Icedemo
Now give permissions to Icedemo to all the folders.
/opt/liquidsoap/Playlists/
/var/log/liquidsoap/
/etc/icecast2/web/
/etc/icecast2/admin/
Use the following script to give permissions.
cd /opt
sudo chown -Rv icedemo:icedemo liquidsoap/
cd /var/log/
sudo chown -Rv icedemo:icedemo liquidsoap/
cd /etc/icecast2/
sudo chown -Rv icedemo:icedemo web/
sudo chown -Rv icedemo:icedemo admin/
Leave permission the same for the icecast.xml file.
Log in as your user Icedemo
su Icedemo
Once you have done all the above and your script is properly formatted. Something like this.
# !/home/iceadmin/.opam/default/bin/liquidsoap
# log dir
set ("log.file.path","/var/log/liquidsoap/basic-radio.log")
set("scheduler.fast_queues",1)
# Music
music = playlist("/opt/liquidsoap/playlists/RadioOne.m3u",mode="normal",reload_mode="watch")
# Start building the feed with music
radio = music
# Stream it out
output.icecast(%mp3(bitrate=256,samplerate=44100,internal_quality=0,id3v2=true,stereo=true,stereo_mode="stereo"),
name="Radio",
encoding="UTF-8",
host="192.168.2.203",
port=8000,
password="hackme",
icy_metadata="true",
description="Internet Radio Station",
mount="Radio",
mksafe(radio))
We will need to restart each service.
#Restart IceCast
sudo systemctl restart icecast2
#Check status of IceCast
sudo systemctl status icecast2
(Ctrl+Z to go to prompt)
Next, we will need to do the same to Liquidsoap.
#Restart Liquidsoap
sudo systemctl restart radio.liq-liquidsoap
#Check status of Liquidsoap
sudo systemctl status radio.liq-liquidsoap
(After checking the status, you can use (Ctrl+Z to go to prompt))
Your output of the Liquidsoap status should look similar to mine.
Icedemo#linuxcore-03:~/liquidsoap-daemon$ sudo systemctl status radio.liq-liquidsoap
● radio.liq-liquidsoap.service - radio.liq Liquidsoap daemon
Loaded: loaded (/etc/systemd/system/radio.liq-liquidsoap.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2021-05-24 03:47:01 UTC; 4s ago
Docs: http://liquidsoap.fm/
Process: 54678 ExecStart=/home/Icedemo/.opam/default/bin/liquidsoap /home/Icedemo/liquidsoap-daemon/script/radio.liq-run.liq (code=exited, status=0/SUCCESS)
Main PID: 54680 (liquidsoap)
Tasks: 9 (limit: 2251)
Memory: 17.3M
CGroup: /system.slice/radio.liq-liquidsoap.service
└─54680 /home/Icedemo/.opam/default/bin/liquidsoap /home/Icedemo/liquidsoap-daemon/script/radio.liq-run.liq
May 24 03:47:01 linuxcore-03 systemd[1]: Starting radio.liq Liquidsoap daemon...
May 24 03:47:01 linuxcore-03 systemd[1]: Started radio.liq Liquidsoap daemon.
If the above is showing: Active: active (running)
Next, check your Icecast webpage, and you should see your mountpoints, and your radio should be working.
To access the radio station in our example script above would be.
(local) 192.168.2.203:8000/Radio
This will load the player in its own browser window, and your stream should start on its own. If not, click the PLAY button.
That's it; that is how you get your streaming radio to work.
I hope this helps others out.
Wayne

Related

Restore fails on ovirt whether engine is on or off - how to get around

Wondering whether this is an issue;
I had an engine where i created a backup and stored elsewhere. This engine was reinstalled
and the backup file (called file_name) used to restore below. However i get 2 different
errors based on whether engine is on....
[root#dub-mgrfarm113 ~]# service ovirt-engine status
Redirecting to /bin/systemctl status ovirt-engine.service
● ovirt-engine.service - oVirt Engine
Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; disabled; vendor preset:
disabled)
Active: active (running) since Thu 2020-01-16 16:13:41 GMT; 2h 5min ago
Main PID: 5621 (ovirt-engine.py)
CGroup: /system.slice/ovirt-engine.service
├─5621 /usr/bin/python
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output
--systemd=notify st...
└─5661 ovirt-engine -server -XX:+TieredCompilation -Xms1991M -Xmx1991M -Xss1M
-Djava.awt.headless=true -Dsun.rmi.dgc.clien...
Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com systemd[1]: Starting oVirt Engine...
Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16
16:13:40,816+0000 ovirt-engine: INFO _detectJBossV...l=36
Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16
16:13:41,795+0000 ovirt-engine: INFO _detectJBossV...'[]'
Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com systemd[1]: Started oVirt Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[root#dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name
--restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: file_name
log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181932.log
Preparing to restore:
FATAL: Engine service is active - can not restore backup
or off.......
[root#dub-mgrfarm113 ~]# service ovirt-engine stop
Redirecting to /bin/systemctl stop ovirt-engine.service
[root#dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name
--restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: file_name
log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181952.log
Preparing to restore:
- Unpacking file 'file_name'
Restoring:
- Files
FATAL: Can't connect to database 'engine'. Please see
'/usr/bin/engine-backup --help'.
In addition, possibly related, the web page for the server will no longer launch even when
ovirt-engine and httpd services are running while.
Any troubleshooting tips or has anyone seen same before?

"failed to execute command: permission denied" Ubuntu 18.04.3 LTS

Trying to set up a game server for Ark on an old HP ProLiant running Ubuntu (version 18.04.3 LTS, 64-bit). Specs are 72GB RAM, Intel Xeon X5650 # 2.67 GHz x2. I'm learning Ubuntu along the way, so I barely know what I'm doing and realize I could just be making some silly error... but I'm totally lost. I managed to get a lot done thanks to Google, but even Google can't seem to help me anymore.
I've been using multiple guides to help me set it up.
https://ark.gamepedia.com/Dedicated_Server_Setup#Linux_.28via_systemd.29
http://arksurvivalevolved.gamewalkthrough-universe.com/dedicatedservers/linux/Default.aspx
https://survivetheark.com/index.php?/forums/topic/87419-guide-cluster-setup/
I've gone over every step in those guides multiple times and at least managed to get to this point where I'm stuck at this "permission denied" error.
I've tried every solution presented under this Google search: https://www.google.com/search?q=linux+%22failed+to+execute+command%3A+permission+denied%22
Additionally, I've tried executing the command to start the server with and without "sudo".
My guess is that the file it's trying to access is not permissible for some reason, but I can't seem to find a working solution for me.
[Unit]
Description=ARK: Survival Evolved dedicated server
Wants=network-online.target
After=syslog.target network.target nss-lookup.target network-online.target
[Service]
ExecStartPre=/home/kinare/steamcmd +login anonymous +force_install_dir /home/kinare/ark +app_update 376030
ExecStart=/home/kinare/ark/ShooterGame/Binaries/Linux/ShooterGameServer.exe Ragnarok?SessionName="Togerland - PVE Ragnarok"?AltSaveDirectoryName=RagSave?Port=7777?QueryPort=27015 -NoTransferFromFiltering -exclusivejoin -clusterid=Togerland
ShooterGameServer.exe Aberration_P?SessionName="Togerland - PVE Aberration"?AltSaveDirectoryName=AbSave?Port=7779?QueryPort=27017 -NoTransferFromFiltering -exclusivejoin -clusterid=Togerland
WorkingDirectory=/home/kinare/ark/ShooterGame/Binaries/Linux
LimitNOFILE=500000
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s INT $MAINPID
User=steam
Group=steam
[Install]
WantedBy=multi-user.target
Only including 2 of 6 maps that are within the cluster there to save space, hopefully that's enough.
Expected result should be it not failing to start... Error message:
ark-dedicated.service - ARK: Survival Evolved dedicated server
Loaded: loaded (/etc/systemd/system/ark-dedicated.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-10-18 15:35:19 EDT; 56s ago
Process: 6383 ExecStartPre=/home/kinare/steamcmd +login anonymous +force_install_dir /home/kinare/ark +app_update 376030 (code=exited, status=203/EXEC)
Oct 18 15:35:19 togerland-server systemd[1]: Starting ARK: Survival Evolved dedicated server...
Oct 18 15:35:19 togerland-server systemd[6383]: ark-dedicated.service: Failed to execute command: Permission denied
Oct 18 15:35:19 togerland-server systemd[6383]: ark-dedicated.service: Failed at step EXEC spawning /home/kinare/steamcmd: Permission denied
Oct 18 15:35:19 togerland-server systemd[1]: ark-dedicated.service: Control process exited, code=exited status=203
Oct 18 15:35:19 togerland-server systemd[1]: ark-dedicated.service: Failed with result 'exit-code'.
Oct 18 15:35:19 togerland-server systemd[1]: Failed to start ARK: Survival Evolved dedicated server.
your systemd service uses user and group steam
...
User=steam
Group=steam
...
you are starting your ark server from the home of kinare
ExecStart=/home/kinare/ark/ShooterGame/Binaries...
and your system logs says: 'Permission denied':
Oct 18 15:35:19 togerland-server systemd[6383]: ark-dedicated.service: Failed to execute command: Permission denied
does the steam user have permissions to read files in /home/kinare?
You can solve this in a few ways:
give the steam user permissions to read from /home/kinare
# change the group of all files and dirs in /home/kinare to steam
chgrp -R steam /home/kinare
# give the group read rights on all files and dirs /home/kinare
chmod -R g+r /home/kinare
# allow the group to open folders under /home/kinare
find /home/kinare -type d -exec chmod 750 {} \;
use service account
move your ark and steam to the home of the steam user (/home/steam) and change
your unit file as needed. keep in mind that you need change the permissions of
the files in /home/steam. This is preferred, you use a service account instead
of your admin user kinare
change the user and group used in your systemd service file
User=kinare
Group=kinare
ark will now run as the user kinare. This is less preferred, see:
https://unix.stackexchange.com/questions/314725/what-is-the-difference-between-user-and-service-account
hope this helps, good luck

Inconsistent systemd startup of freeswitch

I have two problems running freeswitch from systemd :
EDIT 2 - I have moved the slow start up question to here (Freeswitch pauses on check_ip at boot on centos 7.1) as although they may be related it's probably good as a standalone.
EDIT - I have noticed something else. Look at these next lines captured from the terminal output when running it from there. The gap is 4 minutes but it has been around 10 minutes before. I noticed it because I was trying to find out why port 8021 was taking several minutes to accept the fs_cli connection. Why does this happen? Never happened to me before and I've installed loads of FS boxes. This does the same thing on both 1.7 & todays 1.6.
2015-10-23 12:57:35.280984 [DEBUG] switch_scheduler.c:249 Added task 1 heartbeat (core) to run at 1445601455
2015-10-23 12:57:35.281046 [DEBUG] switch_scheduler.c:249 Added task 2 check_ip (core) to run at 1445601455
2015-10-23 13:01:31.100892 [NOTICE] switch_core.c:1386 Created ip list rfc6598.auto default (deny)
I sometimes get double processes started. Here is my status line after such an occurrence :
# systemctl status freeswitch -l
freeswitch.service - freeswitch
Loaded: loaded (/etc/systemd/system/multi-user.target.wants/freeswitch.service)
Active: activating (start) since Fri 2015-10-23 01:31:53 BST; 18s ago
Main PID: 2571 (code=exited, status=0/SUCCESS); : 2742 (freeswitch)
CGroup: /system.slice/freeswitch.service
├─usr/bin/freeswitch -ncwait -core -db /dev/shm -log /usr/local/freeswitch/log -conf /usr/local/freeswitch/conf -run /usr/local/freeswitch/run
└─usr/bin/freeswitch -ncwait -core -db /dev/shm -log /usr/local/freeswitch/log -conf /usr/local/freeswitch/conf -run /usr/local/freeswitch/run
Oct 23 01:31:53 fswitch-1 systemd[1]: Starting freeswitch...
Oct 23 01:31:53 fswitch-1 freeswitch[2742]: 2743 Backgrounding.
and there are two processes running.
The PID file is sometimes not written fast enough for the systemd process to pick it up, but by the time I see this (no matter how fast I run the command) it's always there by the time I do :
Oct 23 02:00:26 arribacom-sbc-1 systemd[1]: PID file
/usr/local/freeswitch/run/freeswitch.pid not readable (yet?) after
start.
Now, in (2) everything seems to work ok, and I can shut down the freeswitch process using
systemctl stop freeswitch
without any issues, but in (1) it just doesn't seem to do anything.
I'm wondering if the two are related, and that freeswitch is reporting back to systemd that the program is running before it actually is. Then systemd is either starting up another process or (sometimes) not.
Can anyone offer any pointers? I have tried to mail the freeswitch users list but despite being registered I simply cannot get any emails to appear on the list (but that's another problem).
* Update *
If I remove the -ncwait it seems to improve the double process starting but I still get the can't read PID warning, so I'm still sure there's an issue present, possibly around timing(?).
I'm on Centos 7.1, & my freeswitch version is
FreeSWITCH Version 1.7.0+git~20151021T165609Z~9fee9bc613~64bit (git
9fee9bc 2015-10-21 16:56:09Z 64bit)
and here's my freeswitch.service file (some things have been commented out until I understand what they are doing and any side effects they may have) :
[Unit]
Description=freeswitch
After=syslog.target network.target
#
[Service]
Type=forking
PIDFile=/usr/local/freeswitch/run/freeswitch.pid
PermissionsStartOnly=true
ExecStart=/usr/bin/freeswitch -nc -core -db /dev/shm -log /usr/local/freeswitch/log -conf /u
ExecReload=/usr/bin/kill -HUP $MAINPID
#ExecStop=/usr/bin/freeswitch -stop
TimeoutSec=120s
#
WorkingDirectory=/usr/bin
User=freeswitch
Group=freeswitch
LimitCORE=infinity
LimitNOFILE=999999
LimitNPROC=60000
LimitSTACK=245760
LimitRTPRIO=infinity
LimitRTTIME=7000000
#IOSchedulingClass=realtime
#IOSchedulingPriority=2
#CPUSchedulingPolicy=rr
#CPUSchedulingPriority=89
#UMask=0007
#
[Install]
WantedBy=multi-user.target
In the current master branch, take the two files from debian/ directory:
freeswitch-systemd.freeswitch.service -- should go as /lib/systemd/system/freeswitch.service
freeswitch-systemd.freeswitch.tmpfile -- should go as /usr/lib/tmpfiles.d/freeswitch.conf
You probably need to adapt the paths, or build FreeSWITCH to use standard Debian paths.

Redis and memory

I am developing software in an embedded system (512 MB RAM). I'm using redis to take the place of a shared memory between processes inside a django application.
We are talking about 150 values, stored every second, coming from a MODBUS device. They all have the same key and their expire time is 10 minutes.
After some work hours (tipically a day), redis ceases to function, due to memory problems. Can someone help me out?
output of ps aux | grep redis
redis 1934 1.9 2.2 76216 8400 ? Ssl 07:49 10:37 /usr/bin/redis-server 127.0.0.1:6379
redis.info
# Server
redis_version:2.8.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7cc01333adfd1c61
redis_mode:standalone
os:Linux 3.4.79+ armv7l
arch_bits:32
multiplexing_api:epoll
gcc_version:4.6.3
process_id:1934
run_id:be418b5a05b6670bb4bff9c73cc7126589d6b5c8
tcp_port:6379
uptime_in_seconds:33584
uptime_in_days:0
hz:10
lru_clock:858206
config_file:/etc/redis/redis.conf
# Clients
connected_clients:138
client_longest_output_list:54
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:6893800
used_memory_human:6.57M
used_memory_rss:6152192
used_memory_peak:47902480
used_memory_peak_human:45.68M
used_memory_lua:25600
mem_fragmentation_ratio:0.89
mem_allocator:jemalloc-3.0.0
# Persistence
loading:0
rdb_changes_since_last_save:1370469
rdb_bgsave_in_progress:0
rdb_last_save_time:1434611837
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:155
total_commands_processed:3207775
instantaneous_ops_per_sec:55
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:297
evicted_keys:0
keyspace_hits:2141758
keyspace_misses:12495
pubsub_channels:6
pubsub_patterns:0
latest_fork_usec:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:295.85
used_cpu_user:333.85
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Keyspace
db0:keys=2,expires=2,avg_ttl=3185003
db1:keys=937,expires=242,avg_ttl=585637
snippet of redis-server.log
[1969] 18 Jun 22:17:21.819 # Server started, Redis version 2.8.6
[1969] 18 Jun 22:17:21.919 * DB loaded from disk: 0.100 seconds
[1969] 18 Jun 22:17:21.919 * The server is now ready to accept connections on port 6379
[1969] 18 Jun 22:17:21.919 * The server is now ready to accept connections at /var/run/redis/redis.sock
[1969] 19 Jun 09:16:50.444 # Client addr=127.0.0.1:38745 fd=9 name= age=39516 idle=4330 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=13724 oll=415 omem=8509160 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
[1969] 19 Jun 09:20:54.056 # Client addr=127.0.0.1:38759 fd=14 name= age=4713 idle=4331 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=13688 oll=415 omem=8509160 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
[1941] 19 Jun 09:17:17.134 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
Some lines from redis-cli monitor, if someone finds that useful. Being the same keys rewritten over and over again, it puzzles me with the high amount of memory used and all those files descriptors.
http://pastebin.com/rQqThUHF

TCP establishment delays for 3 seconds

Most of time, my rsh cycle is general OK, we could get following logs from rshd:
Aug 19 04:36:34 shmm500 authpriv.info in.rshd[21343]: connect from 172.17.0.40 (172.17.0.40)
Aug 19 04:36:34 shmm500 auth.info rshd[21344]: root#172.17.0.40 as root: cmd='echo 481'
While for some error case, the rsh could success but there are several seconds delay, see the following timestamp:
Aug 19 04:12:24 shmm500 authpriv.info in.rshd[17968]: connect from 172.17.0.40 (172.17.0.40)
Aug 19 04:12:27 shmm500 auth.info rshd[17972]: root#172.17.0.40 as root: cmd='echo 18'
I also found that, for most normal case, the PID increased by 1, while for most error case, PID increasd by 4, see the PID in above logs, seems rshd forks some processes. So would you provide any explanation for why rshd took these several seconds and PID increase.
Our rsh is the old rsh, not ssh, I'm not sure, but seems the rsh is from netkit. And this is an embedded board with busybox, no strace/pstack.
For client side, I just 'rsh 172.17.0.8 pwd', not hostname is used.
Answer the question by myself:
This issue was caused by a frame loss. Either SYN or SYN+ACK in 3-way handshake was dropped at a rare rate for some reason, anyway the client peer didn't get the SYN+ACK within in 3 seconds timeout(this timeout is hardcoded in Linux kernel), then the connect() resent SYN again, and usually successful at the second try.
From the viewpoint of application, we got 3 seconds delay, or even 6 seconds if it failed at the second try.
Other relevant information:
The first log is from tcpd(aka tcp wrapper)
Aug 19 04:36:34 shmm500 authpriv.info in.rshd[21343]: connect from 172.17.0.40 (172.17.0.40)
The second log is from rshd in netkit 0.17
Aug 19 04:36:34 shmm500 auth.info rshd[21344]: root#172.17.0.40 as root: cmd='echo 481'
rsh need two tcp connections, the first is from rsh client to rshd, and the second tcp connection is from rshd to rsh client, which means the rshd is the tcp client. And my issue is frame loss on the second tcp connection.

Resources