Posted by MekkisSomething like liquidsoap, that only makes outgoing connections (rather than accepting connections from the internet) has a much narrower threat profile than most online services.

Mekkis: I fully agree with your analysis.

I have to admit, I nearly swallowed my own teeth when I read that Elrond was running Windows XP SP1. Being Windows XP aside, Elrond was running a years-old unpatched server -- practically a welcome mat for hackers.

But like you said -- there are only outgoing connections, so it would seem that the likelihood of any listener being harmed would be nearly zero. That doesn't mean that a badguy wouldn't try to hijack a music server for means completely unrelated to music, though.

It's just experience from the last couple years has shown me that server admins shouldn't take anything for granted. Exploits like Heartbleed (OpenSSL) and Shellshock (Bash) were simple, but egregiously damaging software bugs in software that were commonly taken for granted as being totally safe. And you could say that neither of those should affect streaming -- and I would also agree. But my point is that we are in an era of rapid cat-and-mouse exploits and fixes, and some of these exploits come from the least expected places. To that end, it's simply a "best practice" to stay current on all security updates.

----------

Elrond: I'll be listening to music, no matter how you go about it. I'm just trying to be a helpful voice of professional experience, nothing more. Unfortunately for my part -- I know very little about the streaming side, but Mekkis certainly seems to be on top of it.

Posted by ElrondYay, I think it's running fine now. I have made it public on the site and we should be able to tune in and listen.

Thank you very much for bringing the music back! I can't say for certain -- but my office plants may have also missed Radio Rivendell.

I understand you're still rebuilding, but I do have one teensy request: Can the service be fixed so I can tune in with my workstation's media software? Right now, all I can seem to get is the Radio Rivendell web player.

But again, thank you for bringing the music back.

Oh, I'm definitely not advocating not keeping up-to-date with patches. In my experience, the hardest part about administering a server is the kind of periodic rebuilds that are needed when updating from one major version to another: if I were running it, I'd get the LTS version, (14) and just keep applying updates for three years, before updating to whatever the latest LTS is when 14 drops out of support. Regarding relays, last month I did setup a shoutcast instance on a server (through the Donate Bandwidth area). It's still up, but I can configure an icecast instance on it if it's more suitable. Arclayn: For your media software, try tuning into http://62.65.107.19:8000/112kbit.ogg.m3u it seems to work.

Posted by MekkisArclayn: For your media software, try tuning into http://62.65.107.19:8000/112kbit.ogg.m3u it seems to work.

I briefly tried it out on Tuesday, but the quality was noticeably lower than the 128K MP3 streams I previously listened to. Today (Wednesday), I'm not getting anything (tried VLC and iTunes on my workstation). Ah well.

I'll be grateful when this part of the infrastructure is back. Having the web browser music window feels clumsy when I also have to use my web browser with my work.

Updated 13 January 2016 (21:16)

Posted by ArclaynI briefly tried it out on Tuesday, but the quality was noticeably lower than the 128K MP3 streams I previously listened to. Today (Wednesday), I'm not getting anything (tried VLC and iTunes on my workstation). Ah well.I'll be grateful when this part of the infrastructure is back. Having the web browser music window feels clumsy when I also have to use my web browser with my work.

Ah, there appears to be an update to that address: http://62.65.107.19:8000/128kbit.mp3

This is working pretty good.

Thanks all for you info and ideas

I will post http://62.65.107.19:8000/128kbit.mp3 in the news section as well, for people to tune in using Winamp, iTunes etc. I will also try the script posting thingy. I think posting the track change with LiquidSoap is the best, then it will be triggered just once, and just when that instant happens. I'll store this on the webserver in a static file, for usage on the website.

Mekkis, what would the http-post look like? I will be posting to www.radiorivendell.com (with some security measures)

So, when I set the history_func to:

def history_func(h)

ignore(http.post("http://44.128.88.104/", data=h"filename"))

end

I received the following:

$ nc6 -l -p 80

nc6: using stream socket

POST / HTTP/1.0

Host: 44.128.88.104

User-Agent: Liquidsoap/1.1.1 (Unix; OCaml 4.01.0)

Content-Length: 64

/music/Josh_Woodward_-_I_Want_To_Destroy_Something_Beautiful.mp3

OK - i reckon I can donate bandwith I have a fairly fast connection (approx 75 down and 20 up) by UK standards and now have a spare Mac Mini running Snow Leopard that I can leave on 24/7. Also unlimited (within limits) data on this package When you get it all up and running again drop me a line and I'll see if I can set up this end. Only (slight) problem is that I don't think I can get a static IP address.

Updated 17 January 2016 (17:32)

Turns out you can do it with a bit more information with: def history_func(h) ignore(http.post("http://44.128.88.104/", data=json_of(h))) end Which results in nicely jsoned fields like: POST / HTTP/1.0 Host: 44.128.88.104 User-Agent: Liquidsoap/1.1.1 (Unix; OCaml 4.01.0) Content-Length: 1054 { "comment:itunpgap": "0", "comment:itunsmpb": "00000000 00000210 000007EC 0000000000BF7504 00000000 00AD90A7 00000000 00000000 00000000 00000000 00000000 00000000", "album": "IDMf004: Length and Brecht /Remixed/", "kind": "{audio=2;video=0;midi=0}", "artist": "Halogen", "decoder": "MAD", "title": "Length and Brecht /Synaecide Remix/", "filename": "/music/Halogen_-_03_-_Length_and_Brecht_Synaecide_Remix.mp3", "temporary": "false", "source": "ccmusic(dot)pls", "tracknumber": "3/4", "initial_uri": "/music/Halogen_-_03_-_Length_and_Brecht_Synaecide_Remix.mp3", "comment:itunes_cddb_1": "28048104+86664+4+150+24603+40597+61936", "status": "playing", "comment:itunnorm": "00000661 00000704 00005051 0000580B 0002245E 000253EB 00008903 0000866D 0002C3A7 0003BA49", "encodedby": "iTunes v7.6.2", "comment:itunes_cddb_tracknumber": "3", "on_air": "2016/01/17 23:39:59", "rid": "1", "comment": "Artwork by www.Goreki.com, visit IDMForums.com for more releases & discussion. Produced by Nathan Schwartz (Permutated).", "genre": "Compilation" } (Please don't comment on my choice of music: it was the first bit of Creative Commons music I could get my hands on)

Updated 17 January 2016 (13:44)

Here is a PHP gist that I wrote up with a couple of functions for obtaining status info from Icecast 2.4+:

https://gist.github.com/infinitydev/2b6344940357264e9293

Features:

- Queries http://server/status-json.xsl for Icecast server status

- Easy array access to all data

- Radio Rivendell stream up/down detection

- Query current track name

- Listener count as sum of all Radio Rivendell streams

- Attempts to fix sometimes broken JSON syntax returned by Icecast 2.4.0

- Supports servers that serve additional stations (other than Radio Rivendell)

- Some demo code that queries multiple servers

Demo output: https://byte-storm.com/test/icecast.php

Maybe it helps bringing stats and server status pages back.

Excellent guys!

I have the following in my script now... This is just a test to see how it works, and not the final solution

def post_now_playing(m)

system("#php now_playing.php >> now.txt"

end

radio = on_metadata(post_now_playing, radio);

However, it doesn't seem to work... if I run "php now_playing.php >> now.txt" in the shell, I'll get the output to the file. But the function doesn't seem to trigger in Liquidsoap, or the command isn't executed properly. Any ideas?

Posted by ElrondExcellent guys!I have the following in my script now... This is just a test to see how it works, and not the final solution def post_now_playing(m)system("#php now_playing.php >> now.txt"endradio = on_metadata(post_now_playing, radio);However, it doesn't seem to work... if I run "php now_playing.php >> now.txt" in the shell, I'll get the output to the file. But the function doesn't seem to trigger in Liquidsoap, or the command isn't executed properly. Any ideas?

You don't need the #.

In liquidsoap, #{expression} within double-quotes means "evaluate expression and insert it here", so in my example, "echo #{quote(h'filename')} > /tmp/nowplaying}" takes the filename of the current playing track (that's what the argument to the function is), and quotes it to remove special characters, before using it as an argument to echo.

In many unix shells, (probably like what system() uses), # is used to comment the line out.

Yay, I got my script to work. Liquidsoap was run as a service, so I had of course give the absolute path to the script. I will go for the script that posts the JSON I think. That will probably be the simplest. And then run the server polling (and posting) script as a cronjob on certain intervals. I'll probably also want to get that script to get the list of servers from radiorivendell.com as well, so there's only one source of servers, with a nice UI that I can edit on the fly. Looking forward getting these things up and running properly

RE: Troubles after security update.

Elrond: Do you have logs from your streaming service (hopefully not in /tmp, as previously discussed)? They can tell you a lot of what went wrong. I'm also willing to help if you need some Linux error message deciphered or whatnot.

Ah, haha, I forgot about the logs We'll see what shows up, will keep an eye on this all day live

Apparently a bug somewhere in Liquidsoap that crashes the application when there's something weird in the meta data tags...

Any ideas or solutions?

…


2016/01/22 17:00:36 music:3 Prepared "/var/rivendell/music/Chris Watts/Chris Watts - March of the Giants.mp3" (RID 7).


2016/01/22 17:02:19 decoder:3 Method "MAD" accepted "/var/rivendell/music/Radio Rivendell Compilation 2/16 - Eternal Glory.mp3".


2016/01/22 17:02:29 music:3 Finished with "/var/rivendell/music/Chris Watts/Chris Watts - March of the Giants.mp3".


2016/01/22 17:02:29 music:3 Prepared "/var/rivendell/music/Radio Rivendell Compilation 2/16 - Eternal Glory.mp3" (RID 6).


2016/01/22 17:10:59 decoder:3 Method "MAD" accepted "/var/rivendell/music/Petite Viking/Hybris/3 - La Foret des Illusions Part II.mp3".


2016/01/22 17:11:09 music:3 Finished with "/var/rivendell/music/Radio Rivendell Compilation 2/16 - Eternal Glory.mp3".


2016/01/22 17:11:09 music:3 Prepared "/var/rivendell/music/Petite Viking/Hybris/3 - La Foret des Illusions Part II.mp3" (RID 1).


2016/01/22 17:11:47 decoder:3 Method "MP4" accepted "/var/rivendell/music/Hetoreyn/Tales of the Thorns/05 The mission begins.m4a".


2016/01/22 17:11:57 music:3 Finished with "/var/rivendell/music/Petite Viking/Hybris/3 - La Foret des Illusions Part II.mp3".


2016/01/22 17:11:57 music:3 Prepared "/var/rivendell/music/Hetoreyn/Tales of the Thorns/05 The mission begins.m4a" (RID 2).


2016/01/22 17:11:57 decoder:3 Method "MAD" accepted "/var/rivendell/music/Derek & Brandon Fiechter/Fantasy Realm/Desert Phoenix.mp3".


2016/01/22 17:11:58 clock.wallclock_main:2 Source 128kbit(dot)mp3 failed while streaming: Failure("Utils.utf_8.next"!


2016/01/22 17:11:58 clock.wallclock_main:3 Raised at file "pervasives.ml", line 20, characters 22-33


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "format.ml", line 1197, characters 8-23


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "format.ml", line 1197, characters 8-23


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "list.ml", line 84, characters 24-34


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "format.ml", line 1197, characters 8-23


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "list.ml", line 55, characters 20-23


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "list.ml", line 55, characters 32-39


2016/01/22 17:11:58 clock.wallclock_main:3 Called from file "list.ml", line 55, characters 20-23


2016/01/22 17:11:58 128kbit(dot)mp3:3 Closing connection...


2016/01/22 17:11:58 map_metadata_4793:3 Inserting missing metadata.


2016/01/22 17:11:58 main:3 Shutdown started!


2016/01/22 17:11:58 main:3 Waiting for threads to terminate...


2016/01/22 17:11:58 128kbit(dot)ogg:3 Closing connection...


2016/01/22 17:11:58 single_4780:3 Finished with "/var/rivendell/sounds/default.ogg".


2016/01/22 17:11:58 jingles:3 Finished with "/var/rivendell/jingles/Legolas.mp3".


2016/01/22 17:11:58 music:3 Loading playlist...


2016/01/22 17:11:58 music:3 Playlist is a directory.


2016/01/22 17:11:58 music:3 Finished with "/var/rivendell/music/Hetoreyn/Tales of the Thorns/05 The mission begins.m4a".


2016/01/22 17:11:58 music:3 Successfully loaded a playlist of 5891 tracks.


2016/01/22 17:11:58 clock.wallclock_main:3 Streaming loop stopped.


2016/01/22 17:11:58 threads:3 Thread "wallclock_main" terminated (0 remaining).


2016/01/22 17:11:58 main:3 Cleaning downloaded files...


2016/01/22 17:11:58 main:3 Freeing memory...


2016/01/22 17:11:58 >>> LOG END

Oh, and the version of Liquidsoap you can get throught apt-get is only 1.1.1 and version 1.2 is on Github. Why haven't they pushed it to Ubuntu directories so I can use the normal installation routine? Or is there a simple and clean way to get this new version?

Posted by ElrondOh, and the version of Liquidsoap you can get throught apt-get is only 1.1.1 and version 1.2 is on Github. Why haven't they pushed it to Ubuntu directories so I can use the normal installation routine? Or is there a simple and clean way to get this new version?

TO: Elrond

RE: Logs

I'm not a LiquidSoap guru, but this intrigues me:

2016/01/22 17:11:58 clock.wallclock_main:2 Source 128kbit(dot)mp3 failed while streaming: Failure("Utils.utf_8.next"!

2016/01/22 17:11:58 clock.wallclock_main:3 Raised at file "pervasives.ml", line 20, characters 22-33

and so on...

I'm making an educated guess that there may be a text encoding problem in your metadata. Your old server was Windows and now you are using Ubuntu. Being that a lot of your music (not to mention your service) is European, some of the text metadata has to be encoded beyond the standard 7-bits of ASCII. Unfortunately, these "extended" text encodings are not all the same. Windows has its own called Windows-1252, also popularly called ANSI (but technically it is not). Ubuntu (like most things Linux) favors UTF-8. That and the message

2016/01/22 17:11:58 clock.wallclock_main:2 Source 128kbit(dot)mp3 failed while streaming: Failure("Utils.utf_8.next"!

strongly implies a relationship with UTF-8.

So, if there are any characters in the metadata having diacritics, Cyrillic, or some other "non-English" symbol, those characters will require at least an 8-bit encoding. Compared between Windows-1252 and UTF-8, they won't be equivalent in binary, and that can cause a data integrity problem which may explain the software crash problem you are experiencing.

e.g. The character: 'á' (Latin small A with grave)

Windows-1252 has this binary encoding: 00000000 1110000

UTF-8 has this binary encoding: 11000011 10100000

So my educated guess is that you need to re-encode your metadata into UTF-8. If those are just plain text files, this command might do it:

iconv -f "windows-1252" -t "UTF-8" (old text file) -o (new text file)

Just replace "(old text file)" and "(new text file)" with actual file names.

RE: Updated LiquidSoap

It's my understanding that Ubuntu prioritizes stability. That means, once Canonical has determined a stable operating environment of software, they will only make small tweaks and changes unless a dire need (such as fixing a security threat) requires greater attention. So, they'll likely stick with LiquidSoap 1.1.1 for now. However, Ubuntu is part of a "family" of Linux distributions based on Debian. That means there is a very high probability that software acquired from a Debian repository will work in Ubuntu, and Debian tends to provide new versions of software before Ubuntu does.

But here's the bad news: It looks like Debian is also still providing version 1.1.1 (looking here: https://packages.debian.org/search?keywords=liquidsoap ). That means you'll have to build v1.2 from source.

Now, a Linux guru will tell you, "hey, that's not hard." But I'm not gonna do that to someone learning Linux. Many Linux gurus have a computer science background, and for them building from source is quite trivial as it is a fundamental of Linux C/C++ programming. But to a Linux novice -- it can be a trial-by-fire experience. I don't know your background, so, I'll leave it at this: Unless you know about make files and compilers, it's probably best to stick with LiquidSoap 1.1.1 until LiquidSoap 1.2 appears in a Debian or Ubuntu repository.

Hope that helps...

Updated 23 January 2016 (11:47)

Turns out you're not the first person who had that issue: https://github.com/savonet/liquidsoap/issues/214 (There's a utf8-recode function there that aledgedly fixes it, too)

If a software that is supposed to run unattended for long periods of time fails regularly due to rather normal things like character encoding issues, I would no longer trust it and try an alternative, because there are most likely more bugs of this kind.

Posted by ElrondOh, and the version of Liquidsoap you can get throught apt-get is only 1.1.1 and version 1.2 is on Github. Why haven't they pushed it to Ubuntu directories so I can use the normal installation routine? Or is there a simple and clean way to get this new version?

Ubuntu, like many other distros, usually comes with the software versions available at the time of feature freeze for a new distro version. Application software is usually not upgraded afterwards for compatibility reasons. Ubuntu also has something called PPA (Personal Package Archives). These are basically software repositories maintained by random people and hosted by launchpad.net. If you are lucky, someone built and uploaded a package with a newer software version. If you decide to trust PPAs from random people, there is a certain risk of breaking things, because those packages are less well tested (if at all) and often they are never upgraded. There does not seem to be a recent PPA for liquidsoap.

If you always want the newest software and don't mind the risk of an upgrade to occasionally break something, you can use a rolling release Linux distro or compile the software yourself. However, for a 24/7 server it's not worth the trouble.