Benutzer-Werkzeuge

Webseiten-Werkzeuge


linux:palava_or_the_hellish_way_through_ruby

Palava in Arch

Summary

This „short“ documentation explains how to install the audio & video communication software Palava on Arch.

Main

Palava consists of two components palava-machine and palava-portal.
Both have different dependencies and can per design run on different servers.

Warranty

There is no warranty that this setup works for any system which provide the dependencies listed below.
I'm just providing these information because it worked for me.
If you have questions you can leave a message here but I decide whether I'll answer and help or not.

At first I tried the package already existing in AUR but unfortunately it is really bad. Here are some of the issues:

  • both components are build within one package thus you can't install them seperately
  • there are missing a lot of dependencies
  • the whole ruby environment is packaged twice
  • it can't be build within a clean chroot
  • the installation paths doesn't follow the Arch filesystem hierarchy
  • it doesn't use the official git repository of palavatv

Palava-Portal

Palava-portal is mainly a build upon javascript, css and html.
Thus the portal only uses static webcontent which is easily cacheable because it only changes on updates.
Therefore technologies called content delivery networks (CDN) like varnish and squid can be used to deliver the portal.

The difference between CDN and normal webserver like apache or nginx is, that they store all cachable objects in memory and deliver them without accessing hard drives.
But CDNs have on bigger flaw. They are using hashes of the URL to find cached objects in there storage system.
Thus they only replace object with new content from their backends when the hash of the called URL is not in ther cache or the caching time has expired.

But the developers of palava succesfully conquered this obstacle by generating unique URLs on every build.
When the content of a file changes the name changes because they add something like a hash to it.
Therefore everytime there is an update the html is calling a new URL which causes the CDN to load and cache the new content.
Nice done guys

build & installation

In general it is pretty simple because the ruby-bundler does all the work and ruby is only needed on build time.

The PKGBUILD I created for palava-portal can be found in AUR.

Build dependencies
  • git
  • ruby
  • ruby-bundler
  • python (for i686 architecture)
  • python2 (for i686 architecture)
Install dependencies
  • none
  • you should have a webserver like apache2 or nginx installed
Build steps
  1. clone git repository:
    git clone --recursive https://github.com/palavatv/palava-portal.git
  2. change into cloned directory:
    cd palava-portal
  3. install all ruby dependencies for the build:
    bundle install --without development:test --deployment --path ./
  4. export environment for replacing the URLs used by palava:
    export PALAVA_BASE_ADDRESS="https://palava.example.com"
    export PALAVA_RTC_ADDRESS="wss://rtc.example.com/info/machine"
    export PALAVA_STUN_ADDRESS="stun:stun.example.com"
  5. build palava-portal:
    bundle exec middleman build

:!: If you are packaging palava-portal you should use an easy to find and unique placeholder instead of an URL :!:
:!: because it's up the user which addresses should be used. :!:
I'm using CH4NGEMEBASE, CH4NGEMERTC and CH4NGEMESTUN and a script (set-palava-url.sh) to replace them.

Install steps
  1. install all dependencies
  2. create needed directories:
    mkdir -p /usr/share/doc/palaca-portal /usr/share/webapps/palava-portal/
  3. copy files:
    install -m644 -t "/usr/share/doc/palava-portal/" LICENSE README.md
    cp -r build/* "/usr/share/webapps/palava-portal/"
  4. replace the placeholders if used, an example script:
    set-palava-url.sh
    #!/bin/bash
     
    # This script searches and replaces the placeholders for urls in palava-portal set on build-time
     
    [ $EUID -ne 0 ] && echo "This script should be run as root." && exit 1
     
    basefiles="$(grep -rlF 'CH4NGEMEBASE' /usr/share/webapps/palava-portal/)"
    rtcfiles="$(grep -rlF 'CH4NGEMERTC' /usr/share/webapps/palava-portal/)"
    stunfiles="$(grep -rlF 'CH4NGEMESTUN' /usr/share/webapps/palava-portal/)"
     
    read -p "Please enter the base-url for your palava installation: " baseurl
    read -p "Please enter the rtc-url for your palava installation: (${baseurl}) " rtcurl
    read -p "Please enter the stun-url for your palava installation: (${baseurl}) " stunurl
     
    echo "replacing base-url with ${baseurl} ..."
    sed -i "s/CH4NGEMEBASE/${baseurl}/g" $(tr '\n' ' ' <<<"${basefiles}")
    echo "replacing rtc-url with ${rtcurl:-${baseurl}} ..."
    sed -i "s/CH4NGEMERTC/${rtcurl:-${baseurl}}/g" $(tr '\n' ' ' <<<"${rtcfiles}")
    echo "replacing stun-url with ${stun:-${baseurl}} ..."
    sed -i "s/CH4NGEMESTUN/${stun:-${baseurl}}/g" $(tr '\n' ' ' <<<"${stunfiles}")
  5. configure your webserver, here an example for nginx:
    http {
        upstream palava_machine {
            server 127.0.0.1:4240;
        }
        server {
            listen 80;
            server_name palava.example.com;
            location / {
                rewrite ^ https://palava.example.com redirect;
            }
        }
        server {
            listen 443 ssl;
            server_name palava.example.com;
    
            ssl on;
            ssl_certificate /etc/ssl/private/example.com-pem.pem;
            ssl_certificate_key /etc/ssl/private/example.com-pem.pem;
    
            root /usr/share/webapps/palava-portal;
    
            location /info/machine {
                proxy_pass http://palava_machine;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_read_timeout 86400;
            }
    
            location / {
                if (-f $request_filename) {
                    break;
                }
                rewrite ^/.+$ / last;
            }
        }
    }

That's it for palava-portal.

Palava-Machine

The machine is fully build upon ruby and many many(75) ruby modules.
That is why building the machine is harder than the portal.

The machine is the brain of palava. It knows all open channels and connected peers by storing them in a redis database.

build & installation

The build and installation of palava-machine is hellish because it needs ruby and 75 modules to run.
For a dependency tree have a look in to the hidden-section („total dependency tree with subdependencies“) below.

The PKGBUILD I created for palava-machine can be found in AUR.

Build dependencies
  • ruby-bundler
  • git
Install dependencies

(all needed ruby-modules are alredy available in my repository)

  • ruby >= 2.0.0
  • redis
  • ruby-em-hiredis
  • ruby-logger-colors
  • ruby-em-websocket
  • ruby-mongo
  • ruby-http_parser.rb-0.5
  • ruby-local_port
  • ruby-daemons
  • ruby-synchronize

total dependency tree with subdependencies

total dependency tree with subdependencies

Here is a plotted picture.

palava-machine-git provides palava-machine
|-ruby
| |-gdbm
| | |-glibc
| | | |-linux-api-headers
| | | |-tzdata
| | | `-filesystem
| | |   `-iana-etc
| | `-bash provides sh
| |   |-readline
| |   | |-glibc
| |   | `-ncurses
| |   |   `-glibc
| |   `-glibc
| |-openssl
| | `-perl
| |   |-gdbm
| |   |-db
| |   | |-gcc-libs
| |   | | `-glibc
| |   | `-bash provides sh
| |   `-glibc
| |-libffi
| | `-glibc
| `-libyaml
|-redis
| `-bash
|-ruby-em-hiredis
| |-ruby
| |-ruby-hiredis
| | `-ruby
| |-ruby-em-spec
| | |-ruby
| | |-ruby-rspec
| | | |-ruby
| | | |-ruby-rspec-core
| | | | `-ruby
| | | |-ruby-rspec-expectations
| | | | |-ruby
| | | | |-ruby-diff-lcs
| | | | | `-ruby
| | | | `-ruby-diff-lcs
| | | `-ruby-rspec-mocks
| | |   `-ruby
| | |-ruby-bacon
| | | `-ruby
| | |-ruby-test-unit
| | |-ruby-eventmachine
| | | `-ruby
| | |-ruby-growl
| | |-ruby-guard-rspec
| | | |-ruby
| | | |-ruby-guard
| | | | |-ruby
| | | | |-ruby-formatador
| | | | | `-ruby
| | | | |-ruby-listen
| | | | | |-ruby
| | | | | |-ruby-inotify
| | | | | |-ruby-rb-fsevent
| | | | | | `-ruby
| | | | | `-ruby-celluloid
| | | | |   |-ruby
| | | | |   `-ruby-timers
| | | | |     |-ruby
| | | | |     `-ruby-rspec
| | | | |-ruby-lumberjack
| | | | |-ruby-pry
| | | | | |-ruby
| | | | | |-ruby-coderay
| | | | | | `-ruby
| | | | | |-ruby-slop
| | | | | | `-ruby
| | | | | `-ruby-method_source
| | | | |   `-ruby
| | | | `-ruby-thor
| | | |   `-ruby
| | | |-ruby-rspec
| | | |-ruby-bundler
| | | | `-ruby
| | | `-ruby-launchy
| | |   |-ruby
| | |   `-ruby-addressable
| | |     `-ruby
| | `-ruby-guard-bundler
| |   |-ruby
| |   |-ruby-guard
| |   |-ruby-bundler
| |   `-ruby-rspec
| `-ruby-rspec
|-ruby-logger-colors
| `-ruby-logger
|   `-ruby
|-ruby-em-websocket
| `-ruby
|-ruby-mongo
| |-ruby
| `-ruby-bson-1
|   `-ruby
|-ruby-http_parser.rb-0.5
| `-ruby
|-ruby-local_port
| |-ruby
| |-ruby-bundler
| |-ruby-rspec
| `-ruby-rubygems-tasks
|   |-ruby
|   |-ruby-rspec
|   `-ruby-yard
|     `-ruby
|-ruby-daemons
| `-ruby
`-ruby-synchronize
  |-ruby
  |-ruby-fog
  | |-ruby
  | |-ruby-builder
  | | `-ruby
  | |-ruby-excon
  | | `-ruby
  | |-ruby-formatador
  | |-ruby-multi_json
  | | `-ruby
  | |-ruby-mime-types
  | | `-ruby
  | |-ruby-net-scp
  | | |-ruby
  | | `-ruby-net-ssh
  | |   `-ruby
  | |-ruby-net-ssh
  | |-ruby-nokogiri
  | | |-libxml2
  | | | |-zlib
  | | | | `-glibc
  | | | |-readline
  | | | |-ncurses
  | | | `-xz
  | | |   `-bash provides sh
  | | |-libxslt
  | | | |-libxml2
  | | | `-libgcrypt
  | | |   `-libgpg-error
  | | |     |-glibc
  | | |     `-bash provides sh
  | | |-ruby
  | | `-ruby-mini_portile
  | |   `-ruby
  | `-ruby-ruby-hmac
  |   `-ruby
  |-ruby-commander
  | |-ruby
  | `-ruby-highline
  |   `-ruby
  `-ruby-chef
    |-ruby
    |-ruby-mixlib-config
    | `-ruby
    |-ruby-mixlib-cli
    | `-ruby
    |-ruby-mixlib-log
    | `-ruby
    |-ruby-mixlib-authentication
    | |-ruby
    | `-ruby-mixlib-log
    |-ruby-mixlib-shellout
    | `-ruby
    |-ruby-ohai
    | |-ruby
    | |-ruby-systemu-2.5
    | | `-ruby
    | |-ruby-yajl-ruby
    | | `-ruby
    | |-ruby-mixlib-cli
    | |-ruby-mixlib-config
    | |-ruby-mixlib-log
    | |-ruby-mixlib-shellout
    | `-ruby-ipaddress
    |   `-ruby
    |-ruby-rest-client
    | |-ruby
    | `-ruby-mime-types
    |-ruby-mime-types-1
    | `-ruby
    |-ruby-json-1.7
    | `-ruby
    |-ruby-yajl-ruby
    |-ruby-net-ssh
    |-ruby-net-ssh-multi-1.1
    | |-ruby
    | |-ruby-net-ssh
    | `-ruby-net-ssh-gateway
    |   |-ruby
    |   `-ruby-net-ssh
    |-ruby-highline
    |-ruby-erubis
    | `-ruby
    |-ruby-diff-lcs
    |-ruby-chef-zero-1
    | |-ruby
    | |-ruby-mixlib-log
    | |-ruby-hashie
    | | `-ruby
    | |-ruby-moneta-0.6
    | | `-ruby
    | |-ruby-json
    | | `-ruby
    | `-ruby-rack
    |   |-ruby
    |   `-ruby provides rubygems
    |-ruby-puma-1
    | |-ruby
    | `-ruby-rack
    `-ruby-pry

To be able to compile and install all ruby-modules I've written some little dirty helper scripts.

Now after all dependencies are available to the system we can start with the build and installation process.

Build steps
  1. clone git repository:
    git clone --recursive https://github.com/palavatv/palava-machine.git
  2. change into cloned directory:
    cd palava-machine
  3. set correct ruby-version:
    sed -i "s/^ruby.*/ruby '$(ruby -v | sed 's/^[^[0-9]*\s\([0-9][^p ]*\).*/\1/g')'/g" Gemfile
  4. install all ruby dependencies for the build:
    bundle install --without development:test --deployment --path ./
  5. change PIDFile path as already reported here:
    sed -i 's#^\(\s\s*monitor:.*\)$#\1\n  dir_mode: :normal,\n  dir: "/run/palava-machine/",#g' "bin/palava-machine-daemon"
Install steps
  1. install all dependencies
  2. create needed directories:
    mkdir -p /usr/share/doc/palaca-machine
  3. copy files:
    install -m644 -t "/usr/share/doc/palava-machine/" LICENSE.txt README.md ChangeLog.md
    cp -r bin lib "/usr/"
  4. install systemd-service file to /usr/lib/systemd/system/palava-machine.service:
    palava-machine.service
    [Unit]
    Description=WebRTC Signaling done in Ruby for palava
    After=network.target
    Requires=redis.service
     
    [Service]
    Type=simple
    PrivateTmp=true
    User=palava-machine
    Group=palava-machine
    NoNewPrivileges=true
    EnvironmentFile=/etc/conf.d/palava-machine
    ExecStart=/usr/bin/palava-machine-daemon start -- $PM_PORT $PM_DBID $PM_ADDRESS $PM_SHUTDOWN $PM_REDIS $PM_MONGO
    ExecReload=/usr/bin/palava-machine-daemon reload
    ExecStop=/usr/bin/palava-machine-daemon stop
     
    [Install]  
    WantedBy=multi-user.target
  5. install config file for systemd-service to /etc/conf.d/palava-machine:
    palava-machine
    ### This is the environment file for the palava-machine systemd-service
     
    # port used by palava-machine
    PM_PORT=4240
    # address palava-machine listens on
    PM_ADDRESS="127.0.0.1"
     
    ### following the defaul values should be enough
    # redis database id
    PM_DBID=5
    # timeout for shuting down palava-machine
    PM_SHUTDOWN=3
    # address of redis-server
    PM_REDIS="localhost:6379"
    # address of mongo-server
    PM_MONGO="localhost:27017"
  6. install systemd-tmpfiles definition to /usr/lib/tmpfiles.d/palava-machine.conf:
    palava-machine.tmpfiles.d
    d /run/palava-machine 0755 palava-machine palava-machine -
    d /var/log/palava-machine 0755 palava-machine palava-machine -
  7. add system user palava-machine is run with:
    grep -E "^palava-machine:" etc/passwd >/dev/null || useradd -s /bin/false -d /dev/null -c "user for running palava-machine" -r -U palava-machine
  8. install needed directories:
    systemd-tmpfiles --create palava-machine.conf
  9. add palava-machine to startup services:
    systemctl enable palava-machine
  10. start palava-machine:
    systemctl start palava-machine

That's it for palava-machine.
Now you can use your palava installation.

Helper Scripts

I already said dirty, diddn't I? ;)

dependencies:

  • pacman
  • bash
  • systemd
  • devtools
  • local repository is stored in /data/repo/pkgs/{i686,x86_64}
  • clean chroots are stored in /data/repo/{i686,x86_64}
build-dependencies.sh
#!/bin/bash
getdep() {
  local deps="" depends="" makedepends="" ret=0 dep=""
  pre+="--"
  # grep all dependencies from PKGBUILD of given package (dirty as it doesn't handle line breaks in the array)
  eval $(grep depends "$1"/PKGBUILD)
  # create array with only the names of all needed dependencies
  deps=($(echo "${depends[@]} ${makedepends[@]}" | sed 's/[>=<][>=<]*[^ ]*//g'))
  for dep in ${deps[@]}
  do
    echo "${pre}>${dep}" >&2
    # check whether dependency already is in a repository
    if pacman -Si "$dep" >/dev/null 2>&1
    then
      [ $ret -eq 0 ] && ret=2
    # check whether an AUR-package exists
    elif cower -i "$dep" >/dev/null 2>&1 || [ -d "$dep" ]
    then
      # download package if it is not already there
      if ! [ -d "$dep" ]
      then
        cower -d "$dep" >/dev/null 2>&1
      fi
      # remove all unneeded deplendencies as they are already integrated in ruby
      sed -i "s/['\"(]rubygems[)'\"]//g;s/['\"(]ruby-rake[)'\"]//g;s/['\"(]ruby-rdoc[)'\"]//g" "$dep"/PKGBUILD
      # check for dependencies of the dependency
      getdep "$dep"
      ret=$?
      echo "${pre}>$ret" >&2
      if [ $ret -eq 2 ]
      then
        if ! pacman -Si "$dep" >/dev/null 2>&1
        then
          # replace/create md5sum for the package
          grep 'md5sums=()' "$dep"/PKGBUILD >/dev/null 2>&1 && sed -i "s/md5sums=()/$(makepkg -g)/g" "$dep"/PKGBUILD
          grep 'md5sums=' "$dep"/PKGBUILD >/dev/null 2>&1 || { cd "$dep" && makepkg -g >> PKGBUILD; cd .. ; }
          # build and add the package to own repository
          makerepopkg.sh -Bp i686,x86_64 -P "$dep"/PKGBUILD >&2
          if [ $? -eq 0 ]
          then
            # sync modified repository files to repo-server
            echo -e 'y\n' | unison -auto repo >&2
            # update local repository databases
            sudo pacman -Sy >&2
            sudo arch-nspawn /data/repo/x86_64/root pacman -Sy >&2
            sudo linux32 arch-nspawn /data/repo/i686/root pacman -Sy >&2
            ret=3
          else
            ret=1
          fi
        fi
      fi
    else
      echo "$dep" && ret=1
    fi
  done
  unset pre
  return $ret
}
for pkg in *
do
  echo "$pkg";
  getdep "$pkg"
  if [ $? -eq 2 ]
  then
    if ! pacman -Si "$pkg" >/dev/null 2>&1
    then
      # replace/create md5sum for the package
      grep 'md5sums=()' "$pkg"/PKGBUILD >/dev/null 2>&1 && sed -i "s/md5sums=()/$(makepkg -g)/g" "$pkg"/PKGBUILD
      grep 'md5sums=' "$pkg"/PKGBUILD >/dev/null 2>&1 || { cd "$pkg" && makepkg -g >> PKGBUILD; cd .. ; }
      # build and add the package to own repository
      makerepopkg.sh -Bp i686,x86_64 -P "$pkg"/PKGBUILD >&2
      if [ $? -eq 0 ]
      then
        # sync modified repository files to repo-server
        echo -e 'y\n' | unison -auto repo >&2
        # update local repository databases
        sudo pacman -Sy >&2
        sudo arch-nspawn /data/repo/x86_64/root pacman -Sy >&2
        sudo linux32 arch-nspawn /data/repo/i686/root pacman -Sy >&2
      else
        echo "Couldn't build $pkg."
      fi
    fi
  fi
done

The following line checks wheather all folders in current directory do exist in the local repository path:

for i in *;do ls /data/repo/pkgs/x86_64/"$i"*pkg* >/dev/null 2>&1 || echo $i;done

Comments



L G O T G
linux/palava_or_the_hellish_way_through_ruby.txt · Zuletzt geändert: 2014/02/23 11:52 von Andrwe Lord Weber

Seiten-Werkzeuge