.. meta:: :description: Unattended Sky Quality Meter Station :keywords: Raspberry pi, SQM, Sky Quality Meter, unattended, PySQM, astronomy :name: Rubén Díez lázaro :http-equiv=Content-Type: text/html; charset=utf-8 ====================================================== ATTENTION: software overpassed notice. ====================================================== The pySQM software is overpassed by "tesstractor". Please don't use pySQM for new setups. Use tesstractor instead. Tesstractor is a software designed to read data from TESS photometers (``__). This software reads data also from Unihedron SQM-LU photometers. You can find the tesstractor software at ``__ ====================================================== OBSOLETE -- Unattended Sky Quality Meter Station (V3). ====================================================== OBSOLESCENCE WARNING ==================== This implementation for an unattended SQM station is now obsolete and I do not recommend its use. My work about unatended Raspberrypi SQM unatended station is based in two principles: 1 - Mount a remote network based filesystem for store SQM data. 2 - A script periodically executed by cron service that ensures that the PySQM is running and the remote filesystem is mounted. The implementation of these two points was done a long time ago using the Raspbian version of that time. Now that version is obsolete and the current one (stretch) uses a new administration tool called “systemd”. With this tool, it is possible implement both points 1 and 2 in a very easy, reliable and secure way… May be a day I will find time to do it… But any way, my old solution is now obsolete and I do not recommend its use. If, for study purposes, you want to consult the old and now obsolete documentation, here is it: Generalities ------------ About light pollution, SQM, and related projects. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As a amateur astronomer, light pollution is a phenomena which I usually must deal. Over the years, more and more light pollution become a growing problem for the activities of amateur and professional astronomers. Many institutions are involved with this issue: - `Globe at Night `__ - `Star Light iniciative `__ (in Spanish) - `Celfosc assotiation `__ (in Spanish) - `Galician Night Sky Brightness Monitoring Network `__ - `Red española de estudios sobre la contaminación lumínica `__ (in Spanish) There is a hardware stuff that is more and more used for measuring the light pollution: The “Sky Quality Meter” from `Unihedron `__. This piece of hardware (basically a photometer) is sold in different varieties, that permit various options for connect it to a computer: - RS232 interface - USB interface - Ethernet interface Many organizations around the world centralizes the light pollutions measurements. With the present project I pretend implement a Raspberry Pi based full Unattended light measurement station suitable for server data to some of these organizations, in particular `REECL SQM network `__. Unattended station concept… ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The idea is to have a 24/7 computer with a SQM attached and serve data to some centralized site. Due the price, size and electrical consumption of this computer is a important facet, a Raspberry Pi based station becomes a attractive solution The Unattended station should be these characteristics: - Should be small and have low power consumption. - The software should run without human interaction. - The station should be robust: the recovery under fails mush be automated. - It must server data to a remote storage, accessible too from the data-agregator organization (as `Red española de estudios sobre la contaminación lumínica `__). - The station should notify administrative information, as a recovery or any detected problem. Mail seems to be the natural mechanism for this. - The system should be remotely accessible for maintenance and configuration: SSH and router port redirection are your friends ;) Pieces of the Puzzle. ~~~~~~~~~~~~~~~~~~~~~ - Hardware platform Raspberry Pi, of course :D - As main software PySQM from `REECL `__: “PySQM the UCM open source software to read, plot and store data from SQM photometers” Nievas Rosillo, Miguel and Zamorano Calvo, Jaime (2014) `UCM eprint (PDF) `__. - SQM. The USB model (SQM-LU) is the more suitable, due the hardware/software combination chosen. - Cloud storage. Should have some characteristics that limit the possibilities: - Must be acceded by the Raspberry Pi, following a independent, standard method. WebDav protocol for remote file system mount is a natural solution. - The `Red española de estudios sobre la contaminación lumínica `__ acts as data-aggregator. It uses a DropBox based cloud storage. DropBox client does not run in Raspberry Pi (it is released as a binary client, which has not support for ARM architecture). - Due these two previous two points, a interface between local mounted remote storage and final DropBox storage is required. This concept is named \``Cloud Storage Broker’’. After some tests, `Storage Made Easy `__ service was selectioned. It can be mounted from Raspberry Pi by WebDav protocol and can syncronize the data with DropBox (among others storage services). - The final cloud-stored (DropBox) data must be accessible throw a direct link (not by an user interacting with a web. Happily, DropBox can do this. - Administrative resources for mail notifications and file share administration. `gmail.com `__ and web administration interface from selected storage service were chosen. For the impatient: downloadable disk system image template ---------------------------------------------------------- You can implement your own Raspberry Pi unattended SQM data collector by `download a image template `__ The ``MD5SUM`` code for this ``zip`` file is ``7e34fecfc57fa28db0a4510c87747fa0`` and for the uncompressed image file is ``90e355d499f15ec70bf99833cd8c7427`` You can “burn” this image to a SD card in the usually way: :: # dd bs=8225280 if=2015-08-25-wheezy-raspbian-PySQM_ready-kernel4.1.6-staticIP192.168.0.251.img of=/dev/mmcblk0; sync or similar. Please also note that this image is configured with a static IP: ``192.168.0.251``: you should edit the file ``/etc/network/interfaces`` for adapt to your local network scheme. The default user is the usually one: ``pi`` with password ``raspberry``. By the way, you will need a SD card with at least 4GB in size. Please note that nevertheless the size of the root file system is quite conservative, so don’t forget increase it to the maximum of your SD card, for example by using the ``raspi-config`` utility after boot your Raspberry Pi. For more information about how to “*writing an image to the SD card*”, please refer to the `official raspberrypi.org site `__ Once the image was burned into a SD card, you mus edit the files in order to adapt it to you case: - ``/etc/network/interfaces``: Adapt it to you network scheme. - ``/etc/profile.d/sqm_environment.sh``: Fix here the ``SQM_NAME`` variable for for related mail information stuff. - ``/etc/nail.rc``: Use here your gmail user and password. Also, at the end of the file, put the mail directions for ``errormail`` and ``infomail`` defined aliases. - ``/home/pi/.davfs2/secrets``: Put here your ``Storage Made Easy`` user and password. - ``/etc/fstab``: Modify for match your data directory name in your ``Storage Made Easy`` account. - ``/home/pi/SQM/config.py``: Modify as explained in ``PySQM`` documentation. - ``/srv/www/index.html``: The dummy web page for monitoring purposes. Known issues: Changes you should make in the downloaded image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After upload the previous pre-configured SD image, check following issues that had been discovered. You should fix them after burn the SD from the pre-configured image for avoid problems… At present time, all known issues are reflected in the system image template. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Setup details: how you can make it ---------------------------------- Get account in ``gmail.com``, ``eu.storagemadeeasy.com`` and ``dropbox.com`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For the proposed solution, a account in `gmail.com `__, `eu.storagemadeeasy.com `__ and `dropbox.com `__ should be created. ``gmail.com`` is needed for notification stuff and as base for create the account in ``eu.storagemadeeasy.com`` and ``dropbox.com``. `eu.storagemadeeasy.com `__ is a cloud storage broker service that have some interesting characteristics: - The storage space can be mounted in a standard manner by using the ``webdav`` protocol. This avoid the use of third part protocols used by others cloud storage services, as ``DropBox`` and similar. - Acts as a cloud storage broker. This is: you can syncronize the data inside `Storage Made Easy `__ with some other cloud storage services, as \`DropBox’, which is the one used by `REECL `__. Loggin in the ``eu.storagemadeeasy.com`` page and create a folder named ``DropBox``. Inside, create another folder for store your SQM data. For example, we will use a folder with the name ``OAF-SQM01`` (inside the ``DropBox`` folder). Also, add your DropBox account as ``cloud provider`` and config the syncronization of the folder ``DropBox`` in Storage Made Easy with the DropBox account… This screenshot show the *DropBox* account is associate to a subfolder in the *Storage Made Easy* service: .. figure:: Raspberry_Pi-static/SME_providers.jpg :alt: SME provider SME provider And this one show configuration details for data syncronitation whith DropBox. Please note that ``use trash for deleted files`` should be disabled for avoid show some spurious file names in DropBox: .. figure:: Raspberry_Pi-static/SME_config.jpg :alt: SME configuration SME configuration Finally, the data is available in DropBox: .. figure:: Raspberry_Pi-static/DropBox_SQM.jpg :alt: DropBox data DropBox data Prepare a system image ~~~~~~~~~~~~~~~~~~~~~~ We will use Raspbian as Raspberry Pi O.S.. You can download the latest `Raspbian image `__ and make you SD by follow `this instructions `__. Assuming you have a usable Raspbian SD, we begin for upgrade the system. From now, I assume you was become root user in your system. For do this, open a console and type: :: $> sudo su You could remove all the desktop environment stuff and get more disk space by do (credits: `RemoveDesktop `__): :: $> apt-get remove xserver* x11-common x11-utils x11-xkb-utils \ x11-xserver-utils xarchiver xauth xkb-data console-setup xinit lightdm \ libx{composite,cb,cursor,damage,dmcp,ext,font,ft,i,inerama,kbfile,klavier,\ mu,pm,randr,render,res,t,xf86}* lxde* lx{input,menu-data,panel,polkit,\ randr,session,session-edit,shortcut,task,terminal} obconf openbox gtk* \ libgtk* scratch tsconf desktop-file-utils In any case, please remember to resize the file system to maximum amount of space provided by the SD. Now, the very first job to do is to update/upgrade the system. For this, type: :: $> apt-get update and, after the process ends: :: $> apt-get upgrade and accept the suggestions… only for be sure, you can also type: :: $> apt-get dist-upgrade For clean the downloaded package, do: :: $> apt-get --yes autoremove $> apt-get --yes autoclean $> apt-get --yes clean A singularity of the Raspbian O.S. is that (for now) the Linux kernel is packet on the same package that the boot firmware, so the kernel packages for the Raspbian is not used: it use the kernel associated with the boot firmware… You can update the kernel/firmware with the command: :: $> rpi-update Install and configure mail ~~~~~~~~~~~~~~~~~~~~~~~~~~ A easy (but not so secure) manner for configure a mail system in the Raspberry Pi for use a external account is installing the ``heirloom-mailx`` package: :: $> apt-get install heirloom-mailx Edit the file ``/etc/nail.rc`` and add at the end some configuration and alias definitions: :: #QMS stuff #Account set smtp-use-starttls set ssl-verify=ignore set smtp=smtp://smtp.gmail.com:587 set smtp-auth=login set smtp-auth-user=YOURUSER@gmail.com set smtp-auth-password=YOURPASS set from="YOURUSER@gmail.com (Friendly Name)" #Alias alias infomail \ mymail@gmail.com \ mail1@example.com \ mail2@example.com alias errormail \ mymail@gmail.com \ mail3@example.com \ mail4@example.com \ mail5@example.com Execute now some commands due (really weak) security considerations: :: $> chown root.mail /etc/nail.rc $> chown 640 /etc/nail.rc $> usermod -a -G mail pi Note that the new group for user ``pi`` take effect only since the next login (as user ``pi``). Install and configure davfs ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The necessary package for mount the cloud storage by using the ``webdav`` protocol must be installed: :: $> apt-get install davfs2 During the installation, you will be prompted about permit webdav mounts to unprivileged user: You should answer “Yes” to this question. After this, this line must be **added** this line to the file ``/etc/fstab``: :: https://webdaveu.storagemadeEasy.com/DropBox/OAF-SQM01 /home/pi/SQM/data davfs _netdev,rw,user,noauto,uid=1000,gid=1000,file_mode=644 0 0 Please note that this line assume that the folder ``OAF-SQM01`` was created throw the ``eu.storagemadeeasy.com`` web interface. Some other name could be used, at your choice. Now, as user ``pi``, create the directory ``/home/pi/.davfs2/`` and inside it the file ``/home/pi/.davfs2/secrets`` whit this content: :: /home/pi/SQM/data "YOURUSER@gmail.com" "YOURSMEPASSWORD" And, for security reasons, do: :: $> chmod 600 /home/pi/.davfs2/secrets An slow and/or transitory network issues can produce some error that could be minimized by edit the file ``/home/pi/.davfs2/davfs2.conf`` and set the option ``use_locks`` to 0 (``use_locks 0``). `Credits about this `__. Also, `PySQM `__ suffers a very poor input/output performance, which produce a bottleneck that results in a high consumption of bandwidth when cloud storage is used. For avoid this issue, add to the end of the file ``/home/pi/.davfs2/davfs2.conf`` the lines: :: use_locks 0 cache_size 128 [/home/pi/SQM/data] ## saving bandwidth: ## 10 minutes delay in file and dir refresh... file_refresh 600 dir_refresh 600 ## 10 minutes delay upload. Hope this save bandwidth delay_upload 600 Please note that in this example, ``use_locks`` and ``cache_size`` are general options for all possible ``davfs2`` mounted filesystem, but the other ones only affect to the filesystem mounted under ``/home/pi/SQM/data``. Installation and configuration of PySQM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lets go with the core of the system: the ``PySQM`` software, developed by the Jaime Zamorano team and that can be found at `its page `__. From the project web page: “PySQM the UCM open source software to read, plot and store data from SQM photometers” Nievas Rosillo, Miguel and Zamorano Calvo, Jaime (2014) `UCM eprint (PDF) `__. Installing necessary dependences. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Assuming you are using a SQM-LU model, install the necessary dependences by running, as ``root`` these commands: :: $> apt-get install python-pip $> apt-get install python-matplotlib $> apt-get install ttf-bitstream-vera $> apt-get install python-dev $> pip install pyephem Please note that some other dependences, as ``python-serial`` and ``python-numpy`` should be yet installed in the default *Raspbian* distribution. Also note that the default *Raspbian* distribution have installed various python versions, but the default one is the 2.7: that is the required version by ``PySQM``. There is a little change about ``matplotlib`` stuff that you should to in order to use the library in a not interactive way: Edit the file ``/etc/matplotlibrc`` and change the line: :: backend : TkAgg by this one: :: backend : Agg A little hack for avoid certain aesthetics issues due a bug in one of the ``matplotlib`` related packages (about the fonts path used). :: $> cd /usr/share/fonts/truetype $> ln -s lyx ttf-lyx Also, lets to get permission to read the port associate to the SQM-LU to the user ``pi``. This is done by the command: :: $> usermod -a -G dialout Note that the new group for user ``pi`` take effect only since the next login (as user ``pi``). Download and configure ``PySQM`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As user ``pi``, in its home, make a new directory named ``SQM`` and go into it: :: $> cd; mkdir SQM; cd SQM All the follow stuff in this section will be made into this ``/home/pi/SQM``. Lets to download the ``PySQM`` software from `PYSQM `__. It can be done directly by the command: :: $> wget http://guaix.fis.ucm.es/sites/default/files/luminica_files/PySQM.tar.gz and uncompress it: :: $> tar -xvzf PySQM.tar.gz some other directories should be create for future work: :: $> mkdir data logs In our working directory (``/home/pi/SQM``) should be appears a file named ``config.py``. Edit it. As a primer step, this line should be added immediately after the very first one… So the **two very first lines** of this file should be: :: #!/usr/bin/env python # -*- coding: utf-8 -*- This not avoid the programmer’s advice: “Non-ASCII characters are not supported in the config.py file”. But, at least, permit you to use utf-8 characters into the coments… :D In the rest of the this file (``config.py``) you must configure your particular configuration, observatory name, position, etc… In ``_device_addr`` you should put ``_device_addr = '/dev/ttyUSB0'`` if you are using a SQM-LU. By the way, we are using UT as system time, so its related variable should be ``_computer_timezone = 0``. About the variable ``_local_timezone`` you should use the timezone for where your system is configured. For UTC, it should be: ``_local_timezone = 0`` About the path for data store, it should point to the same directory where you mount the webdav cloud storage. This is: ``monthly_data_directory = "/home/pi/SQM/data"`` You can maintain the defaults for the others path… Main script ~~~~~~~~~~~ Script ``/home/pi/SQM/data/unattended_sqm.sh``: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As user ``pi``, you must create the following script: :: #!/bin/bash . /etc/profile.d/sqm_environment.sh PYSQMPATH="$(dirname $0)" CLOUDSTORAGEPATH="${PYSQMPATH}/data" LOGFILE="${PYSQMPATH}/logs/unattended_sqm.log" PYSQMOUT="${PYSQMPATH}/logs/pysqm.stdout" PYSQMERR="${PYSQMPATH}/logs/pysqm.stderr" MOUNTCHECK="mountpoint -q ${CLOUDSTORAGEPATH}" PYSQMCOMMAND="python -m pysqm" ## Use timeout command to avoid conection issues MOUNT_TIMEOUT=90 MOUNT_PID_FILE="/var/run/mount.davfs/$(echo ${CLOUDSTORAGEPATH} |sed 's/^\///' |sed 's/\/$//' |sed 's/\//-/g').pid" VALID_PROCESS_STATUS=("S" "R") _MSG="$(date -R -u): ${SQM_NAME}" function killpysqm { for i in $(pgrep -f -x "${PYSQMCOMMAND}"); do kill -TERM $i; wait # just in case: kill -KILL $i done } function execpysqm { cd ${PYSQMPATH} nohup ${PYSQMCOMMAND} >>${PYSQMOUT} 2>>${PYSQMERR} < /dev/null & MSG="======> ${_MSG} --> STARTING PROCESS" echo ${MSG} >>${PYSQMOUT} echo ${MSG} >>${PYSQMERR} MSG="PID = $(pgrep -f -x "${PYSQMCOMMAND}")" echo ${MSG} >>${PYSQMOUT} echo ${MSG} >>${PYSQMERR} } function checkpysqm { # first, check for bad state process and kill them for i in $(pgrep -f -x "${PYSQMCOMMAND}"); do _STATUS=$(ps h o state p $i) _VALID="NO" for _ITEM in ${VALID_PROCESS_STATUS[*]}; do [[ "${_STATUS}" == "${_ITEM}" ]] && _VALID="YES" done if [ "${_VALID}" == "$NO" ];then kill -KILL $i; fi done; wait NUMOFPYSQM=$(pgrep -c -f -x "${PYSQMCOMMAND}"); wait if [ "${NUMOFPYSQM}" == "0" ]; then execpysqm MSG="${_MSG} --> Starting: \"${PYSQMCOMMAND}\"." echo "$MSG" >>$LOGFILE echo "$MSG" |mail -v -s "${_MSG}: Starting" infomail elif [ "${NUMOFPYSQM}" == "1" ]; then MSG="${_MSG} --> \"${PYSQMCOMMAND}\" already running." echo "$MSG" >>$LOGFILE else killpysqm MSG="${_MSG} --> Terminating: too many \"${PYSQMCOMMAND}\" processes." echo "$MSG" >>$LOGFILE echo "$MSG" |mail -v -s "${_MSG}: Terminating" errormail fi } ### BEGIN ${MOUNTCHECK} if [ $? -ne 0 ]; then # if exist a pid file, delete it: rm -f ${MOUNT_PID_FILE} timeout ${MOUNT_TIMEOUT} mount "${CLOUDSTORAGEPATH}" ${MOUNTCHECK} if [ $? -ne 0 ]; then MSG="${_MSG} --> ${CLOUDSTORAGEPATH} mount failed." echo "$MSG" >>$LOGFILE echo "$MSG" |mail -v -s "${_MSG}: Mount Failed" errormail killpysqm else MSG="${_MSG} --> ${CLOUDSTORAGEPATH} mount successfully." echo "$MSG" >>$LOGFILE echo "$MSG" |mail -v -s "${_MSG}: Mount successfully" infomail checkpysqm fi else MSG="${_MSG} --> ${CLOUDSTORAGEPATH} already mounted." echo "$MSG" >>$LOGFILE checkpysqm fi sync This file must be made executable by using the command: :: $> chmod u+x unattended_sqm.sh Script ``/root/bin/check_ip.sh``: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As user ``root``, you must create first the directory: :: $> mkdir /root/bin and then create the following script (``/root/bin/check_ip.sh``) inside: :: #!/bin/bash . /etc/profile.d/sqm_environment.sh LOGFILE=${SQMDIR}/logs/check_ip.log _DATE=$(date -R -u) ## Use timeout command to avoid conection issues _TIMEOUT=3 # An array for list external ip service discovers to probe _IPSERVICES=('http://ipecho.net/plain' 'http://ipv4.icanhazip.com' 'http://ifconfig.me/ip') for _USEDIPSERV in ${_IPSERVICES[*]}; do NOWIP=$(timeout ${_TIMEOUT} wget -qO- ${_USEDIPSERV}); wait if [ ! -z "$NOWIP" ]; then break; fi done if [ -z "$NOWIP" ]; then NOWIP=unset _USEDIPSERV=none fi if [ "${NOWIP}" == "${EXTERNAL_IP}" ]; then echo "${_DATE} from ip discover: ${_USEDIPSERV} --> External IP unchanged: ${EXTERNAL_IP}" >>${LOGFILE} sync else echo "This file contains the last detected external IP" >${LASTIPFILE} echo "Dont edit by hand" >>${LASTIPFILE} echo "${_DATE} from ip discover: ${_USEDIPSERV}" >>${LASTIPFILE} echo >>${LASTIPFILE} echo "${NOWIP}" >>${LASTIPFILE} sync echo "${_DATE} from ip discover: ${_USEDIPSERV} --> New external IP detected: ${NOWIP}" >>${LOGFILE} sync echo "${_DATE}: RaspberryPi ${SQM_NAME} --> New external IP (discover: ${_USEDIPSERV}) detected: ${NOWIP}"| mail -v -s "RasPi ${SQM_NAME} new external IP - ${_DATE}" errormail fi exit 0 This file must be made executable by using the command: :: $> chmod u+x check_ip.sh Using cron for do the system (reasonable) fault tolerant ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For a (reasonable) fault tolerant mechanism, lets use ``cron`` for periodically run the above scripts. As i before said, it check if all is right, and re-run the stuff if it is necessary. As ``root``, the files ``/etc/cron.d/unattended_sqm`` and ``/etc/cron.d/check_ip`` must be created with the following content: File ``/etc/cron.d/unattended_sqm`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ## SQM: unattended_sqm ## Every 10 minutes */10 * * * * pi /home/pi/SQM/unattended_sqm.sh File ``/etc/cron.d/check_ip`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ## SQM: check_ip ## Every 6 hours 0 */6 * * * root /root/bin/check_ip.sh Once created, restart ``cron`` with the command: :: $> /etc/init.d/cron restart Log subsystem ~~~~~~~~~~~~~ The main scripts generates log files. These files could grow indefinitely, so the ``logrotate`` utility can be used for avoid this stuff. For this, create, as ``root`` two files named ``/etc/logrotate.d/unattended_sqm`` and ``/etc/logrotate.d/check_ip`` with the contents: File ``/etc/logrotate.d/unattended_sqm`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: /home/pi/SQM/logs/unattended_sqm.log { daily rotate 8 delaycompress compress noolddir missingok } /home/pi/SQM/logs/pysqm.stderr /home/pi/SQM/logs/pysqm.stdout { weekly rotate 4 delaycompress copytruncate compress noolddir missingok } File ``/etc/logrotate.d/cheick_ip`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: /home/pi/SQM/logs/check_ip.log { weekly rotate 4 delaycompress compress noolddir missingok } Tasks on start up and halt ~~~~~~~~~~~~~~~~~~~~~~~~~~ At this point, we must do that some tasks become to automatically be made at system start up and halt. These tasks should be: - At start up: - Notify the system start up and its external ip address - Initiate the main script - At system halt: - Notify the system halt Dealing with the clock at boot time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Raspberry Pi have not a real internal clock (RTC), so when it is powered of the time recorded in the last shutdown is re-established Nevertheless in a few seconds the ntpd stuff do its work and the system become in time, the use of the recorded time at shutdown at the very early boot time can be a crap. But if we can a network available, the system cat be request a more accurate time at network interface up. This time is not accurate, but is very better that use the recorded time at shutdown. For do this stuff, simply install the ``ntpdata`` package by using the command: :: $> apt-get install ntpdate Defining environment variables ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create the file ``/etc/profile.d/sqm_environment.sh`` with the following content: :: ## SQM environment export LASTIPFILE=/tmp/last_ip.txt export SQMDIR=/home/pi/SQM/ export SQM_NAME="OAF-SQM01" if [ -f ${LASTIPFILE} ]; then EXTERNAL_IP=$(grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' ${LASTIPFILE} |tail -1) fi if [ -z "$EXTERNAL_IP" ]; then EXTERNAL_IP=unset; fi export EXTERNAL_IP The ``rc.local`` file ^^^^^^^^^^^^^^^^^^^^^ Edit the file ``/etc/rc.local``. some extra content must be added at the end of the file. The total content of the file will be as follow: :: #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. # Print the IP address _IP=$(hostname -I) || true if [ "$_IP" ]; then printf "My IP address is %s\n" "$_IP" fi ## Begin of SQM stuff: /root/bin/check_ip.sh . /etc/profile.d/sqm_environment.sh _DATE=$(date -R -u) # The -v (verbose) flag ensure send entire message before the command return su -c "echo \"${_DATE}: RaspberryPi ${SQM_NAME} data collector powered on with external IP: ${EXTERNAL_IP}\"| mail -v -s \"RasPi ${SQM_NAME} POWERED ON - ${_DATE}\" errormail; wait" pi su -c /home/pi/SQM/unattended_sqm.sh pi wait ## End of SQM stuff exit 0 Ensure that this file have permissions ``-rwxr-xr-x`` and ``root`` as owner. The ``rc.local.shutdown`` file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A new file ``/etc/rc.local.shutdown`` must be created with the content: :: #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. ## Begin of SQM stuff: . /etc/profile.d/sqm_environment.sh _DATE=$(date -R -u) # The -v (verbose) flag ensure send entire message before the command return su -c "echo \"${_DATE}: RaspberryPi ${SQM_NAME} data collector shutdown.\"| mail -v -s \"RasPi ${SQM_NAME} SHUTDOWN - ${_DATE}\" errormail; wait" pi wait ## End of SQM stuff: exit 0 Ensure that this file have permissions ``-rwxr-xr-x`` and ``root`` as owner. Modifying and active the ``rc.local`` service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The actual ``rc.local`` service must be modified for the new scheme. Edit the file ``/etc/init.d/rc.local`` for become content this stuff: :: #! /bin/sh ### BEGIN INIT INFO # Provides: rc.local # Required-Start: $all # Required-Stop: $all # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Run /etc/rc.local and /etc/rc.local.shutdown if it exist ### END INIT INFO PATH=/sbin:/usr/sbin:/bin:/usr/bin . /lib/init/vars.sh . /lib/lsb/init-functions do_start() { if [ -x /etc/rc.local ]; then [ "$VERBOSE" != no ] && log_begin_msg "Running local boot scripts (/etc/rc.local)" /etc/rc.local ES=$? [ "$VERBOSE" != no ] && log_end_msg $ES return $ES fi } do_stop() { if [ -x /etc/rc.local.shutdown ]; then [ "$VERBOSE" != no ] && log_begin_msg "Running local boot scripts (/etc/rc.local.shutdown)" /etc/rc.local.shutdown ES=$? [ "$VERBOSE" != no ] && log_end_msg $ES return $ES fi } case "$1" in start) do_start ;; restart|reload|force-reload) echo "Error: argument '$1' not supported" >&2 exit 3 ;; stop) do_stop ;; *) echo "Usage: $0 start|stop" >&2 exit 3 ;; esac After this, for the change do effect, we must do: :: $> /usr/sbin/update-rc.d -f rc.local remove $> /usr/sbin/update-rc.d rc.local defaults Wake up a dummy service for external monitoring (optional) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The system is reasonably robust but, what about the possibility of a network failure?. In this case, is not possible for the system to notify the failure from inside the local network. For be possible this notify of network failure you can enable a dummy service and monitoring it from **outside** the local network using, for example, a service as ``uptimerobot.com``. The best option is wake up a very light web server and serve a dummy page. My personal preference as light web server is ``webfs``. For installing it: :: $> apt-get install webfs After this, create a directory for store the dummy web page and put the appropriate permissions/owner: :: $> cd /srv $> mkdir www $> chown www-data. www Inside this new directory (``/srv/www``), put a ``index.html`` file with a dummy content (for example, information about the system and URLs of shared data) Fix the adequate permissions/owner for this file: :: $> chmod 644 index.html; chown www-data. index.html For last, you must fix some configuration stuff in ``/etc/webfsd.conf`` file. The lines I changed for my case were: :: web_root="/srv/www" web_port="1180" web_index="index.html" After these changes, restart the webfs service: :: $> /etc/init.d/webfs restart Once a dummy web page is available, you can create an account in ``uptimerobot.com`` for monitoring and report issues. Please note: in this example dummy web service I am using the port number ``1180``. You can use whatever you like, but in order to become the service available from outside your local network, you should configure your router for redirect the incoming packages to port ``1180`` to the local IP which the SQM data collector station is configured. Perhaps, you also should do a similar redirection if you want to use the ``SSH' access service from outside your local network. As a suggestion, you can change the``\ sshd\ ``port and the``\ pi\` user password… Physical deployment of the SQM and begin of operations ------------------------------------------------------ The built prototype was installed in the “Observatorio Astronómino de Forcarei (OAF)” (Astronomical Observatory of Forcarei). In the next image, we can see the look of the installed prototype (after take the picture, the support was updated using metalic clamps). The Raspi is inside the building. Note that only three wires are connected to the Raspi (power, network, USB) and only one of them (the USB wire that connec with the SQM) must cross the wall. .. figure:: Raspberry_Pi-static/SQM_Forcarei.jpg :alt: The SQM station The SQM station The data :D ~~~~~~~~~~~ The data can be browsed by share the ``DropBox`` (final cloud storage) data directory. As example for our case you can show the data in this URL: :: https://www.dropbox.com/sh/spze9pleyvg2kjs/AADtv1UDcFBCTtUOrH9Vr0h7a?lst It is also possible get the wole data file in ``zip`` format throw the URL: :: https://www.dropbox.com/sh/spze9pleyvg2kjs/AADtv1UDcFBCTtUOrH9Vr0h7a?dl=1 And also, from you account in ``DropBox`` you can share the URL of the generated graphic. In my case this image can be acceded throw the URL: :: https://www.dropbox.com/s/rw5lvwb80b18r5o/SQM01_Observatorio_de_Forcarei.png?raw=1 By using this URL, the image can be embedded in any web page, as for example: :: http://rdlazaro.info/sqm/ Chagelog since system deployment -------------------------------- Before May 2015 ~~~~~~~~~~~~~~~ Time configuration parameters. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In the first release, the time related parameter were (due my country is in the +1 timezone): :: _computer_timezone = +0 _local_timezone = +1 But, if you configure your Raspi for use UTC time (as i do), these parameters should be: :: _computer_timezone = 0 _local_timezone = 0 According to the ``PySQM`` main author, the parameter ``_computer_timezone`` is not used in recents versions (is a legacy parameter) and ``_local_timezone`` controls how the time axis is labeled in the graphics (hours un UTC, UTC+1 and so on). Change of cloud storage provider and use a broker. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In a early stage `PowerFolder `__ service was used as primary cloud storage, but file access from ``davfs2`` was disabled by *PowerFolder* staff. Strange behavior was detected when accessing the files generated by ``PySQM`` in the file system mounted with the ``WebDAV`` protocol using ``davfs2``. If you editing (eg using ``nano`` or ``vim``) or you read (using eg ``cat`` or ``less``) them, their contens was corrupted, and the data record and graphics generation fails. But it you don’t “touch” the files from the Raspi, the stuff works right. This behavior is produced, according *PowerFolder* staff, because some ``html`` access method related to the ``webdav`` protocol were disables by security reasons at *PowerFolder* server side. *PowerFolder* staff claims that them expect add the ``davfs2`` client to a “white list” that should permit a complete functionality when ``webdav`` filesystem is mounted from Linux… in 2 or 3 months (since May 2015). Due these issues, the primary cloud storage system was changed to `Storage Made Easy `__ Currently, *PowerFolder* staff claims the issue is resolved, but anyway, *PowerFolder* service is not yet used. May 2015 ~~~~~~~~ Join the \`REECL SQM Network’ . ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ At May 2015, the Unattended Sky Quality Meter Station mounted at Forcarei obsevatory becomes join the `REECL SQM Network `__. As commented before, the *PowerFolder* service produced some problems. Looking for an alternative, we found the `Storage Made Easy `__ cloud storage service. This storage service can be mounted by using the *webdav* protocol as described before in this document. But this storage service have **NOT** the other desired characteristics (basically, the use of directs links for share contents). However, *Storage Made Easy* have a surprising capability that compensates this deficiency: some other cloud storage (as for example, DropBox) can be associate to Storage Made Easy accound and synchronize the stored files whith the associate external cloud storage. So the `Storage Made Easy `__ service was used for mount in the Raspi the remote data storege, and a DropBox account was also associate with the `Storage Made Easy `__ contents. So the data can be acceded throw a DropBox account. After these changes, we fulfill all requirements for join the `REECL SQM Network `__. August 2015 ~~~~~~~~~~~ Bandwidth consumption issue. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *PySQM* is a nice software, but it was not designed with performance in mind. It suffers a very poor input/output performance, which produce a bottleneck that results in a high consumption of bandwidth when cloud storage is used. For avoid this issue, you must tunning the ``davfs2' stuff by add the file``/home/pi/.davfs2/davfs2.conf\` these lines: :: [/home/pi/SQM/data] ## saving bandwidth: ## 10 minutes delay in file and dir refresh... file_refresh 600 dir_refresh 600 ## 10 minutes delay upload. Hope this save bandwidth delay_upload 600 December 2015 ~~~~~~~~~~~~~ Some system tuning was made. I hope some of them avoid the issue about remote (``davfs2``) filesystem “frozen”… Logrotate (and other stuff) runs now during the day. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In an usual system, logrotate runs during the night, but due the nature of the SQM measurements, this was changed it to running during the day. We hope that this avoid a possible system overload in the middle of the SQM data recording… Really, ALL processes scheduled at night were rescheduled at day. For do it, chage the file ``/etc/crontab`` from its original (only relevant lines showed): :: 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) to: :: 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 12 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 12 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 12 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) Use a RAM based file system as cache for ``davfs2``: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create the mount point ``/home/pi/.davfs2/cache_tmpfs`` (as user ``pi``) and add this line to ``/etc/fstab`` (edit it as user ``root``): :: tmpfs /home/pi/.davfs2/cache_tmpfs tmpfs rw,size=384M,nr_inodes=5k,noexec,nodev,nosuid,uid=1000,gid=1000,mode=1700 0 0 Mount it and change the cache path at ``/home/pi/.davfs2/davfs2.conf`` (``cache_dir``) parameter. You must also remount the remote filesystem. Increase the cache size and number of entries for files. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ These are fixed by the parameters ``cache_size`` and ``table_size`` in the ``/home/pi/.davfs2/davfs2.conf`` file. After all these changes, the ``/home/pi/.davfs2/davfs2.conf`` looks as: :: use_locks 0 cache_size 256 table_size 4096 cache_dir ~/.davfs2/cache_tmpfs [/home/pi/SQM/data] file_refresh 600 dir_refresh 600 delay_upload 600 Contact ======= If you have any comment or correction, please feel free for send me a e-mail about anything that you consider interesting at ``monje314-2005`` *at* ``yahoo`` *dot* ``es``. Cheers!!!