Here’s another Ceph Manager Dashboard screencast, showcasing how the dashboard displays and reacts to ongoing changes in the Ceph cluster, e.g. when OSDs go down. Enjoy!
I created a short overview screencast of the Ceph Manager Dashboard, to give you an impression of what to expect in the upcoming Nautilus release. Enjoy!
Categories
DevConf.CZ 2019 conference report
Last weekend, I traveled to the beautiful city of Brno in the Czech Republic, to attend at speak at the DevConf.CZ open source conference. I had submitted a talk about our ongoing work on the Ceph Manager Dashboard and was very happy that it got accepted, as I haven’t been to a DevConf before.
The conference took place in the lecture halls of the Faculty of Information Technology at Brno University of Technology, which provided a very nice venue with lots of rooms and infrastructure (the WiFi was a tad bit slow, but usable). Still, some talks were so popular that the rooms were full and the session chairs had to block others from entering (something that other conferences like FOSDEM also suffer from). Fortunately, many of the sessions were recorded, so there is an opportunity to watch them later.
The overall organization of the conference was excellent, the organizers had really thought of everything and it was a pleasure to attend this event from a speaker/participant point of view. There was a cloak room, badges that clearly indicated your preference of being photographed or your preferred level of engagement in conversations with other attendees. Food was also available in good variety (both free and several paid-for additional options provided by food trucks). The organizers had also set up a group chat on Telegram, which was used to spread information to all participants quickly.
My talk Ceph Management and Monitoring with the Dashboard was scheduled for Saturday, 16:00, so I had some time to attend other sessions before and afterwards.
The session schedule was packed and I had a tough time to choose which ones to attend. I ended up with attending the following sessions:
- Growing Your Career: From Tactical to Strategic — The topic sounded interesting, but I only attended this panel discussion for about 20 minutes, as it was too painful for me to watch how a single lapel microphone was frequently handed over from one panelist to another one so they could be heard. Even a handheld mike would have been an improvement…
- A Greybeard’s Worst Nightmare — in this session, Daniel Riek summarized the pains of packaging modern applications, the proliferation of custom solutions for distributing software and libraries and how they could be addressed in the modern world of container technologies. Good food for thought, it certainly resonated well with some experiences we have been going through in the openATTIC and Ceph Manager Dashboard project.
- Open leadership: An invitation to all — this presentation resonated very well with me and I was able to capture a number of good ideas on how to improve my own skills. I liked the style of this session, which was more an open conversation than a head-on presentation. To my knowledge it wasn’t recorded, but the slides are worthwhile checking out.
- Ceph data services in a hybrid cloud world — Sage Weil, founder and lead developer of the Ceph project talked about the challenges of syncing and moving data across multiple clouds and explained some of the existing and future solutions that Ceph brings to the table to resolve these. I found this session very insightful and it was interesting to learn more about the project’s future plans in this area. I guess our work on the Ceph Manager Dashboard isn’t finished any time soon!
- Active/Active NFS Serving over CephFS — Jeff Layton gave a very detailed into his work on making it possible to set up NFS services based on NFS Ganesha with CephFS on the backend in a scalable way, this was very insightful (see his blog post for an example on how to set this up). Unfortunately his live demo failed during the presentation with processes getting stuck, but I actually saw the demo before his session and it certainly impressed me!
Unfortunately I could only attend the conference on Saturday, as I was traveling to and back from the event by train on Friday and Sunday. I wished I could have stayed for more! I will definitely plan to go back there some time.
I took a few pictures during the event, which can be found on Flickr :
More pictures can also be found here and here.
In summary, I had a great time at DevConf.CZ and I can only highly recommend to plan on attending it if you get a chance!
Categories
Pictures from recent Ceph events
I finally got around to publishing a number of photo albums related to Ceph events that I attended in the past few months:
Der Open Source Business Award, kurz OSBAR, ist der Innovationspreis des Open Source Business Alliance Vereins. Auch in diesem Jahr werden wieder innovative und interessante Open Source-Projekte gesucht, die sich für diesen Preis bewerben wollen.
Ich freue mich sehr, dieses Jahr Teil der Jury zu sein und bin schon sehr auf die Einreichungen gespannt. Am 16.11. findet in Berlin der Open Source Day (ODS) der OSB Alliance statt; dort können die Bewerber ihr Projekt persönlich vorstellen. Anschließend läuft die Bewertung der Einreichungen. Am 6.12. werden dann im Rahmen der OPEN-Konferenz in Stuttgart die Gewinner bekannt gegeben.
Die Teilnahme ist sehr einfach und ist eine tolle Gelegenheit, den Bekanntheitsgrad eines Projekts zu steigern. In diesem Sinne: mitmachen, man kann nur gewinnen!
I never thought I’d ever say this, but I must admit that I have become quite a fan of Microsoft’s Visual Studio Code IDE, which is available for a wide range of platforms, including Linux.
However, the installation user experience was still somewhat arcane – one had to manually download the RPM package from their web site to install the software. Once a new version was available, the user receives a notification within the application, which redirects him to the website for downloading and updating the latest RPM manually again.
Therefore it didn’t take long for the community to request the creation of a dedicated yum repository, to keep the package updated automatically.
This issue has now been resolved: a yum repo has now been created. Until github issue 20895 has been fixed, this still requires setting up the required repository configuration manually though, but hopefully this will be addressed soon as well.
Until then, here’s how to enable the yum repo on Fedora Linux (Fedora 25 in my case):
Create a file /etc/yum.repos.d/vscode.repo with the following content:
[code]
name=Visual Studio Code
baseurl=https://packages.microsoft.com/yumrepos/vscode
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
Now you can install the package with the following command:
$ sudo dnf install code
Upon the first run, you will be requested to import the GPG build key used for signing the package.
Once a new updates becomes available, running dnf update will pull and install the updated package automatically.
Thanks to the developers at Microsoft for making this step much easier now!
Update: this solution does no longer work on later versions of Fedora that switched to Wayland instead of X.org by default. If you don’t want to switch back to X.org and you’re using the GNOME desktop environment, you can enable scroll wheel emulation as outlined here.
I’ve been struggling with this for quite some time now, but I finally figured out how to enable scroll wheel emulation for the Logitech Trackman Marble on Fedora Linux 24.
Previously (when I was using Ubuntu Linux), I had a small shell script that defined the required xinput properties. However, this did not work on Fedora, as they use the new libinput framework.
With the change to the libinput subsystem, you can now enable this behavior by creating a file /etc/X11/xorg.conf.d/10-libinput.conf with the following content:
Section "InputClass" Identifier "Marble Mouse" MatchProduct "Logitech USB Trackball" Driver "libinput" Option "ScrollMethod" "button" Option "ScrollButton" "8" EndSection
Magically, this function got enabled as soon as I saved the file, without even having to restart X! I’m impressed.
A while ago, I concluded my first year in the openATTIC-Team at it-novum. We’ve gone through a lot of changes and I am quite proud of what we’ve achieved so far. In many ways, we’re starting this project and product from scratch, and I’m excited to be able to contribute to this effort with my experience from past jobs, and to help shaping the future direction. I feel grateful for having a very enthusiastic and supportive team. It’s also quite satisfying to observe how the seeds that we planted slowly come to fruition, as the project begins to gain traction with a growing user base and developer community.
For more details on some of the key highlights, please see my blog post on the openATTIC blog.
The openATTIC project is currently transitioning to adapting the patch contribution signoff process initially established by the Linux Kernel developers, in which every commit message is “signed” with a special Signed-off-by: tag line, that certifies that the patch contribution meets certain criteria with regards to ownership/copyright and licensing.
The git distributed revision control system (DVCS) used by the Linux Kernel and many other projects actually provides an option --sign that can be passed to the git commit command, which will add the Signed-off-by: line at the end of the commit log message automatically.
However, the openATTIC code base is managed using the Mercurial DVCS, which currently does not offer the same functionality out of the box. Fortunately, Mercurial is written in Python and is very extensible: it is possible to extend its functionality and modify the workflows via Hooks and Extensions.
Much to my surprise, I was not able to find an existing plugin or extension that provides this functionality. So I took this opportunity to hone my Python skills and wrote one myself – signoff.py is a simple Mercurial hook, that will automatically add the signoff tag line below the commit message. Currently it is pretty rudimentary, I still need to read up on how to convert it into a proper Mercurial extension that extends the existing hg commit command, so the signoff process can be enabled by passing it as an option.
To enable it, dowmload the file and save it as signoff.py into the .hg directory of your Mercurial repository and add the following to .hg/hgrc:
[hooks] precommit = python:.hg/signoff.py:sign_commit_message
If you have any suggestions or ideas on how to improve it, please let me know!
Update: the MFC-7460DN printer is now supported via the brlaser open source printer driver by Peter De Wachter. See this post for a much simpler way to configure this printer on Fedora (and other Linux distributions).
I’ve always been a fan of Brother, as their devices usually come with decent support for the Linux OS (at least initially). I have an MFC-7460DN Laser Printer / Scanner in my home office, that worked fine with Ubuntu Linux for the past few years. It’s hooked up to my DSL router’s ethernet switch and acts like a network printer for all of our devices.
Just to keep my mind flexible and to take a look at another Linux distribution for a change, I recently started using Fedora Workstation 23 on my Laptop (a company-issued Lenovo ThinkPad T440s). While the OS installation was painless and all main components like Video, Audio, Networking were detected and configured correctly out of the box, the post-installation of some tools and services required some more effort.
This time, the printing part of the MFC-7460DN took me quite some time to figure out. While Brother provides RPM packages of the drivers, they are 32-bit only, and the instructions hadn’t been updated since Fedora 12. The first thing I had to do was to download two driver RPM packages. I initially started with the newer versions of the drivers, brgenml1cupswrapper-3.1.0 and brgenml1lpr-3.1.0, but somehow did not get them to work at all. I then tried the older packages, mfc7460dnlpr-2.1.0 and cupswrapperMFC7460DN-2.0.4. These installed flawlessly, and a new printer was added to the CUPS configuration automatically.
However, it was configured as a local printer, so I first had to change the existing configuration to talk to the remote LPD port instead. While the printer configuration looked correct and no errors showed up, all print jobs simply disappeared into the bit bucket, without any visible error on the application side. Unfortunately the web-based CUPS administration tool was not much helpful, either – the button View Error Log simply returned a “Not found” error. There was no error log file in /var/log/cups, so I queried the status of the CUPS service via systemd next.
The command systemctl status -l cups then gave me a first hint:
sh: /opt/brother/Printers/BrGenML1//lpd/rawtobr3: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
Since this is a 32-bit binary, it might help to actually install a 32-bit version of the GNU C library! I simply forgot this step, even though it’s documented in the installation instructions. A simple dnf install glibc.i686 got me over this hurdle.
Unfortunately the print jobs still did not reach the printer and disappeared in the void! Checking the CUPS error log again, I now saw this:
/usr/local/Brother/Printer/MFC7460DN/lpd/filterMFC7460DN: line 131: 11660 Done eval cat $INPUT_TEMP 11661 Broken pipe | $PSCONV $PSCONV_OP 11662 Segmentation fault | $BRCONV $BRCONV_OP PID 11602 (/usr/lib/cups/filter/brlpdwrapperMFC7460DN) exited with no errors. PID 11603 (/usr/lib/cups/backend/lpd) exited with no errors.
Hilarious. Oh well, maybe the 32-bit binary is simply too old and crashes in the new environment? Let’s take a look at the full systemd journal with journalctl! This gave me further clues:
cupsd[10951]: /usr/local/Brother/Printer/MFC7460DN/lpd/filterMFC7460DN: line 131: 11660 Done eval cat $INPUT_TEMP
cupsd[10951]: 11661 Broken pipe | $PSCONV $PSCONV_OP
cupsd[10951]: 11662 Segmentation fault | $BRCONV $BRCONV_OP
cupsd[10951]: PID 11602 (/usr/lib/cups/filter/brlpdwrapperMFC7460DN) exited with no errors.
cupsd[10951]: PID 11603 (/usr/lib/cups/backend/lpd) exited with no errors.
cupsd[10951]: time-at-completed=1449759139
cupsd[10951]: Job completed.
cupsd[10951]: Removing document files.
dbus[1190]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
setroubleshoot[11624]: SELinux is preventing brprintconflsr3 from using the execmem access on a process. For complete SELinux messages. run sealert -l 5d873063-1d87-4e82-b
python3[11624]: SELinux is preventing brprintconflsr3 from using the execmem access on a process.
***** Plugin catchall_boolean (89.3 confidence) suggests ******************
If you want to allow cups to execmem
Then you must tell SELinux about this by enabling the 'cups_execmem' boolean.
Do
setsebool -P cups_execmem 1
***** Plugin catchall (11.6 confidence) suggests **************************
If you believe that brprintconflsr3 should be allowed execmem access on processes labeled cupsd_t by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep brprintconflsr3 /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp
OK, so SELinux seems to be getting in the way here. I did as suggested and ran the following commands:
# setsebool -P cups_execmem 1 # grep brprintconflsr3 /var/log/audit/audit.log | audit2allow -M brothermfc7460dn # semodule -i brothermfc7460dn.pp
An lo and behold, the printer started printing! Let’s hope it still does when I reboot the system…
Things I learned and that surprised me:
- The error handling in CUPS completely failed here. There was not a single end-user accessible hint that something went wrong, the print jobs just disappeared in the void.
- The move to
systemdstill has some ripple effects, e.g. the “Not found” error for the missing CUPS error log in the web UI. - Analyzing log files with
journalctlis actually quite convenient. Instead of grepping and tailing multiple logs under/var/log/, these tasks can now be performed using a single tool. - SELinux is still a bitch, even though the hints provided by
setroubleshootdwere quite useful to resolve the issues at hand.