OSS Blog

Me, curl and Dagens Nyheter

In the afternoon of October 1st 2019, I had the pleasure of welcoming Linus Larsson and Jonas Lindkvist into my home in Huddinge, south of Stockholm, Sweden. My home is also my office as I work full-time from home. These two fine gentlemen work for Sweden’s largest morning newspaper, Dagens Nyheter, which boasts 850,000 daily … Continue reading Me, curl and Dagens Nyheter →

Prewarming PostgreSQL I/O caches

PostgreSQL uses shared_buffers to cache blocks in memory. The idea is to reduce disk I/O and to speed up the database in the most efficient way possible. During normal operations your database cache will be pretty useful and ensure good response times. However, what happens if your database instance is restarted – for whatever reason? […] The post Prewarming PostgreSQL I/O caches appeared first on Cybertec.

State of Calibre in Debian

To counter some recent FUD spread about Calibre in general and Calibre in Debian in particular, here a concise explanation of the current state. Many might have read my previous post on Calibre as...

Richard W.M. Jones: libnbd + FUSE = nbdfuse

I’ve talked before about libnbd, our NBD client library. New in libnbd 1.2 is a tool called nbdfuse which lets you turn NBD servers into virtual files. A couple of weeks ago I mentioned you can use libnbd as a C library to edit qcow2 files. Now you can turn qcow2 files into virtual raw files: $ mkdir dir $ nbdfuse dir/file.raw \ --socket-activation qemu-nbd -f qcow2 file.qcow2 $ ls -l dir/ total 0 -rw-rw-rw-. 1 nbd nbd 1073741824 Jan 1 10:10 file.raw Reads and writes to file.raw are backed by the ori[..]

Dirk Eddelbuettel: GitHub Streak: Round Six

Five ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery: This is sometimes called Jerry Seinfeld’s secret to productivity: Just keep at it. Don’t break the streak. and then showed the first chart of GitHub streaking github activity october 2013 to october 2014 And four year ago a first follow-up appeared in this post: github activity october 2014 to october 2015 And three years ago we had a followup github activity october 2015 to october 2016 And two year[..]

R with TensorFlow 2.0 on Debian/sid

I recently posted on getting TensorFlow 2.0 with GPU support running on Debian/sid. At that time I didn’t manage to get the tensorflow package for R running properly. It didn’t need much to get...

Using American Fuzzy Lop on network clients

Previously I’ve fuzzed hivex and nbdkit using my favourite fuzzing tool, Michał Zalewski’s American Fuzzy Lop (AFL). AFL works by creating test cases which are files on disk, and then feeding those to programs which have been specially compiled so … Continue reading →

What is autovacuum doing to my temporary tables?

Did you know that your temporary tables are not cleaned up by autovacuum? If you did not, consider reading this blog post about PostgreSQL and autovacuum. If you did – well, you can still continue to read this article. Autovacuum cleans tables automatically Since the days of PostgreSQL 8.0, the database has provided this miraculous […] The post What is autovacuum doing to my temporary tables? appeared first on Cybertec.

RcppArmadillo 0.9.800.1.0

Another month, another Armadillo upstream release! Hence a new RcppArmadillo release arrived on CRAN earlier today, and was just shipped to Debian as well. It brings a faster solve() method and other goodies. We also switched to the (awesome) tinytest unit test frameowrk, and Min Kim made the configure.ac script more portable for the benefit of NetBSD and other non-bash users; see below for more details. One again we ran two full sets of reverse-depends checks, no issues were found, and the packages was aut[..]

Kevin Fenzi: EPEL8 packages

With the opening up of EPEL8, there’s a lot of folks looking and seeing packages they formerly used in EPEL6/7 not being available and wondering why. The reason is simple: EPEL is not a fixed exact list of packages, it’s a framework that allows interested parties to build and provide the packages they are interested in providing to the community. This means for a package to be in EPEL8, it requires a maintainer to step forward and explicitly ask “I’d like to maintain this in EPEL8” and then build, test and [..]

RIP (for now) Calibre in Debian

The current purge of all Python2 related packages has a direct impact on Calibre. The latest version of Calibre requires Python modules that are not (anymore) available for Python 2, which means that Calibre...

TensorFlow 2.0 with GPU on Debian/sid

Some time ago I have been written about how to get Tensorflow (1.x) running on current Debian/sid back then. It turned out that this isn’t correct anymore and needs an update, so here it...

How PostgreSQL estimates parallel queries

Parallel queries were introduced back in PostgreSQL 9.6, and the feature has been extended ever since. In PostgreSQL 11 and PostgreSQL 12, even more functionality has been added to the database engine. However, there remain some questions related to parallel queries which often pop up during training and which definitely deserve some clarification. Estimating the […] The post How PostgreSQL estimates parallel queries appeared first on Cybertec.

Fixing track_activity_query_size in postgresql.conf

Many of you might have wondered why some system views and monitoring statistics in PostgreSQL can contain incomplete query strings. The answer is that in PostgreSQL, it’s a configuration parameter that determines when a query will be cut off: track_activity_query_size. This blog post explains what this parameter does and how it can be used to […] The post Fixing track_activity_query_size in postgresql.conf appeared first on Cybertec.

Richard W.M. Jones: And another boot sector hack

This is Sakura by Řrřola. I have modified it very slightly to turn it into a boot sector program. You can run it like this. Note it takes a few seconds to start up. nbdkit data data=" 49 192 49 219 185 255 0 190 0 1 191 254 255 189 28 9 176 19 205 16 104 0 160 7 0 198 142 234 186 200 3 137 200 238 66 238 208 232 238 208 232 238 226 240 49 255 214 101 134 5 8 192 117 8 107 199 255 193 248 9 12 64 170 9 255 117 235 140 196 228 64 12 1 107 192 85 128 204 128 80 9 228 120 245 177 255 81 140 198 13[..]

Richard W.M. Jones: Another NBD boot sector hack

I was shown a link to an incredible 64 byte MS-DOS demo called into war by HellMood/DESiRE. It doesn’t actually depend on MS-DOS however, using only BIOS calls and PC hardware, so it was easy to turn it into a boot sector. We can use nbdkit-data-plugin to run it from the command line: nbdkit data data=" 49 192 80 185 255 0 190 0 1 191 254 255 189 28 9 20 19 186 48 3 243 110 205 16 184 79 12 230 64 226 247 31 104 0 165 7 184 205 204 247 231 137 232 128 238 246 246 246 146 44 127 246 234 2 22 108 4 146 [..]

10 years in Japan

Exactly 10 years ago, on October 1, 2009, I started my work at the Japan Advanced Institute of Science and Technology (JAIST), arriving the previous day in a place not completely unknown, but with...

Richard W.M. Jones: How to edit a qcow2 file from C

Suppose you want to edit or read or write the data inside a qcow2 file? One way is to use libguestfs, and that’s the recommended way if you need to mount a filesystem inside the file. But for accessing the data blocks alone, you can now use the libnbd API and qemu-nbd together and this has a couple of advantages: It’s faster and you can open snapshots (which libguestfs cannot do). We start by creating a libnbd handle and connecting it to a qemu-nbd instance. The qemu-nbd instance is linked with qemu’s inter[..]

TeX Live/Debian updates 20190930

TeX Live 2019 has seen already many updates since the initial upload to Debian, most of which I have never reported about. Today I have uploaded a new set of packages, based on the...

When companies use the GPL against each other, our community loses

Two router manufacturers, Cambium and Ubiquiti, are using the GPL as a pawn in their ongoing commercial dispute that has resulted in a lawsuit. Our analysis shows that neither complies with the GPL's most basic provisions to begin with, making the suit a distraction from the GPL's main goal: ensuring users' freedom to modify the software in their devices.

Kevin Fenzi: MUA++ (or on to neomutt)

About 18 months ago or so, I posted about switching my Mail User Agent / mail client from claws-mail to thunderbird: https://www.scrye.com/wordpress/nirik/2017/03/24/mua-or-on-to-thunderbird/ last week, I moved on to a new setup. First let em talk a bit about why I am moving on from thunderbird. Since moving to it thunderbird seems to have gotten slower and slower (no matter how much I compacted mail folders). They made some, IMHO, anoying changes (like in thunderbird 52: “”When replying to a mailing list, [..]

Using “Row Level Security” to make large companies more secure

Large companies and professional business have to make sure that data is kept secure. It is necessary to defend against internal, as well as external threats. PostgreSQL provides all the necessities a company needs to protect data and to ensure that people can only access what they are supposed to see. One way to protect […] The post Using “Row Level Security” to make large companies more secure appeared first on Cybertec.

RcppAnnoy 0.0.13

A new release of RcppAnnoy is now on CRAN. RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours—originally developed to drive the famous Spotify music discovery algorithm. This release brings several updates. First and foremost, the upstream Annoy C++ code was updated from version 1.12 to 1.16 bringing both speedier code thanks to AVX512 instruction (where available[..]

digest 0.6.21

A new version of digest is just now arriving at CRAN (following a slight holdup over one likely spurious reverse dependency error), and I will send an updated package to Debian shortly as well. digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, and spookyhash algorithms) permitting easy comparison of R language objects. It is a fairly widely-used package (currently listed at 795k downloads) as many tasks may involve caching of obj[..]

Karen Sandler keynotes State of the Map Tomorrow!

If you are up late in US/Pacific, up early in US/Eastern, or in Europe, be sure to catch the live stream of Karen Sandler's keynote address at the 2019 State of the Map Conference. The keynote begins at 10:30 EU/Central (08:30 UTC, 04:30 US/Eastern, 01:30 US/Pacific).

Cascade – a turn-based text arcade game

I wrote this game about 20 years ago. Glad to see it still compiled out of the box on the latest Linux distro! Download it from here. If anyone can remember the name or any details of the original 1980s … Continue reading →

Gaming: Puzzle Agent

Two lovely but short puzzle games: Puzzle Agent and Puzzle Agent II, follow agent Nelson Tethers in his quest to solve an obscure case in Scoggins, Minnesota: The erasers factory delivering to the White...

nbdkit supports exportnames

(You’ll need the very latest version of libnbd and nbdkit from git for this to work.) The NBD protocol lets the client send an export name string to the server. The idea is a single server can serve different content … Continue reading →

TeX Services at texlive.info

I have been working over the last weeks to provide four more services for the TeX (Live) community: an archive of TeX Live’s network installation directory tlnet, a git repository of CTAN, a mirror...

curl 7.66.0 – the parallel HTTP/3 future is here

I personally have not done this many commits to curl in a single month (August 2019) for over three years. This increased activity is of course primarily due to the merge of and work with the HTTP/3 code. And yet, that is still only in its infancy… Download curl here. Numbers the 185th release6 changes54 … Continue reading curl 7.66.0 – the parallel HTTP/3 future is here →

Dirk Eddelbuettel: pinp 0.0.9: Real Fix and Polish

Another pinp package release! pinp allows for snazzier one or two column Markdown-based pdf vignettes, and is now used by a few packages. A screenshot of the package vignette can be seen below. Additional screenshots are at the pinp page. This release comes exactly one week (i.e. the minimal time to not earn a NOTE) after the hot-fix release 0.0.8 which addressed breakage on CRAN tickled by changed in TeX Live. After updating the PNAS style LaTeX macros, and avoiding the issue with an (older) custom copy of[..]

ttdo 0.0.3: New package

A new package of mine arrived on CRAN yesterday, having been uploaded a few days prior on the weekend. It extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam. Mark also tweeted about it. The package was written to address a fairly specific need. In teaching STAT 430 at Illinois, I am relying on the powerful PrairieLearn system (developed there) to provides tests, quizzes or homework[..]

AMD Ryzen 9 3900X – nice!

This thing really screams. It’s nice being able to do make -j24 (threads) builds so quickly. Advertisements

FIPS ready with curl

Download wolfSSL fips ready (in my case I got wolfssl-4.1.0-gplv3-fips-ready.zip) Unzip the source code somewhere suitable $ cd $HOME/src $ unzip wolfssl-4.1.0-gplv3-fips-ready.zip $ cd wolfssl-4.1.0-gplv3-fips-ready Build the fips-ready wolfSSL and install it somewhere suitable $ ./configure --prefix=$HOME/wolfssl-fips --enable-harden --enable-all $ make -sj $ make install Download curl, the normal curl package. (in my case I … Continue reading FIPS ready with curl →

Debian Activities of the last few months

I haven’t written about specific Debian activities in recent times, but I haven’t been lazy. In fact I have been very active with a lot of new packages I am contributing to. TeX and...

Gwyn Ciesla: Inkscape 1.0 Beta

Fresh and hot in f32! Come test and enjoy!

pinp 0.0.8: Bugfix

A new release of our pinp package is now on CRAN. pinp allows for snazzier one or two column Markdown-based pdf vignettes, and is now used by a few packages. A screenshot of the package vignette can be seen below. Additional screenshots are at the pinp page. This release was spurned by one of those "CRAN package xyz" emails I received yesterday: processing of pinp-using vignettes was breaking at CRAN under the newest TeX Live release present on Debian testing as well as recent Fedora. The rticles package (w[..]

Cloud providers and telemetry via Qt MQTT

This is a follow up to my previous posts about using Qt MQTT to connect to the cloud. MQTT is a prominent standard for telemetry, especially in the IoT scenario. We are often approached by Qt customers and users on how to connect to a variety of cloud providers, preferably keeping the requirements list […] The post Cloud providers and telemetry via Qt MQTT appeared first on Qt Blog.

Richard W.M. Jones: libnbd and nbdkit man pages online

Let’s seed those search engines … http://libguestfs.org/libnbd.3.html http://libguestfs.org/nbdkit.1.html And yes I know there are a few broken links.

What’s been happening in Ignite

First of all: thanks Dennis Marttinen and Lucas Käldström for helping write this up. It’s been only a bit over a month since Weave Ignite was announced to the world (others talked about it as well). Time to catch up on what happened in the meantime, the team around it has been busy. If you’re […]

PostgreSQL Row Level Security, views and a lot of magic

Row Level Security (RLS) is one of the key features in PostgreSQL. It can be used to dramatically improve security and help to protect data in all cases. However, there are a couple of corner cases which most people are not aware of. So if you are running PostgreSQL and you happen to use RLS […] The post PostgreSQL Row Level Security, views and a lot of magic appeared first on Cybertec.

Dirk Eddelbuettel: RcppArmadillo 0.9.700.2.0

A new RcppArmadillo release based on a new Armadillo upstream release arrived on CRAN, and will get to Debian shortly. It brings continued improvements for sparse matrices and a few other things; see below for more details. I also appear to have skipped blogging about the preceding 0.9.600.4.0 release (which was actually extra-rigorous with an unprecedented number of reverse-depends runs) so I included its changes (with very nice sparse matrix improvements) as well. Armadillo is a powerful and expressive C+[..]

Qt Visual Studio Tools 2.4 RC Released

We have released Qt Visual Studio Tools 2.4 RC (version 2.4.0); the installation package is available in the Qt download page. This version features an improved integration of Qt tools with the Visual Studio project system, addressing some limitations of the current integration methods, most notably the inability to have different Qt settings for each […] The post Qt Visual Studio Tools 2.4 RC Released appeared first on Qt Blog.

Microsoft & exFAT: One Step on a Long Journey

This week Microsoft announced that they would sign off on inclusion of exFAT in upstream Linux. In this blog post, I take a closer look at what this means for patent peace around exFAT.

anytime 0.3.6

A fresh and very exciting release of the anytime package is arriving on CRAN right now. This is the seventeenth release, and it comes pretty much exactly one month after the preceding 0.3.5 release. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, … format to either POSIXct or Date objects – and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples. This relea[..]


Windowing functions and analytics have been around for quite some time and many people already make use of this awesome stuff in the PostgreSQL world. Timeseries are an especially important area in this context. However, not all features have been widely adopted and thus many developers have to implement functionality at the application level in […] The post Timeseries: EXCLUDE TIES, CURRENT ROW and GROUP appeared first on Cybertec.

If you can't stand the heat, get out of the kitchen...

Wow, we had a hot weekend in Cambridge. About 40 people turned up to our place in Cambridge for this year's OMGWTFBBQ. Last year we were huddling under the gazebos for shelter from torrential rain; this year we again had all the gazebos up, but this time to hide from the sun instead. We saw temperatures well into the 30s, which is silly for Cambridge at the end of August. I think it's fair to say that everybody enjoyed themselves despite the ludicrous heat levels. We had folks from all over the UK, and Lars[..]

Gwyn Ciesla: Inkscape – Python 3 in f32+

I’ve just updated Inkscape in f32 to a git snapshot to allow it to move to Python 3. It seems to work well for me, but please test and file bugs. There should be an official release coming to address this, but I don’t want to hold up the removal of Python 2.

PostgreSQL: Trivial timeseries examples

Timeseries are an increasingly important topic – not just in PostgreSQL. Recently I gave a presentation @AGIT in Salzburg about timeseries and I demonstrated some super simple examples. The presentation was well received so I decided to share this stuff in the form of a blog PostgreSQL, so that more people can learn about windowing functions […] The post PostgreSQL: Trivial timeseries examples appeared first on Cybertec.

RcppExamples 0.1.9

A new version of the RcppExamples package is now on CRAN. The RcppExamples package provides a handful of short examples detailing by concrete working examples how to set up basic R data structures in C++. It also provides a simple example for packaging with Rcpp. This releases brings a number of small fixes, including two from contributed pull requests (extra thanks for those!), and updates the package in a few spots. The NEWS extract follows: Changes in RcppExamples version 0.1.9 (2019-08-24) Extended Date[..]