Will Jessop's Writings

Fixing errors installing the charlock_holmes ruby gem redux

will@willj.net (Will Jessop) — Thu, 29 May 2025 10:50:00 +0000

A while back I posted about fixing an error installing the charlock_holmes gem on MacOS, well the error has returned. Here’s the newest fix.

If you get an error that looks something like this:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
/Users/will/.rbenv/versions/3.4.3/bin/ruby extconf.rb
--with-opt-dir\=/usr/local/opt --with-opt-include\=/usr/local/opt/icu4c/include
--with-opt-lib\=/usr/local/opt/icu4c/lib --with-cxxflags\=-std\=c++11
checking for pkg-config for icu-i18n...
["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ",
"-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licui18n"]
checking for pkg-config for icu-io...
["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ",
"-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licuio"]
checking for pkg-config for icu-uc...
["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ",
"-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licuuc"]
checking for -licui18n... yes
checking for unicode/ucnv.h... yes
checking for -lz... yes
checking for -licuuc... yes
checking for -licudata... yes
checking for icu that requires explicit C++ version flag... yes
checking for icu that compiles with c++20 standard... yes
Static linking is disabled.
creating Makefile

current directory:
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
make DESTDIR\= sitearchdir\=./.gem.20250529-47905-r2zqyx
sitelibdir\=./.gem.20250529-47905-r2zqyx clean

current directory:
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
make DESTDIR\= sitearchdir\=./.gem.20250529-47905-r2zqyx
sitelibdir\=./.gem.20250529-47905-r2zqyx
compiling converter.c
In file included from converter.c:2:
In file included from ./common.h:9:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6:
warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37:
warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling encoding_detector.c
In file included from encoding_detector.c:2:
In file included from ./common.h:9:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6:
warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37:
warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling ext.c
In file included from ext.c:1:
In file included from ./common.h:9:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6:
warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37:
warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling transliterator.cpp
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:15:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/utypes.h:39:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/uversion.h:182:11: warning:
nested namespace definition is a C++17 extension; define each namespace
separately [-Wc++17-extensions]
  182 | namespace U_HEADER_ONLY_NAMESPACE {}
      |           ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/uversion.h:180:48: note:
expanded from macro 'U_HEADER_ONLY_NAMESPACE'
180 | #define U_HEADER_ONLY_NAMESPACE
U_ICU_NAMESPACE::U_HEADER_NESTED_NAMESPACE
      |                                                ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:271:38: error: no
template named 'enable_if_t' in namespace 'std'
271 | template<typename T, typename = std::enable_if_t<std::is_same_v<T,
UChar>>>
      |                                 ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:271:55: error: no
template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
271 | template<typename T, typename = std::enable_if_t<std::is_same_v<T,
UChar>>>
      |                                                  ~~~~~^~~~~~~~~
      |                                                       is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29:
note: 'is_same' declared here
22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)>
{};
      |                             ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:8: warning:
constexpr if is a C++17 extension [-Wc++17-extensions]
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |        ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:24: error: no
template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |                   ~~~~~^~~~~~~~~
      |                        is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29:
note: 'is_same' declared here
22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)>
{};
      |                             ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:50: error:
expected unqualified-id
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |                                                  ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:385:16: warning:
variable templates are a C++14 extension [-Wc++14-extensions]
  385 | constexpr bool ConvertibleToU16StringView =
      |                ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:386:10: error: no
template named 'is_convertible_v' in namespace 'std'; did you mean
'is_convertible'?
  386 |     std::is_convertible_v<T, std::u16string_view>
      |     ~~~~~^~~~~~~~~~~~~~~~
      |          is_convertible
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_convertible.h:22:29:
note: 'is_convertible' declared here
22 | struct _LIBCPP_TEMPLATE_VIS is_convertible : public
integral_constant<bool, __is_convertible(_T1, _T2)> {};
      |                             ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:393:5: error:
expected '(' for function-style cast or type construction
  386 |     std::is_convertible_v<T, std::u16string_view>
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
387 | #if !U_CHAR16_IS_TYPEDEF && (!defined(_LIBCPP_VERSION) ||
_LIBCPP_VERSION < 180000)
  388 |     || std::is_convertible_v<T, std::basic_string_view<uint16_t>>
  389 | #endif
  390 | #if U_SIZEOF_WCHAR_T==2
  391 |     || std::is_convertible_v<T, std::wstring_view>
  392 | #endif
  393 |     ;
      |     ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:36: error: no
type named 'enable_if_t' in namespace 'std'
429 |           typename = typename
std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
      |                      ~~~~~~~~~~~~~~^~~~~~~~~~~
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:47: error:
expected ',' or '>' in template-parameter-list
429 |           typename = typename
std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
      |                                               ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:93: error:
expected unqualified-id
429 |           typename = typename
std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
|
^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:36: error: no
type named 'enable_if_t' in namespace 'std'
439 |           typename = typename
std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
      |                      ~~~~~~~~~~~~~~^~~~~~~~~~~
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:47: error:
expected ',' or '>' in template-parameter-list
439 |           typename = typename
std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
      |                                               ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:92: error:
expected unqualified-id
439 |           typename = typename
std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
|
^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:40:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:134:29: error:
no template named 'enable_if_t' in namespace 'std'
  134 |             typename = std::enable_if_t<
      |                        ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:135:23: error:
no template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
  135 |                 (std::is_same_v<decltype(T().data()), const char*>
      |                  ~~~~~^~~~~~~~~
      |                       is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29:
note: 'is_same' declared here
22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)>
{};
      |                             ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:40:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:140:17: error:
use of address-of-label extension outside of a function body
  140 |                 std::is_same_v<decltype(T().size()), size_t>>>
      |                 ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:140:62: error:
expected member name or ';' after declaration specifiers
  140 |                 std::is_same_v<decltype(T().size()), size_t>>>
      |                                                              ^
In file included from transliterator.cpp:5:
In file included from
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:346:40: error: no
template named 'enable_if_t' in namespace 'std'
346 |   template<typename S, typename =
std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:381:40: error: no
template named 'enable_if_t' in namespace 'std'
381 |   template<typename S, typename =
std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:1959:40: error: no
template named 'enable_if_t' in namespace 'std'
1959 |   template<typename S, typename =
std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
3 warnings and 20 errors generated.
make: *** [transliterator.o] Error 1

make failed, exit code 2

Gem files will remain installed in
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9
for inspection.
Results logged to
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/extensions/arm64-darwin-24/3.4.0/charlock_holmes-0.7.9/gem_make.out

/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in
'Gem::Ext::Builder.run'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in
'block in Gem::Ext::Builder.make'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in
'Array#each'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in
'Gem::Ext::Builder.make'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in
'Gem::Ext::ExtConfBuilder.build'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in
'Gem::Ext::Builder#build_extension'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in
'block in Gem::Ext::Builder#build_extensions'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in
'Array#each'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in
'Gem::Ext::Builder#build_extensions'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/installer.rb:844:in
'Gem::Installer#build_extensions'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:76:in
'Bundler::RubyGemsGemInstaller#build_extensions'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:28:in
'Bundler::RubyGemsGemInstaller#install'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/source/rubygems.rb:203:in
'Bundler::Source::Rubygems#install'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:54:in
'Bundler::GemInstaller#install'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:16:in
'Bundler::GemInstaller#install_from_spec'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:130:in
'Bundler::ParallelInstaller#do_install'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:121:in
'block in Bundler::ParallelInstaller#worker_pool'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:62:in
'Bundler::Worker#apply_func'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:57:in
'block in Bundler::Worker#process_queue'
  <internal:kernel>:168:in 'Kernel#loop'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in
'Bundler::Worker#process_queue'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:90:in
'block (2 levels) in Bundler::Worker#create_threads'

An error occurred while installing charlock_holmes (0.7.9), and Bundler
cannot continue.

In Gemfile:
  charlock_holmes
will@lentil ~/clients/Outvote/repos/web-app/master (on master)% gem uninstall charlock_holmes
Gem 'charlock_holmes' is not installed
will@lentil ~/clients/Outvote/repos/web-app/master (on master)% git pull
will@lentil ~/clients/Outvote/repos/web-app/master (on master)% bundle
Fetching gem metadata from https://enterprise.contribsys.com/..
Fetching gem metadata from https://rubygems.org/........
Installing charlock_holmes 0.7.9 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
/Users/will/.rbenv/versions/3.4.3/bin/ruby extconf.rb --with-opt-dir\=/usr/local/opt --with-opt-include\=/usr/local/opt/icu4c/include
--with-opt-lib\=/usr/local/opt/icu4c/lib --with-cxxflags\=-std\=c++11
checking for pkg-config for icu-i18n... ["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ", "-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licui18n"]
checking for pkg-config for icu-io... ["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ", "-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licuio"]
checking for pkg-config for icu-uc... ["-I/opt/homebrew/Cellar/icu4c@77/77.1/include ", "-L/opt/homebrew/Cellar/icu4c@77/77.1/lib", "-licuuc"]
checking for -licui18n... yes
checking for unicode/ucnv.h... yes
checking for -lz... yes
checking for -licuuc... yes
checking for -licudata... yes
checking for icu that requires explicit C++ version flag... yes
checking for icu that compiles with c++20 standard... yes
Static linking is disabled.
creating Makefile

current directory: /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
make DESTDIR\= sitearchdir\=./.gem.20250529-48158-xjue2n sitelibdir\=./.gem.20250529-48158-xjue2n clean

current directory: /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9/ext/charlock_holmes
make DESTDIR\= sitearchdir\=./.gem.20250529-48158-xjue2n sitelibdir\=./.gem.20250529-48158-xjue2n
compiling converter.c
In file included from converter.c:2:
In file included from ./common.h:9:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6: warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37: warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling encoding_detector.c
In file included from encoding_detector.c:2:
In file included from ./common.h:9:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6: warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37: warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling ext.c
In file included from ext.c:1:
In file included from ./common.h:9:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby.h:38:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/ruby.h:27:
In file included from /Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/internal/anyargs.h:83:
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:6: warning: 'RUBY_API_VERSION_MAJOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |      ^
/Users/will/.rbenv/versions/3.4.3/include/ruby-3.4.0/ruby/backward/cxxanyargs.hpp:540:37: warning: 'RUBY_API_VERSION_MINOR' is not defined, evaluates to 0 [-Wundef]
  540 | #if (RUBY_API_VERSION_MAJOR * 100 + RUBY_API_VERSION_MINOR) >= 301
      |                                     ^
2 warnings generated.
compiling transliterator.cpp
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:15:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/utypes.h:39:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/uversion.h:182:11: warning: nested namespace definition is a C++17 extension; define each namespace separately
[-Wc++17-extensions]
  182 | namespace U_HEADER_ONLY_NAMESPACE {}
      |           ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/uversion.h:180:48: note: expanded from macro 'U_HEADER_ONLY_NAMESPACE'
  180 | #define U_HEADER_ONLY_NAMESPACE U_ICU_NAMESPACE::U_HEADER_NESTED_NAMESPACE
      |                                                ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:271:38: error: no template named 'enable_if_t' in namespace 'std'
  271 | template<typename T, typename = std::enable_if_t<std::is_same_v<T, UChar>>>
      |                                 ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:271:55: error: no template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
  271 | template<typename T, typename = std::enable_if_t<std::is_same_v<T, UChar>>>
      |                                                  ~~~~~^~~~~~~~~
      |                                                       is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29: note: 'is_same' declared here
   22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)> {};
      |                             ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:8: warning: constexpr if is a C++17 extension [-Wc++17-extensions]
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |        ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:24: error: no template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |                   ~~~~~^~~~~~~~~
      |                        is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29: note: 'is_same' declared here
   22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)> {};
      |                             ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:273:50: error: expected unqualified-id
  273 |     if constexpr (std::is_same_v<UChar, char16_t>) {
      |                                                  ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:385:16: warning: variable templates are a C++14 extension [-Wc++14-extensions]
  385 | constexpr bool ConvertibleToU16StringView =
      |                ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:386:10: error: no template named 'is_convertible_v' in namespace 'std'; did you mean 'is_convertible'?
  386 |     std::is_convertible_v<T, std::u16string_view>
      |     ~~~~~^~~~~~~~~~~~~~~~
      |          is_convertible
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_convertible.h:22:29: note: 'is_convertible' declared here
   22 | struct _LIBCPP_TEMPLATE_VIS is_convertible : public integral_constant<bool, __is_convertible(_T1, _T2)> {};
      |                             ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:37:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:393:5: error: expected '(' for function-style cast or type construction
  386 |     std::is_convertible_v<T, std::u16string_view>
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  387 | #if !U_CHAR16_IS_TYPEDEF && (!defined(_LIBCPP_VERSION) || _LIBCPP_VERSION < 180000)
  388 |     || std::is_convertible_v<T, std::basic_string_view<uint16_t>>
  389 | #endif
  390 | #if U_SIZEOF_WCHAR_T==2
  391 |     || std::is_convertible_v<T, std::wstring_view>
  392 | #endif
  393 |     ;
      |     ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:36: error: no type named 'enable_if_t' in namespace 'std'
  429 |           typename = typename std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
      |                      ~~~~~~~~~~~~~~^~~~~~~~~~~
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:47: error: expected ',' or '>' in template-parameter-list
  429 |           typename = typename std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
      |                                               ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:429:93: error: expected unqualified-id
  429 |           typename = typename std::enable_if_t<!std::is_pointer_v<std::remove_reference_t<T>>>>
      |                                                                                             ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:36: error: no type named 'enable_if_t' in namespace 'std'
  439 |           typename = typename std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
      |                      ~~~~~~~~~~~~~~^~~~~~~~~~~
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:47: error: expected ',' or '>' in template-parameter-list
  439 |           typename = typename std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
      |                                               ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/char16ptr.h:439:92: error: expected unqualified-id
  439 |           typename = typename std::enable_if_t<std::is_pointer_v<std::remove_reference_t<T>>>,
      |                                                                                            ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:40:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:134:29: error: no template named 'enable_if_t' in namespace 'std'
  134 |             typename = std::enable_if_t<
      |                        ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:135:23: error: no template named 'is_same_v' in namespace 'std'; did you mean 'is_same'?
  135 |                 (std::is_same_v<decltype(T().data()), const char*>
      |                  ~~~~~^~~~~~~~~
      |                       is_same
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/is_same.h:22:29: note: 'is_same' declared here
   22 | struct _LIBCPP_TEMPLATE_VIS is_same : _BoolConstant<__is_same(_Tp, _Up)> {};
      |                             ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:40:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:140:17: error: use of address-of-label extension outside of a function body
  140 |                 std::is_same_v<decltype(T().size()), size_t>>>
      |                 ^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/stringpiece.h:140:62: error: expected member name or ';' after declaration specifiers
  140 |                 std::is_same_v<decltype(T().size()), size_t>>>
      |                                                              ^
In file included from transliterator.cpp:5:
In file included from /opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/translit.h:27:
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:346:40: error: no template named 'enable_if_t' in namespace 'std'
  346 |   template<typename S, typename = std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:381:40: error: no template named 'enable_if_t' in namespace 'std'
  381 |   template<typename S, typename = std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
/opt/homebrew/Cellar/icu4c@77/77.1/include/unicode/unistr.h:1959:40: error: no template named 'enable_if_t' in namespace 'std'
 1959 |   template<typename S, typename = std::enable_if_t<ConvertibleToU16StringView<S>>>
      |                                   ~~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
3 warnings and 20 errors generated.
make: *** [transliterator.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/charlock_holmes-0.7.9 for inspection.
Results logged to /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/extensions/arm64-darwin-24/3.4.0/charlock_holmes-0.7.9/gem_make.out

  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:76:in 'Bundler::RubyGemsGemInstaller#build_extensions'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:28:in 'Bundler::RubyGemsGemInstaller#install'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/source/rubygems.rb:203:in 'Bundler::Source::Rubygems#install'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:54:in 'Bundler::GemInstaller#install'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:16:in 'Bundler::GemInstaller#install_from_spec'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:130:in 'Bundler::ParallelInstaller#do_install'
/Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:121:in 'block in
Bundler::ParallelInstaller#worker_pool'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
  <internal:kernel>:168:in 'Kernel#loop'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
  /Users/will/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/bundler-2.4.22/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'

An error occurred while installing charlock_holmes (0.7.9), and Bundler cannot continue.

In Gemfile:
  charlock_holmes

Try installing using a newer c++ standard:

gem install charlock_holmes -v=0.7.9 -- --with-opt-dir=/usr/local/opt --with-opt-include=/usr/local/opt/icu4c/include --with-opt-lib=/usr/local/opt/icu4c/lib --with-cxxflags=-std=c++17
Building native extensions with: '--with-opt-dir=/usr/local/opt --with-opt-include=/usr/local/opt/icu4c/include --with-opt-lib=/usr/local/opt/icu4c/lib --with-cxxflags=-std=c++17'
This could take a while...
Successfully installed charlock_holmes-0.7.9
1 gem installed

Final robot photos

will@willj.net (Will Jessop) — Tue, 06 Aug 2024 10:00:00 +0100

I spent several months iterating on the design and build of a ping pong ball collecting robot back in about 2014 but never posted the final photos of it anywhere, these are the final photos I took of the finished robot.

You should use the Ruby on Rails logger block syntax

will@willj.net (Will Jessop) — Wed, 31 Jul 2024 17:00:00 +0100

Summary

Passing strings to the Rails logger methods (eg. Rails.logger.info(…)) causes unecessary object allocations, and if you’re calling methods to generate data for your log messages then it can cause unecessary CPU work too. In this post I’ll show you how to use block syntax when using your logger with Rails to save object allocations and CPU time.

The Context

When you’re logging in Ruby on Rails log messages sent with a level above the currently configured log level will not be logged. The default in production is to log at info or below, so anything logged at debug level will not be output to the logs:

debug  ⬆️ Ignored
-----------------
info   ⬇️  Logged
warn
error
fatal
unknown

The log level can be set in the environment configs, from config/environments/production.rb:

config.log_level = ENV.fetch("RAILS_LOG_LEVEL", "info")

The log level can also be changed at any time, including in the Rails console, by setting the log level directly:

> Rails.logger.level = :warn
=> :warn

The problem - Object allocations for unused logs

Let’s say you want to log some debug data for an API call that a user is making. You might do something like this:

Rails.logger.debug "Processed API request to host #{somehost} for user \
  #{user.name} (#{user.id}): status: #{request_status} user_details: #{user.to_json}"

You push this code to production, it’s useful to have a way to debug user API issues just by setting the Rails log level to debug from info using the RAILS_LOG_LEVEL environment variable, you can always set it back to info when you’re done. Let’s look at what this statement allocates when the application is logging at the debug level:

# Set the log level
> Rails.logger.level = :debug
=> :debug

# Do the logging
> stats = AllocationStats.trace {
	Rails.logger.debug "Processed API request to \
      host #{somehost} for user #{user.name} (#{user.id}): status: #{request_status} \
      user_details: #{user.to_json}"
}
Processed API request to host for user Slartibartfast (42): status: DENIED user_details: {"id":42,"name":"Slartibartfast","email":"s.bartfast@magrathea.biz","favourite_colour":"blue"}
=> [#<AllocationStats::Allocation:0x000000015489fe18 @object=#<Proc:0x000000015471e9e0 activesupport-7.1.3.4/lib/active_support/broadcas...

# Get an allocation count, source paths shortened for brevity
> puts stats.allocations.group_by(:sourcefile, :sourceline, :class).to_text
                           sourcefile                             sourceline                      class                      count
----------------------------------------------------------------- ----------  ---------------------------------------------  -----
activesupport-7.1.3.4/lib/active_support/broadcast_logger.rb             231  Proc                                               1
(irb)                                                                     45  Array                                              1
ruby/3.3.0/json/common.rb                                                303  JSON::Ext::Generator::State                        1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb         188  String                                             4
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                 23  ActiveSupport::JSON::Encoding::JSONGemEncoder      1
(irb)                                                                     13  String                                             1
(irb)                                                                     45  String                                             2
<internal:timev>                                                         226  Time                                               2
ruby/3.3.0/json/common.rb                                                305  String                                             1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                 92  Hash                                               1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                 77  Hash                                               1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb         186  Hash                                               1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb          78  Hash                                               1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                 33  Hash                                               1
ruby/3.3.0/logger/log_device.rb                                           42  String                                             1
activesupport-7.1.3.4/lib/active_support/logger.rb                        38  String                                             2
ruby/3.3.0/logger/log_device.rb                                          128  File::Stat                                         1

This is fine! We’re logging some data, the Ruby VM is doing work and allocates objects. Let’s look at what happens when we change the log level to info when we’re done debugging:

# Set the log level
> Rails.logger.level = :info
=> :info

# Do the logging
> stats = AllocationStats.trace {
	Rails.logger.debug "Processed API request to \
      host #{somehost} for user #{user.name} (#{user.id}): status: #{request_status} \
      user_details: #{user.to_json}"
}
=> [#<AllocationStats::Allocation:0x000000015489fd28 @object=#<Proc:0x000000015471e648 activesupport-7.1.3.4/lib/active_support/broadcas...

# Get an allocation count, source paths shortened for brevity
> puts stats.allocations.group_by(:sourcefile, :sourceline, :class).to_text
                           sourcefile                              sourceline                      class                      count
-----------------------------------------------------------------  ----------  ---------------------------------------------  -----
activesupport-7.1.3.4/lib/active_support/broadcast_logger.rb              231  Proc                                               1
(irb)                                                                      48  Array                                              1
/Users/will/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb           303  JSON::Ext::Generator::State                        1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb          188  String                                             4
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                  23  ActiveSupport::JSON::Encoding::JSONGemEncoder      1
(irb)                                                                      13  String                                             1
(irb)                                                                      48  String                                             2
/Users/will/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb           305  String                                             1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                  92  Hash                                               1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                  77  Hash                                               1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb          186  Hash                                               1
activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb           78  Hash                                               1
activesupport-7.1.3.4/lib/active_support/json/encoding.rb                  33  Hash                                               1

This code didn’t log anything beacuse the log level is set to info and we called the debug log level method, however we still allocated almost as many objects!.

Why this happens

When we call Rails.logger.debug with a string, say "Hello #{name}, here's some #{generated_data}!", the string is created as an object, including any interpolation that is required which may include methods that are called to retrieve or generate the data which also allocates objects. This happens before the string is passed to the method regardless of the current log level, it is up to the logger when passed this now created string to decide if it is going to log it or not. That’s wasteful!

The solution, passing a block to the logger

Let’s run that debug code again but this time pass a block to the logger:

# Set the log level
> Rails.logger.level = :info
=> :info

# Do the logging
> stats = AllocationStats.trace {
	Rails.logger.debug { "Processed API request to \
      host #{somehost} for user #{user.name} (#{user.id}): status: #{request_status} \
      user_details: #{user.to_json}" }
}
=> [#<AllocationStats::Allocation:0x000000015489ffa8 @object=#<Proc:0x000000015471ea58 /Users/will/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.3.4/lib/active_support/broadcas...

# Get an allocation count, source paths shortened for brevity
> puts stats.allocations.group_by(:sourcefile, :sourceline, :class).to_text
                          sourcefile                          sourceline  class  count
------------------------------------------------------------  ----------  -----  -----
activesupport-7.1.3.4/lib/active_support/broadcast_logger.rb         231  Proc       2
(irb)                                                                 51  Array      1

That’s a lot better, we’re still not logging anything but the allocations are now significantly reduced. Ideally we’d allocate zero objects when logging nothing, but allocating three is a lot better than what we were allocationg before.

This works because when you pass a block to the logger method (debug in this case) the block will only be called if the log level is currently being logged. We had set the log level to info which meant that the block passed to the debug method was never called, and the object allocations to generate the string never happened.

Why should I care?

The examples given here are admittedly trivial (and a little contrived to show the difference), and for an application that doesn’t log much or doesn’t get much traffic then the benefit is going to be tiny, but the effort involved in making the change to block syntax is really small, it’s not detrimental to the code readability either. Small applications can eventually grow into large ones, a handful of requests per second can change to thousands, and if that happens the block syntax can make a difference and you’ll likely be glad you made it a habit.

Finding Postgres rows too large for BTree indexes

will@willj.net (Will Jessop) — Sat, 17 Feb 2024 17:25:00 +0000

We are currently migrating our main application database from AWS Aurora Postgres to Google AlloyDB Postgres (it’s a cost decision) and I’m using Google’s database Migration tool to do the work. At some point last week the migration failed on Google’s side with an error:

index row size 2712 exceeds btree version 4 maximum 2704 for index index_name_column_name

I’ve not seen this before, but it was easy to find the problem rows affecting this specific index/column:

SELECT * FROM table_name WHERE octet_length(column_name) > 2704

Simple enough, out popped 5 rows where the data in column_name was too large. This data was junk so NULLing it out fixed the bad data I knew about, but I wanted to make sure that there weren’t more indexes with this issue, replicating our database to AlloyDB takes time and egress cost from AWS so fixing any known issues before we do this makes a lot of sense. To do this I wrote this Ruby program. The program scans through all column combinations for each table where the combined data could be too large for an index and outputs a warning if the data it finds is too large, allowing you to then run a select to get the specific data that caused the problem.

# This code is MIT licensed

require "pg"

dsn = ARGV[0]
conn = PG.connect(dsn)

# So our postgres array gets converted to a Ruby array
conn.type_map_for_results = PG::BasicTypeMapForResults.new(conn)

# I've seen various figures for the maximum index entry size including 2712
# bytes, 2713 bytes, and block_size/3 (which for the default block size of 8 kB
# is ~2730 bytes), however 2704 bytes is the figure given by the error from
# AlloyDB, so is the one we're using here.
#
# You might need to change this value if you are using a different block size.
max_index_size = 2704

# Create a cache for column size lookups. Column sizes will be in two types:
#
#   1. "static" types, eg. int. These won't change no-matter what data is stored
#      in them so we can just return the size for the type, eg. int => 4 bytes.
#   2. "dynamic" types, eg. text. The octet_length of these will change
#      depending on what is stored in them, and so we can't store a fixed
#      integer byte size and need to scan the database to find the max size. As
#      such rather than storing a fixed size we will store the query to find the
#      max length.
#
# We cache the values as we will be doing the opereration per index, and for the
# same table a column can appear in multiple indexes, but the static/dynamic
# nature of the column will not changed so we can save a little work by cacheing
# the result.
@column_type_length_cache = {}

# Query to find all indexes in the 'public' schema that contain columns that are
# "dynamic", eg. text or jsonb expressions. Indexes that contain "static" data
# types, eg. integer alone aren't interesting as it's unlikely that you will
# exceed the max BTree index entry size with things like UUIDs or integers
# alone.
query_indexes = <<~SQL
  SELECT
    t.relname AS table_name,
    ix.relname AS index_name,
    ARRAY_AGG(a.attname) AS column_name,
    CASE
        WHEN i.indexprs IS NOT NULL THEN pg_get_expr(i.indexprs, i.indrelid)
        ELSE NULL
    END AS index_expression
  FROM
    pg_class t
    JOIN pg_catalog.pg_index i ON t.oid = i.indrelid
    JOIN pg_catalog.pg_class ix ON ix.oid = i.indexrelid
    JOIN pg_attribute a ON a.attrelid = t.oid
      AND a.attnum = ANY (i.indkey)
    JOIN pg_am am ON am.oid = ix.relam
  WHERE
    i.indexrelid IN (
      -- Select any indexes that contain at least one field type we care about,
      -- excluding field types that we know, when used alone, will not be larger
      -- than an index entry, unless you are doing something especially fruity.
      SELECT
        distinct(c.oid)
      FROM
      pg_catalog.pg_class c
        JOIN pg_catalog.pg_index i ON c.oid = i.indexrelid
        JOIN pg_catalog.pg_namespace ns ON c.relnamespace = ns.oid
        JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid
      WHERE
        relkind = 'i'
        AND indisprimary = FALSE
        AND ns.nspname = 'public'
        AND a.atttypid NOT IN (
          16,   -- boolean
          20,   -- bigint
          21,   -- smallint
          23,   -- integer
          1082, -- date
          1114, -- timestamp without timezone
          2950  -- uuid
        )
      )
    AND am.amname = 'btree' -- I am only concerned about btree types as that was the error we are dealing with
  GROUP BY table_name, index_name, index_expression
SQL

indexes_result = conn.exec(query_indexes).map(&:values)

def column_octet_length_query(conn, column)
  "max(octet_length('#{conn.quote_ident(column)}'))"
end

def expression_octet_length_query(expression)
  "max(octet_length(#{expression}))"
end

def fetch_column_type_length(conn, table_name, column_name)
  cache_key = "#{table_name}_#{column_name}"

  # We can have multiple indexes on the same table that reference the same
  # column so let's cache the lookups for a column
  @column_type_length_cache[cache_key] ||= begin
    query = <<~SQL
      SELECT
        a.atttypid,
        t.typlen
      FROM pg_attribute a
      JOIN pg_type t ON a.atttypid = t.oid
      JOIN pg_class c ON a.attrelid = c.oid
      WHERE c.relname = $1 AND a.attname = $2
    SQL

    result = conn.exec_params(query, [table_name, column_name])
    typlen = result.first['typlen'].to_i

    typlen.positive? ? typlen : column_octet_length_query(conn, column_name)
  end
end

indexes_result.each do |table_name, index_name, column_names, index_expression|
  index_expression ||= ""
  begin
    regular_column_size_queries = column_names.map { |cn| fetch_column_type_length(conn, table_name, cn) }
    expression_size_queries = index_expression.split(", ").map { |e| expression_octet_length_query(e) }
    all_size_queries_string = (regular_column_size_queries + expression_size_queries).join(' + ')
    compound_size_query = "SELECT #{all_size_queries_string} AS max_size FROM #{conn.quote_ident(table_name)}"

    max_size = conn.exec(compound_size_query).first["max_size"].to_i

    if max_size > max_index_size
      puts "\033[31mWARNING: #{table_name} (#{index_name}): #{column_names.join(', ')}: Maximum Size: #{max_size} bytes exceeds the max index size\033[0m"
    end
  rescue PG::InsufficientPrivilege
    # We have some tables that I can't access with the user this runs as, but I don't care about them.
    puts "Permission denied for table #{table_name}"
  end
end

conn.close

You run it using:

$ ruby script_name.rb your_postgres_dsn

It may take quite a lot of time to run, it effectively sequential scans all of your tables where there is a row that could be larger than the allowed row size for a BTree index, and as such you may want to run it against a replica and/or out of hours to avoid production impact, but other than that it’s safe. For us it did find about a hundred rows in another table that were too large for the index allowing us to fix these before re-starting the migration process.

WARNING: some_other_table (some_index_name_idx): user_id, email, phone, approved: Maximum Size: 28586 bytes exceeds the max index size

Yeah, some people had suspiciously long emails. We’re adding size constraints on the columns that were missing them now. If you get a warning like this then you can craft a query to find the problem rows, eg.

SELECT * FROM some_other_table WHERE octet_length(email) + octet_length(phone) > 2704

Some highlights:

The program will sum the octet_length of all fields in an index, so if you have an index with two columns in it and it’s only the sum of the octet_lengths that will push the row over the BTree row size this program will still find the issue.
The program prunes indexes and tables where there are no indexes that could be large enough to be over the limit, effectively cutting the number of table scans it needs to do. If you’ve got an index with 677 int fields in it then the assumption this relies on will break down, but I’ve never seen that, and never want to.

Some caveats:

Table names and column names are escaped, but don’t run this if you have untrusted third-parties who can control any expressions you have in your database. For instance if you’re indexing into JSON with a user supplied key, eg. (comments ->> 'user_supplied_value'::text). You’re probably not doing this, I’ve never seen it, but just be aware.
This program finds rows that violate the size constraints, it doesn’t find situations where a row could violate the size limit if you were to put in the maximum amount of data allowed by the types.
I believe this program works if you are indexing into multiple JSONB fields in one index, but I’ve not tested it with those as we only have indexes into a single JSONB field, eg:
```
"index_name" btree (some_id, (comments ->> 'some_key'::text))
```

Please let me know if you run this and have success and/or issues, or if you come up with a more efficient way of doing the same thing.

Measuring IO timings when using EXPLAIN ANALYSE in Postgres

will@willj.net (Will Jessop) — Tue, 19 Sep 2023 23:21:00 +0100

In an earlier post about using BUFFERS with EXPLAIN ANALYSE I mentioned that there were a couple of options to EXPLAIN ANALYSE that were not widely known, but I only talked about one, the BUFFERS option. In this article I will dig into that second option; IO timing tracking.

IO timing tracking

The second tweak to the default EXPLAIN usage is using IO timing tracking. You can set this per session:

set track_io_timing = on;

As usual the Postgres docs do a thorough job explaining what this setting does:

Enables timing of database I/O calls. This parameter is off by default, as it will repeatedly query the operating system for the current time, which may cause significant overhead on some platforms. You can use the pg_test_timing tool to measure the overhead of timing on your system. I/O timing information is displayed in pg_stat_database, in the output of EXPLAIN when the BUFFERS option is used, in the output of VACUUM when the VERBOSE option is used, by autovacuum for auto-vacuums and auto-analyzes, when log_autovacuum_min_duration is set and by pg_stat_statements. Only superusers and users with the appropriate SET privilege can change this setting.

However simply put for our use case if you turn this setting on then when you run an EXPLAIN (ANALYSE, BUFFERS) you will get the time in milliseconds that the IO operations in each part of your query took to perform, for example using one of the queries from the previous post:

EXPLAIN (ANALYSE, BUFFERS)
SELECT
	events.*
FROM
	events
	INNER JOIN activity_logs ON events.activity_logs_id = activity_logs.id
WHERE
	activity_logs.user_id = 83463
	AND events.date in ('2022-12-04')
	AND (events.event_start IS NOT NULL
		AND events.event_end IS NOT NULL
		AND events.duration != 0
		OR events.exception_id IS NOT NULL)
ORDER BY
	date ASC,
	events.event_start DESC NULLS LAST;

	Sort  (cost=10157.36..10157.37 rows=1 width=166) (actual time=2290.762..2290.763 rows=0 loops=1)
  Sort Key: events.event_start DESC NULLS LAST
  Sort Method: quicksort  Memory: 25kB
  Buffers: shared hit=1166 read=3117
  I/O Timings: read=3221.979
  ->  Nested Loop  (cost=1.13..10157.35 rows=1 width=166) (actual time=2290.757..2290.758 rows=0 loops=1)
        Buffers: shared hit=1166 read=3117
        I/O Timings: read=3221.979
        ->  Index Scan using index_activity_logs_on_user_id_and_event_start_and_event_end on activity_logs  (cost=0.56..325.89 rows=82 width=4) (actual time=0.018..14.964 rows=315 loops=1)
              Index Cond: (user_id = 83463)
              Buffers: shared hit=33 read=116
              I/O Timings: read=137.986
        ->  Index Scan using index_events_on_activity_logs_id on events  (cost=0.57..119.89 rows=1 width=166) (actual time=7.221..7.221 rows=0 loops=315)
              Index Cond: (activity_logs_id = activity_logs.id)
              Filter: ((((event_start IS NOT NULL) AND (event_end IS NOT NULL) AND (duration <> '0'::double precision)) OR (exception_id IS NOT NULL)) AND (date = '2022-12-04'::date))
              Rows Removed by Filter: 11
              Buffers: shared hit=1133 read=3001
              I/O Timings: read=3083.993
Planning:
  Buffers: shared hit=146
Planning Time: 0.427 ms
Execution Time: 2290.813 ms

As with the BUFFERS option the timings are cumulative at each node in the hierarchy of the explain output.

This isn’t something that I’ve personally found particularly useful, I suspect it might be useful when diagnosing slow fetches if you’re storing data on different backend data stores across tables, but it’s interesting to see.

Buffer analysis when using EXPLAIN ANALYSE in Postgres

will@willj.net (Will Jessop) — Fri, 15 Sep 2023 23:49:00 +0100

Many people are familiar with running EXPLAIN and EXPLAIN ANALYSE to diagnose Postgres performance issues, but there are a couple of useful options available when running these reports that aren’t always commonly known that you might find useful. The first of these is the BUFFERS option.

In this article I will explain how to measure buffer usage in Postgres using the BUFFERS option, and how I used it to help verify that a query optimization to massively reduce the IO demands of a problem query had worked.

The BUFFERS option

The BUFFERS option can be used when running a simple EXPLAIN, but really comes into it’s own when running an EXPLAIN ANALYSE. In summary BUFFERS will give you data on the buffer usage of a query. The postgres docs give a good description of what the data you get tells you:

Include information on buffer usage. Specifically, include the number of shared blocks hit, read, dirtied, and written, the number of local blocks hit, read, dirtied, and written, the number of temp blocks read and written, and the time spent reading and writing data file blocks and temporary file blocks (in milliseconds) if track_io_timing is enabled. A hit means that a read was avoided because the block was found already in cache when needed. Shared blocks contain data from regular tables and indexes; local blocks contain data from temporary tables and indexes; while temporary blocks contain short-term working data used in sorts, hashes, Materialize plan nodes, and similar cases. The number of blocks dirtied indicates the number of previously unmodified blocks that were changed by this query; while the number of blocks written indicates the number of previously-dirtied blocks evicted from cache by this backend during query processing. The number of blocks shown for an upper-level node includes those used by all its child nodes. In text format, only non-zero values are printed. It defaults to FALSE.

https://www.postgresql.org/docs/current/sql-explain.html

Cutting down buffer usage can be a great way to decrease IO resource usage for a query.

An example

Let’s look at an example query. This is similar to one that I optimized for a client and the exact SQL/data structure isn’t important. Note the EXPLAIN (ANALYSE, BUFFERS) line at the start of the query:

EXPLAIN (ANALYSE, BUFFERS)
SELECT
	events.*
FROM
	events
	INNER JOIN activity_logs ON events.activity_logs_id = activity_logs.id
WHERE
	activity_logs.user_id = 83463
	AND events.date in ('2022-12-04')
	AND (events.event_start IS NOT NULL
		AND events.event_end IS NOT NULL
		AND events.duration != 0
		OR events.exception_id IS NOT NULL)
ORDER BY
	date ASC,
	events.event_start DESC NULLS LAST;

Sort  (cost=10157.36..10157.37 rows=1 width=166) (actual time=2290.762..2290.763 rows=0 loops=1)
  Sort Key: events.event_start DESC NULLS LAST
  Sort Method: quicksort  Memory: 25kB
  Buffers: shared hit=1166 read=3117 -- (0)
  ->  Nested Loop  (cost=1.13..10157.35 rows=1 width=166) (actual time=2290.757..2290.758 rows=0 loops=1)
        Buffers: shared hit=1166 read=3117  -- (1)
        ->  Index Scan using index_activity_logs_on_user_id_and_event_start_and_event_end on activity_logs  (cost=0.56..325.89 rows=82 width=4) (actual time=0.018..14.964 rows=315 loops=1)
              Index Cond: (user_id = 83463)
              Buffers: shared hit=33 read=116 -- (2)
        ->  Index Scan using index_events_on_activity_logs_id on events  (cost=0.57..119.89 rows=1 width=166) (actual time=7.221..7.221 rows=0 loops=315)
              Index Cond: (activity_logs_id = activity_logs.id)
              Filter: ((((event_start IS NOT NULL) AND (event_end IS NOT NULL) AND (duration <> '0'::double precision)) OR (exception_id IS NOT NULL)) AND (date = '2022-12-04'::date))
              Rows Removed by Filter: 11
              Buffers: shared hit=1133 read=3001 -- (3)
Planning:
  Buffers: shared hit=146
Planning Time: 0.427 ms
Execution Time: 2290.813 ms

There’s a lot to take in here, but the difference you’re looking for compared to a regular EXPLAIN output are the lines that start with Buffers: … I’ve marked them all in the explain output above with the labels (0), (1), (2) and (3). They look like this:

Buffers: shared hit=1166 read=3117

This line is telling us that 1166 blocks were satisfied by the shared cache, and 3117 blocks were read from disk.

What is a block?

Blocks are chunks of data of a certain size. The size is configurable, but is typically left at the default of 8192 bytes. You can check this on your server:

=> show block_size;
 block_size
------------
 8192
(1 row)

Taking this block size value and going back to our Buffers report:

Buffers: shared hit=1166 read=3117

We can see here that 1166 blocks (9,551,872 bytes) were provided by the shared cache, but 3117 blocks (25,534,464 bytes) had to be read from disk. This was the data reported from the location I marked with (0), but there are other locations too, marked with (1), (2) and (3). The EXPLAIN command breaks down buffer usage by section, and the report is cumulative, so the buffer usage reported in sections (2) and (3) when added together equal the buffer usage reported at the parent node in section (1), and so on.

The breakdown by section means that you can see where the buffer usage is highest and target your optimization in that area. For instance in the example query you can see that more than 95% of the buffer usage happens in section (3), the Index Scan using index_events_on_activity_logs_id and so this area would be a good place to start investigating optimizations.

Why does it matter if we go to disk instead of the shared cache in memory?

Disk (SSD) read speeds are substantially slower than reads from memory, in the region of 450 MB/s for an SSD compared to around 13,000 MB/s for memory (varying depending on the specific hardware). The "Latency numbers every programmer should know" table, originally by Peter Norvig shows the difference well.

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs
Read 1 MB sequentially from memory ..... 250,000 ns  = 250 µs
Round trip within same datacenter ...... 500,000 ns  = 0.5 ms
Read 1 MB sequentially from SSD ...... 1,000,000 ns  =   1 ms (Assuming ~1GB/sec SSD)
Disk seek ........................... 10,000,000 ns  =  10 ms
Read 1 MB sequentially from disk .... 20,000,000 ns  =  20 ms
Send packet CA->Netherlands->CA .... 150,000,000 ns  = 150 ms

Before we get into what this buffers data in the explain output is useful for, let’s run the explain again with the same parameters to see what happens:

EXPLAIN (ANALYSE, BUFFERS)
SELECT
	events.*
FROM
	events
	INNER JOIN activity_logs ON events.activity_logs_id = activity_logs.id
WHERE
	activity_logs.user_id = 83463
	AND events.date in ('2022-12-04')
	AND (events.event_start IS NOT NULL
		AND events.event_end IS NOT NULL
		AND events.duration != 0
		OR events.exception_id IS NOT NULL)
ORDER BY
	date ASC,
	events.event_start DESC NULLS LAST;

Sort  (cost=10157.36..10157.37 rows=1 width=166) (actual time=6.150..6.151 rows=0 loops=1)
Sort Key: events.event_start DESC NULLS LAST
Sort Method: quicksort  Memory: 25kB
Buffers: shared hit=3577
->  Nested Loop  (cost=1.13..10157.35 rows=1 width=166) (actual time=6.145..6.145 rows=0 loops=1)
			Buffers: shared hit=3577
			->  Index Scan using index_activity_logs_on_user_id_and_event_start_and_event_end on activity_logs  (cost=0.56..325.89 rows=82 width=4) (actual time=0.020..0.350 rows=315 loops=1)
						Index Cond: (user_id = 83463)
						Buffers: shared hit=45
			->  Index Scan using index_events_on_activity_logs_id on events  (cost=0.57..119.89 rows=1 width=166) (actual time=0.018..0.018 rows=0 loops=315)
						Index Cond: (activity_logs_id = activity_logs.id)
						Filter: ((((event_start IS NOT NULL) AND (event_end IS NOT NULL) AND (duration <> '0'::double precision)) OR (exception_id IS NOT NULL)) AND (date = '2022-12-04'::date))
						Rows Removed by Filter: 11
						Buffers: shared hit=3532
Planning:
Buffers: shared hit=148
Planning Time: 0.445 ms
Execution Time: 6.198 ms

What you can see here from the buffer data is that Postgres no-longer has to go to disk. The data that was read in the previous query is now in the shared cache so we’re fetching all our data from memory, there is no read=nnnn reported:

Buffers: shared hit=3577

This results in a much faster query time, the original query took 2291ms and the cached one 6ms ⚡️. Unfortunately even though this query was much faster the second time round we don’t always have the luxury of running queries that are cached so we often need to optimize for the cold cache situation, plus if we can reduce the buffers usage of the query we can make it even faster and less resource intensive still, plus if we can trim the data pulled from the shared cache down too that will also help with performance.

So, what can we use this information for? The query I’m using in the example is similar to one I optimized for a client that was the highest IO usage of all queries on their production server at nearly 16% if all IO used.

When we’re trying to improve the performance of this query we can make it faster, and that would be great by itself and is often the main goal of a performance optimization, but we really want to actually verify that any changes we make have reduced the IO which is the larger problem here; we really want to stick to the performance optimization mantra of “measure, change, measure”. With the buffers data we can see what difference we’re making, if any, to the performance of the query as we change it. Here’s the same query on a cold cache but with an optimization:

EXPLAIN (ANALYSE, BUFFERS)
SELECT
	events.*
FROM
	events
	INNER JOIN activity_logs ON events.activity_logs_id = activity_logs.id
WHERE
	activity_logs.user_id = 348927
	AND events.date in ('2022-12-04')
	AND activity_logs.event_start > '2022-12-10'
	AND (events.event_start IS NOT NULL
		AND events.event_end IS NOT NULL
		AND events.duration != 0
		OR events.exception_id IS NOT NULL)
ORDER BY
	date ASC,
	events.event_start DESC NULLS LAST;

Sort  (cost=1491.66..1491.67 rows=1 width=166) (actual time=1.855..1.856 rows=0 loops=1)
  Sort Key: events.event_start DESC NULLS LAST
  Sort Method: quicksort  Memory: 25kB
  Buffers: shared hit=2 read=2
  ->  Nested Loop  (cost=1.13..1491.65 rows=1 width=166) (actual time=1.850..1.851 rows=0 loops=1)
        Buffers: shared hit=2 read=2
        ->  Index Scan using index_activity_logs_on_user_id_and_event_start_and_event_end on activity_logs  (cost=0.56..51.70 rows=12 width=4) (actual time=1.850..1.850 rows=0 loops=1)
              Index Cond: ((user_id = 348927) AND (event_start > '2022-12-10'::date))
              Buffers: shared hit=2 read=2
        ->  Index Scan using index_events_on_activity_logs_id on events  (cost=0.57..119.99 rows=1 width=166) (never executed)
              Index Cond: (activity_logs_id = activity_logs.id)
              Filter: ((((event_start IS NOT NULL) AND (event_end IS NOT NULL) AND (duration <> '0'::double precision)) OR (exception_id IS NOT NULL)) AND (date = '2022-12-04'::date))
Planning:
  Buffers: shared hit=148
Planning Time: 0.427 ms
Execution Time: 1.897 ms

The specific optimization isn’t relevant here (it’s the addition of the condition AND activity_logs.event_start > '2022-12-10' which I highlighted), I’ll talk about that in another article, but what you can see if you compare to the earlier uncached query plan is that the “Execution Time” is far smaller at 1.9ms, down from 2290ms, but also the amount of data fetched from disk (the “read” buffers) is far smaller too at 2 blocks, or 16,384 bytes, down from 25,534,464 bytes before.

Buffers: shared hit=1166 read=3117

vs:

Buffers: shared hit=2 read=2

That’s a reduction of over 24 MB, a huge IO amount of IO to trim from the query! With this data we have a pretty good idea that we’ve made a significant difference to the IO problem.

Conclusion

At scale IO can be a significant performance bottleneck for databases and optimizing problem queries that cause large amounts of IO can yield significant benefits. Adding the BUFFERS option when using EXPLAIN ANALYSE can give you a great way to identify problem areas of a query, and to verify when you’re done that you’ve actually solved the issue.

If you’ve found this interesting you might be interested in second part of this article, Measuring IO timings when using EXPLAIN ANALYSE in Postgres.

Converting a wordpress.com dump to Hugo, including images

will@willj.net (Will Jessop) — Tue, 12 Sep 2023 13:30:00 +0100

There are a few “official” options for migrating wordpress content to Hugo, but they are all either cumbersome (requiring installing wordpress in docker) or don’t handle images. I didn’t want to spend too much time on this and definitely wanted images to be converted so wrote a quick program to handle the conversion for me.

In the end I created a somewhat rough, but functional program to do the job and it was successful.

The requirements:

Extracts tags where there are any on the original post and adds them to the front Hugo matter
Maintains the original post date
Converts the HTML to Markdown

Maintains the directory structure, with images copied to the directory that the post’s index.md is in:

basedir/
  content/
    posts/
      Some-post-title/
        index.md
        images/
          post-image-1.jpg
          post-image-2.png
     A-different-post-title/
      index.md
      images/
        post-image-for-this-other-arcicle.jpg

There are some caveats:

Posts require some fixup

There’s a little bit of tweaking that I’ve had to do to Hugo front matter or the posts, but the program got me mostly all the way. These things could likely be fixed in the code and it would be worth doing so if I was going to run the program more than once, but I’m not so it was faster to fix things up manually.
Doesn’t write the summary

I had to write this manually, but I was only dealing with 10s of posts. If you want to use this then you might want to add something to generate a summary.
Doesn’t always get the image URLs right in the markdown (but it’s easily fixable)

The code left a few images URLs in the posts pointing to the old wordpress site. It still copied over the images so it was a really simple find and replace to fix these up.
Doesn’t do any error checking, just panics

It didn’t error so I didn’t put in the time to handle errors as it was a one-off program, it worked for me.
Skips image only posts

This was delberate, but you might want to add that feature.

Anyway, the code got me 90% of the way to where I wanted the posts to get so worked well for what I wanted it to do and I’d call it a success. I’ve put the code up on Github, I likely won’t need to run this code again, but if it’s useful to you please feel free to submit PRs.

Reverting Firefox's recent URL bar suggestion order change

will@willj.net (Will Jessop) — Mon, 11 Sep 2023 17:27:00 +0100

A recent update to Firefox changed the order of suggestions when you started typing in the URL bar (or “address bar”). It used to be that you’d get recently visited URLs first but the change caused search suggestions to appear first. I almost always navigate to sites I know by typing the first few letters of the domain name and almost never used the search suggestions so this change cause consternation.

The URL bar suggestions after Firefox "Improved" the experience in a recent update and changed the suggestions order

Fearing for the longevity of my down arrow key caused the wear of all the extra pressing to get down to the recently visited URLs section I searched for a solution. It turns out that you can change the order of the suggestions in the config. To do so:

Open a new Firefox tab
Type about:config
Accept any warnings (yolo)
Type browser.urlbar.showSearchSuggestionsFirst into the search field
Toggle the value

You shouldn’t need to restart Firefox for the change to take effect, and should now have your URL history back where it was before the search suggestions.

URL bar suggestion order, now fixed

Fitting two hard drives and an SSD in a Dell OptiPlex 7010 SFF

will@willj.net (Will Jessop) — Sun, 28 May 2023 00:00:00 +0000

Update!

It turns out that this is a Dell OptiPlex 7010 SFF, not a USFF (as advertised by the eBay seller)!

A Dell OptiPlex 7010 SFF

I recently bought a Dell OptiPlex 7010 Small Form Factor PC from eBay. I was Intending on squeezing two of the spare Seagate Barracuda 4TB 3.5" hard drives I have in it with an SSD boot disk and installing TrueNAS to use it as a file store. It’s a small, quiet machine and should be ideal, but when I was doing my research into what I should buy I think I got the SFF (Ultra Small Form Factor) confused with another Dell OptiPlex machine and it turns out that the SFF model I got can only officially fit a single 2.5" SSD in it, and no 3.5" drives at all. Whoops.

Opening up the machine it turns out that there was actually a 3.5" drive bay and the eBay seller used a tiny amount of hot glue to just glue a 2.5" SSD into the bay rather than using an adapter. Quality. This didn’t help much though, there was still no way of fitting the number of drives I wanted in the machine as it came. Anyway, after removing the SSD, the official Dell drive mounts and the slim CDRW drive I took a look at the situation and figured I could probably rescue this with a bit of 3d printing. Any day that requires you to solve a problem with a bit of 3d printing is generally a good day.

Caveat!

This works for my Seagate Barracuda 4TB drives. They’re slightly thinner than some of the other drives I’ve got, if yours are thicker then it might work for yoiu or it might not.

Risers of the Machine

View inside a Dell OptiPlex 7010 SFF, there isn't much room

The motherboard. Red circles are candidates for risers, the turquoise square is where the CPU heat sink and fan are

Once the existing mounts had been removed I could see that there were four screws holding the motherboard in place, and an existing riser, all of which could be re-purposed as places to screw a riser which would allow me to mount drives suspended over the RAM. See the image where I’ve marked the locations of these holes with a red circle. The turquoise square is about where the CPU heatsink and fan are, but the heatsink is low profile and draws the heat towards the back of the machine closer to the IO ports where the fan is.

The Curse of the Imperial Menace!

Metric is the best and only measurement system anyone should use, but unfortunately PC manufacturers seem to have settled on imperial for at least the bolt hole sizes I need to use which are 6-32 UNC. The risers I need to incorporate into this design are fairly long too at nearly 40mm. That’s about M3.5 * 40mm in metric, but I couldn’t find any M3.5 risers of the right height, and I can buy 6-32 UNC bolts on eBay easily enough so I bought some of those. You can probably use M3.5 as a substitute.

To Fusion360!

These days my 3d designing tool of choice is Fusion 360 so I opened it up and set to work. Much cursing, and a fair few scrapped prototypes later I came up with this:

3d design isn't my day job

It’s definitely not going to win any awards for beautiful design, and it’s got some problems, but it works. I could certainly have improved the design further, but I was running out of patience with 3d printing prototypes I had to throw away, and time on my Fusion 360 trial (damn you Autodesk!). Anyway, it’s ugly, it’s flawed, but it works.

The finished mount in action

Here’s the first part of the install showing the “base” of the mount. The SSD is slung underneath with M3 bolts * 12mm, because that was what I had. I should have made the recess for the bolt heads a bit deeper so the bolt heads were further away from the underside of the bottom hard drive, but though it’s tight it doesn’t touch so I’m leaving it.

There is about 5mm clearance between the bottom of the mount and the RAM, and enough clearance between the SSD and the CPU cooler for wires to pass comfortably. The SSD could be mounted about 5mm higher, closer to the bottom of the first hard drive, but I only had 12mm bolts in M3.5 and so had to print spacers.

There are captive nuts on the top-left of this image underneath the base, nearest the front of the case. These were exactly the right size for the nuts I was using, which meant that the nuts stayed in right up until I used any force at all on them like when I was trying to screw a bolt into them. This was infuriating so I epoxied the nuts in place. If I was to do another version of this I’d make the nut traps tighter so the nuts could be safely held in place with a press fit.

The two bolts you can see with two nuts on them used as spacers, these are a flaw in the design. In order to prevent the 50mm bolts I was using from bottoming out I could have used shorter bolts and printed recesses, but I didn’t want to buy more imperial bolts I was never going to use again. I could have printed spacers, but the nuts worked and they were there. In the end I had to remove one of the nuts from the bottom-left most bolt so it didn’t foul the hard drive, but it didn’t bottom out even then so it worked out.

Here you can see the 3.5" hard drives in place. There is a gap between the two drives to allow for air flow.

Here you can see how tight it is near the case fan, the tolerances in this design are pretty tight. I couldn’t shift the hard drives further away from the front of the case/the case fan because the latch in the removable panel on the case prevents them from moving backwards in the case.

The front of the case looking in through the old CDRW slot. It doesn’t look like it but there is a mm or two clearance between the top of the top hard drive and the case when the lid is on.

The same view but with the lit fitted.

Parts list

If you want to make this mount exactly as I did then you will need these parts, or metric equivalent. Just bear in mind that it could be improved, so if you’ve got access to a 3d design program and the willingness it could be better. Let me know if you release an improved model and I’ll link it here.

4 x 6-32 UNC * 50mm bolts
5 x 6-32 UNC nuts (or 2 x if you print out spacers for the risers, see first image and description in “The finished mount in action”)
4 x M3 * 12mm flat head bolts for the SSD
10 x standard HD mounting screws (8 for the HDs, 2 for the base)
SATA cables
A SATA power splitter

The SSD bolts looks like this:

Printing instructions

The design is printed in parts. Print one each of all parts you see in the image above, except you’ll need four of the risers (the long thing sticking down on the bottom left of the image) and four of the SSD mounting spacers if you’re using the same m3 bolts as me. Adjust or omit these if you have different length M3 bolts.

Any 3d printer should work, but all this was printed on a Prusa i3 MK3S, if you’re looking for your own 3d printer then the Prusa i3 MK4 would be a great option.

Things I’d do differently

The HD Screws aren’t really long enough for going through a nut and two layers of 3d print, so where I’ve used them to connect the base of the mount the print is a little thin to allow for all the thread to pass through and bite into the thread in the nut. I could have used different screws/bolts or could likely have done this without any screws at all with an interlocking design, but this works and I don’t want to re-design it.
Make the SSD mounting bolt head recesses very slightly deeper.
Made the nut traps tighter, the nuts were loose which made installation a pain.
Re-designed to use less 3d printing and to take advantage of the HDs inherant stiffness. The HDs are made of metal, and I didn’t need anywhere near as much 3d printed structure to keep them in place, I could have relied on the inherant stiffness of the drives more.
Made the risers tighter so they didn’t fall off the long bolts on installation.
Made the base installable first then the HDs bolt onto that, as it stands there’s a fiddly installation process and you have to remove pretty much the whole thing to change a drive.
The riser closest to the power button interferes with the HD if you offset it from the base the correct amount, I’d re-design that (see first picture and description under “The finished mount in action”).

Things to test later

I’ve fully assembled the mount, powered it on, installed TrueNAS and started using it as a file store, so it works. A couple of things I’d like to test at some point but haven’t got round to:

Heat

Adding more things into the case means less airflow. I think there’s still enough (below the mount, round the side near the case fan, between the hard drives) that it’s not going to overheat, but I’ll be monitoring the heat using the TrueNAS UI.

Power usage

The PSU is 64W I think, but seems to be fine driving two mirrored 5400 RPM drives and an SSD. I left my power monitor on my boat though so will test when I get the chance.

Downloads

The Fusion 360 design file:

Dell Optiplex 7010 SFF HD Mount.f3d

STL:

Base
Front top
Back top
Riser (print x 4)
SSD Offset (print x 4)

Conclusion

I’ve said it before, but it’s worth repeating. This mount works just fine, the drives are well fixed in place, the cables stay in, the case lid closes. However, designing stuff isn’t my day job and this design has a number of issues that if someone with more time and who is better at 3d design wanted to build on then it could be greatly improved. If I were to print a new version of this I’d probably alter the design quite a lot.

All that said, if you want to print this, and have the parts, then I don’t see any reason why it shouldn’t also work for you, it’s working for me.

Let me know if you use this or improve it! Contact details on the homepage.

Running a specific knapsack test set locally

will@willj.net (Will Jessop) — Wed, 26 Apr 2023 15:00:00 +0000

Sometimes you have a test order issue CI that you want to debug locally, and to do that you need to only run the tests that knapsack runs in the relevant shard, in the right order. Here’s how to do that!

Grab CI_NODE_TOTAL and CI_NODE_INDEX from the test output, it’s likely near the top:

Grab the seed value, it’s probably going to be near the bottom of the output:

Run the command with the interpolated values you got above

 $ RAILS_ENV=test CI=true CI_NODE_TOTAL=16 CI_NODE_INDEX=4 bundle exec rake "knapsack:rspec[--seed 43092 --format documentation]"

You might also want to add --fail-fast to the options to cut short the test suite when you hit your error:

 $ RAILS_ENV=test CI=true CI_NODE_TOTAL=16 CI_NODE_INDEX=4 bundle exec rake "knapsack:rspec[--seed 43092 --format documentation --fail-fast]"

You can even rspec bisect across this set of tests:

 $ OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES RAILS_ENV=test CI=true CI_NODE_TOTAL=16 CI_NODE_INDEX=13 bundle exec rake "knapsack:rspec[--seed 16837 --bisect]"

Here I added OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES as there is bug/issue with Ruby on OS X, you should be able to remove that environment variable on Linux.

Will's YouTube sailing channel recommendations

will@willj.net (Will Jessop) — Thu, 15 Sep 2022 00:00:00 +0000

I started watching YouTube sailing videos in about 2014 when I discovered Sailing la Vagabonde. I’d never really considered that sailing was something I could be interested in, I didn’t know anything about it at all. I remember being on holiday with my wife in Greece a few times and seeing sailing yachts moored up in the harbours, and I think there was a curiosity about the boats, and the people eating what I probably assumed were fancy lunches on them. I must have seen yachts in the sea, or in harbours or marinas, but I don’t really remember. They were just part of the view, not something to be interacted with. A different world. Billionaire bosses and boat shoes.

Anyway, I ignored sailing, and sailing ignored me, until one day a cooking show I watched linked to la Vagabonde. I was suddenly given a peek behind the curtain, a view into a previously unknown world and I was pretty curious, so I kept watching. These people seemed… normal, approachable. I started to consume the la Vagabonde back catalogue, and was curious if there were more people making these videos. A quick google later and I’d found Delos, and after a couple of tries (their early videos were hard to get through) I started working my way through their back catalogue too, and so started my interest in sailing that has lasted 8 years, and has now seen us buy our own boat.

Over those years I’ve gathered a number of YouTube subscriptions and figured I’d share them here, save people having to dig through the less great content out there to find some gems.

SV Delos - Try to push through the earlier videos, this one of the most widely respected sailing channels out there, but the early videos don’t do them justice.
Sail Life - First several years are a re-fit (which was very well filmed and I loved), but they’re sailing now.
Ran Sailing
Sailing Uma
Sailing Millennial Falcon
Erik Aanderaa
Sailing Yacht Florence
Sailing Project Atticus
Christian Williams
Sailing cos I love it
How to Sail Oceans
Sailing Sweet Ruca
Calico Skies Sailing
Magic Carpet
Parlay Revival
Sailing La Vagabonde - I don’t watch them as much anymore, they do a lot less stuff I’m interested in.
Gone with the Wynns
Sailing Soulianis
Sam Holmes Sailing
MJ Sailing - Currently doing a rather dry, no pun untended, boat build, but see old videos.
Sailing Kittiwake - They seem to have stopped posting now
Sailing Ruby Rose - Currently rather dry Catamaran videos, but the earlier stuff was good.

Some of these are pure sailing, some more “lifestyle” sailing. I like each for different reasons. Some I tolerate and only watch when I’ve run out of other stuff, but see what you like. Bear in mind not everything these people do is a good idea, but it’s a pretty easy way of immersing yourself in the scene without any outlay.

Full-height Github "checks" box in Firefox

will@willj.net (Will Jessop) — Tue, 16 Aug 2022 12:00:00 +0000

The scrolling checks box on Github really irritates me. Follow these instructions to make it full height on Firefox.

1. Take this CSS:

.branch-action-item.open > .merge-status-list-wrapper > .merge-status-list, .branch-action-item.open > .merge-status-list {
  max-height: fit-content !important;
}

2. Tell Firefox to use the new CSS

Follow the guide here for adding it to the custom CSS. And done, full height checks box!

Effective Github Notifications

will@willj.net (Will Jessop) — Tue, 16 Aug 2022 00:00:00 +0000

Github notifications can be overwhelming, but with effective email filtering they can be turned from an encumbrance to being highly valuable. Here’s some filters to get you started (in Mail.app on a mac) to make them more useful.

Luckily Github provides us with the tools to make this filtering relatively easy. All email notifications sent from Github are automatically CCed to a Github email address that describes what they are, and this is the hook we use to create our filters.

All email addresses here are @github.com, even though in some examples they are hidden, and these rules can be added to the GMail web UI too if needed.

Flag high priority notifications

Mark assignments, review requests, mentions or author actions as high priority and flag them. This will put them at the top of your inbox:

Change the colour of medium priority items so they stand out

Comments, your activity, and when a team mentions you are in is mentioned.

Hide low priority notifications

Things you might want to go and check once in a while but aren’t directly involved in we can put in a folder so they don’t demand attention throughout the day.

What you should end up with after doing this is an inbox that shows you all the things that need your attention or you’d be interested, but hides the everyday torrent of Github spam in a folder that you can check if you want later.

Simple Sourdough Bread

will@willj.net (Will Jessop) — Wed, 13 Jan 2021 02:53:53 +0000

I make sourdough bread usually at least once a week and sometimes post pictures of the result to Twitter and get asked pretty frequently about my technique. There’s a lot written about sourdough bread on the Internet these days, and plenty of YouTube videos on the subject too, but my technique is slightly different to many of them in that it is simple and takes the smallest amount of time possible while still getting great results. I don’t have the time to dedicate an entire week to the creation of a loaf of bread, I haven’t the patience to softly sing to the dough five times a day, and will not be spending several hours gently cupping the proving basket whispering encouraging thoughts long into the night.

I have other stuff to do. So, my technique reqires the minimum amount of time. In total a loaf using my technique takes about:

Day 1: about 10 minutes total.
Day 2: 40 minutes to bake + oven pre-heat time.

Even this seems like a long time, but bear in mind that a lot of this time is spent waiting, for instance the baking and oven warming which is time you can spend doing other important stuff like making lunch, or endlessly doom-scrolling through Twitter.

Though it’s fashionable to add your entire life story as the preamble to any recipe posted on the Web these days, let’s draw a line under this introduction and move on to the method. If you’re pushed for time to make bread then you’re not going to want to read about how a particularly nutty focaccia I ate when I was four set me on the path to posting marginally iterative recipes that I’ve made my own “with a twist” many years later.

The starter

You’re going to need a starter. If you’ve not got one already, head over to my article on making one, otherwise, read on!

The Method

You will need:

250g strong white bread flour
7g salt (about 1 teaspoon for table salt)
115g water (I just experimented and got to this point, vary the water for the flour you use)
1 or 2 teaspoons honey (optional)
150g starter

I recommend starting off with white bread, it seems easier to make, but for wholemeal replace the flour above with:

200g strong white bread flour
50g wholemeal flour
30g mixed grains and seeds

Day 1

Stir the starter before use in case it has separated at all. If any water has accumulated, stir that in too. Put all the ingredients into a bowl and mix. Once mixed you need to knead it. I use a stand mixer (kitchenaid) on setting 2 for 5 minutes. Hand kneading works too, but it’s harder to fiddle with your phone while you do it and you’ll need to knead for more like 10 minutes.

While the bread is kneading, or afterwards if you’re doing it by hand, top up the starter with 75g flour and 75g water. If you used more than 150g starter, for instance if you made a larger loaf, alter these quantities respectively. Mix, cover with your cloth and put back in the fridge.

After kneading do a few stretch and folds (see elsewhere for description), then leave the dough in the bowl covered with a slightly damp tea towel at room temperature for several hours, eight is good to aim for. I often make bread sometime between 11 AM and early evening, then deal with it at about 10PM, but it’s pretty robust. You can leave it a longer or shorter time and it should be fine. Don’t leave it out overnight, it will develop a dry crust. If you forget and do leave it out overnight though, don’t worry about it, you can still use it, it might just not be as good.

That evening, take the dough and do some “stretches”. Take one side of the dough, stretch it over to the other side and attach. rotate the dough, repeat. You should start to feel the dough slightly stiffen or harden. This step doesn’t take too long, just get a feel for it.

Now, put it in the fridge. I use a banneton, they’re about £12 on amazon, but you can probably use some other dish. Flour the inside of it, drop the dough top of the loaf side down in the receptacle, cover, place in the fridge overnight.

These steps in total will take about 10 minutes for the initial mix and a few minutes for the step in the evening.

Day 2

A finished loaf.

The next day at some point (there’s no hard rules about how long to leave it), take the dough out of the fridge, pre-heat the oven to 250°C with a lidded dutch oven inside or similar, lined with baking paper.

When the oven is up to temperature take the dough out of the banneton, score it to help it rise (I find a serrated knife works best for this) then plop into the hot cooking pot. Put the lid on, put it in the oven then turn the oven down to 200 (if you forget this step you will end up with a sourdough rock) and set a timer for 20 minutes.

When 20 minutes is up, remove the lid and set the timer for 20 minutes again. When that timer is up it’s done, take everything out of the oven and place the bread on a wire cooking rack. Or just balance it on the handle of the cooking pot, this isn’t a cooking show, no-one cares.

Leave it to mostly cool down before you cut it to prevent it from drying out as the steam escapes.

Tips and examples

Keeping it

The bread is best on the day of baking, but leave it to cool before eating it. If you want to keep it longer than one day I recommend slicing it and freezing it. It makes great toast later.

Tips

Use good quality strong or very strong white bread flour if you’ve got it. It’s harder to get it to work with “fancy”, darker or lower quality flours. I use this: https://www.shipton-mill.com/bulk-shop/canadian-strong-white.htm
If you use an enamelled dutch oven you may destroy the enamel over time in the hot oven, use a cheaper le crueset knock-off, or one without enamelling.
Don’t stress timings too much, a few hours either way is probably fine on any of the steps. It should work around your schedule.
Keep the starter in the fridge.
Different flours need different amounts of water, you just have to experiment.

Examples

There’s some photos of what the end result should look like on my almost never updated instagram:

https://www.instagram.com/willj9000

or twitter, for example:

https://twitter.com/will_j/status/1265263352244289537/photo/1

If you want to make more than one loaf, or a larger loaf

Start to feed the starter with 50/50 flour and water a few days in advance until you have enough starter to take out the amount you will need while still leaving about 150g that you can then feed and put back in the fridge.

Making a Sourdough Starter

will@willj.net (Will Jessop) — Wed, 13 Jan 2021 02:53:52 +0000

The absolutely best way of making a starter if you don’t already have one is to not make one at all. If you can, ask someone who already has one for a bit of theirs. Most people will be happy to donate a bit, and this is a far shorter way of starting. There’s nothing heroic about creating your own, it’s mostly just a pain in the butt. If you’ve not got one though, here’s how make your own.

Already made your starter and want to make bread? Head over to the bread recipe!

Things you will need

My sourdough container, banana for scale.

To make a starter you’ll need only four things.

Flour
Water
Something to keep it in
Patience

That’s it! nothing fancy required. For the most reliable results I recommend using good quality, fresh flour. Don’t use the stuff that’s a year out of date you found at the back of the cupboard. It might work, but I’ve had more luck with good quality fresh flour.

The container should be a glass, plastic or non-reactive metal, about one banana in height (see image). It should be very clean and free from dirt or any cleaning product residue. You will also need some sort of cloth to cover it, and an elastic band to keep the cloth in place.

The Method

Make sure to be exact with the measurements, no eyeballing quantities or using weird measurements like “cups” or anything like that. Also, use a set of digital scales with 1g accuracy if you can, they make baking much easier.

Sourdough progression. Inactive starter on the left, some activity in the middle, the same starter 24 hours later.

Day 1

Mix 50g water (around tap temperature is fine) with 50g flour until you get a smooth paste. Cover with a cloth and leave out at room temperature somewhere it will not get too hot or cold for about 24 hours, so not in the fridge or a sunny windowsill.

Day 2

Mix 50g flour and 50g water as on day 1 into the existing mixture, cover and leave for another 24 hours

Day 3 through 7

Repeat the process from day 2.

You should start to see some bubbling activity on about day 4 and if everything went right you should have a starter after about a week. You should end up with something that smells slightly sour and a bit like yoghurt.

It might take a little bit longer, it’s going to depend on the flour and temperature. If it’s not doing anything by about two weeks or it smells like it’s gone off then you might need to restart.

Once it’s done, keep the cloth cover on and put it in the fridge until you need it. It will keep about a week without attention in a cold fridge.

A working starter.

Use a cloth lid for your starter to allow it to breath.

Maintaining your starter

As long as you keep taking some out and then topping it up, the starter should last forever.

I found that my starter keeps best when kept in the fridge as it turns over too fast when any warmer.

When you use some of the starter, replace what you took with 50/50 water and flour. For example, if you take 150g starter to make a loaf of bread, replace what you took with 75g water and 75g flour. Try to keep the ratios exact. If you bias towards more water for instance each time you’re going to end up with wet bread, but bear in mind that over time your starter will suffer from evaporation, so you may need to add a little extra water every once in a while.

If you think you might want to make a larger, or multiple loaves in a few days, just add more water and flour over a couple of days until you’ve got the amount of starter you will need without taking too much.

When you take the starter out ideally it will be fairly stiff and coagulated, see about 30s into this video (you don’t have to stir it for ages, I was just fiddling). Ignore the rest of the video, my technique has changed in some spots, see the bread recipe for the updated method.

The starter should be good to use every day, but if you don’t need to make bread for a while it should keep in the fridge for over a week without use. When it’s been left for longer than that without being refreshed you can still make bread with it, but it gets harder to get good results. If you leave it too long though it might die and you’ll need to start from scratch.

If your starter has sat for a while then you will find it will become more liquid, this is OK, you can still make bread with it, it just might not be as good. If there is liquid sitting on top of the starter, mix it in. If you pour it out your starter will be unbalanced.

Using the T67XX CO₂ Sensor Module in Go

will@willj.net (Will Jessop) — Mon, 30 Nov 2020 11:24:00 +0000

So, you bought four T67XX CO₂ sensors and only once they were delivered started to work out how to get values from them huh? Well I’m right there with you because so did I. and I carefully deposited the results in a new Go lib. Announcing the Go T67XX CO₂ sensor library! This post will give you some details about what these sensors can do, then introduce the library and how to use it.

Unit specs

I got my devices (I think mine are T6713) from AliExpress, and there are specs on the page there, but some interesting specs to highlight would be:


Measurement Range	0 to 5000 ppm
Accuracy	400-5000 ppm ± 75 ppm or 10% of reading, whichever is greater
Signal Update	Every 5 second
Response Time	< 3 minutes for 90% step change typical
Warm Up Time	< 2 minutes (operational), 10 minutes (maximum accuracy)
Connection	I²C slave up to 100 kHz, UART @ 19200 Baud w/Modbus support
Power Requirements	4.5-5.5 VDC, Peak 200mA (155mA typical), Average 25mA (20mA typical)

This sensor is likely good enough for most home uses like just ensuring you’re getting enough fresh air during the day, and this is what I’m using it for:

There is more data available on the datasheet.

Connecting it up

I’m using it connected to an I²C bus on my Raspberry Pi 4, so although there are a few modes of connecting the sensor that’s the one I’m focussing on. To do this. I connected the pins as follows:

Pin	Function	Raspberry Pi 4 pin
1	SDA	3
2	SCL	5
3	V++	4
4	GND	6
5	PWM	Not connected
6	CTRL/TEST	6 (Also GND)

Whatever you do, don’t connect them backwards as this sensor seems to be particularly sensitive to being misconnected (I did this and accidentally let the magic smoke out of one of them). I used a level shifter to switch the 3.3v lines to 5v because I didn’t have any 4.7k resistors and this is what worked for me, though the unit should work at 3.3v, from the datasheet:

There is an internal pull up resistor on pin 1 of the I2C interface. Customer will need to provide an external pull up resistor on pin 2 with a recommended value of 4.7k. I2C interface can operate at both 3.3V and 5V logic levels.

Using the sensor with the T67XX Go library

I’m not going to go into the details of how the library works, there are docs and examples over at the T67XX Github page, but at a high level, given a connection to the I²C bus it can read CO₂ data and perform management tasks. Here’s a simple program that waits for the sensor to achieve full accuracy then reads CO₂ values from it every 10 seconds:

package main

import (
	"log"
	"os"
	"time"

	"github.com/wjessop/t67xx"
	"golang.org/x/exp/io/i2c"
)

const (
	// We can change the sensor address on the bus if we want to, but it defaults
	// to 0x21
	t67XXSensorAddress = 0x21
)

func main() {
	// Open an i2c bus that we can pass to the driver
	device, err := i2c.Open(&i2c.Devfs{Dev: "/dev/i2c-1"}, t67XXSensorAddress)
	if err != nil {
		log.Fatalf("Couldn't open the T67XX sensor at %x, error was %v", t67XXSensorAddress, err)
	}

	// Create the driver
	driver := &t67xx.T67XX{}
	driver.Device = device

	// For now the library needs a logger to be provided. It needs to satisfy the
	// following interface:
	//
	// type Logger interface {
	// 	Debug(...interface{})
	// 	Debugf(string, ...interface{})
	// 	Fatalf(string, ...interface{})
	// }
	log := log.New(os.Stderr, "T67XX", log.LstdFlags)
	driver.SetLogger(log)

	// EnableABC enables the ABC calibration. From the datasheet:
	//
	//   "ABC LOGIC™ Automatic Background Logic, or ABC Logic™, is a patented
	//    self-calibration technique that is designed to be used in applications where
	//    concentrations will drop to outside ambient conditions (400 ppm) at least
	//    three times in a 7 days, typically during unoccupied periods. Full accuracy
	//    to be achieved utilizing ABC Logic™. With ABC Logic™ enabled, the sensor will
	//    typically reach its operational accuracy after 24 hours of continuous
	//    operation at a condition that it was exposed to ambient reference levels of
	//    air at 400 ppm CO2. Sensor will maintain accuracy specifications with ABC
	//    Logic™ enabled, given that it is at least four times in 21 days exposed to
	//    the reference value and this reference value is the lowest concentration
	//    to which the sensor is exposed. ABC Logic™ requires continuous operation of
	//    the sensor for periods of at least 24 hours.
	//
	//    Note: Applies when used in typical residential ambient air. Consult Telaire
	//    if other gases or corrosive agents are part of the application environment."
	if err := driver.EnableABC(); err != nil {
		log.Fatal("Could not enable ABC calibration on the sensor", err)
	}

	// Create a signal channel that will be closed when the sensor reaches full accuracy
	accuracyChan := make(chan interface{})

	go func(driver *t67xx.T67XX) {
		// Sleep in the background until the sensor has been powered up long enough
		// to achieve full accuracy.
		err := driver.SleepUntilFullAccuracy()
		if err != nil {
			log.Fatal("Error sleeping until full accuracy", err)
		}

		// Close the signal channel then exit the goroutine as we no-longer need it.
		close(accuracyChan)
	}(driver)

	// Now we can read the CO₂ readings in a loop, taking care to discard any
	// spurious readings.
	for {
		select {
		case <-accuracyChan:
			// A successful read on the closed channel indicates that the sensor is
			// now fully accurate.
			co2Reading, err := driver.GasPPM()
			if err != nil {
				log.Fatal(err)
			}

			// The sensors I have sometimes give spurious readings. Let's discount them.
			// Adjust these values based on the baseline CO₂ reading you expect. The max is
			// the measurement limit according to the datasheet, but i've seen values well
			// over 10,000.
			if co2Reading > 5000 || co2Reading < 200 {
				log.Printf("Reading of %d from CO₂ sensor was out of allowed bounds", co2Reading)
			} else {
				log.Printf("Got CO₂ reading of %d from CO₂ sensor", co2Reading)
			}
		default:
			log.Print("Skipping CO₂ reading as the sensor has not yet achieved full accuracy")
		}

		time.Sleep(10)
	}
}

That’s it! There’s a few more functions that the library provides, such as enabling calibration and setting the I²C bus address, you can check them out at the repo.

Custom 3D printed Blue Snowball desk stand

will@willj.net (Will Jessop) — Wed, 25 Nov 2020 04:59:00 +0000

Introducing, an all-new, super-sleek Blue Snowball microphone desk stand!

It’s very nice Will, but why?

The Problem

I inherited a unwanted Blue Snowball from an old employer several years ago, and it’s been a really fantastic addition to my day to day kit, especially as the number of video calls has increased in recent years. The only problem I had with the microphone started when I upgraded my monitors to some portrait orientation screens. These come almost all the way down to the desk, and as a result the (in my opinion rather ugly) original Blue Snowball microphone stand rather got in the way, even at it’s shortest setting.

I was able to “fix” the problem of the microphone being in the way by laying the microphone flat. This wasn’t ideal, it took up a lot more room, looked bad, and made it sound like I was lying down. Still, it worked for long enough.

The Solution

I’m always looking for a problem where the solution is “3D print something”, and this seemed like a perfect fit. An hour with Fusion 360 later and I had a stand I liked.

I’m really happy with it, it fulfils all the criteria I had when I started designing it, and I’m particularly happy with the way it looks:

Is as minimalistic as possible. I don’t want clutter.
It’s also fairly elegant in terms of looks (in my opinion of course), I have to look at this thing every day.
Fits the screw threads on the Blue Snowball mic. I want to position it myself and don’t want it swivelling around if a cable gets nudged.
Rises as low as possible above the desk. I don’t want it blocking the view of my monitors.
Allows room for the microphone to be tilted without the cable fouling on the desk.

Print one yourself

I’ve uploaded the STL files for you to print yourself. There’s the v3 design which is the one I use, and the v5 design. The two versions are very similar, the difference is only that the v3 version has a curved underside, and the v5 version has been designed to not require a support structure while printing. V3 is my preferred version and is probably less prone to wobble.

If you are printing these yourself due to the fine thread you will likely want to use a fairly high quality print setting.

Welcome to My New Blog

will@willj.net (Will Jessop) — Wed, 24 Jun 2020 23:30:40 +0100

Welcome to my new site! If you want to see my old writings you can still find them at my Wordpress blog site which is still up. I may at some point attempt to copy the posts over from there, but I don’t have time right now.

Anyway, thanks for checking out my new site!

Writing to an Lcd Panel Using Go and a Raspberry Pi

will@willj.net (Will Jessop) — Wed, 24 Jun 2020 02:09:43 +0100

In my last post I wrote about how I rigged up a Raspberry Pi to monitor temperature, humidity and air pressure. In this post I will talk about how I added a multiline LCD display into the mix to display the same data on a multiline LCD display.

Why tho

Why indeed! It’s not particularly useful, I know it’s too hot in my office already just by, well, feeling the air, and I can always check out the Grafana dashboard I put together to get the latest data if I need to know an exact figure, which I don’t. It’s quite interesting to know how hot it is in here (my wife definitely agrees on this) and just being able to know by glancing at an LCD display is quite convenient.

But that’s all irellevant. I did it because it was fun and having an LCD display stuff is cool.

The display

The display is an LCM1602 20 character by 4 line LCD. Attached to the back of the LCD is a “backpack”. This backpack connects to an I²C bus, for example like the ones on either an Arduino or Raspberry Pi.

Making the dislay show text

This was the fun part of the challenge. There are some Python code and libraries and at least one C++ library out there that work mostly OK, but I don’t want to use Python, I want to use Go.

So, rather than attempt to start from scratch and interpret the datasheet I decided try to convert some of the Python code to Go. This turned out to be pretty easy, even though I don’t really know Python at all.

What I ended up with was my Go LCM1602 LCD driver project. This library is pretty simple to use, as can be seen from the example code:

package main

import (
	lcd "github.com/wjessop/lcm1602_lcd"
)

func main() {
	// Configure this line with the device location and address of your device
	lcdDevice, err := i2c.Open(&i2c.Devfs{Dev: "/dev/i2c-1"}, 0x27)
	if err != nil {
		log.Fatal(err)
	}
	defer lcdDevice.Close()

	lcdDisplay, err = lcd.NewLCM1602LCD(lcdDevice)
	if err != nil {
		log.Fatal(err)
	}

	// Write a string to row 1, position 0 (ie, the start of the line)
	if err := lcdDisplay.WriteString("Hello World!"), 1, 0); err != nil {
		log.Fatal(err)
	}

	// Write a string to row 2, position 7
	if err := lcdDisplay.WriteString("(>'.'<)", 2, 7); err != nil {
		log.Fatal(err)
	}

	if err := lcdDisplay.Clear(); err != nil {
		log.Fatal(err)
	}
}

The only feature not documented here is the WritePaddedString method:

// WritePaddedString writes a string to the LCD at the given row, but pads the
// row to effectively wipe any remaining characters off the line. This is handy
// for when you want to repeatedly display data on a line, but it could be of
// varying length, and you don't want to blank the entire screen each time.
func (l *LCM1602LCD) WritePaddedString(message string, row int, startPosition byte) error {
	return l.WriteString(fmt.Sprintf("%-20v", message), row, startPosition)
}

Once I’d written the library it was a simple job to incorporate it into the climate monitoring program I’d created, and you can see this in the full working example I uploaded to GitHub.

That’s it!

I’ll probably keep the LCD displaying climate data in my office, as I’m basically addicted to knowing what the temperature and humidity is now. If you decide to get your own LCD screen, make sure you grab one that has the backpack, as you’ll have a hard time using it if it doesn’t have one.

Just for fun, here’s a photo of the back of the LCD with the backlight shining through the circuit board:

Humidity, Temperature and Pressure Sensing on a Raspberry Pi With Go

will@willj.net (Will Jessop) — Tue, 23 Jun 2020 20:26:10 +0100

(click for a higher resolution image)

Background

A few months ago I bought a couple of lemon trees to grow in my conservatory, but most excitingly for me, a Buddha’s hand tree. This tree was fairly expensive, and I wanted to maximise it’s chances of surviving and hopefully thriving. Citrus trees are quite sensitive to extremes of temperature (they obviously don’t like cold, but apparently don’t like excess heat either) and also humidity, preferring an atmosphere of around 50% relative humidity or more.

It was for this reason that I decided to start experimenting with climate monitoring.

The Hardware

I was originally planning on making some sensor units as described in Aaron Paterson’s ESP8266 + Plantower AQ sensor project, so I ordered a load of parts and PCBs to make those, but a lot of the parts for these sensor units are ordered from AliExpress, and so have delivery times in the order of weeks to months, rather than days, so given that my Buddha’s Hand tree had already arrived I decided to get some BME280 temperature, humidity and pressure sensors from Amazon as they don’t cost too much and would allow me to get a jump start on checking that my trees were safe.

These BME280 sensors use the I²C bus, which makes them really easy to connect to, and read, from an Arduino or Raspberry Pi. I have both hanging around, but the Raspberry Pi is easier to get something quick running because I can write my code in a higher level language and it comes with Wifi onboard. Using the raspberry pi meant that the equipment list I needed was small.

Required equipment

Raspberry Pi
BME280 sensor
Wire to connect the sensor to the Raspberry pi
Power supply for the Pi

Wiring the sensor was pretty easy. You can find Raspberry Pi pinout diagrams on the Web, and you just need to connect:

VIN to DC Power 5v (pin 4)
GND to Ground (pin 6)
SCL to SCL1 (pin 5)
SDA to SDA1 (pin 3)

That’s it! No complex electronics required for this setup.

The Software

I decided on using an Influx DB to store the data, and to graph the results in Grafana (as Aaron does for his monitoring) as they seemed like a good fit, and I didn’t want to spend too much time looking at alternatives as my trees were going unmonitored. They are also incredibly easy to connect to each other to see the time-series data in Influx DB in Grafana which helps a lot. There is handily an existing guide to installing Influx DB and Grafana on a Raspbery Pi 4 already available, which made things a bit quicker.

For the software to grab the sensor readings and put the data into Influx I used Go for a number of reasons:

I like writing it
It’s efficient. Raspberry Pis are faster than they used to be, but they’re still somewhat constrained
It’s really easy to deploy to a Pi
It’s really simple to make work with systemd

In reality you can use whatever language you like, Ruby or Python would be fine as this isn’t a highly CPU or memory intensive application, but you are likely to want to choose a language that someone has already written a driver for the sensor in as that’s going to cut down the amount of time you have to spend writing the software a lot.

It just so happens that a couple of people have written Go libraries for the BME280, I chose this one. Putting it to use it’s fairly simple to get the data out of the sensor and print it to the terminal:

package main

//
// Get sensor data from the BME280 and print it to the terminal
//

import (
	"fmt"
	"log"

	"github.com/maciej/bme280"
	"golang.org/x/exp/io/i2c"
)

func getBME280() *bme280.Driver {
	device, err := i2c.Open(&i2c.Devfs{Dev: "/dev/i2c-1"}, 0x76)
	if err != nil {
		log.Fatal(err)
	}

	driver := bme280.New(device)
	err = driver.InitWith(bme280.ModeForced, bme280.Settings{
		Filter:                  bme280.FilterOff,
		Standby:                 bme280.StandByTime1000ms,
		PressureOversampling:    bme280.Oversampling16x,
		TemperatureOversampling: bme280.Oversampling16x,
		HumidityOversampling:    bme280.Oversampling16x,
	})

	if err != nil {
		log.Fatal(err)
	}

	return driver
}

func main() {
	bme280 := getBME280()
	defer bme280.Close()

	response, err := bme280.Read()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf(
		"Temp: %.1fC, Press: %.1fhPa, Hum: %.1f%%\n",
		response.Temperature,
		response.Pressure,
		response.Humidity,
	)
}

To make this code work for yourself, write it to a file, compile it on your local machine:

GOOS=linux GOARCH=arm GOARM=7 go build sensor.go -o sensor

Copy it over to your Raspberry Pi, wherever that happens to be:

$ scp sensor pi@192.168.1.120:~/
tmp 100% 2096KB   4.4MB/s   00:00

Then SSH into your Raspberry Pi and after you have made sure that i2c is enabled run the binary as root:

# sudo ./sensor
Temp: 22.4C, Press: 1000.0hPa, Hum: 54.8%

Sweet! The next step is to get the data into Influx DB so we can log it and graph it with Grafana. To do that I used the Influx DB API provided by the Influx creators themselves. This looks something like this:

package main

import (
	"context"
	"flag"
	"fmt"
	"os"
	"time"

	influxdb2 "github.com/influxdata/influxdb-client-go"
	lcd "github.com/wjessop/lcm1602_lcd"
	"golang.org/x/exp/io/i2c"
)

const (
	databaseName = "climate"
	locationName = "conservatory"
)

func main() {
	// create new client with default option for server url authenticate by token
	client := influxdb2.NewClient(
		fmt.Sprintf("http://127.0.0.1:8086", influxhost),
		"climate-writer:<password goes here>",
	)

	// user blocking write client for writes to desired bucket
	writeAPI := client.WriteApiBlocking("", databaseName)

	var ctx = context.Background()

	now := time.Now()

	// Write a single datapoint to Infux
	p := influxdb2.NewPoint(
		"stat",
		map[string]string{"unit": "temperature", "location": locationName},
		map[string]interface{}{"value": response.Temperature},
		now,
	)

	if err = writeAPI.WritePoint(ctx, p); err != nil {
		log.Fatal(err)
	}

	// Repeat the above Influx write for humidity and pressure
}

I have pushed a full working example including reading the sensor data and sending to Influx DB to Github. It also includes optional writing of the climate data to an LCD panel.

That is pretty much it! Place your Raspberry Pi with it’s sensor in the place you want to be monitored, and it will gather the data for you. Creating a Grafana dashboard is left as an exercise for the reader.

The Finished Device

Here is the working sensor and Pi in place in my conservatory, running the Go program, Influx DB and Grafana. I have now added a second Pi in my study that reports it’s sensor readings to the Influx DB on this machine too.

Improvements

Raspberry Pis are fairly expensive, and have way more processing power than is needed to read sensor data, and they draw more power too. I plan on eventually putting together the boards in Aarons project so I can put a few more of these around my house relatively cheaply, and with more sensor types. The Raspberry Pi is a great stop-gap though, and fine for a one-off. I may still run Influx and Grafana on a Pi.

Because the Raspberry Pi was meant to be temporary I’ve left it bare, I’ll 3d print some cases for everything when my 3d printer arrives.

In my next post I discuss how I added an LCD display to show the climate data.

Error installing libv8 on Mac OS Catalina

will@willj.net (Will Jessop) — Mon, 17 Feb 2020 16:23:53 +0000

I got an error installing the libv8 gem on Catalina:

$ gem install libv8 -v '3.16.14.19'
Building native extensions.  This could take a while...
ERROR:  Error installing libv8:
	ERROR: Failed to build gem native extension.
    current directory: /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/ext/libv8
/Users/will/.rbenv/versions/2.4.5/bin/ruby -r ./siteconf20200217-34155-18bho1v.rb extconf.rb
creating Makefile
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/disable-building-tests.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/disable-werror-on-osx.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/disable-xcode-debugging.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/do-not-imply-vfp3-and-armv7.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/do-not-use-MAP_NORESERVE-on-freebsd.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/do-not-use-vfp2.patch
Applying /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/patches/fPIC-for-static.patch
Compiling v8 for x64
Using python 2.7.16
Using compiler: c++ (clang version 11.0.0)
Unable to find a compiler officially supported by v8.
It is recommended to use GCC v4.4 or higher
Beginning compilation. This will take some time.
Building v8 with env CXX=c++ LINK=c++  /usr/bin/make x64.release ARFLAGS.target=crs werror=no
GYP_GENERATORS=make \
	build/gyp/gyp --generator-output="out" build/all.gyp \
	              -Ibuild/standalone.gypi --depth=. \
	              -Dv8_target_arch=x64 \
	              -S.x64  -Dv8_enable_backtrace=1 -Dv8_can_use_vfp2_instructions=true -Darm_fpu=vfpv2 -Dv8_can_use_vfp3_instructions=true -Darm_fpu=vfpv3 -Dwerror=''
  CXX(target) /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/vendor/v8/out/x64.release/obj.target/preparser_lib/src/allocation.o
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
In file included from ../src/allocation.cc:33:
../src/utils.h:33:10: fatal error: 'climits' file not found
#include
         ^~~~~~~~~
1 warning and 1 error generated.
make[1]: *** [/Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/vendor/v8/out/x64.release/obj.target/preparser_lib/src/allocation.o] Error 1
make: *** [x64.release] Error 2
/Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/ext/libv8/location.rb:36:in `block in verify_installation!': libv8 did not install properly, expected binary v8 archive '/Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/vendor/v8/out/x64.release/obj.target/tools/gyp/libv8_base.a'to exist, but it was not found (Libv8::Location::Vendor::ArchiveNotFound)
	from /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/ext/libv8/location.rb:35:in `each'
	from /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/ext/libv8/location.rb:35:in `verify_installation!'
	from /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19/ext/libv8/location.rb:26:in `install!'
	from extconf.rb:7:in `'
extconf failed, exit code 1
Gem files will remain installed in /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/gems/libv8-3.16.14.19 for inspection.
Results logged to /Users/will/.rbenv/versions/2.4.5/lib/ruby/gems/2.4.0/extensions/x86_64-darwin-19/2.4.0/libv8-3.16.14.19/gem_make.out

The Solution

The solution was as follows:

$ brew install v8@3.15
$ gem install libv8 -v '3.16.14.19' -- --with-system-v8
Building native extensions with: '--with-system-v8'
This could take a while...
Successfully installed libv8-3.16.14.19
1 gem installed

Fixing the ruby 'CoreFoundation/CFString.h' file not found install error

will@willj.net (Will Jessop) — Wed, 01 May 2019 15:00:33 +0000

I hit this error installing the latest Ruby using rbenv:

$ rbenv install 2.6.3
ruby-build: use openssl from homebrew
Downloading ruby-2.6.3.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.bz2
Installing ruby-2.6.3...
ruby-build: use readline from homebrew
BUILD FAILED (OS X 10.14.4 using ruby-build 20190423)
Inspect or clean up the working tree at /var/folders/r7/kjzbwmx533b20hcf1_s9kc9c0000gn/T/ruby-build.20190501131413.33977
Results logged to /var/folders/r7/kjzbwmx533b20hcf1_s9kc9c0000gn/T/ruby-build.20190501131413.33977.log
Last 10 log lines:
compiling error.c
compiling eval.c
compiling file.c
compiling gc.c
file.c:23:10: fatal error: 'CoreFoundation/CFString.h' file not found
#include
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [file.o] Error 1
make: *** Waiting for unfinished jobs....

The problem is missing headers. To re-install just run this command:

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

Now, Ruby should install just fine:

$ rbenv install 2.6.3
ruby-build: use openssl from homebrew
Downloading ruby-2.6.3.tar.bz2...
-\> https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.bz2
Installing ruby-2.6.3...
ruby-build: use readline from homebrew
Installed ruby-2.6.3 to /Users/will/.rbenv/versions/2.6.3 ``

Fixing errors installing the charlock_holmes ruby gem

will@willj.net (Will Jessop) — Wed, 13 Feb 2019 05:50:52 +0000

The charlock_holmes gem can be a PITA to install on a mac, this is what worked for me. If you get this error:

gem install charlock_holmes -v '0.7.7'
Building native extensions. This could take a while...
ERROR:  Error installing charlock_holmes:
ERROR: Failed to build gem native extension.

current directory: /Users/will/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/gems/charlock_holmes-0.7.7/ext/charlock_holmes
/Users/will/.rbenv/versions/2.6.1/bin/ruby -I /Users/will/.rbenv/versions/2.6.1/lib/ruby/2.6.0 -r ./siteconf20190213-34122-e636v8.rb extconf.rb
checking for -licui18n... yes
checking for -licui18n... yes
checking for unicode/ucnv.h... no

***************************************************************************************
*********** icu required (brew install icu4c or apt-get install libicu-dev) ***********
***************************************************************************************
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/Users/will/.rbenv/versions/2.6.1/bin/$(RUBY_BASE_NAME)
--with-icu-dir
--without-icu-dir
--with-icu-include
--without-icu-include=${icu-dir}/include
--with-icu-lib
--without-icu-lib=${icu-dir}/lib
--with-icui18nlib
--without-icui18nlib
--with-icui18nlib
--without-icui18nlib

To see why this extension failed to compile, please check the mkmf.log which can be found here:

/Users/will/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0-static/charlock_holmes-0.7.7/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /Users/will/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/gems/charlock_holmes-0.7.7 for inspection.
Results logged to /Users/will/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0-static/charlock_holmes-0.7.7/gem_make.out

First, make sure you have installed the required dependencies:

brew install xz icu4c

Now, install charlock_holmes:

gem install charlock_holmes -v=0.7.7 -- --with-opt-dir=/usr/local/opt --with-opt-include=/usr/local/opt/icu4c/include --with-opt-lib=/usr/local/opt/icu4c/lib --with-cxxflags=-std=c++11

Basic pasta sauce

will@willj.net (Will Jessop) — Tue, 14 Feb 2017 00:56:16 +0000

Someone asked on Twitter how I made my pasta sauce, so here is the recipe! The base of my Pasta sauce is largely this recipe, except I mostly don’t add mozzarella:

At it’s simplest I can make the whole thing for one person in 19 minutes, so it’s great for lunch.

Ingredients

85g dried pasta per person (50g per child). You can use whatever you want, I prefer spaghetti.
1 punnet of good tomatoes, about 225/250g (small ones, eg. baby plum, sunstream etc.)
Small bunch of basil or parsely.
Salt & Pepper
1 clove of garlic
1 chilli, I don’t de-seed
Olive oil

Method

To start, boil some water in a medium pan. Add a decent amount of salt. While the water is coming up to the boil prepare the other ingredients.

First, chop the tomatoes, you can leave the skins on. I chop mine in half, I prefer the texture. It is also quicker. Finely chop the garlic and chilli. Roughly chop the basil or parsley.

As soon as the water comes to the boil, add your pasta. Now heat some oil over a medium heat in a small frying pan. Add in the chilli and garlic. Make *really* sure not to burn the garlic. Before the garlic browns add in the tomatoes then a good pinch of salt. Stir around for a few seconds. Now add in some water. It’s really easy to add in too much, you want just enough to cover the base of the pan, any more and you’ll end up with soup. You can always add more if you need to.

Next, add the basil or parsley. Stir through. When the pasta is al dente drain it retaining some of the cooking water and add to the sauce, stir around. Add pasta water if needed. You should still be able to work out what each of the individual parts of your sauce are made of. If you can’t you cooked it too long and/or added too much water.

At this point you can stir in mozzarella, I mostly don’t.

Slap it on a plate, grate parmesan over it and eat it.

Notes

With practice I’ve got this down to 19 minutes for the simple version, you might be faster if you can boil water faster than me. Adding more stuff makes it slower obviously.

Only get good tomatoes. They have to taste of something, if they don’t taste of anything your sauce will be crap. If you can’t get good fresh ones (or they have a gazillion air miles on them) use a good tin. It’s not as good and doesn’t work as well with the parsley or mozzarella versions, but it’s better than the alternative.

This sauce is great by itself, but you can add to it as you want. Some black pitted olives from a jar are great. If I’ve got other veg lying around, or more than one person eating I will add that too, depending on the density of the vegetable and how finely it is chopped depends when I add it. Chop it finer to cook it quicker. A good addition to bulk it up is a sweet pointed pepper & celery soffritto. One pointed pepper and 2 celery sticks is about the right balance with one punnet of tomatoes. Fennel and black olive are great too. Just be careful not to overdo it. There’s enough sauce with just the tomatoes for one person, less is more when adding extra ingredients.

If you do add more veg, try to keep the tomatoes per person ratio the same.

In the photo above I went a bit crazy and added:

sweet pointed pepper & celery soffritto
1 small bulb of fennel, thinly sliced.
Bacon*

It was really a bit too much.

*Bacon! Aubergine is great for adding some a bulky umami flavour in-place of meat. Chop some aubergine into ~1cm cubes or strips, one handful should be enough, think proportional to the other ingredients. Put on some foil, sprinkle salt on, create a parcel (crimp the edges to keep the water in) and oven cook at ~180°C (?, experiment) until just soft then add to the sauce.

I think that’s it! Nail the basic sauce then experiment on top of that and you’ve got a handy and versatile base!

The security of a system is as strong as it's weakest component

will@willj.net (Will Jessop) — Fri, 15 Apr 2016 12:47:57 +0000

The padlock that had seized shut on my back gate was proving hard to remove, the hardened steel U was not yielding to my hacksaw. I noticed some rivets underneath that turned out to be soft steel. 10 minutes of sawing and I was in.

Hardened steel U

Soft steel rivets on the bottom

After removing the rivets, hammering off the bottom plate

The bottom plate revealed, lock mechanism removed

Lock open!

Random graph

will@willj.net (Will Jessop) — Fri, 25 Sep 2015 22:27:43 +0000

Random graph, from playing around with RGL in Ruby.

I like it, but it's pointless

How to find if two nodes are connected in an RGL graph

will@willj.net (Will Jessop) — Fri, 25 Sep 2015 16:39:53 +0000

Say you have a graph like this:

A simple graph

How do you find out if there is a path between any of the two nodes? By using a breadth-first search:

require 'rgl/implicit'
require 'rgl/traversal'

vertices = ["one", "two", "three"]

g = RGL::ImplicitGraph.new do |g|
  g.vertex_iterator { |b| vertices.map{|v| b.call(v) } }
  g.adjacent_iterator { |x, b| b.call( vertices[(vertices.index(x) - 1).abs] ) }
  g.directed = true
end

t = g.bfs_search_tree_from("one")

puts t.has_vertex?("two")   # true
puts t.has_vertex?("three") # false

I got my pan tilt servos working

will@willj.net (Will Jessop) — Tue, 29 Jul 2014 22:59:02 +0000

I’m making a simple camera gimbal and tonight I wired up the right analog stick on my xbox controller to control the pan/tilt servos:

Tomorrow I will print out a prototype gimbal structure.

Testing Github commit hook one liners

will@willj.net (Will Jessop) — Mon, 09 Jun 2014 13:24:11 +0000

A couple of ruby one-liners for dumping out the content of a github commit hook payload:

For hooks that use type application/json:

$ ruby -rpp -rjson -rsinatra -e 'set :port, 8000; post "/*" do; pp JSON(request.body.read); end'

For hooks that use type application/x-www-form-urlencoded:

$ ruby -rpp -rjson -rsinatra -e 'set :port, 8000; post "/*" do; pp JSON(params[:payload]); end'

AWS IAM Policy for allowing s3cmd to sync to an S3 bucket

will@willj.net (Will Jessop) — Fri, 18 Apr 2014 17:41:47 +0000

It’s a good idea to set an IAM access policy for anything that accesses AWS using your account details, I wanted to do this for s3cmd syncing a local directory to an s3 bucket. There are a number of posts on setting up the IAM policy for s3cmd already but none of the examples worked for me, I got a 403 permission denied error when running the s3cmd sync command.

After some digging it turns out that s3cmd now tries to set an ACL on the files it uploads, and this needs to be specifically allowed in the ACL. I’m guessing that it didn’t in the past, hence the now incorrect IAM advice. So here is the new working IAM policy, complete with the s3:PutObjectAcl permission added:

(See jrantil’s comment below on wether s3:ListAllMyBuckets is needed in this instance)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1397834652000",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Sid": "Stmt1397834745000",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::bucketname",
        "arn:aws:s3:::bucketname/*"
      ]
    }
  ]
}

Update!

This post was imported from my original blog where there were comments, and these may be relevant to anyone finding this page now:

Comments on the original blog post

Ping-Pong Press!

will@willj.net (Will Jessop) — Mon, 16 Dec 2013 18:24:25 +0000

My robot was written about recently on the Raspberry Pi foundation blog (now seemingly removed 😢, but still available on archive.org), and now on the Pololu blog too, the place I originally got my tracks from!

Robot at RubyConf

will@willj.net (Will Jessop) — Mon, 09 Dec 2013 14:14:11 +0000

I took my robot to RubyConf. It’s not really Ruby related, but I figured it would go down like a robot at a programmers conference… So I took the robot apart, packed it away and headed to Miami:

The view from my hotel balcony

First job, putting the robot back together again. I was happy it made it through security. It was the first time I’d flown with a small toolkit.

The robot laid bare

I was carrying the robot around most of the first day, people constantly asked to try it out

It worked well on the shiny floor of the lobby. You can just about see the new 3d printed lifter arm:

The balls went everywhere

Totally coincidentally Ron Evans also did a great talk on Artoo and the newly announced Gobotio.

Programmers love robots!

Prototype lifter arm

will@willj.net (Will Jessop) — Sat, 02 Nov 2013 22:04:05 +0000

I added a prototype lifter arm to my robot yesterday. It needs re-doing (it’s unreliable and heavy) but it works for now.

I’m using a new motor driver circuit for it as I burned out the SN754410 motor driver I was using, the lifter motor seems to peak briefly at about 1.5 amps when starting to lift the arm, and that’s out of the chips range. I stayed up late making a new motor control circuit with a L298N that can provide a lot more power.

Here is the robot picking up 12 ping-pong balls. I was driving, Morwenna was filming:

Here is the same run seen through the on-board camera. Almost all of the run was done watching the live feed rather than watching the robot itself:

Here is the paper strip-board design for the L298N circuit.

Designing tHe motor driver circuit for stripboard

Testing the circuit out on breadboard:

Wire everywhere, as often happens when I'm doing electronics projects

The finished circuit. You can see it bolted to the back of the robot in the video.

First robot remote driving test

will@willj.net (Will Jessop) — Fri, 01 Nov 2013 02:16:09 +0000

I programmed some remote control software using a Golang receiving program on the robot and a ruby control client using my gamepad ruby gem and an Xbox 1 controller. It worked OK. It was a bit jerky, there’s no PWM so no acceleration, it’s either go or stop; anything not totally rigid on the robot wobbles. Also the position of the camera doesn’t show enough of the robot so it’s hard to get a real idea of where the robot is.

I was filming, the robot was being controlled my my wife, Morwenna, from upstairs.

The robot is also prone to shed a track if the “half turn” is used too much, that is one track forwards or backwards, the other one stationary. I can fix this in software if I can work out a way to do PWM on the robot that doesn’t run the Raspberry Pi CPU.

Rejects and prototypes

will@willj.net (Will Jessop) — Thu, 31 Oct 2013 13:25:19 +0000

Prototypes and rejects from the 3d printer. All part of the design process for my robot:

Rejected and prototype 3D printed componenets

New robot base design and completed prototype arm assembly

will@willj.net (Will Jessop) — Thu, 31 Oct 2013 13:16:49 +0000

I spent all day at hacman Tuesday and Wednesday printing and tweaking robot parts. I put it all together when I got home. This is the front, the lifter arm has holes to allow me to screw in different attachments, I’ve still not designed that part of the robot yet:

The robot, mostly assembled

Here is the back view, you can see the lifter arm gear housing (the curved structure) and the motor that drives it, also the battery (underneath the ball hopper)

The rear of the robot with the hopper in place

The lifter arm gear system looking through where the hopper would be:

The robot without the hopper in place

I need to design an attachment for the end of the lifter arm. Here I’ve created a simple one using meccano. It’s too heavy, the bolts catch on the floor and it dumps the ping-pong balls short of the hopper. Other than that it’s perfect. You can see the camera with capped lens that I’ve just wedged into the structure temporarily:

A prototype lifter arm made from Meccano

Next-up: better lifter attachment design, a couple of re-prints with tweaks and some software.

Testing the new battery + lifter arm circuit

will@willj.net (Will Jessop) — Mon, 28 Oct 2013 23:02:01 +0000

First test of the Raspberry Pi running off battery power with no external supply at all. In the picture the 5000mAh battery is the object at the back with “5000” written on it. My multimeter is measuring the voltage output. One of the Raspberry Pi (the thing with the red light on it) output pins is connected through the MotorPiTX board (the yellow circuit board on-top of the Raspberry Pi) to the breadboard.

The breadboard contains two circuits, the top-right cluster of components is the circuit that turns the output of the Raspberry Pi into something relevant to the motor driver in the bottom left, the bottom left cluster of components is the motor driver itself.

The Pi is pulsing the output causing the motor driver to spin the motor. The micro-switch just on the very bottom of the picture is one of the limit switches for the ball lifter arm.

A relatively tidy electronics bench for once

An assembled ping-pong robot hopper

will@willj.net (Will Jessop) — Thu, 24 Oct 2013 01:27:44 +0000

Today I printed out and glued together the first iteration of the ping-pong ball hopper that takes up most of the space of the robot.

The assembled hopper

Full of ping-pong balls, I can get 12-14 in without packing them manually. That seems like enough for now, but a larger hopper would be good.

The hopper loaded with ping-pong balls

Mounted on the robot. You can see the completed camera mount and Raspberry Pi mounted on the side in the foreground:

The ping-pong ball hopper mounted on the robot

The reverse angle

Testing new motor mounts

will@willj.net (Will Jessop) — Thu, 24 Oct 2013 01:22:00 +0000

I needed to re-design the motor mounts on the robot base, they didn’t hold the motors securely enough and tended to shift around.

The new mounts are better fitting and include a small wedge that fits between one of the gearbox plates where there are no cogs.

You can see the small gearboxes built in to the motors

The new motor mounts are a snug fit

One of the great things about 3d printing is being able to pull out part of a design and iterate over in isolation, then when you’re done incorporate it back into your main design.

Raspberry Pi camera mount + lens holder

will@willj.net (Will Jessop) — Thu, 24 Oct 2013 01:15:37 +0000

I printed out the front half of my Raspberry Pi camera module mount. It serves two purposes, providing a way of mounting the camera on my robot, and giving me a place to attach the wide-angle lens I’d bought earlier.

The new Raspberry pi camera holder

The new Raspberry pi camera holder rear view

New Pi mount, and testing the power requirements

will@willj.net (Will Jessop) — Sun, 20 Oct 2013 00:06:46 +0000

Testing the robot power requirements

Before I pick a battery I need to know what power requirements my robot is going to have so I took it into the hackspace to test. I hooked it up to one of the power supplies then ran all the systems, so Streaming video off the Raspberry Pi over wireless whilst running the motors. I weight it down with a large motor (the big black thing) and then provided resistance, it peaks at about 1.6 amps.

In the video you can also see the new side mounted Raspberry Pi and the base of the ping-pong ball hopper.

Detail view of the side mounted Raspberry Pi and the base of the ping-pong ball hopper:

Raspberry Pi mount and ping-pong ball hopper base

Lenses

will@willj.net (Will Jessop) — Fri, 18 Oct 2013 16:17:20 +0000

The Raspberry Pi camera doesn’t have a field of view that is wide enough for my robot, I’d have to mount it really far from my robot to see enough context to make it drivable so I’ve bought some wide angle iPhone lenses to try out.

Wide-angle and fish-eye lenses

Here’s a photo taken using the fish-eye lens held up to my iPhone:

The view through the fish-eye lens

First prototype robot base

will@willj.net (Will Jessop) — Mon, 30 Sep 2013 20:08:56 +0000

The robot base design from my previous post printed OK. Here it is printing:

Robot base 3D printing

The finished print. The two small separate objects are the motor covers:

Completed robot base print

The assembled robot, including Raspberry Pi and MotorPiTX:

Assembled robot base

The underside of the robot:

Even robots have a bottom

OpenSCAD robot base design

will@willj.net (Will Jessop) — Tue, 24 Sep 2013 01:23:15 +0000

I’ve re-designed a prototype robot mount in OpenSCAD as Sketchup wasn’t up to the job. It took a while, you have to program the model in a text editor so there’s lots of trial and error. Will have to wait to print it to see if this design will work.

Prototype robot base designed in OpenSCAD

I’ve made some improvements over the original design, most notably thicker side walls and motor mounts, more accurate measurements and added the cross-pieces with a Raspberry Pi mount. Here’s the original Sketchup design for comparison:

Robot base designed in Sketchup

Sketchup was great, really easy to use, the only problem was that it seemed to create invalid model files.

Trying out the prototype caterpillar track mount

will@willj.net (Will Jessop) — Mon, 23 Sep 2013 18:09:32 +0000

It works, but it needs more refinement.

Testing the new motor mount

Printing out a test caterpillar track and motor mount

will@willj.net (Will Jessop) — Mon, 23 Sep 2013 14:38:44 +0000

Printing a motor / caterpillar track mount detail

The 3d printer at work

Designing a track mount

will@willj.net (Will Jessop) — Sat, 21 Sep 2013 22:42:44 +0000

Designing a track mount for my pololu caterpillar tracks. Sketchup is trying very hard to prevent me.

Pololu track mount design in progress

Secret project - MotorPiTX motor board

will@willj.net (Will Jessop) — Sun, 08 Sep 2013 00:51:32 +0000

I soldered together my MotorPiTX motor controller today and fitted it to the robot:

The robot with MotorPiTX board

It’s a hell of a lot neater than my original effort, and only trails one wire. It goes forwards and backwards now too. To the left of the blue relay near the top you can see my homemade heatsink attached to the voltage regulator:

Pro-spec heatsink

The robot in action, being driven by a Go program on the Raspberry Pi:

http://www.youtube.com/watch?v=hIabB-AURT0

Next up, getting the camera working, and figuring out a better power supply/battery.

Secret project

will@willj.net (Will Jessop) — Sat, 07 Sep 2013 12:41:02 +0000

One of my new motors. It's about 10mm in diameter

I’ve started work on a top-secret project. I can’t really hide the fact that it’s going to be a robot, but I’m not going to say what it is, at least not just yet.

So, last night I was designing a 3d printed mount for the tiny 3-6V motors I bought and I started to wonder if I could cobble something together using my old technic lego. I dug out the lego, but on top of that was my dusty old meccano set, even better!

WIthin a short amount of time I had some motor mounts and a frame made, including tensioning springs for the caterpillar tracks. All that was left was to take it for a spin. I hooked it up to my Raspberry Pi via my Custard Pi breakout board, a ULN2803A and a custom voltage regulator circuit.

Motor mount

Seeing if it drives in a straight line:

Hooked up to the Raspberry Pi, controlled by microswitches. You can see the top of the Custard Pi poking out over the mess of wires that is my breadboard and the cheap wireless dongle/antenna I got from eBay. The voltage regulator circuit makes an appearance being dragged along behind:

MotorPiTX kit

Right now it only goes forwards because I didn’t have the circuitry for anything else, but I got my MotorPiTX in the mail this morning so that will change soon.

go-piglow, a lib for controlling the piglow in Golang

will@willj.net (Will Jessop) — Tue, 03 Sep 2013 23:26:43 +0000

A Piglow glowing

A few days ago I got a Piglow. It’s a fairly useless but fun addon board for the Raspberry Pi that has 18 individual user controllable LEDs arranged in Arms/Legs/Tentacles (whatever you want to call them).

There are example programs out there to control the LEDs, but they are all in Python, and on my Pi they are all fairly slow so I wrote my own lib for Go: https://github.com/wjessop/go-piglow

The API is fairly strigthtforward, this sample program just turns on and off some of the LEDS:

package main

import (
    "github.com/wjessop/go-piglow"
    "log"
)

func main() {
    var p *piglow.Piglow
    var err error

    // Create a new Piglow
    p, err = piglow.NewPiglow(); if err != nil {
        log.Fatal("Couldn't create a Piglow: ", err)
    }

    p.SetLED(0, 255) // Set LED 0 to 255 (max brightness)
    p.SetLED(1, 128) // Set LED 1 to half brightness
    err = p.Apply(); if err != nil { // Apply the changes
        log.Fatal("Couldn't apply changes: ", err)
    }
}

The lib API allows for controlling individual LEDs, the colour rings, the tentacles, or to display a value bar-graph style on each tentacle.

I wrote some more complex example programs to go with the lib to demo these capabilities. A simple program to flash the LEDs, a CPU meter that displays 1, 5 and 15 minute load average on each of the tentacles, and the most fun, a disco program, here is me demonstrating them:

Right now i’m running this program on my Pi to slowly fade between the colour rings.

Creating BERT dicts in Go

will@willj.net (Will Jessop) — Sun, 10 Feb 2013 02:09:06 +0000

I’ve been learning Go recently and have written a program to connect to an existing service (written in Ruby) that sends and receives messages serialised as BERT terms.

I’m posting this partly because I had quite a lot of fun figuring it out and partly to document creating BERT dicts in Go should anyone else need to do this in the future and hit the same issues I did.

Why BERT?

I’m a big fan of BERT. It’s compact, flexible, and there are good libs available for serialisation/de-serialisation. So far I’ve exclusively been using the bert gem (written by Tom Preston-Werner, author of the BERT spec).

Creating BERT dicts

One of the great features of BERT is the complex types it supports, including dicts. The equivalent to a dict in Ruby would be a hash, in Go a map. They are really simple to create in Ruby:

require 'bert'
BERT.encode({"key" => "val"})
=> "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keym\x00\x00\x00\x03valj"

We can pull this apart and see exactly what the bert gem did to our data. Let’s dump the string to an array of 8-bit unsigned integers:

BERT.encode({"key" => "val"}).unpack("C*")
=> [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 104, 2, 109, 0, 0, 0, 3, 107, 101, 121, 109, 0, 0, 0, 3, 118, 97, 108, 106]

It’s hard to see exactly what happened, but with the BERT docs and the erlang External Term Format docs we can see how the hash got encoded.

magic| tuple |  atom   |       bert     |            |    dict        |  list 1 elem    |      list      |   atom  |      key      |   atom  |      |  val       | nil | nil
131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 108, 0, 0, 0, 2, 100, 0, 3, 107, 101, 121, 100, 0, 3,       118, 97, 108, 106, 106

If the formatting of that breakdown is messed up here’s a raw gist that may be clearer.

What you can see here are what the bytes represent (you can see the breakdown of each data type on the External Term Format docs). This is great, but why write a blog post just about dicts? Well, they’re easy to create in Ruby:

BERT.encode(:complex => {"key" => [:data, {:structures => "are easy to serialise"}]})
=> "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\acomplexh\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keyl\x00\x00\x00\x02d\x00\x04datah\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\nstructuresm\x00\x00\x00\x15are easy to serialisejjjj"

but it’s not so obvious in Go, and I hit some issues when trying to create them.

Serialising to BERT in Golang

Serialising data to BERT/BERP in Go is pretty easy for simple cases using the gobert lib:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, bert.Atom("foo"))
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

This gives us:

0 0 0 7 131 100 0 3 102 111 111

If we run that through the Ruby lib decoder we get:

> BERT.decode([131, 100, 0, 3, 102, 111, 111].pack("C*"))
 => :foo

(The Ruby bert lib decodes atoms to symbols).

Serialising to BERT dicts in Golang

However, there is a little more effort involved serialising more complex data structures, in particular dicts, as I found out.

You might have thought that you could just pass in a map:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message := map[string]string{"key1": "val1", "key2": "val2"}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, message)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

We get the output:

0 0 0 1 131

Well, that doesn’t work. What you end up with is a one byte long BERP. It seems that gobert doesn’t automatically serialise maps. No problem, we’ll build one up manually. A quick look at the BERT documentation shows the format of a dict:

“Dictionaries (hash tables) are expressed via an array of 2-tuples representing the key/value pairs. The KeysAndValues array is mandatory, such that an empty dict is expressed as {bert, dict, []}. Keys and values may be any term. For example, {bert, dict, [{name, «“Tom”»}, {age, 30}]}.”

So let’s create this special structure manually.

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")}
    message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")}
    keys_and_values := []bert.Term{message1, message2}

    dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, dict)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

The result:

0 0 0 51 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 104 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51

It looks better, but it doesn’t decode, using Ruby:

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 104, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51].pack("C*"))
TypeError: Invalid dict spec, not an erlang list

We’re still missing something. Let’s compare the output of the Ruby bert lib to the output of gobert for the same data structure:

> BERT.encode({:key1 => :val1, :key2 => :val2}).unpack("C*")
 => [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 50, 106]

We’re definitely missing some data in the gobert output.

If you follow along the byte sequences you can see that they start off the same until the 18th byte. In the Ruby output this is ‘108’, or LIST_EXT. In the gobert output it’s 104, a SMALL_TUPLE_EXT. We can see where this difference happens in encode.go in the gobert lib (in the writeTag func):

case reflect.Slice:
    writeSmallTuple(w, v)
case reflect.Array:
    writeList(w, v)

Let’s decode the BERT data to see where the diversion happens in the underlying data structures:

magic| tuple  |  atom   |       bert       |   atom   |    dict
  131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116

We can see that the “bert” and “dict” atoms are encoded the same, but the keys_and_values array is getting encoded as a SMALL_TUPLE_EXT by gobert when we wanted a LIST_EXT. If we look back at the gobert code we can see that the decision to use SMALL_TUPLE_EXT over LIST_EXT is dependent on a slice or array being present. We can use the go “reflect” package to look at the arrays/slices we are creating and see what they are:

package main

import (
    "fmt"
    "reflect"
    "github.com/sethwklein/gobert"
)

func main() {
    array := [2]bert.Term{}
    slice := []bert.Term{}

    array_val := reflect.ValueOf(array)
    slice_val := reflect.ValueOf(slice)
    fmt.Printf("array is a: %v\n", array_val.Kind())
    fmt.Printf("slice is a: %v\n", slice_val.Kind())
}

array is a: array
slice is a: slice

The fix

So, in order to fix our data structure to get gobert to correctly encode the dict we need to change the keys_and_values slice to an array:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")}
    message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")}
    keys_and_values := [2]bert.Term{message1, message2} // Now an array

    dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, dict)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

The result:

0 0 0 55 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 108 0 0 0 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51 106

But more importantly, can we decode the data we encoded?

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51, 106].pack("C*"))
 => {:key1=>:val1, :key2=>:val3}

Yes!

Creating animated gifs with imagemagick

will@willj.net (Will Jessop) — Tue, 24 Apr 2012 10:49:42 +0000

In my last post I linked to some animated gifs I made on my mac. It’s pretty easy if you have imagemagick installed.

Simply get a bunch of images together and use the convert command:

convert -resize 900x -loop 0 -delay 10 DSC_8264.JPG DSC_8265.JPG DSC_8266.JPG DSC_8267.JPG DSC_8268.JPG DSC_8269.JPG DSC_8270.JPG jon_walk.gif

The last argument is the output filename. The options should be self explanatory, but you can check them up in the man page if you want to tweak the values.

Railsberry animated gifs

will@willj.net (Will Jessop) — Mon, 23 Apr 2012 16:05:30 +0000

I put up some photos of Railsberry here, but Flickr doesn’t work with animated gifs, so here they are. Click the images to get the animated versions.

Riding the Railsberry Unicorn:

Jon Leighton getting closer:

Josh Kalderimis misbehaving:

Paula Deen riding things

will@willj.net (Will Jessop) — Thu, 02 Feb 2012 22:00:10 +0000

I made some Paula Deen riding… pictures. Because I can.

Testing http over socket connections with socat

will@willj.net (Will Jessop) — Fri, 02 Dec 2011 22:00:55 +0000

Sometimes I need to test an http server that is listening on a unix socket. It’s really easy to do this using socat, but the socat man page is pretty long. Here it is for anyone who needs it in the future, and me when I inevitably forget.

In this case the server is unicorn, but this will work for any http server listening on a socket, for instance thin. The lines beginning with “–>” are lines I typed (the 4 lines at the start), remove the “–>” when you try this.

$ socat - UNIX-CONNECT:/u/apps/app/shared/sockets/unicorn.sock,crnl
-->GET /session/new HTTP/1.1
-->Host: thehostname.com
-->X-Forwarded-Proto: https
-->
HTTP/1.1 200 OK
Date: Fri, 02 Dec 2011 14:37:23 GMT
Status: 200 OK
Connection: close
Strict-Transport-Security: max-age=31536000
Content-Type: text/html; charset=utf-8
X-UA-Compatible: IE=Edge,chrome=1
ETag: "2346c47c7cb3bc37729e42fc8b20c821"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _x_session=blablabla; path=/; HttpOnly; secure
X-Request-Id: c0a374f460d1b1205df450ab77dd2328
X-Runtime: 0.159219

<!DOCTYPE html>
<html lang="en" data-behavior="wallpaper">
<head>
etc.

For those interested in the relevance of the crnl option at the end of the socket path, this from the man page:

Converts the default line termination character NL (’n’, 0x0a)

to/from CRNL (“rn”, 0x0d0a) when writing/reading on this

channel (example). Note: socat simply strips all CR characters.

Cloud email service price comparison

will@willj.net (Will Jessop) — Wed, 30 Nov 2011 21:27:42 +0000

Larger interactive versions of all the graphs on this page are available here. Update: Added Mailgun to the graphs.

Earlier this year I posted a price comparison between Sendgrid, and the then newly available Amazon SES.

Tim Falls commented on the post saying that Sendgrid had updated their pricing:

Since this post was published, we have released a new pricing structure *and* a new service tier that offers more email for less + a feature set and pricing model that you will find very competitive with SES.

That was back in June, so it’s about time I produced an updated comparison. First, lets look at the difference between the old and new Sendgrid prices:

Comparison of old and new Sendgrid prices

Overall the up-front plan prices, and prices for email over allowance have remained the same, but email allowance within each plan has increased. The exception is the Silver plan where email over allowance has increased by $0.0001/email. New to the lineup is the Lite plan.

More interesting is how these new prices compare to the competitors. I’ve added in Amazon SES, and Postmark too:

Sendgrid, Postmark and Amazon SES price comparison

The most notable differences here are the inclusion of Postmark, and the the Sendgrid Lite plan that shadows Amazon SES. I’d guess this was added purely to compete with Amazon. As in my last post it is hard to see what is going on with smaller numbers of emails being sent, here’s a zoom on the origin:

Price comparison for small numbers of emails sent

Here you can see the Sendgrid Lite plan shadowing Amazon and the Postmark costs heading up rapidly.

Conclusion

It seems Sendgrid have just added an ‘Amazon SES’ plan to pull back any customers that would have chosen SES based on price. It’s probably a good move, and it will allow easy transition into their more ‘premium’ plans if you sign up and later decide to change plan.

Given the advertised features of Postmark compared to the price it seems hard to consider using them. They seem to have some fairly well known customers though, so if anyone has used Postmark leave a comment with how that is working out for you.

So which email cloud provider should you use? Use the graphs I made, but price is only going to be one factor, so check what each provider offers. I’ve linked to all the pricing pages below.

Price sources

Should I switch from Sendgrid to Amazon SES?

will@willj.net (Will Jessop) — Tue, 25 Jan 2011 21:53:07 +0000

Update: A new comparison with updated Sendgrid prices, and Postmark is available here “Sendgrid, Postmark and Amazon SES price comparison”).

Probably yes, at least if price is your main concern and you are just concerned with sending email and not with extras. I wanted to see just how the Amazon SES prices stacked up against (that I am aware of) the next cheapest provider, Sendgrid so I graphed it:

Cost comparison for Amazon SES/Sendgrid

SendGrid can’t be too happy with that, in short at no point is it better to go with SendGrid over SES if you are only taking price into account. Of course SendGrid have value-add over just plain email sending, you decide if it’s worth the premium, but for me the only feature I’d want would be the ‘Whitelabel’ option, and Amazon SES has that included.

Note that you get 2000 emails per day free with Amazon SES if you send from an Amazon EC2 instance, but at this scale there is very little visible difference in cost. I thought it would be useful to take into account the cost of an EC2 instance, even if you have your main server elsewhere you could run your email processing on a micro or small EC2 machine to take advantage of the 2000 free emails per day, here’s a zoom in on the origin:

Cost comparison for Amazon SES/Sendgrid + EC2 instance cost

So, there is no point in spinning up an EC2 instance to take advantage of the 2000 free emails per day.

I will be interested in SendGrid’s response to this. Possibly lowering prices? For me certainly their value-add isn’t worth the extra cost over Amazon SES.

Rewriting URL params in nginx

will@willj.net (Will Jessop) — Sat, 07 Mar 2009 14:30:37 +0000

I came across this problem recently, a customer was moving to Ruby on Rails from another framework/language (.NET I think) and needed to re-write a bunch of URLs. Some needed the query parameters rewriting too. One example was rewriting the old search path, so the old URL:

http://somedomain.com/OldSearchPath.aspx?qry=things&page=4

would become:

http://somedomain.com/search?query=things&page=4

This should be fairly simple except for the qry parameter needed to be changed to query. A bit of googling didn’t turn up much but with some experimentation I came up with this using the pre-populated nginx $args variable:

location /OldSearchPath.aspx {
  if ($args ~* qry=(.+)) {
    set $args query=$1;
  }
}
rewrite ^.+$ /search redirect;

It even leaves the other parameters intact, so the pagination will still work.

Tricks of the Trade

will@willj.net (Will Jessop) — Thu, 12 Apr 2007 19:04:33 +0000

Interesting article:

…I recently asked readers for their “tricks of the trade,” and was amazed by the response. It seems every profession is rich with clever little occupational secrets.

Here is one:

Juggler

With any routine under seven minutes (which is almost all of them), you only really need one thing: a good closer. And there are only two things you really need to know about a great closer. First, it needs to be impressive. That sounds obvious, but most beginning jugglers think “difficult” and “impressive” are synonymous. Your closer must look hard, but there’s no real reason it has to be hard. Secondly, you should intentionally blow your closer on the first two tries. If you get it on the first try it looks too easy, but if you “miss” it a few times it looks harder and builds tension.

A great 'site unavailable' message

will@willj.net (Will Jessop) — Fri, 14 Jul 2006 16:49:02 +0000

Seen on Fish4:

Thank you for coming to Fish4 this morning.

Unfortunately the fish4 website is unavailable due to the failure of a very expensive piece of Sun hardware (Sun 6900). A Sun engineer is at the data centre but didn’t think to bring the replacement part with him.

We hope the sun engineer will have repaired the hardware by 12pm, and will update this estimate if necessary as the work progresses.

Here is a screengrab of the error:

99 bottles of Ruby beer on the wall

will@willj.net (Will Jessop) — Wed, 05 Jul 2006 18:32:24 +0000

A while back a friend and I had a 99 bottles of beer on the wall one-liner competition in Ruby. I just found this in my home directory and so am claiming it as my entry:

def b(c)"#{c} bottle#{c>1?"s":""} of beer on the wall"end;99.downto(1){|x|$>0?x-1:99)}.

It weighs in at a hefty 206 characters and I know it is possible to get it smaller than this, but it is way too much of a time-sink for me to try.

Fixing the 'Memory card error' on the Canon IXUS 400 / S3

will@willj.net (Will Jessop) — Sun, 12 Feb 2006 16:29:42 +0000

The day after my daughter Holly was born my Canon IXUS 400 camera started giving a ‘Memory card error’ and refused to save anything to the compact flash card. I assumed that this meant that there was an error with the memory card so I bought a brand new one and was slightly cheesed off when the IXUS gave the same ‘Memory card error’ message after installing it.

A bit of googling revealed that a number of people have had this error with the Canon IXUS s3 and that the problem was solved by opening the case and removing the internal battery. The specs of the IXUS 400 are similar to that of the S3 and the chassis are practically identical, so I guessed they may well share a fair few components, and therefore problems. I was not that keen on the idea of doing this however until I discovered that Canon wanted to charge me nearly £100 to fix it (including the cost of postage).

It seems a bit cheeky asking for that amount of money to fix a problem that is effectively a design flaw, so I went for the DIY option and it seems to have worked, I have only taken about 20 photos, but as the camera would not allow any to be taken before, this is a significant improvement!

For anyone wanting to do the same here are some photos of what to look for. This one is the back of the camera with the case taken off.

A naked Canon IXUS 400

And this one is of the front of the camera. The battery is under the circled bit of metal film which was stuck down and had to be peeled off to get to the battery which is mounted in a clip just underneath.

The battery is under the metal film which can be peeled back carefully

The battery popped out really easily with a small flat screwdriver. I put the battery back in, put the camera back together and the camera worked (I had to format the memory card first).

Some tips if you are going to try this:

Have a very small flat and cross head screwdriver.
Make a note of where the screws came from, many are different lengths.
Remember which order the body panels and parts came off.

Don’t attempt this if you are not confident in doing so, if you break your camera it is entirely your fault, and this will probably invalidate your warranty (mine had expired already). This whole experience has shaken my confidence in Canon cameras, I will be looking at alternative makes for my next purchase.

New hard disc

will@willj.net (Will Jessop) — Wed, 09 Nov 2005 21:24:40 +0000

I bought a new hard disc. It probably doesn’t work anymore, but then it did only cost me £5.20. I don’t care what anyone says, I like it. It is hanging behind my desk at work.

Here is the eBay description:

14" hard disk platter from the 1970s

From the days when programmers didn’t eat quiche!

This platter from a hard disk drive came from a Hull University mainframe in 1978. The disk is 14 inches (35.5cm) in diameter and 1.5mm thick.

It is made from a hard aluminium alloy coated with magnetic material. One side is covered with a random collection of supermarket price and product labels of the period. Even though it has been used as a rather lethal frisbee, it is still perfectly flat.

And here are some photos: