↑ NewBlog
Last week, I finally got the Lagrange geminispace browser installed on Ubuntu 24.04 LTS (as it was missing from its repository) and decided to use the Flatpak from the skyjake.fi developer rather than an unofficial Snap or a direct compilation from the source, as not all of the required source is on Github (though it was a bit depressing and ironic to use a GUI client and bloated Flatpak to serve such minimal text pages). Lagrange↗ and Offpunk↗ are the two clients I tend to use to browse Gemini and Spartan protocol small-Internet (smolnet) pages, but even the Offpunk version in that Ubuntu repo is rather old and not new enough to support Spartan. So the Numbat, while Noble, turned out to be a smolnet desert when compared with the rolling-release Void Linux.
But after installing flatpaks of both Lagrange 1.18.8 and 1.19 and then trying to open a .jpg file from my spartan site at spartan://greatfractal.com, it failed with an "Error Loading Image", which was odd since it worked last month. I then tried using the non-Flatpak version 1.18.5 in Void↗, but it failed with the same error. Strangely, Offpunk in Void displayed the image properly, although it has a neat way of using Chafa↗ to display images using big, colored ANSI blocks/pixels directly in the terminal which confused me at first, as I was not sure if there may have been a progressive JPEG rendering issue at play, in which the client was only getting the first part of the JPEG.
It turned out that this was indeed what was happening, although not perceptible in Offpunk, yet it caused direct failures in Lagrange. Furthermore, in Lagrange, it would sometimes work and display the image without error. Later troubleshooting determined that neither browser was getting the full file, and the file sizes were often smaller than the original, reduced in byte size in semi-random ways. For when the bytes did not match, they often did not match in a stepped, somewhat repeatable pattern.
The byte-loss forced me to reexamine my Lua 5.1 code for my Louia server, which I haven't done in months, and when I was refreshing myself with the terse Luasocket documentation↗ regarding how the TCP client:send() function worked, I found this:
Note: Output is not buffered. For small strings, it is always better to concatenate them in Lua (with the '..' operator) and send the result in one call instead of calling the method several times.
Ha! That was exactly what I was doing for any non-gmi files that were downloaded if they exceeded the size of my user-set DOWNLOADBUFFER constant which I defaulted to 16 KiB, sending several small strings instead...
When the files to download are larger than 16 KiB, I had created a chunking system to just read one chunk from the file into memory, send that chunk, read the next chunk, etc., so as to not require a lot of RAM in my live variables, especially with multiple co-routines in use.
However, that would not explain why one of my 27 K JPEGs that exceeded this limit did work when the 167 K one did not, and they were both progressive JPEGs created in GIMP.
At first, I thought that it was the repeated calls to the client:send() function that were causing a buffering issue and allowing bytes to drop, per the Luasocket note above, but then I thought about what "not buffered" actually meant. In my client:receive() code in Louia 1.0, I turned off the blocking mode in the client to allow my co-routines to work, which meant that I had to handle any lack of buffering in my own code (I knew this), but... I never thought about having to do what with the client:send() function, too.
It turns out that when I turned off blocking for the client, it turned off blocking for both receives and sends, so the sends also needed to be checked for successfully-sent bytes and handled appropriately if they were not successful in one shot due to a delay in the buffer (and TCP buffers tend to be small in Linux by design, too, which helps to prevent TCP-congestion-control-related bufferbloat, which I described in 2021.
Well this was a different situation than just my singular client:receive() function as I had numerous client:send() functions that send both short response strings back to the client, serve .gmi text pages, serve .jpg downloads, etc. So in addition to my singular co-routine function, I had to create a second Lua function called "safesend" to loop and perform this check, then replace all of my client:send(data) functions with safesend(client,data) functions to intercept that call each time.
It is live and working well so far, no bytes seem to be dropped, and there are no failures in opening JPEGs with Lagrange now, so I released the fix in Louia 1.1 earlier today. Interestingly, this issue would have affected all files served by my server, but the reason I didn't notice this was because: (1) my .gmi pages were both short amounts of text and sent in one shot (no chunking applied); (2) it was intermittent and often worked when the TCP buffers were not full; (3) it was masked by JPEG progressive rendering; and (4) I never actually tried adding a file that needed byte-accurate transfer, like a .tgz tarball. I did finally add a .tgz file to the server (the Louia-1.1.tgz source tarball itself, which I renamed to Louia-1_1.tgz on my spartan server since I have STRICTMODE enabled which disables multiple periods in the filename for path-traversal protection). It is amusingly ironic and recursive in that I had an issue in my download code that would have been immediately noticed if I served .tgz files to download, and here I'm posting the fix itself to be downloaded in its first .tgz file.
While I unfortunately had to add another function to my code, it should not noticeably slow down the fast Lua 5.1 or LuaJIT interpreter-compiler, since files smaller than the DOWLOADBUFFER and .gmi files (which make up most of the site) are sent in one shot without chunking, and the first check will often be successful (since those files tend to be smaller and fit in the limited TCP buffers), eliminating the need to loop and do further checks.
I am surprised at the current lack of interest in Gemini and Spartan sites these days, as I post Spartan gemlogs more often than I post HTTP weblogs. Part of this may have to do with the fact that the Latin and Greek names "Gemini" and "Spartan" are also the names of different corporate tech technologies that have nothing to do with these modern, but still-obscure smolnet protocols↗, so you have to do more complex web searches to find them in the noise. Ironically, Google's Gemini 2.0 AI Overviews will, not-so-amusingly, also intercept those searches...
Gemini↗ and Spartan↗ are great small-Internet protocols that are fun to use and browse. Spartan, in particular, is often left out of site aggregators and sometimes gets a bad rap because its lightweight, TLS-free protocol makes creating multi-user sites difficult, but the point is that a protocol this simple does not need to do that (heck, we're trying to get away from the central, big tech sites that dominate the HTTP web). You just serve your own Spartan site, then read and bookmark other sites that interest you like the early WWW. Instead of a "public square" or quadrangle, like how a folksonomy↗ is to a taxonomy, it naturally creates a folksforum of a more-complex polygonal geometry.