MB

▄▄▄·▪ ▐ ▄ ▄▄▄ . ▐█ ▄███ •█▌▐█▀▄.▀· ██▀·▐█·▐█▐▐▌▐▀▀▪▄ ▐█▪·•▐█▌██▐█▌▐█▄▄▌ .▀ ▀▀▀▀▀ █▪ ▀▀▀

seq 1 4 | xargs -I% sh -c 'echo "Fetching page %" >&2; curl -s "https://www.last.fm/charts/weekly?page=%"' \ | pup 'td.weeklychart-name a.weeklychart-cover-link attr{href}' \ | sed 's|^|https://www.last.fm|' \ | xargs -I {} sh -c 'echo "Processing artist URL: $1" >&2; curl -s "$1" | pup "h1.header-new-title, div.header-new-background-image"' _ {} \ | pup 'json{}' \ | jq '[ .. | objects | select(.tag=="h1" or .itemprop=="image") ] as $list | [ range(0; $list|length; 2) | { name: $list[.].text, img: $list[. + 1].content } ]' \ | tee artists.json \ | jq -r '.[].img' \ | xargs -n 1 sh -c 'echo "Downloading image: $1" >&2; wget -q "$1"' _ && \ jq -j '.[] | .img |= (split("/")[-1]) \ | @json, "\u0000"' artists.json \ | xargs -0 -I{} sh -c 'img=$(printf "%s" "$1" \ | jq -r .img); name=$(printf "%s" "$1" \ | jq -r .name); echo "Adding $name"; curl -s -X POST -H "Authorization: Bearer $TOKEN" -F "img=@$img" -F "json={\"name\":\"$name\"}" http://localhost:3000/api/artists/add' _ {}

func (Image) Hooks() []ent.Hook { return []ent.Hook{ hook.On( func(next ent.Mutator) ent.Mutator { return hook.ImageFunc( func(ctx context.Context, m *gen.ImageMutation) ( ent.Value, error) { _, hasRelease := m.ReleaseID() _, hasArtist := m.ArtistID() if hasRelease == hasArtist { if hasRelease { return nil, fmt.Errorf("but both were provided") } else { return nil, fmt.Errorf("neither was provided") } } return next.Mutate(ctx, m) }) }, ent.OpCreate|ent.OpUpdate|ent.OpUpdateOne, ), } }

I have spent far too much time designing a custom ID type for my current project. I wanted to use it as the primary key in a SQLite database, which imposes some constraints. Specifically, the ID must fit within 63 bits, since SQLite only supports signed integers and I want to avoid negative values. (Technically, negative IDs would work, but they’re not ideal.)

You might be thinking, “Why not just use a BLOB as the primary key? That gives you much more flexibility.” And that’s a fair point, but I am intentionally avoiding that because of how SQLite handles its hidden rowid. When you use an integer as the primary key, SQLite internally aliases it to the rowid, which makes operations significantly faster. Using a BLOB would remove that performance advantage and make the database larger.

So the next step is choosing the bit layout. The first bit is unused to prevent negative values. Then I went with 43 bits for a Unix millisecond timestamp. This gives me 278 years of ranges, should be plenty. Using the default Unix epoch this will work until the year 2248. It will outlive me so that is more than enough.

The remaining 20 bits are random, which gives 1_048_576 possible values. I am using random values because I don’t want to keep track of state (as with an autoincrement), and my current system can handle collisions. It is still possible to swap approaches down the road while keeping the already generated IDs. 1_048_576 Sounds like a lot, but this gives a 1% chance of a collision occurring when only generating 146 IDs. Then again, those IDs would need to be generated within the same millisecond. I am not expecting that much volume.

Bit  | 63 (MSB) | 62 ... 20 | 19 ... 0 |        some        1JTWRZPBJ4DSE
-----|----------|-----------|----------|        example     1JTWRZSPA1NBS
Use  | Unused   | Timestamp | Random   |        IDs         1JTWRZTQSE9G6
Size | 1 bit    | 43 bits   | 20 bits  |        ->          1JTWRZVY5R7RT

The reason for using a timestamp in the leading bits is to minimize B-tree rebalances. As time advances, the generated IDs grow in sequence, allowing the B-tree to insert new entries without reorganizing older pages. By contrast, a completely random primary key (like a UUID v4) forces the B-tree to rebalance frequently, which can significantly degrade database performance.

Finally, the string representation: I chose Crockford’s Base32 (without the check digit). Just 13 characters to represent a int64. To me it’s practically perfect from a technical standpoint, and I like how the IDs look. I know aesthetics shouldn’t matter, but this is my project. So I set the rules, and I want things to look cool and and have some aesthetic appeal. Looks way better than those stupid UUIDs.

One final note, please store your IDs in a binary presentation (BLOB or integer). It hurts me every time I seed a ID stored as its string representation. It is way slower and waists storage. It mainly happens with UUIDs, most people don’t realize it actually is a binary ID and not a string. Even the spec states it but I guess people just don’t read it.

Still alive, just busy. First post of the year… yay I guess.

I have been working on my fork of Navidrome that will have some audio and music analysis with the help of Essentia, the MVP is almost done. Because fuck Spotify, I am done with their bullshit. But I still want my cool data, so I am making it myself, all open source ofcourse.

Navidrome is written in Golang but most analysis libraries are written in C/C++ or Python, so gRPC was the solution because I am not writing a wrapper. My first time working with gRPC, it’s really nice once you get it all set up. But the setup can be a pain, like WTF are .pyi files. I have never seen them before, they are interface files so you have your types available (this is because Python Protobuf does some weird runtime C thing that processes the proto files or something). It has been nice to use, apart from setup. The end-to-end types are just a huge plus, you just know that they will work.

Python multiprocessing has also been a journey. Tested all types of pools but they always just deadlocked or didn’t run. In the end I just made my own worker pool with each worker having its own process. And they share a multiprocessing-safe queue for input and output. Sort of like I would do in Golang with channels instead of queues. And this dead simple approach worked first try of course, after everything I tried. But I am just happy that it works now. Still feels weird to see python use almost 100% of your CPU on all cores.

One year anniversary coming up for MB.

for chunk := range slices.Chunk(allSimpleTracks, 100) { ids := make([]spotify.ID, len(chunk)) for i, a := range chunk { ids[i] = a.ID } f, err := client.GetAudioFeatures(ctx, ids...) if err != nil { return nil, err } fullTracks := make([]*spotify.FullTrack, len(ids)) for subChunk := range slices.Chunk(ids, 50) { full, err := client.GetTracks(ctx, subChunk, spotify.Limit(50)) if err != nil { return nil, err } fullTracks = append(fullTracks, full...) } for i := range len(ids) { allTracks[i] = &FullerTrack{ Track: fullTracks[i], Features: f[i], } } }

DO $$ DECLARE cols text; query text; BEGIN SELECT string_agg(quote_ident(name) || ' text', ', ') INTO cols FROM ( SELECT name FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o ORDER BY priority DESC ) AS o; BEGIN EXECUTE 'DROP MATERIALIZED VIEW IF EXISTS device_analysis_summary'; query := format(' CREATE MATERIALIZED VIEW device_analysis_summary AS SELECT * FROM crosstab( ''SELECT d.deveui, da.name, da.value FROM devices d LEFT JOIN device_analyses da ON d.deveui = da.device_id ORDER BY d.deveui, da.name'', ''SELECT name FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o ORDER BY priority DESC'' ) AS ct(deveui text, %s); ', cols); EXECUTE query; EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'Error creating materialized view: %', SQLERRM; ROLLBACK; RETURN; END; END $$;