https://pine32.be - © pine32.be 2024
Welcome! - 64 total posts. [RSS]
A funny little cycle. [LATEST]


Search 64 posts with 31 unique tags


#1731188466


[ dev | meta_raid ]

I am making a scraper for Spotify meta data. My testing numbers indicates that I could scrape 100% of Spotify in less then a week, something feels wrong.

INFO Stats per minute id=0 request=204 tracks=2451
INFO Stats per minute id=2 request=193 tracks=2086
INFO Stats per minute id=1 request=212 tracks=2392

#1730630725


[ dev | golang | meta_raid ]

Golangs new integrators came in handy for request pagination. I know the code is not optimal but is very readable and it is just for a proof of concept. I am try to get Spotify metadata in bulk. Hopefully I won’t get IP banned, fingers crossed.

for chunk := range slices.Chunk(allSimpleTracks, 100) {
	ids := make([]spotify.ID, len(chunk))
	for i, a := range chunk {
		ids[i] = a.ID
	}

	f, err := client.GetAudioFeatures(ctx, ids...)
	if err != nil {
		return nil, err
	}

	fullTracks := make([]*spotify.FullTrack, len(ids))
	for subChunk := range slices.Chunk(ids, 50) {
		full, err := client.GetTracks(ctx, subChunk, spotify.Limit(50))
		if err != nil {
			return nil, err
		}
		fullTracks = append(fullTracks, full...)
	}
	for i := range len(ids) {
		allTracks[i] = &FullerTrack{
			Track:    fullTracks[i],
			Features: f[i],
		}
	}
}