▄▄▄·▪ ▐ ▄ ▄▄▄ . ▐█ ▄███ •█▌▐█▀▄.▀· ██▀·▐█·▐█▐▐▌▐▀▀▪▄ ▐█▪·•▐█▌██▐█▌▐█▄▄▌ .▀ ▀▀▀▀▀ █▪ ▀▀▀
https://pine32.be - © pine32.be 2024
Welcome! - 64 total posts. [RSS]
A funny little cycle. [LATEST]
#1728318672
Apart from a few minor glitches (which have been fixed) the scraper and scheduler runs fine. The frontend is also coming along nicely, needs a few more pages and then the CSS. And I also need to figure out how to load a dynamic amount of columns from a materialized view, shouldn’t be to hard but I want to make it fault tolerant. I don’t know what I want to do regarding design. But I know some body that maybe wants to help me, fingers crossed.
#1727195369
Python (scraper) rewrite is done. Almost no dependencies now. Reduced the Docker image from 1.2 GB to less then 100MB. Feels a lot better to update and modify to. Now time for the fronted webserver.
beautifulsoup4==4.12.3
requests==2.32.3
python-dotenv==1.0.1
psycopg==3.2.2
psycopg-binary==3.2.2
#1726927465
The amount of cursed SQL that I am writing just to keep it in pure SQL. It would be way faster to just make the query in Python. Anyway… Corap rewrite is coming along nicely.
DO $$
DECLARE
cols text;
query text;
BEGIN
SELECT string_agg(quote_ident(name) || ' text', ', ')
INTO cols
FROM (
SELECT name
FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o
ORDER BY priority DESC
) AS o;
BEGIN
EXECUTE 'DROP MATERIALIZED VIEW IF EXISTS device_analysis_summary';
query := format('
CREATE MATERIALIZED VIEW device_analysis_summary AS
SELECT *
FROM crosstab(
''SELECT d.deveui, da.name, da.value
FROM devices d
LEFT JOIN device_analyses da ON d.deveui = da.device_id
ORDER BY d.deveui, da.name'',
''SELECT name
FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o
ORDER BY priority DESC''
) AS ct(deveui text, %s);
', cols);
EXECUTE query;
EXCEPTION
WHEN OTHERS THEN
RAISE NOTICE 'Error creating materialized view: %', SQLERRM;
ROLLBACK;
RETURN;
END;
END $$;
#1726662585
Time to rewrite Corap finally, starting with the scheduler. The current docker images is more then 1 GB. Going to remove a lot of dependencies. Also going to rewrite the fronted, learned a lot about Golang sins starting that project.