How: 3h bus ride from Montanita to Guayaquil (Ecuador).  Modern, clean bus.  Two of us,  a bit of a hangover, but mostly awake (13:00).  Keeping our bags in the overhead storage space.  Nobody was walking around, as far as I remember.  A man tried to help me with putting the bag into the storage space, but I stopped him and did that myself.  After the ride, one of the passengers passed me the bag, when I was standing up.

Outside of the bus, I noticed the bag is too light and too soft — the netbook is gone.  I thought I left it in Montanita, have excluded the possibility of theft on the bus, seemed too difficult… but then I noticed the loss of the cell phone, and everything clicked together… the man with the guinea pig (distraction), the helpful passenger…

Fuck, fuck, fuck.  I´m lucky they left the passport.

Before I figured things out and notified the police (a helpful lady from the tourist information office was kind enough to translate), the bus was gone to another destination… I bet the driver knew what´s going on, but the friendly cops couldn´t do anything more.

I mark this Technical, for it´s a good lesson about backups.  The last one I (probably) can recover is from 2006, and that´s just the most critical stuff (my SSH and PGP keys, etc).

The phrase ¨epic fail¨ somehow doesn´t seem appropriate.

Not sent to builders — this will require more testing and rebuilding of whatever depends on libperl.so and the directories, which usually results in a temporary mess.  I’m not going to do that: EBUSY.

Packages for i686 (just bare perl & perl-dirs) can be found at:

poldek -s http://carme.pld-linux.org/~radek/perl-512-i686/

Notes:

• suidperl has been removed
• microperl is currently broken (does anyone care?  if yes, speak up)
• @INC order has been changed to the one PLD has since v5.8.0 (site, vendor, priv), therefore rendering the -INC patch obsolete; please be testing it (perl -le 'map print, @INC')

See perl5120delta.pod for the release notes.  Official release information is here.

All domain registrars I know of provide the service, but:

1. the interface is usually too crude and simplistic (I do want the TXT, SRV, IPv6 support and overall fine-grained control over the zone’s content),
2. I want to manage the domains from different registrars in one place,
3. I don’t trust them; they’re in the business of making money from registering domains, not providing a reliable DNS service.

FreeDNS has been the perfect solution so far: operated by Piotr `Beeth’ Kucharski, whom I trust, and hosted on SGH’s servers, an entity I have trusted.

Recently, SGH has demanded a copy of the database, and when Piotr refused, they have blocked his access and took over the service, claiming rights to the data.  Trust FAIL.

If you’re keeping your domains there, it’s time to find an alternative.  Frankly, I don’t know of one at the moment.

Update: FreeDNS::42.

Source link (in Polish) with information about the issue: http://42.pl/freedns.html; the comments on Wykop are also useful.  The only thing I can infer from the discussion is that someone at SGH is up to something evil.

Enter XChat and its Perl scripting capability.

I wrote a simple plugin to look up the IP address in the Maxmind GeoIP lookup database, you can find it here: xchat-geo.pl.  See “perldoc xchat-geo.pl” for the installation instructions.  Usage:

/geo some_nickname
[17:33] ==> some_nickname | US San Francisco CA 37.7525 -122.3194  n=somenick@some.host

\set trashindexes '( select s.schemaname as sch, s.relname as rel, s.indexrelname as idx, s.idx_scan as scans, pg_size_pretty(pg_relation_size(s.relid)) as ts, pg_size_pretty(pg_relation_size(s.indexrelid)) as "is" from pg_stat_user_indexes s join pg_index i on i.indexrelid=s.indexrelid left join pg_constraint c on i.indrelid=c.conrelid and array_to_string(i.indkey, '' '') = array_to_string(c.conkey, '' '') where i.indisunique is false and pg_relation_size(s.relid) > 1000000 and s.idx_scan < 100000 and c.confrelid is null order by s.idx_scan asc, pg_relation_size(s.relid) desc )'
\set missingindexes '( select src_table, dst_table, fk_name, pg_size_pretty(s_size) as s_size, pg_size_pretty(d_size) as d_size, d from ( select distinct on (1,2,3,4,5) textin(regclassout(c.conrelid)) as src_table, textin(regclassout(c.confrelid)) as dst_table, c.conname as fk_name, pg_relation_size(c.conrelid) as s_size, pg_relation_size(c.confrelid) as d_size, array_upper(di.indkey::int[], 1) + 1 - array_upper(c.conkey::int[], 1) as d from pg_constraint c left join pg_index di on di.indrelid = c.conrelid and array_to_string(di.indkey, '' '') ~ (''^'' || array_to_string(c.conkey, '' '') || ''( |$)'') join pg_stat_user_tables st on st.relid = c.conrelid where c.contype = ''f'' order by 1,2,3,4,5,6 asc) mfk where mfk.d is distinct from 0 and mfk.s_size > 1000000 order by mfk.s_size desc, mfk.d desc )'

The first one displays unused indexes.  Why would you care?

1. wasted RAM and disk space,
2. longer restore,
3. slower INSERT, UPDATE and DELETE statements.

Running this is useless after a fresh restore (or after calling pg_stat_reset(), or — in case you’re running something ancient, like 8.2 or older, with stats_reset_on_server_start=on — after restarting the server).

It excludes unique indexes and the ones on columns being foreign keys (used when deleting / updating rows in the tables the FK points to).

Tables smaller than 1MB are excluded as well.

Example usage:

radek=# :trashindexes limit 5;
  sch   |  rel   |            idx            | scans |   ts    |   is
--------+--------+---------------------------+-------+---------+---------
 public | tracks | i_tracks_theme            |     0 | 15 MB   | 4664 kB
 public | tracks | i_tracks_album_theme      |     0 | 15 MB   | 9696 kB
 public | tracks | i_tracks_theme_album      |     0 | 15 MB   | 6616 kB
 public | albums | i_albums_theme            |     0 | 5296 kB | 1336 kB
 public | albums | i_albums_collection_theme |     0 | 5296 kB | 1664 kB
 public | albums | i_albums_theme_collection |     0 | 5296 kB | 1664 kB
(5 rows

Columns: schema name, table name, index name, number of index scans, table size, index size.

The second one suggests missing indexes.

Imagine a large table (large enough that you wouldn’t like any sequential scans on it) with a foreign key to another table.  Now, when you DELETE a referenced row from the second table, Postgres has to enforce the constraint somehow (raise an exception, delete the row from the first table or set the FK column(s) to NULL).  For that, the first table has to be scanned somehow, to find out if any action is needed.

It’s not worth it if the referencing table is small, or if it’s not-so-small but always in RAM, or if the rows in the second table are never updated or deleted.  The query omits small tables using an arbitrary number; tweak for your needs.

Example usage:

msawi11=# set search_path to "$user"; -- demonstrate lack of separate columns for schemas
SET
msawi11=# select * from :missingindexes x where src_table ~ 'promoc|usl';
    src_table    |     dst_table     |         fk_name         | s_size  |   d_size   | d
-----------------+-------------------+-------------------------+---------+------------+---
 public.promocje | public.operatorzy | promocje_fk_operator_id | 4736 kB | 8192 bytes |
 public.uslugi   | public.taryfy     | uslugi_fk_taryfa_id     | 1032 kB | 552 kB     | 2
(2 rows

Last three columns: source table size, destination table size, distance.  If “distance” is not null, it means there already is an index starting with the FK columns; “distance” constains the number of extra columns.

In this example, “uslugi.taryfa_id” is a FK to the “taryfy” table.  There is an unique index on uslugi (taryfa_id, foo, bar), so the distance = 2.

The textin(regclassout(oid)) calls are there for Pg 8.2 compatibility (lacks a regclass::text cast).  Indented query looks like this:

select src_table, dst_table, fk_name, pg_size_pretty(s_size) as s_size, pg_size_pretty(d_size) as d_size, d
    from (
        select
                distinct on (1,2,3,4,5)
                textin(regclassout(c.conrelid))  as src_table,
                textin(regclassout(c.confrelid)) as dst_table,
                c.conname                        as fk_name,
                pg_relation_size(c.conrelid)     as s_size,
                pg_relation_size(c.confrelid)    as d_size,
                array_upper(di.indkey::int[], 1) + 1 - array_upper(c.conkey::int[], 1) as d
        from pg_constraint c
        left join pg_index di on di.indrelid = c.conrelid
                and array_to_string(di.indkey, ' ') ~ ('^' || array_to_string(c.conkey, ' ') || '( |$)')
        join pg_stat_user_tables st on st.relid = c.conrelid
        where c.contype = 'f'
        order by 1,2,3,4,5,6 asc
    ) mfk
    where mfk.d is distinct from 0 and mfk.s_size > 1000000
    order by mfk.s_size desc, mfk.d desc;

Note: don’t blindly add indexes basing on the results of this query.

A new beefy machine has been bought, postgresql.conf got reviewed independently by several seasoned DBAs, some performance tests were run… and we figured it’s going to be OK.

On the big day, the database started swapping and grinded to a halt.  After a long night of adding hacks to the application and more RAM to the machine, we got it to +- working state, but the performance was far from acceptable.

Then, Szymon the Project Manager has asked the Right Question™: why are the idle backends not freeing the memory, actually?  This got me back to studying the config file and I noticed “temp_buffers = 8GB”.  We all missed it before — the default is also 8, but MB, not GB.  Also, I thought the allocated memory is being freed after the temporary table is dropped…

Wrong.  The temp_buffers memory is not being freed.  It’s being kept until the end of the session.

So, when the users log in, they usually run a heavy “what’s going on today” query, which includes a function creating a large temporary table.  Each backend has been allocating ~1GB of RAM for it and keeping it through the day.

Epic fail, but with a happy ending — everything is humming nicely now. :-)

• doesn’t help much with 8.3 and 8.2 databases,
• doesn’t let me run aggregates on these,
• the view is cluttered by table comments,
• can’t be sorted by size.

So, my ~/.psqlrc contains these two lines:

$ tail -n2 .psqlrc
\set rtsize '(select table_schema, table_name, pg_relation_size( quote_ident( table_schema ) || \'.\' || quote_ident( table_name ) ) as size, pg_total_relation_size( quote_ident( table_schema ) || \'.\' || quote_ident( table_name ) ) as total_size  from information_schema.tables where table_type = \'BASE TABLE\' and table_schema not in (\'information_schema\', \'pg_catalog\') order by pg_relation_size( quote_ident( table_schema ) || \'.\' || quote_ident( table_name ) ) desc, table_schema, table_name)'
\set tsize '(select table_schema, table_name, pg_size_pretty(size) as size, pg_size_pretty(total_size) as total_size from (:rtsize) x order by x.size desc, x.total_size desc, table_schema, table_name)'

Both queries return the same data; “:rtsize” outputs raw numbers (mnemonic: raw table size), and “:tsize” formats them with pg_size_pretty().  Usage and the benefits will be best illustrated by examples; this one shows the 5 largest tables:

redmine=# :tsize limit 5;
 table_schema |      table_name       |  size  | total_size
--------------+-----------------------+--------+------------
 public       | journals              | 192 kB | 360 kB
 public       | issues                | 128 kB | 248 kB
 public       | wiki_content_versions | 48 kB  | 304 kB
 public       | journal_details       | 48 kB  | 152 kB
 public       | attachments           | 32 kB  | 88 kB
(5 rows)

Sum of table sizes from the “log” schema, without pricing-related tables; note the “x” after “:rtsize” — since the macro is expanded to “( select ... )“, it has to be aliased:

msawi11=# select pg_size_pretty( sum(size)::int ), pg_size_pretty( sum(total_size)::int )
          from :rtsize x where table_schema='log' and table_name !~ 'ceny';
 pg_size_pretty | pg_size_pretty
----------------+----------------
 18 MB          | 32 MB
(1 row)

Useful.

It’s a pity PostgreSQL doesn’t come with such views in the standard installation; psql’s code would have been much simpler, some incompatibilities of \X commands could have been avoided, and other client application developers wouldn’t have to copy/paste code from “psql -E” output…

Update: more ~/.psqlrc tricks.

First, see the documentation on DISTINCT ON; it’s a PostgreSQL-only extension.

Consider a table structure: a collection has many albums, an album has many tracks.  Both an album and a track can have an optional theme attribute.  The task: find the list of tracks from a (single) given collection, and for each track, suggest one other album with the same theme from the same collection.

A full example with create table / create index statements is here: pg-view-distinct-on.sql.  This example could be solved with a GROUP BY instead of a DISTINCT ON (), but in my real query it wouldn’t work so great.

Let’s start with a simple view, giving us all the tracks from all collections:

create or replace view v_test as
    select a.collection_id, c.id as c_id, t.*
    from tracks      t
    join albums      a on a.id = t.album_id
    join collections c on c.id = a.collection_id
    left join albums r on
            r.id <> a.id and r.id <> t.album_id  -- we don't want the same album
        and r.collection_id = a.collection_id
    --  and r.collection_id = c.id               -- or would this be better...?
        and r.theme = t.theme;

This is very nice, but duplicates the “track” rows, if more than one album with the same theme is found.  My approach looked like this:

create or replace view v_test_bad as
    select distinct on (id) *
    from v_test
    order by id;

Now, “select * from v_test_bad where collection_id = 42” gives the correct results.  The problem is, it takes way too long — see the query plan.  Indexes are not used at all.

Why, wtf?

The magic answer is: since we’re sorting the result (order by), then picking the first row from the group defined by the list of columns we’re sorting with (order by id, distinct on (id)), the planner doesn’t push the WHERE condition downwards, to the v_test view.  It doesn’t figure out, that pushing (using) it there won’t make any difference to the results.

Since there is no usable search condition in the v_test view, it has to do sequential scans, and then filter out the results… on bigger tables and longer sorting conditions that would take hours to weeks.

Solution:

create or replace view v_test_good as
    select distinct on (id, c_id, collection_id) *
    from v_test
    where collection_id = c_id
    order by id, c_id, collection_id;

Adding columns to the “order by” and “distinct on” allows it to push the conditions on these columns “downwards”.  The “collection_id = c_id” part is a hint, which lets us avoid specifying both columns in the outer WHERE to get index scans on both [edit: now I'm not sure where I saw benefit in that].  The query time is now 4 miliseconds instead of 4 seconds; compare the new plan for “select * from v_test_good where collection_id = 12“.

Obvious, when you think about it.  Not so obvious with a complex query joining a dozen tables, though.

Example calls:

radek=# select build_insert( 'foo', 'a=>1, b=>2009-01-02, c=>"foo bar baz", d=>NULL'::hstore );
                                    build_insert
-------------------------------------------------------------------------------------
 INSERT INTO foo ( a, b, c, d ) VALUES ( '1', '2009-01-02', 'foo bar baz', DEFAULT )
(1 row)

radek=# select build_update( 'foo', 'a=>1, b=>2009-01-02, c=>null'::hstore, '{a,b}'::text[] );
                     build_update
------------------------------------------------------
 UPDATE foo SET c=NULL WHERE a='1' AND b='2009-01-02'
(1 row)

radek=# select build_delete( 'foo', 'a=>1, b=>2009-01-02, c=>null'::hstore, '{c}'::text[] );
          build_delete
---------------------------------
 DELETE FROM foo WHERE c IS NULL
(1 row)

You should quote the table / schema name (the first argument) with quote_ident(), like this:

quote_ident( 'public' ) || '.' || quote_ident( 'foo' )

Tests and comments most welcome; here’s the code:

create or replace function build_insert( _table text, _record hstore ) returns text as $$
declare
	_sql	text;
	_values	text;
	_tmp	record;
begin
	for _tmp in ( select * from each( _record ) order by key ) loop
		_sql := coalesce( _sql || ', ', 'INSERT INTO ' || _table || ' ( ' )
			|| quote_ident( _tmp.key );
		_values := coalesce( _values || ', ', 'VALUES ( ' )
			|| coalesce( quote_literal( _tmp.value ), 'DEFAULT' );
	end loop;
	return _sql || ' ) ' || _values || ' )';
end;
$$ language plpgsql immutable strict;

comment on function build_insert( text, hstore ) is
  'build an INSERT statement from the hstore, the table name should be quoted';

create or replace function build_update( _table text, _record hstore, _pk text[] ) returns text as $$
declare
	_sql	text;
	_where	text;
	_tmp	record;
begin
	for _tmp in ( select * from each( _record - _pk ) order by key ) loop
		_sql := coalesce( _sql || ', ', 'UPDATE ' || _table || ' SET ' )
			|| quote_ident( _tmp.key ) || '='
			|| coalesce( quote_literal( _tmp.value ), 'NULL' );
	end loop;
	for _tmp in ( select * from each( _record => _pk ) order by key ) loop
		_where := coalesce( _where || ' AND ', '' )
			|| quote_ident( _tmp.key )
			|| coalesce( '=' || quote_literal( _tmp.value ), ' IS NULL' );
	end loop;
	return _sql || ' WHERE ' || _where;
end;
$$ language plpgsql immutable strict;

comment on function build_update( text, hstore, text[] ) is
  'build an UPDATE statement, pass the list of primary key columns in the last argument';

create or replace function build_delete( _table text, _record hstore, _pk text[] ) returns text as $$
declare
	_where	text;
	_tmp	record;
begin
	for _tmp in ( select * from each( _record => _pk ) order by key ) loop
		_where := coalesce( _where || ' AND ', '' )
			|| quote_ident( _tmp.key )
			|| coalesce( '=' || quote_literal( _tmp.value ), ' IS NULL' );
	end loop;
	return 'DELETE FROM ' || _table || ' WHERE ' || _where;
end;
$$ language plpgsql immutable strict;

comment on function build_delete( text, hstore, text[] ) is
  'builds a DELETE statement with WHERE columns specified as the last argument';

Edit: noted the need for quote_ident() and hstore-new.

Result, after the reboot, finishing the upgrade and rpm -Va:

That’s right, it’s just gkrellm and nothing else (not counting the obligatory useless, utterly annoying gnome-keyring-daemon, the volume control applet and pulseaudio).  Same with the “Gnome failsafe” session.

So, business as usual; it’s not like I expected it to work anyway.

Ctrl+Alt+F1, “DISPLAY=:0 xterm” and I could run gnome-panel and metacity.  No luck with working Alt+Tab or the gnome-terminal:

$ gnome-terminal
Xlib:  extension "Generic Event Extension" missing on display ":0.0".
[...]
Failed to contact the GConf daemon; exiting.

The stunning result:

Incomplete window controls, missing icons. Meh.

To waste time for debugging, or switching; that is the question…

Anyway, I’m looking forward to 2.28.  We had the locked keyboards already, now the broken desktop… screen blinking in rainbow colours and “cat /dev/urandom > /dev/dsp” is likely to be next, I suppose.