Finding duplicate entries in MySQL database

On upgrading my Moodle installation from version 1.9 to 2.1 the upgrade process (initiated from command line) exited with the following error:

!!! Error reading from database !!!
!! Illegal mix of collations (utf8_general_ci,IMPLICIT) and (utf8_unicode_ci,IMPLICIT) for operation '='
SELECT po.id AS oldpage_id, po.pagename AS oldpage_pagename, po.version, po.flags,
                   po.content, po.author, po.userid AS oldpage_userid, po.created, po.lastmodified, po.refs, po.meta, po.hits, po.wiki,
                   p.id AS newpage_id, p.subwikiid, p.title, p.cachedcontent, p.timecreated, p.timemodified AS newpage_timemodified,
                   p.timerendered, p.userid AS newpage_userid, p.pageviews, p.readonly, e.id AS entry_id, e.wikiid, e.course AS entrycourse,
                   e.groupid, e.userid AS entry_userid, e.pagename AS entry_pagename, e.timemodified AS entry_timemodified,
                   w.id AS wiki_id, w.course AS wiki_course, w.name, w.summary AS summary, w.pagename AS wiki_pagename, w.wtype,
                   w.ewikiprinttitle, w.htmlmode, w.ewikiacceptbinary, w.disablecamelcase, w.setpageflags, w.strippages, w.removepages,
                   w.revertchanges, w.initialcontent, w.timemodified AS wiki_timemodified,
                   cm.id AS cmid
              FROM wiki_pages_old po
              LEFT OUTER JOIN wiki_entries_old e ON e.id = po.wiki
              LEFT OUTER JOIN wiki w ON w.id = e.wikiid
              LEFT OUTER JOIN wiki_subwikis s ON e.groupid = s.groupid AND e.wikiid = s.wikiid AND e.userid = s.userid
              LEFT OUTER JOIN wiki_pages p ON po.pagename = p.title AND p.subwikiid = s.id
              JOIN modules m ON m.name = 'wiki'
              JOIN course_modules cm ON (cm.module = m.id AND cm.instance = w.id)

Summed up — an “Illegal mix of collations” was occurring while querying the table wiki_pages_old. A quick search found a possible solution. Unfortunately (for now) I’m not really sure if it helped, since the error persisted, but it is worth of making a note of it until being sure (by migrating the production server).

So I went in with phpMyAdmin and changed the collation on the table manually, which didn’t help either, since it didn’t change the collation of the individual fields. So I changed the fields’ collation manually, except for one field, which — after changing the collation — started to report a duplicate entry. After some trial and error I’ve found a solution.

From the table dump file I’ve deleted the offending key and after importing the table into the database I’ve searched for the offending duplicate entries.

The following MySQL query, from MySQL forum, finds the duplicate entries.

SELECT t1.* FROM t1 INNER JOIN (
SELECT colA,colB,COUNT(*) FROM t1 GROUP BY colA,colB HAVING COUNT(*)>1) as t2
ON t1.cola = t2.cola and t1.colb = t2.colb;

In my case the upper query changes into this:

SELECT wiki_pages_old.* FROM wiki_pages_old INNER JOIN (
SELECT pagename,version,wiki,COUNT(*) FROM wiki_pages_old GROUP BY pagename,version,wiki HAVING COUNT(*)>1) as wiki_t2
ON wiki_pages_old.pagename = wiki_t2.pagename and wiki_pages_old.version = wiki_t2.version and wiki_pages_old.wiki = wiki_t2.wiki;

I’ll write some more details after migrating the production server.