Prev: FAQ 4.15 How can I take a string and turn it into epoch seconds?
Next: FAQ 4.30 How do I capitalize all the words on one line?
From: PerlFAQ Server on 15 Jun 2010 12:00
This is an excerpt from the latest version perlfaq6.pod, which
comes with the standard Perl distribution. These postings aim to
reduce the number of repeated questions as well as allow the community
to review and update the answers. The latest version of the complete
perlfaq is at http://faq.perl.org .

--------------------------------------------------------------------

6.23: How can I match strings with multibyte characters?

Starting from Perl 5.6 Perl has had some level of multibyte character
support. Perl 5.8 or later is recommended. Supported multibyte character
repertoires include Unicode, and legacy encodings through the Encode
module. See perluniintro, perlunicode, and Encode.

If you are stuck with older Perls, you can do Unicode with the
"Unicode::String" module, and character conversions using the
"Unicode::Map8" and "Unicode::Map" modules. If you are using Japanese
encodings, you might try using the jperl 5.005_03.

Finally, the following set of approaches was offered by Jeffrey Friedl,
whose article in issue #5 of The Perl Journal talks about this very
matter.

Let's suppose you have some weird Martian encoding where pairs of ASCII
uppercase letters encode single Martian letters (i.e. the two bytes "CV"
make a single Martian letter, as do the two bytes "SG", "VS", "XX",
etc.). Other bytes represent single characters, just like ASCII.

So, the string of Martian "I am CVSGXX!" uses 12 bytes to encode the
nine characters 'I', ' ', 'a', 'm', ' ', 'CV', 'SG', 'XX', '!'.

Now, say you want to search for the single character "/GX/". Perl
doesn't know about Martian, so it'll find the two bytes "GX" in the "I
am CVSGXX!" string, even though that character isn't there: it just
looks like it is because "SG" is next to "XX", but there's no real "GX".
This is a big problem.

Here are a few ways, all painful, to deal with it:

# Make sure adjacent "martian" bytes are no longer adjacent.
$martian =~ s/([A-Z][A-Z])/ $1 /g;

print "found GX!\n" if $martian =~ /GX/;

Or like this:

@chars = $martian =~ m/([A-Z][A-Z]|[^A-Z])/g;
# above is conceptually similar to: @chars = $text =~ m/(.)/g;
#
foreach $char (@chars) {
print "found GX!\n", last if $char eq 'GX';
}

Or like this:

while ($martian =~ m/\G([A-Z][A-Z]|.)/gs) { # \G probably unneeded
print "found GX!\n", last if $1 eq 'GX';
}

Here's another, slightly less painful, way to do it from Benjamin
Goldberg, who uses a zero-width negative look-behind assertion.

print "found GX!\n" if $martian =~ m/
(?<![A-Z])
(?:[A-Z][A-Z])*?
GX
/x;

This succeeds if the "martian" character GX is in the string, and fails
otherwise. If you don't like using (?<!), a zero-width negative
look-behind assertion, you can replace (?<![A-Z]) with (?:^|[^A-Z]).

It does have the drawback of putting the wrong thing in $-[0] and $+[0],
but this usually can be worked around.



--------------------------------------------------------------------

The perlfaq-workers, a group of volunteers, maintain the perlfaq. They
are not necessarily experts in every domain where Perl might show up,
so please include as much information as possible and relevant in any
corrections. The perlfaq-workers also don't have access to every
operating system or platform, so please include relevant details for
corrections to examples that do not work on particular platforms.
Working code is greatly appreciated.

If you'd like to help maintain the perlfaq, see the details in
perlfaq.pod.
 | 
Pages: 1
Prev: FAQ 4.15 How can I take a string and turn it into epoch seconds?
Next: FAQ 4.30 How do I capitalize all the words on one line?