How to distinguish human languages by letter frequency histogram?

in , / with No comments /

How to find without dictionaries a language of a text sample? It can be accomplished by comparing frequencies of letters in a sample and in known languages. For example in polish 'a' letter is about 0.0551146 of all letters, in french it's 0.049458 and in german 0.0434701.

I created a small program in C++ that takes as an argument path to file with unknown language and print, how this language differs from languages that he knowns. The lowest result is the best match. Frequencies of letters in known languages are computed from files in samples directory.

Below is the output for checked sample and this sample (in French):

bash-3.2$ ./a.out test3.txt 
difference between Polish language: 0.0965482
difference between French language: 0.0442431
difference between German language: 0.0945827

A simple trick to avoid "goto" statement

in , , / with 2 comments /

Error handling in C is sometimes done by using goto and labels. One of examples in the Linux kernel:

static struct avc_node *avc_alloc_node(void)
{
    struct avc_node *node;

    node = kmem_cache_zalloc(avc_node_cachep, GFP_ATOMIC);
    if (!node)  
        goto out;

    INIT_RCU_HEAD(&node->rhead);
    INIT_HLIST_NODE(&node->list);
    avc_cache_stats_incr(allocations);

    if (atomic_inc_return(&avc_cache.active_nodes) > avc_cache_threshold)
        avc_reclaim_node();
   
out:
    return node;
}

There's a trick with the usage of do/while construction that allows to write the same without using goto label. The idea is presented on below:

// with goto label
if (!bar()) {
    goto error;
}
someValue = baz();
error:
// with do/while construction
do {
    if (!bar()) {
        break;
    }
    someValue = baz();
} while (0);

Easier error handling by using automatic code generation [Perl interpreter]

in , , / with 2 comments /

Perl interpreter has an interesting mechanism to checks if function arguments have correct values. Below is its example from util.c:

char *
Perl_delimcpy(register char *to, 
      register const char *toend, 
      register const char *from, 
      register const char *fromend, 
      register int delim, I32 *retlen)
{
    register I32 tolen;

    PERL_ARGS_ASSERT_DELIMCPY;

    /* ... */
}

Body of PERL_ARGS_ASSERT_DELIMCPY is in proto.h:

#define PERL_ARGS_ASSERT_DELIMCPY   \
    assert(to); assert(toend); assert(from); assert(fromend); assert(retlen)

What makes those macros interesting, is that they are automatically generated by a perl script (regen.pl). They are created from a data stored in embed.fnc, a sample entry of this file is presented below:

: Used in util.cp
op  |void   |example_function   |NN const char *name|STRLEN len|

The script needs to be executed after any change in embed.fnc.

Interesting and weird.

Remote control of devices by using PC Speaker

in , , / with 2 comments /

This is one of my older projects that aims to control external devices directly, by using the PC Speaker port. It can control multiple devices at one go, but they must be powered one at a time. I made it a couple of years ago and it was working fine, but due to mistake during plugging it literally burned ;(

Now, when both cheap USB chips are available and PC speaker is unused (or integrated with sound card) it doesn't make sense, to build it, but this is still an interesting concept.

CMS written entirely in PL/pgSQL

in / with 1 comment /

In one of my previous jobs, I heard that it would be inspiring, to write a CMS with all logic stored and executed on database side. Every request would be performed in stored procedures, theirs final result would be HTML page sent back to the user.

I will present, how to do it with PotgreSQL, Apache, Linux and a bit of bash scripting. The script is a cheat because I promised to write everything in stored procedures, but as you will see, it's just a gateway to the database, there isn't any logic.

Subscribe to: Posts (Atom)