A bastard file is a file whose teeth configuration is in between a rough or coarse file and a 'second cut' file. In precise terms therefore a bastard file is one which 'is one cut finer than that of a coarse file'.

In common language a bastard is an illegitimate child, one born out of wedlock. However it may also allude to something that is unusual, irregular or disproportionate. It is this latter meaning that is employed in the term bastard file, not the former.
When used for a file the word bastard means irregular, in other words, a file that can neither be classified as a 'coarse file' or as a 'second cut file'.

it was born from a non-legitimized union

Although dictionary etymologies are virtually nonexistent, a plausible origin can be extracted from historical usage. "Bastard file" in current usage is a file of coarse cut (as opposed to a "finishing" file. However, its technically precise definition is "a file one cut finer than a "coarse file." Files are classed as "coarse," "second cut" and "smooth," from coarsest to finest. Thus, a "bastard file" is a cut in between a "coarse" and a "second cut." The word "bastard" functions here in its meaning as "irregular." So, a "bastard file" is a file that is neither "coarse" nor "second cut."