Where did the 'Bastard File' get its odd name?

A bastard file is a file whose teeth configuration is in between a rough or coarse file and a 'second cut' file. In precise terms therefore a bastard file is one which 'is one cut finer than that of a coarse file'.

In common language a bastard is an illegitimate child, one born out of wedlock. However it may also allude to something that is unusual, irregular or disproportionate. It is this latter meaning that is employed in the term bastard file, not the former.
When used for a file the word bastard means irregular, in other words, a file that can neither be classified as a 'coarse file' or as a 'second cut file'.

In a metal file (bastard), there is only a singular direction of cutting rows, and not an intersecting row in another direction. Like a single factor producing the product without the complementary intersecting other row (or pair directions) necessary in a wood file. A single direction produces a bastard.

it was born from a non-legitimized union

Although dictionary etymologies are virtually nonexistent, a plausible origin can be extracted from historical usage. "Bastard file" in current usage is a file of coarse cut (as opposed to a "finishing" file. However, its technically precise definition is "a file one cut finer than a "coarse file." Files are classed as "coarse," "second cut" and "smooth," from coarsest to finest. Thus, a "bastard file" is a cut in between a "coarse" and a "second cut." The word "bastard" functions here in its meaning as "irregular." So, a "bastard file" is a file that is neither "coarse" nor "second cut."

The mill file with grooves in one direction was bastardised by grooves being cut in the other direction in the same way that a coat of arms was bastardised for illigitimate offspring by the addition of a diagonal "bend sinister" across the coat of arms.